Customizing the Tokenization Config

1. In the Administration Console, go to Index > Linguistics > Tokenizations.
   • Click an existing tokenization config to edit.
   • Click Add tokenization config to create a new one.
2. Set up the tokenizers as described in Using Native Tokenizers.
3. When finished, click Apply.
4. Reindex your data.

1. In the Administration Console, go to Index > Analysis, then select your analysis pipeline (for example, ap0).
2. From the Tokenization config list, select a different configuration.
3. Click Apply.
4. Reindex your data.

This tokenization config is only used on metas processed by document or semantic processors that were manually configured in the analysis pipeline.

1. In the Administration Console, go to Index > Data Model > Semantic Types tab.
2. Expand the semantic type you want to modify, or create a new one.
3. In the semantic type:
   1. Make sure that the Tokenize option is selected.
   2. From the Tokenization config list, select a different configuration.
4. Click Apply.
5. Reindex your data.

All properties using this semantic type tokenize the corresponding meta using this tokenization configuration, regardless of the tokenization config specified for its associated analysis meta.

1. In the Administration Console, go to Index > Data processing, then select your analysis pipeline (for example, ap0).
2. Select the Mappings tab.
3. Under the Mapping sources list, click the mapping you want to modify.
4. From the Tokenization config list, select a different configuration.
   Note: If this mapping was produced by a data model, you need to click Customize before you can modify any mapping settings.

5. Click Apply.
6. Reindex your data.

The meta for this mapping is always be tokenized using the tokenization config specified here, regardless of the tokenization config specified for its analysis pipeline or for its semantic type (if any).