About Creating Additional Tokenization Configurations

A tokenization configuration specifies which tokenizers to use when Exalead CloudView analyzes incoming documents at index-time. It also specifies how to tokenize queries at search-time.

By default, Exalead CloudView uses tok0 as the tokenization configuration for converting text into tokens. However, if you create additional tokenization configs, you must specify them explicitly in Data Model > Semantic Types and Data Processing.

This page discusses:

See Also
Using Native Tokenizers
Using Basis Tech Tokenizer
Customizing the Tokenization Config

When do you Need to Create a New Tokenization

You may want to create a new tokenization config to define:

  • A certain character as a separator, or a character as NOT being a separator. See Character Overrides.

  • A specific pattern to be a word only, instead of a word with separators. For example, to make sure C++ is always indexed as the token C++. See Pattern Overrides.

Specify a tokenization config for a specific index mapping or semantic type, when you know that a certain meta contains characters that need to be interpreted differently than other alphanumeric metas. Typical examples are identifiers like user IDs, model numbers and product codes.

Index-Time and Search-Time Tokenization

When you specify a new tokenization config for document analysis at index-time, you must also specify this same tokenization config for interpreting queries at search-time.

You can specify a default tokenization config for a search logic, as well as an “exception” tokenization config for specific prefix handlers. For more information, see Specify a Tokenization Configuration for Prefix Handlers.

Indexing is both morphological and ngram-based. This gives Exalead CloudView a kind of fallback mechanism. For example, "cjk" (that is,, Chinese Japanese Korean) in Linguistics > Tokenizations > Advanced > Form indexing triggers the indexing of isolated characters but we still index words made of several characters output by tokenizers. This way, in case of different tokenizations between indexing and querying for the same text (either because of a different context or because of a tokenization error), we can still find documents containing the sequence of characters.

Which Tokenization Config takes Precedence?

A tokenization config that is defined for a specific:

  • semantic type overrides the one defined for its corresponding analysis pipeline.

  • meta mapping overrides the one defined for its corresponding semantic type.

For example:

  • The analysis pipeline ap0 uses the default tok0 tokenization config. Any text that is processed by processors added directly to this analysis pipeline is tokenized with tok0.

  • The text semantic type uses tok1, and the text, location, and product_name metas in the data model are set up to use this semantic type. In other words, tok1 overrides tok0 for these three metas.

  • However, the product_name meta has special tokenization requirements, so its index mapping only uses a third config, tok2, which overrides tok1.