Using Basis Tech Tokenizer

Exalead CloudView also offers Extended Languages, provided by a third-party linguistic platform, Basis Technology’s Rosette. Basis Tech is only available if you have installed the Extended Languages add-on.

You can define a Basis Tech tokenizer for a specific language only. Basis Tech tokenizers generally offer richer semantic processing for Asian, Middle-eastern, and African languages than the native Exalead CloudView tokenizers.

Note: For the complete list of supported languages, see the CloudView Supported Languages datasheet.

This task shows you how to:

See Also
Using Native Tokenizers
About Creating Additional Tokenization Configurations
Customizing the Tokenization Config

Install the Extended Languages Add-On

This gives you access to the Basis Tech tokenizer. This add-on is available as a separate license.

See Install Add-ons in the Exalead CloudView Administration Guide.
  1. From the Administration Console, go to Help > License.
  2. Check that the Extended Languages feature is available.

Enable Extended Languages

Context:

Create one Basis Tech tokenizer for each language to be tokenized.

Important: There are additional steps for setting up Basis Tech tokenizers for Chinese, Japanese, German, Dutch, or Norwegian. See steps 5 and 6.

Before enabling Extended Languages, keep in mind the following:

  • While providing more in-depth text analysis for certain languages, Basis Technology’s analysis typically requires more RAM.

  • Named Entities extraction for languages with Extended Languages enabled are determined by Basis Technology’s tokenization rules instead of Exalead CloudView’s:

    Basis Technology uses a tokenization based on the Unicode standard word boundaries definition (http://www.unicode.org/reports/tr29/#Word_Boundaries) with a few customizations.

    The main difference with Exalead CloudView's tokenization is that Basis Tech tokenization does not split text on certain punctuation when they are not preceded and followed by blank spaces.

    • colons: Basis Tech keeps together 12:30pm. Standard tokenization produces three tokens 12 + : + 30pm
    • periods: I.B.M. or 12.33 both produce a single token
    • apostrophes: a single token for can't or 1970's

    This may affect your downstream processing in the semantic pipeline. For example, when an ontology has been compiled with Exalead CloudView's default tokenization, terms made of those characters do not match.

  1. In the Administration Console, go to Index > Linguistics > Tokenizations.
  2. Click your tokenization config (for example, tok0).
  3. In Tokenizers, click Add tokenizer.
    1. For Type, select Basis Tech, and then click Accept.
    2. For Language, select the language to tokenize.
  4. Repeat the previous step for each language you want to tokenize.
  5. If adding Basis Tech tokenizers for Japanese or Chinese, delete the default NETVIBES Japanese and Chinese tokenizers

  6. If adding Basic Tech tokenizers for German, Dutch, or Norwegian:
    1. Click the Standard tokenizer.
    2. Clear the German, Dutch, or Norwegian disagglutiner option.



  7. Click Apply.
  8. Reindex.

Enable Lemmatization with Basis Tech

With Exalead CloudView’s native tokenization, we only index the original words. At query time this adds an OR expansion for lemmatization.

For example: q=alsacien is expanded to q=alsacien OR alsacienne OR alsaciens OR alsaciennes.

When using BasisTech to tokenize a language, you must index the lemmatized form of the word in addition to the original form. Indeed, Basis Tech does not include the resource that associates a form to all its expansions.

  1. In Data Processing > Semantic Processors, add a Lemmatizer semantic processor to your analysis pipeline.

    For more information, see Configure Lemmatization Manually.

  2. Go to Index > Linguistics > Tokenizations.
  3. Click your tokenization config (for example, tok0).
  4. On the Advanced tab, add the following tag and matching mode:
    1. Tag: lemma
    2. Matching Mode: 2
  5. Click Apply.
  6. Reindex.

All lemma semantic annotations, which store the lemmatized form of the token, are indexed with a kind=2, meaning they are indexed in the same way as the original normalized form.