Best Practices to Increase Indexing Speed

Enhance your Data Model configuration:

• Reduce the number of properties in your data model.

  Above 50 properties, consider using dynamic properties. For more information, see Creating Dynamic Properties in the Exalead CloudView Configuration Guide.

• Do not set Compress content (in Index > Data Model > Properties > Other advanced options) for properties < 200 bytes.

Enhance your Data Processing configuration:

• Remove everything unnecessary in the analysis pipeline.
• By default, a LanguageDetector is applied to your meta. If you already know the language, push a meta named language with the corresponding 2-letter ISO code instead of using the meta.
• By default, a NGramsExtractor in Semantic Processors is used to do advanced multiword spell check. If you do not need advanced spell check, remove it to accelerate indexing.
• In your java custom doc processor, use getMeta(name) rather than iterating on all metas.

Enhance your Connectors configuration:

• Check that your connector sends documents per batch at least 20 per 20:
  • Explicitly using the addDocumentList PAPI command.
  • Or via a BufferedPushAPI.
  • Or by using the Buffering PushAPI Filter.
• Multithread your connector but stay below the number of logical cores. More threads would trigger too much CPU system.
• If you have more than 20 metas and no parts, use the MetaData Compaction PushAPI Filter, which will group them as a binary blob.
  • In this case, at the top of your Data Processing > Analysis pipeline, just after the MimeDetector, you must have the native_textextractor - Text Extractor (text,html,exalead) which will uncompress this blob.
  • Also remove the Standard Parts Merger.

Enhance your Build Group and index Tuning configuration:

• If you have good I/Os, you can set more slices (up to the number of logical cores).
• Set a number of Analyzers around the number of logical cores and proportional to your number of slices.
• If you do not manage to consume all your cores on machines with more than 16 cores, consider splitting your content into 2 build groups.
• Trigger jobs on RAM threshold only to do the biggest jobs possible (avoid sync(), setCheckpoint(sync=true), triggerIndexingJob).
• Perform only 1 indexing job per hour.
• If you have made a lot of updates, run cvadmin cvdebug indexing full-compact-dih after your big batches of remove.