Tuning
Basic Tuning
In the Administration Console, you can adjust:
-
The number of aggregation threads in Consolidation > Advanced Settings. For example, if you set it to
4, you get 4 transformation workers, 4 aggregation workers, and 4 forwarders (* by number of forward rules), all potentially running in parallel for an incremental batch. -
Your commit conditions to fit your current scenarios
Get the Initial Scan Recommended Settings
- In Consolidation > Commit triggers, specify a commit trigger based on No. of tasks to 500,000 tasks.
- Specify a commit trigger based on Inactivity set to 1 task and 60s of inactivity.
- Add an Aggregation trigger based Inactivity set to 1 task and 60s of inactivity (so your aggregation starts at the end of your initial push).
Get the Incremental Scan Recommended Settings
- In Consolidation > Commit triggers, specify a commit trigger based on No. of tasks to 50,000 tasks.
- Specify a commit trigger based on Inactivity set to 1 task and 60s of inactivity.
- Adjust your aggregation triggers to fit your required freshness.
Advanced Tuning
In your <DATADIR>/config/Consolidation.xml file, you can add an AdvancedConfig node to ConsolidationConfig to tweak internal queues used during aggregation. It might increase throughput with more buffering, but you must take it into account in your sizing.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<conso:ConsolidationConfigList version="0" xmlns:bee="exa:exa.bee"
xmlns:edit="exa:com.exalead.editor.v10" xmlns:index="exa:com.exalead.mercury.mami.indexing.v10"
xmlns:conso="exa:com.exalead.mercury.mami.consolidation.v10" xmlns:config="exa:exa.bee.config>
<conso:ConsolidationConfig name="cc0_standard" nbThreads="4"
maxNativeMemoryConsumptionThreshold="enabled" maxNativeMemoryConsumptionInMB="2048">
...
<conso:AdvancedConfig>
<conso:AdvancedAggregationConfig impactQueueSize="8" aggregationQueueSize="8"
forwardQueueSize="800" />
</conso:AdvancedConfig>
</conso:ConsolidationConfig>
</conso:ConsolidationConfigList>
Default values for AdvancedAggregationConfig are:
-
Impact Queue size / Aggregation Queue size = number of threads * 2
-
Forward Queue size = number of threads * 200