Consolidation

This section lists the elements you can use to configure Consolidation.

This page discusses:

ConsolidationConfig

  • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig
  • Consolidation Server configuration. The Consolidation Server collects documents from a push API. The document goes through the following transformation steps before being sent to the Indexing Server:
    1. Transformation processors
    2. Aggregation processors
  • Attributes:
    Name Type Default value Description
    name string Identifies this consolidation configuration uniquely
    nbThreads int 4 Number of threads to allocate during the transformation and aggregation phases.
    maxNativeMemoryConsumptionThreshold enum(disabled, enabled, auto) enabled When reaching the memory consumption value specified, transformation is stopped and transformed documents are committed. Then the transformation starts again.
    • Enabled: Commits when the memory usage reaches the threshold value specified (by default, 2048 MB).
    • Auto: Commits when the memory usage reaches 2048 MB.'
    maxNativeMemoryConsumptionInMB int 2048 The maximum of native memory the transformation service can allocate. Reaching this limit triggers a commit.
  • Nested elements:
    Name Type Description
    CommitTriggerConditions com.exalead.mercury.mami.indexing.v10.CommitTriggerCondition*
    AggregationTriggerConditions com.exalead.mercury.mami.indexing.v10.CommitTriggerCondition*
    AdvancedConfig com.exalead.mercury.mami.consolidation.v10.AdvancedConfig
    AggregationForwardProcessorConfigList com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfigList
    AggregationProcessorConfigList com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList
    TransformationProcessorConfigList com.exalead.mercury.mami.consolidation.v10.TransformationProcessorConfigList

AdvancedConfig

  • com.exalead.mercury.mami.consolidation.v10.AdvancedConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as ConsolidationConfig)
  • Attributes:
    Name Type Default value Description
    sequentialCompaction boolean True
  • Nested elements:
    Name Type Description
    AdvancedAggregationConfig com.exalead.mercury.mami.consolidation.v10.AdvancedAggregationConfig
    CompactPolicies com.exalead.mercury.mami.indexing.v10.CompactPolicies
    DocumentStoreWarnConfig com.exalead.mercury.mami.consolidation.v10.DocumentStoreWarnConfig

AdvancedAggregationConfig

  • com.exalead.mercury.mami.consolidation.v10.AdvancedAggregationConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AdvancedConfig (as AdvancedConfig)
  • Attributes:
    Name Type Default value Description
    impactQueueSize int 8
    aggregationQueueSize int 8
    forwardQueueSize int 1600
    aggregatedVerticesCacheSize int 65536 Deprecated since V6R2016xR4, a dedup on impact detection results is now performed before running the aggregation
    vertexCacheSize int 65536
    impactDetectionResultCacheMaxFileSizeInBytes int 10485760
    logCountImpactedVertices int -1 Possible values:
    • -1: disabled (default value)
    • 0: log all
    • N: log only if the number of vertices impacted is greater than or equals to N
    metaCountLogThreshold int -1 Possible values:
    • -1: disabled (default value)
    • N: log only if the number of values for a meta is greater than or equals to N
    disableCheckpointForwarding boolean
    progressionSaveThresholdInMS long 3600000

CompactPolicies

  • com.exalead.mercury.mami.indexing.v10.CompactPolicies
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AdvancedConfig (as AdvancedConfig)
  • Attributes:
    Name Type Default value Description
    synchronous boolean By default, compaction jobs are asynchronous. If set, compacts will be done synchronously just after imports.
    maxParallelFullCompacts int Limit the number of full compacts in parallel, can be useful when you don't have too much disk space available. 0 means no limit.
    type enum(mmap, pagecache) mmap Specifies which I/O mode is used while compacting. ( Value can be null or one of
    • mmap
    • pagecache
    )
    maxPageCacheSizeMB int 32 If the policy uses the PageCache mode, it specifies the max cache size.
    pageCachePageSizeKB int 8 If the policy uses the PageCache mode, it specifies the page size.
    priorityCompactThreshold int 48 When compacting a slot gen0-gen1, consider as a priority compact if gen1-gen0 < priorityCompactThreshold. Default is 48. (0: disabled)
    lowPriorityCompactNbThreads int 2 Number of threads to use for a compact having low priority (0: all available threads).
    highPriorityCompactNbThreads int Number of threads to use for a compact having high priority (0: all available threads).
  • Nested elements:
    Name Type Description
    AutoCompactPolicy com.exalead.mercury.mami.indexing.v10.AutoCompactPolicy* Specifies the auto-compact policies.

NumberOfSlotsBasedCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.NumberOfSlotsBasedCompactPolicy
  • Compaction policy based on a fixed number of slots for a given number of generations.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.CompactPolicies (as CompactPolicies)
  • Attributes:
    Name Type Default value Description
    component string
    arity int 4 Specifies the number of slots of the same length required to compact.
    maxSlotSizeMb long 5000 If a slot reaches this size; it will never be used by the next automatic compaction processes.
  • Nested elements:
    Name Type Description
    FullCompactPolicy com.exalead.mercury.mami.indexing.v10.FullCompactPolicy

MaxSizeFullCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.MaxSizeFullCompactPolicy
  • A FullCompactPolicy that compacts all slots into one whenever the "tail" of small slots exceeds a certain ratio of the large first slot. This policy is appropriate when auto-compacts are restricted to slots under a certain size for performance reasons. In this case, a full optimization can occasionally be triggered to purge the deletes. If not, the deletes occurring in later slots would never be deleted, incurring performance costs at query-time and extra disk space consumption.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.AutoCompactPolicy (as AutoCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.LowLatencyCompactPolicy (as LowLatencyCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.NoCompactPolicy (as NoCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.NumberOfSlotsBasedCompactPolicy (as NumberOfSlotsBasedCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.SlotsLogSizeBasedCompactPolicy (as SlotsLogSizeBasedCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.SlotsSizeBasedCompactPolicy (as SlotsSizeBasedCompactPolicy)
  • Attributes:
    Name Type Default value Description
    percentage int 100 Minimum percentage to launch a full compaction. Compacts all slots into one whenever the "tail" of small slots exceeds a certain percentage of the large first slot. Eg: with percentage=100, when cumulated size of all slots except biggest is higher than size of the biggest slot, a full compact is triggered.
    minSlots int 2 Minimum number of slots before triggering a full compact.

ArityBasedFullCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.ArityBasedFullCompactPolicy
  • A FullCompactPolicy that compacts all slots into one whenever the "tail" of slots with smaller arities exceeds together a certain arity. The idea is that the arity-based policy guarantees occasional full-compaction, but the time interval between full-compaction increases exponentially. This add-on policy caps the increase at a certain arity, and schedules full-compacts at regular intervals afterwards. This policy is appropriate when auto-compacts are managed per generation-arity. In this case, a full optimization can occasionally be triggered to purge the deletes. If not, the deletes occurring in later slots would never be deleted, incurring performance costs at query-time and extra disk space consumption.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.AutoCompactPolicy (as AutoCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.LowLatencyCompactPolicy (as LowLatencyCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.NoCompactPolicy (as NoCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.NumberOfSlotsBasedCompactPolicy (as NumberOfSlotsBasedCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.SlotsLogSizeBasedCompactPolicy (as SlotsLogSizeBasedCompactPolicy)
    • com.exalead.mercury.mami.indexing.v10.SlotsSizeBasedCompactPolicy (as SlotsSizeBasedCompactPolicy)
  • Attributes:
    Name Type Default value Description
    maxArity int 256 Whenever the long tail total arity reaches maxArity, a full compact is scheduled. The "long tail" are the slots whose span has an arity inferior to this parameter. This is generally a multiple of the auto-compact Arity policy arity parameter.
    minSize long Slots below this size are considered neglectable.

SlotsSizeBasedCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.SlotsSizeBasedCompactPolicy
  • Compaction policy based on size that produces slots with similar size. When N consecutive slots have a size below targetSizeForCompactionMB, it performs a compaction if:
    • N is at least minArity AND
      • The N+1 slot makes the size above targetSizeForCompactionMB OR
      • The size is above minSizeForCompactionMB
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.CompactPolicies (as CompactPolicies)
  • Attributes:
    Name Type Default value Description
    component string
    targetSizeForCompactionMB int 200 Targeted size for a compacted slot.
    minSizeForCompactionMB int 50 Minimum size required to compact.
    minArity int 2 Minimum number of slots required to compact.
  • Nested elements:
    Name Type Description
    FullCompactPolicy com.exalead.mercury.mami.indexing.v10.FullCompactPolicy

SlotsLogSizeBasedCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.SlotsLogSizeBasedCompactPolicy
  • A CompactPolicy that tries to compact slots into levels of exponentially increasing size, where each level has fewer slots than the value of the compact factor. Whenever extra slots (beyond the compact factor upper bound) are encountered, all slots within the level are compacted.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.CompactPolicies (as CompactPolicies)
  • Attributes:
    Name Type Default value Description
    component string
    compactFactor int 10 Determines how often slots are compacted. With smaller values, less RAM is used while indexing, and searches on unoptimized indices are faster, but indexing speed is slower. With larger values, more RAM is used during indexing, and while searches on unoptimized indices are slower, indexing is faster. Thus larger values (greater than 10) are best for batch index creation, and smaller values (lower than 10) for indices that are interactively maintained.
    minSize long 1048576 A size setting type which sets the minimum size for the lowest level slots. Slots below this size are considered to be on the same level (even if they vary drastically in size) and will be merged whenever there are mergeFactor for them. This effectively truncates the "long tail" of small slots that would otherwise be created into a single level. If you set this too large, it can greatly increase the merging cost during indexing (if you flush many small slots).
    maxSize long 9223372036854775807 A size setting type which sets the largest slot that may be merged with other segments.
  • Nested elements:
    Name Type Description
    FullCompactPolicy com.exalead.mercury.mami.indexing.v10.FullCompactPolicy

LowLatencyCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.LowLatencyCompactPolicy
  • Compacts when the size of all small slots is above the average large slot size, or when the number of slots is above nbLargeSlots + maxNbSmallSlots.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.CompactPolicies (as CompactPolicies)
  • Attributes:
    Name Type Default value Description
    component string
    nbLargeSlots int 8 The number of large slots to keep.
    maxNbSmallSlots int 8 Maximum number of small slots allowed. As soon as this limit is reached, small slots are compacted together.
    gatherSmallsAtTheEnd boolean True
    contiguousCompact boolean
  • Nested elements:
    Name Type Description
    FullCompactPolicy com.exalead.mercury.mami.indexing.v10.FullCompactPolicy

NoCompactPolicy

  • com.exalead.mercury.mami.indexing.v10.NoCompactPolicy
  • Compact policy that does not perform any compact.
  • Parent elements:
    • com.exalead.mercury.mami.indexing.v10.CompactPolicies (as CompactPolicies)
  • Attributes:
    Name Type Default value Description
    component string
  • Nested elements:
    Name Type Description
    FullCompactPolicy com.exalead.mercury.mami.indexing.v10.FullCompactPolicy

DocumentStoreWarnConfig

  • com.exalead.mercury.mami.consolidation.v10.DocumentStoreWarnConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AdvancedConfig (as AdvancedConfig)
  • Attributes:
    Name Type Default value Description
    thresholdInBytes long 104857600
    enableAtTransformation boolean True
    enableAtAggregation boolean True

AggregationForwardProcessorConfigList

  • com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfigList
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as ConsolidationConfig)
  • Nested elements:
    Name Type Description
    AggregationForwardProcessorConfig com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfig*

AggregationForwardProcessorConfig

  • com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfigList (as AggregationForwardProcessorConfigList)
  • Attributes:
    Name Type Default value Description
    pushAPIServer string Specify the target Push API server
    documentTypes string Comma-separated list of document types to forward. Leave empty to match all documents. Deprecated since V6R2015x.SP4, replaced by ForwardedDocumentTypeList
    triggerIndexing boolean True If set to true (default behavior), it will trigger an indexing job on your remote Push API server when the aggregation job is complete
  • Nested elements:
    Name Type Description
    ForwardedDocumentTypeList com.exalead.mercury.mami.consolidation.v10.ForwardedDocumentTypeList

ForwardedDocumentTypeList

  • com.exalead.mercury.mami.consolidation.v10.ForwardedDocumentTypeList
  • List of forwarded document types
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationForwardProcessorConfig (as AggregationForwardProcessorConfig)
  • Nested elements:
    Name Type Description
    ForwardedDocumentType com.exalead.mercury.mami.consolidation.v10.ForwardedDocumentType*

ForwardedDocumentType

  • com.exalead.mercury.mami.consolidation.v10.ForwardedDocumentType
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ForwardedDocumentTypeList (as ForwardedDocumentTypeList)
  • Attributes:
    Name Type Default value Description
    type string

AggregationProcessorConfigList

  • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList
  • List of aggregation processors that yield aggregated documents
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as ConsolidationConfig)
  • Nested elements:
    Name Type Description
    CommonProcessorConfig com.exalead.mercury.mami.consolidation.v10.CommonProcessorConfig* Declare an aggregation processor to be used by the Consolidation Server.

TransformationProcessorConfig

  • com.exalead.mercury.mami.consolidation.v10.TransformationProcessorConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList (as AggregationProcessorConfigList)
  • Attributes:
    Name Type Default value Description
    source string
    name string Identifies this processor uniquely in the processing step
    description string Description of this processor
    enabled boolean True
    mime string Mime-type of the language used to define the processor. As of now, supported types are:
    • text/x-groovy
    • text/x-jar
    perfMonitored boolean Show the processor statistics in the Monitoring Console.
    context string Indicates the context in which the processing operation will occur
    • create/update
    • delete
    code string Processor source code run on add/update operations.

CustomJavaTransformationProcessor

  • com.exalead.mercury.mami.consolidation.v10.CustomJavaTransformationProcessor
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList (as AggregationProcessorConfigList)
  • Attributes:
    Name Type Default value Description
    source string
    name string Identifies this processor uniquely in the processing step
    description string Description of this processor
    enabled boolean True
    mime string Mime-type of the language used to define the processor. As of now, supported types are:
    • text/x-groovy
    • text/x-jar
    perfMonitored boolean Show the processor statistics in the Monitoring Console.
    context string Indicates the context in which the processing operation will occur
    • create/update
    • delete
    classId string
    pluginName string Deprecated since V6R2016x.R2, not used by the product so not replaced.
  • Nested elements:
    Name Type Description
    config exa.bee.KeyValue*

AggregationProcessorConfig

  • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfig
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList (as AggregationProcessorConfigList)
  • Attributes:
    Name Type Default value Description
    name string Identifies this processor uniquely in the processing step
    description string Description of this processor
    enabled boolean True
    mime string Mime-type of the language used to define the processor. As of now, supported types are:
    • text/x-groovy
    • text/x-jar
    perfMonitored boolean Show the processor statistics in the Monitoring Console.
    context string Indicates the context in which the processing operation will occur
    • create/update
    • delete
    code string Processor source code run on add/update operations.

CustomJavaAggregationProcessor

  • com.exalead.mercury.mami.consolidation.v10.CustomJavaAggregationProcessor
  • No documentation for this element.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.AggregationProcessorConfigList (as AggregationProcessorConfigList)
  • Attributes:
    Name Type Default value Description
    name string Identifies this processor uniquely in the processing step
    description string Description of this processor
    enabled boolean True
    mime string Mime-type of the language used to define the processor. As of now, supported types are:
    • text/x-groovy
    • text/x-jar
    perfMonitored boolean Show the processor statistics in the Monitoring Console.
    context string Indicates the context in which the processing operation will occur
    • create/update
    • delete
    classId string
    pluginName string Deprecated since V6R2016x.R2, not used by the product so not replaced.
  • Nested elements:
    Name Type Description
    config exa.bee.KeyValue*

TransformationProcessorConfigList

  • com.exalead.mercury.mami.consolidation.v10.TransformationProcessorConfigList
  • List of transformation processors that yield transformed documents
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as ConsolidationConfig)
  • Nested elements:
    Name Type Description
    CommonTransformationProcessorConfig com.exalead.mercury.mami.consolidation.v10.CommonTransformationProcessorConfig* Declare a transformation processor to be used by the Consolidation Server.

NumberOfTasksBasedCommitTriggerCondition

  • com.exalead.mercury.mami.indexing.v10.NumberOfTasksBasedCommitTriggerCondition
  • Triggers a commit after the specified No. tasks has been processed. The No. of tasks calculation is executed each time a batch of documents received, to avoid performance penalties. You might therefore have a bit more than the specified No. of tasks analyzed.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as AggregationTriggerConditions)
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as CommitTriggerConditions)
  • Attributes:
    Name Type Default value Description
    nbTasks int The number of tasks

SizeBasedCommitTriggerCondition

  • com.exalead.mercury.mami.indexing.v10.SizeBasedCommitTriggerCondition
  • Triggers a commit when the Max size (MB) is reached.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as AggregationTriggerConditions)
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as CommitTriggerConditions)
  • Attributes:
    Name Type Default value Description
    maxSizeMB int Max size threshold in MB

RAMUsageCommitTriggerCondition

  • com.exalead.mercury.mami.indexing.v10.RAMUsageCommitTriggerCondition
  • Triggers a commit when RAM usage reaches the limit.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as AggregationTriggerConditions)
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as CommitTriggerConditions)
  • Attributes:
    Name Type Default value Description
    maxRAMUsageInMB int Max RAM usage in MB

PeriodicCommitTriggerCondition

  • com.exalead.mercury.mami.indexing.v10.PeriodicCommitTriggerCondition
  • Commits every N seconds after the first push order done after the last commit.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as AggregationTriggerConditions)
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as CommitTriggerConditions)
  • Attributes:
    Name Type Default value Description
    delayS long Time in seconds between two commits.

InactivityCommitTriggerCondition

  • com.exalead.mercury.mami.indexing.v10.InactivityCommitTriggerCondition
  • Inactivity-based condition. This condition is triggered when:
    • there is no new data for the specified time period
    • AND at least the specified No. tasks has been analyzed.
  • Parent elements:
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as AggregationTriggerConditions)
    • com.exalead.mercury.mami.consolidation.v10.ConsolidationConfig (as CommitTriggerConditions)
  • Attributes:
    Name Type Default value Description
    numberOfTasks int Minimum number of tasks to trigger a commit.
    inactivityTimeS long After N seconds of no indexing activity, it is defined as inactive.

KeyValue

  • exa.bee.KeyValue
  • No documentation for this element.
  • Parent elements:
    • exa.bee.KeyValue (as KeyValue)
    • com.exalead.mercury.mami.consolidation.v10.CustomJavaAggregationProcessor (as config)
    • com.exalead.mercury.mami.consolidation.v10.CustomJavaTransformationProcessor (as config)
  • Attributes:
    Name Type Default value Description
    key string The name of the key
    value string
    type string
    description string
  • Nested elements:
    Name Type Description
    KeyValue exa.bee.KeyValue*