How the Consolidation Server Fits into Exalead CloudView

A Exalead CloudView installation is made up of one or several build groups, with connectors feeding the build groups with documents. The Consolidation Server allows you to define consolidation rules for documents before pushing them into the build group Indexing Server.

The Consolidation Server therefore fits before the build group Indexing Server or before another Consolidation Server if a specific forward rule indicates to do so. You can view it as a transformation phase between source connectors and the Indexing Server.

For each connector, you can choose to enable consolidation. You can therefore use the Consolidation Server for a set of connectors and not for other connectors as shown in the following diagram.



The following diagram illustrates the consolidation workflow within the Consolidation Server, when connectors push documents.



Step

Description
1

Connectors push an addDocuments bulked order through HTTP to the Consolidation Server:

  • Documents arrive in the Consolidation Server
  • Source URIs are added to the CDIH
2

Documents go through the transformation processing.
3

As soon as commit conditions are met (see yellow star on the diagram) OR when one of the source connectors sends a synchronization order, all changes are persisted to disk in the Consolidation storage. Its purpose is to store transformed documents and the updated object graph.
4

As soon as aggregation conditions are met (see red star on the diagram) the Impact detection is launched on new object graphs. It detects the nodes of the existing graph that have to be aggregated again.
5

Once the impact detection is complete, aggregation processing can be launched for all detected nodes. The processing depend on their type and the action context.
6

As soon as a document is aggregated, it is pushed to the target defined in the existing forward rules.

The target can be an Indexing Server or another Consolidation Server.
Note: The reception order of ADD/DELETE operations for a given document is respected all along the processing chain. For example, if a connector sent an ADD order for document and then a DELETE order, the Consolidation Server will also send an ADD order and a DELETE order to the Indexing Server.