Indexing Process

Step	Description
1. Index metadata	The indexing process begins when the MQL index client runs a streaming query based on the config.xml file stored in the database. The config.xml file defines the host, ports, types, and fields to index. This process collects the metadata and submits it to the CloudView server through HTTP. The index uses the business object's ID or OID as the reference ID.
2. Is there file data?	The indexing process then checks for file data. If there are files checked into a business object, the indexing process determines which FCS to submit a file indexing ticket. This ticket queues an entry in the FCS. The config.xml file defines the stores, formats, and extensions to index. A property on the store defines the preferred location to avoid WAN access if the file is synchronized to a local location.
3. Fetch files for processing from the FCS	As the queue is processed on each FCS, the FCS does a file checkout and calls the file convert server running on that host with the path of the file to index. The file convert server must be on the same LAN as the FCS. The file convert server is an executable and is a different process than the FCS.
4. Scrub file content and send result to CloudView server	The FCS calls the convert, gets the converted binary, and sends a document to CloudView through the PushAPI.

In the image below, an indexing process runs with a streaming query returning 500 OIDs at a time. The selectables defined in the index definition are fetched for those 500 business objects across 4 threads, and the selectable data is returned and posted to the CloudView server.

To index information from an FCS, 3DSpace Index requires each FCS to support remote indexing of the files within the stores and locations that it manages. This means a file convert server must be installed and running on the same machine as each FCS.

The local file converts files into IDX format and directly sends them to the CloudView server. Remote indexing does not require a file to be checked out before it can be indexed into the CloudView server. This prevents network traffic created by file check outs.

The format.file.synchronized selectable determines how files are grouped from all the stores and locations. These groups are submitted in batch jobs to the controlling FCS. The FCS takes the groups and creates an "envelope" with XML syntax for the file convert server to process. Metadata continues indexing in the foreground as the file convert server sends the groups to the CloudView server. This process can occur at the same time across many FCSs. A queue system on each FCS prevents overload.

For large datasets, you can obtain a significative performance gain (x10), by performing the following procedure for the initial indexing:

1. In the CloudView Administration Console, go to Index > Consolidation.
2. Deactivate the aggregation trigger condition (every 30s and every 50000 tasks).
3. Scan the documents.
4. Click Force aggregation.
   Note: Before running this action, all Database documents must be pushed to the Consolidation server.
5. When running force aggregation, enter eno:co in prompt window. After all connections are indexed, enter eno:bo and then eno:acl. This step completes the indexing of all documents.

You can run these commands either manually or programmatically (using the CloudView MAMI in Java).