Indexing Options

About the Indexing Process

The Indexing process contains a file-based queue with pending jobs. This queue protects the server from losing index requests if it fails or restarts.

Remember that:

You can start only one indexing process at a time.
You do not need to start an indexing process on each FCS. However, each FCS must be running at indexing time since a request may be sent to that FCS to index file data.
The process does not require on-demand synchronization since files are read and indexed at the FCS server, and file index data is sent to the central CloudView server.
When running multiple instances of the Live Collaboration Server against the same database (for example, in load-balancing configuration), you do not need to run an indexing process on each server.
Regardless of the number of deployed machines, you only need to execute one indexing process at a time. The config.xml file information is updated centrally in your database and in the CloudView server.

Baseline Indexing Process

This indexing process runs a complete index on your database. This is the initial indexing process you must run when you first set up 3DSpace Index.

The baseline indexing process is a multithread Java logic that can index millions of objects (metadata) quickly. Indexing is performed on one vault at a time, for all vaults defined in config.xml, and optionally on the MQL command line.

You can run a new baseline index at any time. It is usually required after very significant schema changes or config.xml file changes. During a subsequent baseline index, the entire database of searchable information is cleared and then populated.

To run a baseline indexing, use the following MQL command:

start searchindex mode FULL [vault VAULTPATTERN];

During baseline indexing, the crawler creates data sets and partitions:

One data set for each lattice type (BO/REL) of each vault.
Each data set can have one or more partitions. A partition has the following characteristics:
- It is a list of key-values that is immutable and independent.
- It has a maximum number of 1M physicalids of only one BO/Rel Type.
- Each partition has a type associated with it, which can be one of init, success or error. On initial creation, each partition is of type init. Once the crawling is complete, it creates a success partition, or an error partition, or both.

Important: If indexing fails, there is a recovery mechanism which allows the crawler to resume using partitions. For more information, see Resume Indexing After Failure.

Partial Indexing Process

This indexing process runs a partial index of your database. The index refreshes using the modifications that happened since the last baseline/partial indexing job.

Partial indexing is event-based. There are two approaches to run partial indexing:

By default, 3DSpace Index uses the eventmonitor-based approach. It listens for events in the kernel during partial indexing. 3DSpace Index adds an AdvancedSearch eventmonitor call that uses the eventmonitor to determine what objects to reindex.

Warning: The AdvancedSearch eventmonitor instance is internal to 3DSpace Index. Monitor and manage it only with help from customer support. For more information, use the help eventmonitor MQL command.

To activate this mode, set the CONSOLIDATIONSERVER eventmonitor parameter to true in the config.xml file:

<CONSOLIDATIONSERVER eventmonitor="true" />

Run a partial indexing process against each vault that contains modified or deleted objects, or objects that have moved to a different vault. If a vault is not defined in the config.xml file and contains modified objects, the partial indexing process does not index these modifications.

There is also a timestamp-based approach that uses checkpoints.
To activate this mode, set the CONSOLIDATIONSERVER eventmonitor parameter to false in the config.xml file:
```
<CONSOLIDATIONSERVER eventmonitor="false" />
```

To run partial indexing, whatever the chosen approach, use the MQL command:

start searchindex mode PARTIAL [vault VAULTPATTERN];

Note: To repair the index if you met some issues, you can also run partial indexing with the following MQL command:

start searchindex mode partial modifed DATE

This command enforces the DATE timestamp given as input for crawling. It does not use checkpoints, and there is no need to turn on/off the eventmonitor to use it.

Update Indexing Process

This indexing process allows you to make minor changes to the existing CloudView server index without running a new baseline index.

You can modify the config.xml file and update only affected objects in the CloudView server index. This indexing process only updates already indexed objects.

Use this indexing process to update the existing CloudView server index with new types, attributes, and fields. If you specify a new type in the command, all previous records of that type are replaced.

To run an update, use the following MQL command:

udpate searchindex type TYPE field FIELD [files];

Where:

TYPE is the name of the type affected by the new field.
FIELD is the new field that is updated for the type in the CloudView server.
You can optionally add the [files] parameter to index files for the type and field parameters entered. You can only use the [files] parameter if the following parameters are active in the config.xml: FORMATS, STORES, EXTENSIONS.

Specify Crawling Options

To reduce the overall indexing time in large databases, the crawler allows multiple MQLs to run simultaneously, where each MQL crawls a subset of the database.

The crawler provides partitioning modifiers such as !clear or aclonly.

Important: These options do not improve indexing time considerably for smaller databases.


Crawler partitioning modifier	Description
`start searchindex mode full vault <vaultname> !clear;`	This command: Crawls all types (and their subtypes). Computes access lists for each object and crawls all access lists. Does not clear out the existing index.
`start searchindex mode full aclonly;`	This command: Indexes all access lists only. Does not clear out the existing index. When you index business objects and connections using partitioning modifiers, execute this command only after all business objects are indexed.
`update searchindex;`	An option to update a specific attribute for a type/relationship type in the index. For more information, use the help searchindex command.
`status searchindex;`	This command: Reflects the current status of each of the crawlers started using the new partitioning modifiers. Includes a verbose option to provide more details about the crawlers. No longer shows the FCS queue status, unless you used the verbose option. For more information, use the help searchindex command.
`clear searchindex;`	You can run this command with the `statsonly` option, to clear the status of all crawlers running with and without partitioning modifiers. For more information, use the help searchindex command.
`modify searchindex;`	This command has an option to specify a checkpoint (timestamp) for the specified partitioning modifier. For more information, use the help searchindex command.

Resume Indexing After Failure

The 3DSpace Index has an indexing recovery mechanism to resume the crawler from where it left off during baseline indexing, and save time and efforts.

Important: Use this recovery mechanism for failed baseline indexing only. It does not work for the partial indexing and the update indexing processes.


Command	Description
`resume searchindex`	Processes all `init` and `error` partitions, and ACLs.

The indexing recovery mechanism expects the dataset creation of each vault to be complete. You can verify the dataset status section of each vault using the status searchindex verbose command. For example:

MQL>status searchindex verbose;
vplm -  Crawler: Complete    Index: Complete    last object modified: 11/23/2020 1:30:21 PM    
last successful index: 11/23/2020 12:32:41 PM    last indexation start: 11/23/2020 12:32:41 PM    
last indexation end: 11/23/2020 12:35:26 PM 
crawled objects: 19998    crawled connections: 19998    duration: 165 seconds
vplm - businessobject dataset: Complete    number of objects: 19998    last modified: 11/23/2020 12:26:17 PM    
relationship dataset: Complete    number of connections: 19998    last modified: 11/23/2020 12:26:21 PM
eService Production -  Crawler: Complete    Index: Complete    last object modified: 11/23/2020 1:40:42 PM    
last successful index: 11/23/2020 12:26:01 PM    last indexation start: 11/23/2020 12:26:01 PM    
last indexation end: 11/23/2020 12:28:42 PM    
crawled objects: 19998    crawled connections: 19998    duration: 161 seconds
eService Production - businessobject dataset: Complete    number of objects: 19998    
last modified: 11/23/2020 12:26:05 PM    relationship dataset: Complete    number of connections: 19998    
last modified: 11/23/2020 12:26:09 PM
eService Administration -  Crawler: Stopped    Index: Stopped    last object modified:     
last indexation start: 11/23/2020 12:32:30 PM    last indexation end: 11/23/2020 12:32:41 PM    
crawled objects: 0    crawled connections: 0    duration: 11 seconds
eService Administration - businessobject dataset: Complete    number of objects: 19998    
last modified: 11/23/2020 12:26:12 PM    relationship dataset: Complete    number of connections: 19998    
last modified: 11/23/2020 12:26:14 PM
https://vdevpril0171plp.dsone.3ds.com:453/fcs/servlet/fcs :
For remote queue :   JOBS directory: /home/data/RTV/R423relhme1devpril0171/apache-tomcatNoCAS/temp/FILEINDEXING/JOBS
JOBS in Queue: 0

Important: If there are pending FILEINDEXING jobs in the FCS queue, use the timestamp-based partial indexing to process them. Run a start searchindex mode partial modified <time-stamp> with a <time-stamp> that is earlier than the last successful partial indexing time.