Verifying the Indexing Status

The MQL command status searchindex [vault VAULTPATTERN] displays for each vault:

• The modified date of the last object indexed
• The number of objects processed
• The start and end time of the indexing process
• The status of the index (Complete, Running, Stopped, or Cancel)

Example:

MQL<5>status searchindex;
eService Sample: Complete last object modified: 11/27/2018 9:13:10 PM  number of crawled objects: 14 duration: 9 seconds
eService Administration: Complete last object modified: 11/27/2018 9:13:18 PM  number of crawled objects: 2903 number of crawled 
connections: 779708 duration: 392 seconds
eService Production: Complete last object modified: 3/4/2018 10:17:16 AM number of crawled objects: 4438 number of crawled 
connections: 779436 duration: 2172 seconds
vplm: Complete  last object modified: 12/16/2017 2:23:19 PM number of crawled objects: 1259428 number of crawled connections: 
779436 duration: 4226 seconds
SIXW: Complete last object modified: 11/27/2017 5:25:44 PM number of crawled objects: 1081 number of crawled connections: 1397     
duration: 32 seconds
acl: Complete last object modified: 9/8/2018 4:58:35 PM  number of crawled objects: 3934 duration: 1 seconds

You can add the verbose attribute to the MQL command, that is, status searchindex verbose, to get more details about:

• The search index status.
• The dataset creation status, if datasets were created. It reflects the crawler's status, with the business objects and connections of every input vault.

Important: If the status of any vault is Stopped, the counts of crawled objects and crawled connections is generally 0. This typically means that an error occurred during the crawling of this vault. In such a case, correct all the errors reported by the crawler and the CloudView logs, for all the failed objects and connections. You can get them using the validate searchindex reports, as described in Count and Verify Missing Objects. Once this first correction is done, use the validate repair scripts to repair the problematic objects and connections. If the failed objects and connections were a result of baseline indexing, you can also use the resume searchindex command (see Resume Indexing After Failure).

MQL>status searchindex verbose;
vplm -  Crawler: Complete    Index: Complete    last object modified: 11/23/2020 1:30:21 PM    
last successful index: 11/23/2020 12:32:41 PM    last indexation start: 11/23/2020 12:32:41 PM    
last indexation end: 11/23/2020 12:35:26 PM 
crawled objects: 19998    crawled connections: 19998    duration: 165 seconds
vplm - businessobject dataset: Complete    number of objects: 19998    last modified: 11/23/2020 12:26:17 PM    
relationship dataset: Complete    number of connections: 19998    last modified: 11/23/2020 12:26:21 PM
eService Production -  Crawler: Complete    Index: Complete    last object modified: 11/23/2020 1:40:42 PM    
last successful index: 11/23/2020 12:26:01 PM    last indexation start: 11/23/2020 12:26:01 PM    
last indexation end: 11/23/2020 12:28:42 PM    
crawled objects: 19998    crawled connections: 19998    duration: 161 seconds
eService Production - businessobject dataset: Complete    number of objects: 19998    
last modified: 11/23/2020 12:26:05 PM    relationship dataset: Complete    number of connections: 19998    
last modified: 11/23/2020 12:26:09 PM
eService Administration -  Crawler: Stopped    Index: Stopped    last object modified:     
last indexation start: 11/23/2020 12:32:30 PM    last indexation end: 11/23/2020 12:32:41 PM    
crawled objects: 0    crawled connections: 0    duration: 11 seconds
eService Administration - businessobject dataset: Complete    number of objects: 19998    
last modified: 11/23/2020 12:26:12 PM    relationship dataset: Complete    
number of connections: 19998    last modified: 11/23/2020 12:26:14 PM
https://vdevpril0171plp.dsone.3ds.com:453/fcs/servlet/fcs :
For remote queue :   JOBS directory: /home/data/RTV/R423relhme1devpril0171/apache-tomcatNoCAS/temp/FILEINDEXING/JOBS
JOBS in Queue: 0

In the config.xml file, if you define the <DATASETS/> section with a concurrency attribute value higher than 1, then every spawned MQL process will have its own log directory and file:

• On Linux, this mechanism uses the directory pointed by <MX_TRACE_FILE_PATH>/sxi/concurrent_crawler_logs/<tenant_name>.
• On Windows, this mechanism creates log files in the same directory as the <MX_TRACE_FILE_PATH> environment variable. The filename starts with <timestamp>_<datasetid>_<partitiontype>_<integer_suffix>.

This mechanism ensures that all partitions crawled by each MQL process can be tracked and logged correctly.

You can validate the index objects and relationships with the validate searchindex [vault PATTERN] [type|relationship] [interval START_TIME END_TIME] command.

Option	Description
vault PATTERN	Specifies the vault to validate. Use the PATTERN argument to specify a pattern of vault names, including a wildcard '*' character. For example, if there were two vaults named v1 and v2, instead of issuing the command twice, once for each vault, you could issue it once using vault v*. This clause is optional.
type|relationship	Specifies whether to validate business objects only or connections only. Important: You cannot use both arguments in the same command.
interval START_TIME END_TIME	Specifies a time interval to validate only objects or connections modified in that interval. START_TIME and END_TIME must be in MX_NORMAL_DATETIME_FORMAT. This option validates all access lists in the Database and the index. It makes comparisons object by object. If an object is missing in the Database and another object of the same type is missing in the index, the logs report that the first object is missing in the Database and the second object is missing in the index. This can have a slight performance impact.

The validate searchindex command takes the 3DSpace Index High Availability configuration into account and produces separate validation reports for the main and secondary indexes.

This command generates a tcl script that you can use to repair the index. This tcl script contains the physical ids of all business objects and connections that are missing in the index. Once you run the script, the event monitor logs these business objects and connections as custom events, and triggers their incremental indexing.

You can find all the logs and repair scripts in the logs/sxi/validate/<tenantname>/validate_<timestamp>.zip. The generated logs are:

• validate_searchindex_counts*.log
• validate_searchindex_objects*.log
• validate_searchindex_repair*.log contains the oids of business objects or connections that you need to repair.

Executing the validate searchindex; command multiple times, generates multiple zip files under logs/sxi/validate/<tenantname>. Each zip file name has a timestamp indicating when its generation date. For High Availability cases, the zip includes:

• The main index validate logs.
• The secondary index validate logs.
• The main index repair script.
• The secondary index repair script.

To control the default behavior of the validate searchindex; command, you can use system environment variables. Add them into the mxEnv.sh file (for Linux) or the enovia.ini file (for MS Windows).

Environment variable	Description
MX_VALIDATE_MISSING_OBJECT_LIMIT	Configure this variable to specify the maximum number of objects tolerated in the index for a specific type. By default, the limit is 10,000 objects for a specific type. In the index, when the number of objects for a specific type is below the limit, the validate searchindex command validates the missing objects and logs their physicalid in the validate_searchindex_objects.log. This log file contains a list of all physical ids missing in the index and the database. If the number of objects exceeds the limit, the validate_searchindex_objects.log entries for this type will not be generated." It however continues to give results in COUNT logs, that is, Count of Indexed BO Vs Count of BO in the database. Note: The MX_VALIDATE_MISSING_OBJECT_LIMIT variable limit does not affect COUNT logs at all.
MX_VALIDATE_MAX_OBJECT_VALIDATIONS	This variable is useful for large indexes. Configure it to specify the maximum number of indexed objects of a specific type and the indexed access lists that can be validated. For example, if you specify the variable to MX_VALIDATE_MAX_OBJECT_VALIDATIONS=1000000, the validate searchindex command validates 1,000,000 objects of a specific type and 1,000,000 access lists. In more detail, it: Queries the index to retrieve a maximum of 1 million objects of a specific type from the index (which may have, for example, 2 million objects for that specific type.) Validates if these indexed objects (which may be a subset of all of the objects of this type in the index) are consistent against all objects of this type in the DB. If the number of objects for a type in the database is higher (for example, 5 million objects for that specific type), we cannot really fully validate this specific type as it depends on which indexed objects were retrieved. In such cases, it is recommended to increase this variable's limit to 5 million and validate all the indexed objects of this type. Note: If undefined, the default value for this environment variable is 1,000,000. This variable does not have any impact on validate COUNT logs. The MX_VALIDATE_MISSING_OBJECT_LIMIT variable value has an impact on this variable while reporting missing objects in the validate object logs. MX_VALIDATE_FORCE_OBJECT_VALIDATIONS=true overrides this variable.
MX_VALIDATE_MAX_FILEINDEXING_VALIDATIONS	Configure this variable to specify the maximum number of objects having missing files that can be validated. By default, the validate searchindex command validates file indexing up to 5,000 validations.
MX_VALIDATE_FORCE_OBJECT_VALIDATIONS	By default, the variable value is false, which means that the validation depends on the limit specified for the MX_VALIDATE_MISSING_OBJECT_LIMIT variable. Defining this variable to true forces the validation of missing objects in the index, bypassing the limit specified for the MX_VALIDATE_MISSING_OBJECT_LIMIT variable.

To enable tracing during indexing, use the following MQL command:

trace type verbose,MQL,JPO filename mxtrace.log;

The output is a file in STUDIO_PATH or MX_TRACE_FILE_PATH (if defined in enovia.ini).

During indexing, MatrixException errors may occur because of bad or corrupted data. When the process encounters a MatrixException, the process logs the error and continues. You can find the list of errors in the log file called 3DXServer_SLF4J_*.log in MatrixInstall/logs. The log file contains information about the MatrixException thrown, including the IDs of the objects and the object types. After each indexing process, look for this log file and correct any errors.

Unused or obsolete access lists can affect the performances of the indexing crawler.

When building the index, it is important to evaluate whether security elements are correctly indexed. The search engine uses indexed security tokens to apply security rules, as established in the policies and P&O choices made for a particular deployment.