Unexpected Search Behavior

This task shows you how to:

See Also
Identify the Cause of the Index Crash

Search for Issues in Document Processing

Follow the steps below to identify document processor issues.

Step 1 - to Add log Information

To display detailed chunks in the log.log file, you must first add a Debug Processor element to your document processors list.

  1. From the Administration Console, go to Index > Data processing > Pipeline name.
  2. In the Document Processors tab, click Other in the Processor types menu.
  3. Drag the Debug Processor element to the end of the processors list.

Example: <Debug> tags are displayed:

[2013/09/23-09:54:11.584] [info] [AnalyzerThread-bg0-default_model-1] [analysis.debug] uri:
 C:\Users\E7G\Downloads\traffic.csv8 source: RATP did: 57 slice: 0: DebugProcessor: dumping 
 C:\Users\E7G\Downloads\traffic.csv8DebugProcessor: dumping C:\Users\E7G\Downloads\traffic.csv8:
<DebugChunk type="TextChunk" ctx="ville" deleted="false" part="null" value="Paris" score=0 language="xx">
</DebugChunk>
<DebugChunk type="TextChunk" ctx="arrondissement" deleted="false" part="null" value="1" score=0 
language="xx"> </DebugChunk>
<DebugChunk type="TextChunk" ctx="source" deleted="false" part="null" value="RATP" score=0 language="xx">
</DebugChunk>

Step 2 - to Submit Document Using cvdebug

Submit your document to test processors in your pipeline: 

cvconsole cvdebug> analysis analyze path=<PATH_TO_DOCUMENT>

For example, submit a .CSV file.

cvconsole cvdebug> analysis analyze path=/tests/myfile.csv

The output is a mapping of contexts and chunk values.

<TestAnalysisPipelineOutput xmlns="com.exalead.indexing.analysis.v10" documentProcessorsTimeUS=1000 
semanticAndMappingTimeUS=0>
  <##default>
    <DocumentProcessorsOutput xmlns="com.exalead.indexing.analysis.v10">
      <Document>
        <Document xmlns="com.exalead.ndoc.v10">
          <element>
            [
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="source"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="sourceTest"/>,
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="uri"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="C:\Users\E7G\Downloads\traffic2.csv"/>,
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="extracted_mime"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="text/plain"/>,
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="mime"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="text#plain"/>,
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="docsrc"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="txt"/>,
             <Context xmlns="com.exalead.ndoc.v10" language="xx" name="text"/>,
             <ScoreContext xmlns="com.exalead.ndoc.v10" value=0/>,
             <Chunk xmlns="com.exalead.ndoc.v10" value="Rang,Reseau,Station,Trafic,Correspondances,
c1,c2,c3,c4,Ville,Arrondissement1,MΘtro,GARE DU NORD,&quot;48,146,629&quot;,4,5,0,0,0,Paris,102,
MΘtro,SAINT-LAZARE,&quot;46,790,941&quot;,3,9,12,13,14,Paris,8"/>,
          </element>
        </Document>
      </Document>
    </DocumentProcessorsOutput>
  </##default>
  <##default>
    <UnmappedContexts xmlns="com.exalead.indexing.analysis.v10">
      <StringValue>
        [
         <StringValue xmlns="exa.bee" value="docsrc"/>,
         <StringValue xmlns="exa.bee" value="extracted_mime"/>,
         <StringValue xmlns="exa.bee" value="source"/>
        ]
      </StringValue>
    </UnmappedContexts>
  </##default>
</TestAnalysisPipelineOutput>

Search for Issues in Semantic Processor

You can follow the steps below to identify semantic processor issues.

Step 1 - Display Semantic Processors for Each Document

To display detailed information on semantic processing in an HTML file, you must first add a Debug Processor element to your semantic processors list.

Important: The HTML output is verbose. You can use sample data to avoid using large amounts of disk space during indexing.

  1. From the Administration Console, go to Index > Data processing > Pipeline name.
  2. In the Semantic Processors tab, drag the Debug Processor to the end of the proces­sors list.
  3. In the Input from field, specify the HTML file in which information is logged.

Example: all semantic processing applied to field 327 of document [0000000031217F90]



Step 2 - Submit Text or Document

Submit text through the semantic pipeline to display all running processors:
  • Using a single word:
  • cvconsole cvdebug> semantic annotate language=en context=text value=”WORD”
  • Using a text block (.TXT file):
  • cvconsole cvdebug> semantic annotate-file language=en path=<PATH_TO_TXT_FILE>

Example: submit the word ‘test’ 

cvconsole cvdebug> semantic annotate language=en context=text value=”test”

The output displays tokens tagged with annotations:

<AnnotatedToken id="0" kind="TOKEN_SEP_PUNCT" lang="en" token="ΓÇ¥" offset="0" />
<AnnotatedToken id="1" kind="TOKEN_ALPHA" lang="en" token="test" offset="3" >
  <Annotation TID="7" kind="LOWERCASE" id="0" nbTokens="1" display="test" displayKind="lower" 
trustLevel="0" />
  <Annotation TID="8" kind="NORMALIZE" id="1" nbTokens="1" display="test" displayKind="norm" 
trustLevel="0" />
  <Annotation TID="20" kind="phonetic" id="2" nbTokens="1" display="T.E.S.T" displayKind="norm" 
trustLevel="100" />
  <Annotation TID="23" kind="relatedTermsPreprocessor_staticLemma" id="3" nbTokens="1" display="test"
displayKind="norm" trustLevel="100" />
</AnnotatedToken>
<AnnotatedToken id="2" kind="TOKEN_SEP_PUNCT" lang="en" token="ΓÇ¥" offset="7" />
<AnnotatedToken token="ö" kind="PUNCT" lang="en" offset="0 ">
</AnnotatedToken>
<AnnotatedToken token="test" kind="ALPHA" lang="en" offset="1">
  <Annotation displayForm="test" displayKind="lowercase" tag="LOWERCASE" nbTokens="1" trustLevel="0" />
  <Annotation displayForm="test" displayKind="normalized" tag="NORMALIZE" nbTokens="1" trustLevel="0" />
  <Annotation displayForm="T.E.S.T" displayKind="normalized" tag="phonetic" nbTokens="1" 
trustLevel="100" />
  <Annotation displayForm="test" displayKind="normalized" tag="relatedTermsPreprocessor_staticLemma"
 nbTokens="1" trustLevel="100" />
</AnnotatedToken>
<AnnotatedToken token="ö" kind="PUNCT" lang="en " offset="5 ">
</AnnotatedToken>

Step 3 - Display All Processors

Display the list of semantic processors in the analysis pipeline using cvdebug: 

cvconsole cvdebug> semantic dump-pipe

The output (extract) displays each semantic processor and resources:

<Processor>
  [
    <Normalizer xmlns="com.exalead.mot.components" resource="exalead.891136526.normalizer.resource"
 trustLevel="0" 
normalizeCJ="true"/>,
    <GermanTokenizer xmlns="com.exalead.mot.components" 
normalizerResource="exalead.891136526.normalizer.resource" 
splitPolicy="keepToken3" useCustomNormalizationTID="true" stickTokens="false" 
resource="exalead.subtokenizer.de.resource"/>,
    <DutchTokenizer xmlns="com.exalead.mot.components" 
normalizerResource="exalead.891136526.normalizer.resource" 
splitPolicy="keepToken3" useCustomNormalizationTID="true" stickTokens="false"
 resource="exalead.subtokenizer.nl.resource"/>,
    <NorwegianTokenizer xmlns="com.exalead.mot.components" 
normalizerResource="exalead.891136526.normalizer.resource" 
splitPolicy="keepToken3" useCustomNormalizationTID="true" stickTokens="false" 
resource="exalead.subtokenizer.no.resource"/>,
    <JapaneseCharDetector xmlns="com.exalead.mot.components"/>,
    <CJKProcessor xmlns="com.exalead.mot.components"/>
  ]
</Processor>