Managing Semantic Annotations

This section explains how to manage semantic annotation with the Annotation Manager or with custom code.

This page discusses:

See Also
Create a Resource File from the Administration Console
Appendix - Configure Semantic Processors
Manage Resources in cvadmin

Manage Annotations with the Annotation Manager

The Annotation Manager allows you to perform several operations on annotations under the right conditions. You can use it to copy, select, and remove annotations.

The Annotation Manager configuration consists of a list of operations.

Important: There is no define order of the execution of operations. If you really care about operation ordering, you must add several annotation managers to the semantic pipe.

Copy Annotation

You can copy a source annotation along with its display form, display kind, and trust level to a target annotation.

Option

Example

Copy without condition

For example, to ignore the distinction between famous and nonfamous people.

<Copy annotation="NE.famousperson" target="NE.person"/>

Copy with a condition

For example, to copy famous people to the nonfamous people annotation, unless they are block listed.

<Copy annotation="NE.famouspeople" target="NE.people" unless="blocklisted"/>

Remove Annotation

You can remove the occurrences of an annotation under the right conditions.

Option

Example

Remove without condition

For example, to remove all end-of-sentences:

<Remove annotation="sbreak" />

Remove an annotation if it overlaps with another one

For example, to remove a person's name annotation when it spreads over two sentences:

<Remove annotation="NE.person" ifOverlapWith="sbreak" />

Remove an annotation if the annotated text span matches that of another one

For example, to remove a person's name annotation when the text is block listed:

<Remove annotation="NE.person" ifMatchWith="blocklist.person” />

An ontology matcher upstream or any other semantic processor can set the annotation blocklist.person. Both annotations must start and end exactly on the same tokens.

Remove an annotation if the annotated text span and display form match those of another one

For example, we want to implement a block list with a fine granularity:

<Remove annotation="title.approx" ifMatchWith="blocklist.title" displayFormsMustMatch="true"/>

If an ontology containing a title package matches professor on the text processor using approximation

<pkg path="title"> <Entry> <Form value="professor" /> </Entry> </pkg>

... the annotation is removed if the annotation (blocklist.title, "professor") occurs at the very same place, thus block listing the specific approximation.

Keep the first occurrence of an annotation and remove all others

For example, to keep only the first organization occurrence in title and text:

<KeepFirst annotation="NE.organization" contexts="title,text"/>

Keep the longest leftmost of a set of overlapping annotations and remove all others

<KeepLongestLeftMost annotations="NE.person,NE.place,NE.organization" interTags="false"/>

With interTags set to false, one annotation per tag is kept.

Select Annotation

You can select the most frequent annotations and store the results as document annotations.

Option

Example

Select the most frequent values in a document for a given annotation

For example, we want to select the 5 places that occur the most in a document and store them in selectedPlaces document annotations.

<SelectMostFrequentValue annotation="NE.place" documentAnnotation="selectedPlace" howMany="5" truncate="true"/>

If there are more than 5 most frequent places, the resulting list is arbitrarily truncated since truncate="true" guarantees that no more than 5 annotations are ever reported.

Select the most frequent annotation in a document among a list

<SelectMostFrequentAnnotation annotations="NE.organization,NE.place,NE.person" documentAnnotation="selectedAnnotation"/>

The most frequent annotation is used to output a selectedAnnotation document annotation whose value is one of the annotations from the list.

Select annotations depending on an index field (context) priority

For example, we want to select an annotation from the “title, text” contexts, by first looking within the title context and then, if the annotation is not found, looking within the text context:

<SelectByContexts annotation="NE.person" contexts="title,text" documentAnnotation="selectedAnnotation" firstOnly="false"/>

With firstOnly set to false, all occurrences of NE.person annotations in the specified contexts are reported.

Use Regular Expressions

You can use regular expressions for all annotation parameters.

Set enableRegexp to true in the <AnnotationManager> object (default is false).

Example of Annotation Manager XML Configuration File

<AnnotationManager xmlns="exa:com.exalead.linguistic.v10">
  <Copy annotation="NE.famousperson" target="NE.person"/>
  <Copy annotation="NE.famousperson" target="NE.person" unless="blocklist"/>
  <Remove annotation="NE.famousperson" ifOverlapWith="sbreak" />
  <Remove annotation="NE.person" ifOverlapWith="sbreak" />
  <Remove annotation="NE.person" ifMatchWith="blocklist.person" />
  <Remove annotation="title.approx" ifMatchWith="blocklist.title" displayFormsMustMatch="true"/>
</AnnotationManager>

Manage Annotations with Custom Code

In most semantic-oriented projects, you need to manipulate (filter, combine, replace, etc.) the semantic annotations set by your semantic processors before sending them to the index.

The easiest way to do this is to add semantic processors into the Document Processor pipeline, transforming the annotations into metas (also known as chunks). Then you can manipulate them using either:

  • custom java code via the JavaDocumentProcessor

  • standard document processors such ReplaceValues or ConcatenateValues.

Example: Index Term Occurrences in a Document

Let us say you want to send to the index the number of times a term is matched in the document from an existing list of terms.

Recommendation: Use the OntologyMatcher to detect all terms. Go through it using a SemanticPipeDocumentProcessor. Convert the semantic annotations into metas (or chunks) and use custom java code to count them.

Create a List of Terms

This ontology annotates each term of the list with the "myterms" annotation. 

<Ontology xmlns="exa:com.exalead.mot.components.ontology">
  <Pkg path="myterms">
    <Entry display="Term 1">
      <Form level="normalized" />
    </Entry>
    <Entry display="Term 2">
      <Form level="normalized" />
    </Entry>
    <Entry display="Term 3">
      <Form level="normalized" />
    </Entry>
    <!-- [...] -->
  </Pkg>
</Ontology>

Modify the Analysis Pipeline

Add the following configuration to the end of your document processor Analysis Pipeline.



Each "myterms" semantic annotation is converted into a meta (or chunk), that the Document Processors can manipulate.

The XML representation of the SemanticPipeDocumentProcessor configuration looks like: 

<SemanticPipeDocumentProcessor 
  annotations="myterms" // Comma-separated semantic annotations that will be converted into metas   
  topLevelAnnotationsOnly="false" // only convert document annotations?
  disabled="false" name="SemanticPipeDocumentProcessor.0">
  <OntologyMatcher resourceDir="/path/to/myterms.bin" disabled="false" name="OntologyMatcher.0"/>
</SemanticPipeDocumentProcessor>

In the JavaDocumentProcessor, count the number of metas (or chunks) named "myterms". Add the count to a new "nbTerms" meta (the mapping of this "nbTerms" meta to the index is not detailed here). 

import com.exalead.pdoc.ProcessableDocument;
import com.exalead.pdoc.analysis.DocumentProcessingContext;
import com.exalead.pdoc.analysis.StandardDocumentProcessor;
import com.exalead.pdoc.Meta;

public class JavaDocumentProcessorTemplate extends StandardDocumentProcessor {
    @Override
    public void process(DocumentProcessingContext context, ProcessableDocument document)
      throws Exception {
        int count = 0;
        for (Meta m : document.getMetas("myterms")) {
            ++count;
        }
        document.addMeta("nbTerms", String.valueOf(count));
    }
}