Rules Matcher (Rule-Based)

If the matching rules for this processor depend on phonetic, stem, or lemma matching, you must add the corresponding processor above this one in the pipeline.

For example, if your rules require phonetic forms, place the Phonetizer processor above this processor in the analysis pipeline.

Rules matching is based on an XML file that lists patterns to identify. It is defined using regexp-like expressions.

• The XML file may contain several rules. Each rule has a priority and a specific annotation.

• Use priority to disambiguate multiple matches for the same content.

  The best match is determined by the criteria below, applied in this order:

  1. Leftmost match

  2. Longest match

  3. Highest priority value, where 0 = lowest priority.

  4. The order in which the rules are defined in the rules file.

• The annotation marks the matched tokens.

For example, a Named-Entity matcher could tag people's names with the annotation (NE, people) and then the Rules Matcher tags the first names with (sub, 1) and the last names with (sub, 2) thus allowing match normalization.

In this example, we want to identify email addresses, long dates, and short dates in French and English. We use the Rules Matcher to create an annotation for a document when it identifies an email address, based on the character sequence defined in the Rules XML file.

<TRules xmlns="exa:com.exalead.mot.components.transducer">
   <!-- 1st rule tags emails with the annotation NE.email -->
  <TRule priority="0">
    <MatchAnnotation kind="NE.email"/>
    <Seq>
    <TokenRegexp value="[0-9A-Za-z_]+"/>
    <Noblank/>
    <Iter min="0" max="4">
      <Or>
        <Word value="-" level="exact"/>
        <Word value="." level="exact"/>
      </Or>
      <Noblank/>
      <TokenRegexp value="[0-9A-Za-z_]+"/>
      <Noblank/>
    </Iter>
    <Word value="@" level="exact"/>
    <Noblank/>
    <TokenRegexp value="[0-9A-Za-z_]+"/>
    <Noblank/>
    <Iter min="0" max="6">
      <Or>
        <Word value="-" level="exact"/>
        <Word value="." level="exact"/>
      </Or>
      <Noblank/>
      <TokenRegexp value="[0-9A-Za-z_]+"/>
      <Noblank/>
    </Iter>
    <Word value="." level="exact"/>
    <Noblank/>
     <Or>
      <TokenRegexp value="[A-Za-z][A-Za-z]"/>
      <Word value="gov"/>
     <!-- government entities in the US -->
      <Word value="com"/>
     <!-- commercial entities -->
      <Word value="net"/>
     <!-- network providers -->
      <Word value="org"/>
     <!-- non-profit organizations -->
      <Word value="edu"/>
     <!-- educational institutions -->
      <Word value="info"/>
    <!-- informative websites -->
      <Word value="biz"/>
     <!-- business -->
      <Word value="pro"/>
     <!-- business use by qualified professionals -->
      <Word value="name"/>
    <!-- individuals' real names, nicknames, pseudonyms -->
      <Word value="aero"/>
    <!-- aviation-related businesses -->
      <Word value="asia"/>
    <!-- region of Asia, Australia, and the Pacific -->
      <Word value="cat"/>
     <!-- Catalan language and culture -->
      <Word value="coop"/>
    <!-- cooperatives -->
      <Word value="int"/>
     <!-- international treaty-based organizations -->
      <Word value="jobs"/>
    <!-- employment-related sites -->
      <Word value="mil"/>
     <!-- US Department of Defense -->
      <Word value="tel"/>
     <!-- publishing contact data -->
      <Word value="museum"/>
  <!-- museums -->
      <Word value="travel"/>
  <!-- travel industry -->
    </Or>
  </Seq>
  </TRule>
 <!-- 2nd rule tags dates with the annotations NE.date.full and NE.date.compact -->
  <TRule priority="9">
  <!-- All that's been matched by the captures, French way -->
  <MatchAnnotation kind="NE.date.full" value="%1 %2 %3 %4"/>
  <!-- Don't use day of week -->
  <MatchAnnotation kind="NE.date.compact" value="%2/%3/%4"/>
  <!-- matches dates like Samedi 31 janvier, 2004 or 16th of November 2003 -->
  <Seq>
    <!-- 1st capture: optional week day -->
    <Opt>
      <Sub no="1">
        <!-- Day names ontology, taken from Named Entities matcher -->
         <Annotation kind="exalead.nlp.date.days"/>
      </Sub>
      <Opt>
 <Word value=","/>
 </Opt>
      <Opt>
 <Word value="the"/>
 </Opt>
    </Opt>
    <!-- 2nd capture: day number/ordinal -->
        <Sub no="2">
      <Or>
        <!-- day ordinal : 16, 28, 15th, 1st,3rd ... -->
        <Annotation kind="exalead.nlp.date.ordinals"/>
        <!-- day number with no ordinals: 16, 28, 1 ... -->
        <TokenRegexp value="0?[1-9]"/>
        <TokenRegexp value="[12][[:digit:]]" />
        <TokenRegexp value="3[01]" />
     </Or>
    </Sub>
   <Opt>
      <Or>
        <Word value="of"/>
        <Word value="-"/>
      </Or>
    </Opt>
    <!-- 3rd capture: month name -->
    <Sub no="3">
      <Annotation kind="exalead.nlp.date.months"/>
    </Sub>
    <Opt>
 <Word value="-"/> </Opt>
    <!-- 4th capture: year -->
    <Sub no="4">
      <Or>
        <!-- year like 06 -->
        <TokenRegexp value="[[:digit:]]{2}"/>
        <!-- full year [1000, 2999] -->
        <TokenRegexp value="[12][[:digit:]]{3}" />
      </Or>
    </Sub>
  </Seq>
  </TRule>
</TRules>

Do the following:

• Use the <Digit> operator to match tokens made of a sequence of digits instead of a regular expression. Indeed, it uses the token kind computed by the tokenizer and is therefore more efficient.

• Use a normalizer; it is likely that you need a normalizer somewhere upstream in the analysis pipeline because <Word> and <TokenRegexp> operators may match against the lowercase or normalized forms of tokens.

• Use the <Noblank> operator if you do NOT want to skip spaces. The rules matcher skips spaces and tabs so that the rules are not littered with hundreds of references to blanks. It is however possible to assert that there is no blank token between two tokens at a precise position in the stream with the operator <Noblank>.

Avoid the following:

• Using the <Near> operator too often. It uses repeat operators, and so matches the longest possible match. This can be costly in terms of compilation time and RAM consumption, so you must try to keep A and B as simple as possible and limit NEAR overlapping.

The Rules Matcher has the following caveats:

• It does not report overlapping or embedded matches; the earliest and longest match is reported. If there are ambiguities, the tokens matched by the highest-priority rules are kept. If there are two or more rules with similar priorities, the first rule in the declaration order has highest priority. For the repeat operators (<Opt>, <Iter>) and <Near>, the longest possible match is preferred.

• The operator <Sub> allows for sub expression matches retrieval. It has two attributes kind and value defining the annotation emitted each time the sub expression matches. These "numerical subs" are useful in match normalization. They are defined so that concatenating the text they have matched in increasing order of their value yields normalized matches. For example, people's names detection rules could mark first names with (sub, 1) and last names with (sub, 2) thus giving equal results after concatenation for "John Smith" and "Smith John". Of course, these submatch annotations are only emitted when the overall pattern matches.

• During the compilation, the RulesMatcher performs a number of optimizations, in particular for Boolean ORs that are, whenever possible, replaced with a single regular expression that indicates if any of the conditions in the list match. For example, the following example is automatically rewritten to the second simplified expression, which is more efficient. The same thing is done with words <word> are transformed to regular expressions.

  <Or>
    <TokenRegexp value="0?[1-9]"/>
    <TokenRegexp value="[12][[:digit:]]" />
    <TokenRegexp value="3[01]" />
  </Or>
  <TokenRegexp value="0?[1-9]|[12][[:digit:]]|3[01]"/>

• The <And> operator does not impose that lengths of submatches be equal. A match is found if all its subpatterns match, irrespective of the length of the matches. The following example matches even if one annotation does not have the same length as another annotation, provided that they are both present on the same token. The match length is the length of the longest annotation. For example:

  <And>
    <Annotation kind="some"/>
    <Annotation kind="other"/>
  </And>

The Rules Matcher has the following limitations:

• The window size is 200 tokens. This is the maximum length of a match.

• The upper bound for the operator <iter> must remain as low as possible as it is costly in terms of resources. The maximum is limited to 128.

• UNICODE is not handled and matching is done on UTF-8 strings without specific processing (at the byte level). Consequently:

  • Accentuated characters do not match if they are used in case-insensitive or normalized mode.

  • The dot wildcard matches a byte and therefore not all UTF-8 characters.

We now need to configure the Rules Matcher processor to map the NE.email annotation to a category facet that represents the email address. This allows the document to be related to all email addresses found in it.

1. In the Administration Console, select Index > Data processing > Pipeline name > Semantic Processors.

2. On the Mappings subtab, click Add mapping source.

   1. Name: Enter the annotation name that you created in the rules file, for example, NE.email for the sample file above.

   2. Type: select Annotation.

3. (Optional) In Input from field of the mapping, restrict the mapping so it only applies to a subset of comma-separated metas (also known as contexts) associated with this annotation.

4. Click Add mapping target and add a category target.

5. Modify the category-mapping properties.

   For example, the Create categories under this root property must be modified to Top/Email in our example.

6. Go to Search > Search Logics > Your_Search_Logic > Facets and add a category group.

   1. Click Add facet and enter the name to display in the Mashup UI Refinements panel.

   2. For Root, enter the value you have entered for Create categories under this root in step 4, for example, Top/Email.

7. Click Apply.