Manage Resources in cvadmin

Yyou can edit linguistic and semantic resources, without compiling and publishing them manually, using the Resource Manager.

The Resource Manager publishes semantic resource files to specified roles, which are organized into groups to update interrelated resource files in unison.

The Resource Manager:

  • compiles resources

  • assigns versions to resources

  • publishes resources, including to multiple hosts

  • converts resource formats, such as from XLS to XML

This section explains how to use the Resource Manager by taking the Ontology Matcher semantic processor as an example. The procedure is the same, however, for all resource types.

This page discusses:

See Also
Create a Resource File from the Administration Console
Managing Semantic Annotations
Appendix - Configure Semantic Processors

Edit the ResourceManager.xml File

The first step is to edit the <DATADIR>/config/ResourceManager.xml file. In this file, you define groups of resources and which roles to publish these resources to.

Tip: Grouping resource files enables you to keep dependant resources together (for example, if a Rules Matcher depends on the Ontology Matcher's results). This ensures consistent updates since the group is published as a unit.

The default configuration includes the following resource groups:

  • indexing, which targets the analyzers

  • search, which targets the searcher

For example, to include an OntologyMatcher in the semantic pipeline:

  1. Insert an ontology resource in the indexing ResourceGroup as shown in the following ResourceManager.xml example:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<res:ResourceManagerConfig resourceDir="data:///build/resources" version="0" 
xmlns:res="exa:com.exalead.mercury.mami.resources.v10" xmlns="exa:exa.bee" 
xmlns:config="exa:exa.bee.config">
  <res:ResourceGroup waitOnSync="false" roles="Analyzer,Searcher,Dictionary" name="content-filtering">
    <res:Resource type="rt-blacklist" name="rt-blacklist"/>
    <res:Resource type="rt-whitelist" name="rt-whitelist"/>
    <res:Resource type="categories-blacklist" name="synthesis-blacklist"/>
    <res:Resource type="facets-blacklist" name="facets-blacklist"/>
    <res:Resource type="stopwords" name="stopwords"/>
    <res:Resource type="spellcheck-blacklist" name="spellcheck-blacklist"/>
    <res:Resource type="spellcheck-whitelist" name="spellcheck-whitelist"/>
  </res:ResourceGroup>
  <res:ResourceGroup waitOnSync="false" roles="Analyzer" name="indexing">
    <res:SemanticResource type="ontology" name="onto" tokenizationConfig="tok0"/>
    <res:SemanticResource type="rules" name="rules" tokenizationConfig="tok0"/>
  </res:ResourceGroup>
  <res:ResourceGroup waitOnSync="false" roles="Searcher" name="search" />
</res:ResourceManagerConfig>

    Once you have edited your ResourceManager.xml file, apply the configuration:

  2. Open the API Console.
  3. Select Manage.
  4. Search for the applyConfiguration method.
  5. Click Send.

Upload, Compile, and Publish the Ontology

Once you have created your own resource file (called ontology.xml in this example), you must upload this file to the Resource Manager with the cvadmin command-line tool.

  1. Go to the <DATADIR>/bin directory.

  2. Run the following command in the cvadmin command-line tool to verify that the resources are properly added in the ResourceManager.xml file:

    ./cvadmin resources list-resources group=group_name_of_your_resource

    You can see your resource in the list.

  3. Upload your resource file, with the following cvadmin command: 

    cvconsole cvadmin> resources upload path=<path/to/your/file/ontology.xml> resource=<name_of_your_resource>
 [publish=true]

    Where <name_of_your_resource> is the name previously defined in the ResourceManager.xml file.

    On the private cloud only: By using the optional argument publish=true, the file is uploaded, compiled (if required) and published in one command. If you want to separate these actions, after uploading your file, run: 
    cvconsole cvadmin> resources publish group=<name_of_the_group>

    This publishes all the resources in the specified group. If the last upload date is later than the last compile date, then this also proceeds to a compilation.

Tip: Errors may occur because the resource file (here ontology.xml) is not correct. Since the Resource Manager is part of the gateway, you can check the gateway's log file for causes of a compilation failure.

The resources published are available under resourcemanager://group_name/resource_name

Supported Resource Types

Resource

Type value

RESOURCE_NAME (for samples)

Annotation Manager

type="annotation-manager"

n/a

Ontology Matcher

type="ontology"

n/a

Sentiment Analyzer

type="sentiment"

n/a

Rules Matcher

type="rules"

n/a

Fast Rules

type="fastrules"

n/a

Semantic Extractor

type="semantic-extractor"

n/a

Lemmatizer

type="lemmas"

n/a

Synonyms expansion module

type="synonyms"

n/a

Related terms blacklist

type="rt-blacklist"

rt-bl0

Suggest allow list

type="suggest-whitelist"

suggest-wl0

Suggest block list

type="suggest-blacklist"

suggest-bl0

Query block list

type="query-blacklist"

query-bl0

Spellcheck block list

(See Configure Spellcheck Resources in dictionary.xml)

type="spellcheck-blacklist"

spellcheck-bl0

Spellcheck allow list

(See Configure Spellcheck Resources in dictionary.xml)

type="spellcheck-whitelist"

spellcheck-wl0

CSV-compliant

type="csvraw"

(no compilation, no validation)

n/a

Custom

type="raw"

(no compilation, no validation)

n/a

XML-compliant

type="xmlraw"

(no compilation, no validation)

n/a

Configure Spellcheck Resources in dictionary.xml

For spellcheck block list and allow list resources, you need to perform an extra step. After declaring the resources, you must also edit the dictionary.xml.

  1. Declare the two resources handling the allow and block listing. In this example, we add them to the "content-filtering" group of the Resource Manager. 

    <res:ResourceManagerConfig resourceDir="data:///build/resources" version="1338967974000" 
xmlns:res="exa:com.exalead.mercury.mami.resources.v10" xmlns="exa:exa.bee" 
xmlns:config="exa:exa.bee.config">
  <res:ResourceGroup waitOnSync="false" roles="Analyzer,Searcher,Dictionary" name="content-filtering">
    <res:Resource type="spellcheck-blacklist" name="spellcheck-bl0"/>
    <res:Resource type="spellcheck-whitelist" name="spellcheck-wl0"/>
  </res:ResourceGroup>
</res:ResourceManagerConfig>

  2. Since the global dictionary is in charge of spell checking, you need to edit the dictionary.xml configuration file to bind the resources. 

    <DictionaryConfig xmlns="exa:com.exalead.mercury.mami.linguistic.v10" version="1338967974673" 
maxWords="126000000" maxRelatedTerms="10000000"
minNbDocsForRelatedTerm="10" maxWordCooccurrencesForSpellcheck="1260000000" automaticLanguagesRatio="true"
 maxInputsBeforeBuild="0" 
tokenizationConfig="tok0" preAllocatedInterpretationPipelines="2" preAllocatedSpellCheckPipelines="2"        
<!-- specify resource names declared for the "content-filtering" group -->
spellCheckBlacklistResource="spellcheck-bl0" spellCheckWhitelistResource="spellcheck-wl0">
<!-- [...] -->
</DictionaryConfig>
  3. Apply the configuration.
  4. Create the allow list and block list CSV files. For sample formats, see Access Sample Block List and Allow List Formats.

  5. Upload, compile, and publish your resources to the Resource Manager using the following cvadmin command: 

    cvconsole cvadmin> resources upload path=/path/to/spellcheck_blacklist.csv resource=spellcheck-bl0
 publish=true
cvconsole cvadmin> resources upload path=/path/to/spellcheck_whitelist.csv resource=spellcheck-wl0
 publish=true

Access Sample Block List and Allow List Formats

These resource formats are available using the resources get-sample command of the cvadmin command-line tool, where RESOURCE_NAME is the name of the sample.

  1. Go to the <DATADIR>/bin directory.

  2. Run the following command in the cvadmin command-line tool:

    cvconsole cvadmin> resources get-sample resource=RESOURCE_NAME

    See Supported Resource Types for RESOURCE_NAMES.

    For example: 

    cvconsole cvadmin> resources get-sample resource=spellcheck-bl0

    Returns: 

    # language, level, expression "en", "normalized", "exalead"

Convert Semantic Resources to Other Formats

Several cvadmin and cvdebug commands are available to convert resources to different formats. Go to <DATADIR>/bin/ to start them.

To output/convert

Start the command...

A compiled ontology resource to XML

cvconsole cvdebug > linguistic dump-ontology path="<PATH TO COMPILED ONTOLOGY>"

An ontology resource from XML to XLS

cvconsole cvadmin > linguistic convert-ontology-from-xml-to-xls input=/tmp/onto.xml output=/tmp/out.xls

For example, the following ontology:

<Ontology xmlns="exa:com.exalead.mot.components.ontology"> <Pkg path="top.custom"> <Entry display="Mickey Mouse"> <Form value="Mickey M." level="normalized"/> <Form value="Mouse M." level="exact" /> </Entry> </Pkg> </Ontology>

generates an out.XLS file with a single sheet, named " top.custom" that contains:

  • Display: Mickey Mouse, Form: Mickey M., Lang: xx, Level: normalized
  • Display: Mickey Mouse, Form: Mouse M., Lang: xx, Level: exact

An ontology resource from XLS to XML

cvconsole cvadmin> linguistic convert-ontology-from-xls-to-xml input=/tmp/out.xls output=/tmp/out.xml

An ontology resource from SKOS to XML

cvconsole cvadmin> linguistic convert-ontology-from-skos-to-xml input=/tmp/out.xls output=/tmp/out.xml

A suggest resource from SKOS to XML

cvconsole cvadmin> linguistic convert-suggest-from-skos-to-xml input=/tmp/out.xls output=/tmp/out.xml

A synonym resource from SKOS to XML

cvconsole cvadmin> linguistic convert-synonyms-from-skos-to-xml input=/tmp/out.xls output=/tmp/out.xml