UC-6: Incremental Scan - Propagating Arc Changes

Another interesting feature of the Consolidation Server is the ability to propagate any arc changes on related views.

This task shows you how to:

Before you begin: We assume that previous UCs have been completed.

Step 1 - Set the Country Connector to Incremental Mode

  1. In the Administration Console, go to Connectors and click the country JDBC connector.
  2. In Query parameters:
    1. For Synchronization mode, select Query-based incremental synchronization
    2. For Initial Query, enter: select country_id, ico_status, name, timestamp from countries
    3. For Checkpoint query, enter: select max(timestamp) from countries
    4. For Incremental variable, enter: TIMESTAMP
    5. For Incremental query, enter: select country_id, ico_status, name, timestamp from countries where timestamp > "$(TIMESTAMP)"
  3. Click Apply.

Step 2 - Create Organization from Countries

Configure the Transformation Processor

  1. Go to Index > Consolidation
  2. Add a new transformation processor:
    1. Select Groovy as format
    2. For Name, enter Countries
    3. Click Accept
  3. For Source connector, select country
  4. Replace the default code by the following one: 

    // Process all nodes
process("") {
 // Link country documents to the correct organization depending on its membership status
 if (it.metas.getValue("ico_status").equals("Member"))
 {
        // create the organization document.
        // This is a managed document, meaning that if no more links are pointing to this object,
        // it deletes itself automatically
        organization = createDocument("organization_ICO", "organization")
        organization.metas.org_id="ICO"
        organization.metas.name="International Coffee Organization"
        organization.directives.datamodel_class = "organization"

        // It is required to "yield" created documents explicitly if they should be pushed to 
        // the aggregation step
        yield organization
        // create the link to the created document
        it.addArcTo("isMemberOf", "organization_ICO");
        } else {
        // create the organization document.
        // This is a managed document, meaning that if no more links are pointing to this object,
        // it deletes itself automatically
        organization = createDocument("organization_NONE", "organization")
        organization.metas.org_id="NONE"
        organization.metas.name="not member"
        organization.directives.datamodel_class = "organization"

        // It is required to "yield" created documents explicitly if they should be pushed to 
        // the aggregation step
        yield organization
        // create the link to the created document
        it.addArcTo("isMemberOf", "organization_NONE");
 }
}

Organization documents are generated from countries. If you delete countries, they are deleted too, automatically.

Configure the Aggregation Processor

  1. Add an aggregation processor:
    1. Select Groovy as format
    2. For Name, enter Organization_UC_6
    3. Click Accept
  2. Replace the default code by the following one: 

    // Process nodes having the “organization” type
process("organization") {
 // Log the content of the document passing through this processor
 log.info "Organization: " + it

 // Add all members of Countries to Organization
 it.metas.members +=
 // Get all paths of related country nodes
 match(it, "-isMemberOf[country]") *.last() // fetch last node
 .collect{n-> n.metas.getValue("name") }
 it.metas.number +=
 // Get all paths to related country nodes
 match(it, "-isMemberOf[country]").size();
}

  3. Save and apply the configuration.

Step 3 - Rescan the Country Connector and Check What Is Indexed

  1. Go to the Home page and under the connectors list, click Clear documents for the country JDBC connector.
  2. Once the clear operation is done, click Scan for the country JDBC connector.
    Wait for data to be fully indexed.
  3. Check that if you click Scan once again for the country JDBC connector, nothing more is pushed to the index.
  4. Go to the analytics page: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/analytics_v1
  5. Select the ICO Membership tab. The tab displays the members count and a list of members.



    Note: You can also check existing arcs in the Index > Consolidation > Introspect tab.

The following graphic shows what we achieved on the object graph at step 3. Arcs (of type isMemberOf) are added to a managed document (called organization_ICO) linked to countries that are part of the ICO.

Step 4 - Update the Membership of a Country

Before you begin: For this operation, you need to access the server.
  1. Go to the <INPUTDIR> containing the coffee sample data.
  2. Change the membership of a country in the coffee database, for example, Brazil.
    1. In your command-line tool, run sqlite3 ./coffee.db
    2. Run the following commands one after the other: 

      delete from countries where country_id="Brazil";
insert into countries(country_id, name, ico_status) values ("Brazil", "Brazil", "Non Member");
.exit

    Note: The insert statement adds the current timestamp to the record automatically. The JDBC connector uses it to detect this modification.

Step 5 - Rescan the Country Connector and Check What Is Indexed

  1. Go to the Home page and under the connectors list, click Scan for the country JDBC connector.
    Wait for data to be fully indexed.
  2. Open the Mashup UI application: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/searchcountry_v3
  3. Search for Brazil, and click its see details link.
    • In the detail page, the ICO Status is Non Member.
    • If you select the Trade volume per Year (Kg) > Table tab, every trade now has its membership updated to Non Member.



  4. Go to the analytics page: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/analytics_v1
  5. Choose the ICO Membership tab.

Brazil is not present in the list anymore.