UC-7: Generating Child Documents

When flattening data, it is sometimes useful to be able to generate multiple documents from a parent document. These child documents are not pushed by any source but are interesting to simplify queries performed later on the index.

This task shows you how to:


Before you begin: We assume that previous UCs have been completed.

Step 1 - Create Child Documents from Organization with an Aggregation Processor

  1. Add an aggregation processor:
    1. Select Groovy as format
    2. For Name, enter Organization_UC_7
    3. Click Accept
  2. Replace the default code by the following one:

    // Process nodes having the “organization” type
    process("organization") {
     // Log the content of the document passing through this processor
     log.info "Child creation for organization: " + it
    
     // Find the top country for import trade per year
     trades =
     // Get all paths to related country nodes
     match(it, "-isMemberOf[country].import[trade]") *.last()  // fetch last node
     year_top = [:].withDefault() { [:].withDefault() {0} }
     // Big Integer
     def bInt = 0G;
     year_top_volume = [:].withDefault() { bInt }
    
     // Build the child collection
     trades.each {
     trade -> if (year_top[trade.metas.getValue("year")]["volume"] < trade.metas.getValue("volume")
    .toInteger()) {
     year_top[trade.metas.getValue("year")]["volume"] = trade.metas.getValue("volume").toInteger();
     year_top[trade.metas.getValue("year")]["country"] = trade.metas.getValue("country_id");
     year_top_volume[trade.metas.getValue("year")] += trade.metas.getValue("volume").toInteger();
       }
     }
     // Caution! Before pushing any new document, remove existing child documents, if any.
     // This operation is yielded automatically.
     deleteDocumentChildren(it, "/year_import/");
    
     // create child documents
     year_top.each { key, value -> 
          log.info "Year:" + key + " - " +
              value["volume"] + " - " + value["country"] +
              " - " + year_top_volume[key] ;
          child = createChildDocument(
                it, // root
                '/year_import/' + key, // child URI
                "ico_trade" // type
          );
          // Add metas to the child document
         child.metas.parent_identifier = it.getUri();
         child.directives.datamodel_class = "ico_trade";
         // directly set with the type defined in createDocument but it can be overridden if needed
         child.metas.org_id = it.metas["org_id"];
         child.metas.year = key;
         child.metas.country_id = value["country"];
          child.metas.volume = value["volume"];
          child.metas.globalvolume = year_top_volume[key];
          yield child;
     }
    }

  3. Save and apply the configuration.

Step 2 - Relaunch the Organization Aggregation and Check What Is Indexed

  1. Go to the Home page.
  2. Click Force aggregation and enter organization as type.
  3. Open the following Mashup UI application page: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/analytics_v2
  4. Select the ICO Membership tab.

You can now see the Top Import country and Global import volume for each year.

Step 3 - Change the Membership of a Country

Before you begin: For this operation, you need to access the server.
  1. Go to the <INPUTDIR> containing the coffee sample data.
  2. Change the membership of a country in the coffee database, for example, USA.
    1. In your command-line tool, run sqlite3 ./coffee.db
    2. Run the following commands one after the other:

      delete from countries where country_id="USA";
      insert into countries(country_id, name, ico_status) values ("USA", "USA", "Non Member");
      .exit

    Note: The insert statement adds the current timestamp to the record automatically. The JDBC connector uses it to detect this modification.

Step 4 - Rescan the Country Connector and Check What Is Indexed

  1. Click Scan for the country JDBC connector.
    Wait for data to be fully indexed.
  2. Go to the analytics page: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/analytics_v2
  3. Select the ICO Membership tab.

USA is not displayed in the Top Import country column anymore.