UC-5: Incremental Scan - Propagating Node Changes

One of the most interesting features of the Consolidation Server is the ability to propagate any node change on related views.

In the coffee sample, we provide an extra year of trade values. When adding new trade values, we want to be sure that countries information is updated accordingly (UC-2 and UC-3).

This task shows you how to:

Before you begin: We assume that previous UCs have been completed.

Step 1 - Set the Trades Connector to Incremental Mode

  1. In the Administration Console, go to Connectors and click the trades JDBC connector.
  2. In Query parameters:
    1. For Synchronization mode, select Query-based incremental synchronization
    2. For Checkpoint query, enter: select max(year) from trade
    3. For Incremental variable, enter: YEAR
    4. For Incremental query, enter: select country_id, type, volume, year from trade where year > "$(YEAR)"
  3. Click Apply.

Step 2 - Rescan the Trades Connector and Check What Is Indexed

  1. Go to the Home page and under the connectors list, click Clear documents for the trades JDBC connector.
  2. Once the clear operation is done, click Scan for the trades JDBC connector.
    Wait for data to be fully indexed.

Check that if you click Scan once again for the trades JDBC connector, nothing more is pushed to the index.

Step 3 - Add a New Year of Trades

In our example, we are going to add the year 1999 to the coffee database.

Before you begin: For this operation, you need to access the server.
  1. Go to the <INPUTDIR> containing the coffee sample data.

    You can find the following files: coffee.db and trades_1999.csv.

  2. Import the year 1999 into the coffee database:
    1. In your command-line tool, run sqlite3 ./coffee.db
    2. Run the following commands one after the other: 

      .separator ";"
.import trades_1999.csv trade
.exit

Step 4 - Rescan the Trades Connector and Check What Is Indexed

  1. Go to the Home page and under the connectors list, click Scan for the trades JDBC connector.
    Wait for data to be fully indexed.
  2. Open the Mashup UI application: http://<HOSTNAME>:<BASEPORT>/mashup-ui/page/searchcountry_v3
  3. Check that countries now have 1999 as last import year.