Indexing Event Data

You can push event orders, serialized in JSON files in an external S3, into an index unit. For this type of ingestion, the source can only be an S3, and the target an index unit.

Events can add or remove data in an index unit.

Event Type Supports Event Orders
Data Events

Add or replace Item

Delete Item

Delete Item with predicate

This task shows you how to:

See Also
Main Workflows
JSON Source for Event Data
Configuring a Lineage with Datasets Governance
Configuring the Data Storage

Add a New Pipeline

Define one pipeline per Data Model class.

Before you begin: You have created an index unit.
  1. From the left panel, select Storage.
  2. Click Add pipeline.
    The Add new pipeline dialog box opens to let you create a new pipeline or import an existing pipleine (see Exporting and Importing a Pipeline Configuration.)
  3. Click Create pipeline.
  4. Enter a Pipeline name.
  5. For Pipeline type, select Index Event Data.
  6. Click Create.
    The General information page opens.
    Recommendation: Click Save after any editing operation.

Configure the Source

Once the pipeline is created, configure the source, that is the connection to the storage on which Data Factory Studio must fetch source files.

  1. Select the Source tab.
  2. Configure the Storage parameters.

    Parameter Description
    Bucket name

    Defines in which S3 bucket Data Factory Studio fetches source files.

    Folder path prefix

    Defines the folder path where Data Factory Studio must fetch source files.

    Important: Do not forget to add a slash / at the end of the path.

  3. Optional: If lineage with Datasets Governance is activated, you see a Source Lineage section on the page. In that case, you must select a catalog previously defined in Datasets Governance from Source Catalog.

    Important: This is one of the steps to configure the interaction with Datasets Governance. For more info, see Configuring a Lineage with Datasets Governance.

  4. Configure the Scheduling, that is the refresh frequency of the source to fetch.

    By default, Data Factory Studio refreshes data every minute. You can specify longer intervals, like every 10 minutes, every hour, every day, etc. You can also specify the source scheduling to run continuously, which means that each time an S3 fetch request comes to an end, a new one starts automatically.

    You can define your own custom source scheduling using a Quartz Cron Expression. For example:

    • "0 0 13 * * ?" indicates that you want to schedule the processor to run at 1:00 PM every day.
    • "0 20 14 ? * MON-FRI" indicates that you want to schedule the processor to run at 2:20 PM from Monday to Friday.
    • "0 15 10 ? * 6L 2011-2017" indicates that you want to schedule the processor to run at 10:15 AM, on the last Friday of every month, between 2011 and 2017.
    For more information on Quartz format, see https://www.quartz-scheduler.org/documentation/

  5. Click Save.

Configure the Processing

Advanced users, already familiar with Semantic Graph Index Data Queries can optionally define a MAP expression in the Processing tab.

The expression input must be a tuple, called record, with your data source values. For more information, see Appendix - About Processing.

Configure the Destination

Define in which Data Model class the ingestion pipeline must push fetched data.

Notes: You cannot use the 3DSpace as destination target. No pipeline can modify it.

  1. Select the Destination tab.
  2. Enter the Index unit where you want to push data.
  3. Click Save.

Run the Pipeline Configuration

Once the configuration is complete, you can apply it to start indexing documents.

  1. Select the Run tab.
    The running status of the indexing displays at the top of the screen.
  2. To start the pipeline indexing, click Run.

    You can stop the indexing at any time and restart it when ready.

    Important: Data Factory Studio blocks the Run action if there is any unsaved modification.