Configuring a Lineage with Datasets Governance

You can configure Data Factory Studio to work with the dataset catalogs defined in Datasets Governance.

This task shows you how to:

See Also
Main Workflows
Indexing Tabular Data
Indexing Event Data
Copying Object Data
Configuring the Data Storage

Prepare the Data Source Lineage in Datasets Governance

First prepare the data source lineage in Datasets Governance, by creating the catalog that will contain the datasets resulting from the Data Factory Studio pipeline (as explained in the next section).

  1. Open Datasets Governance.
  2. Optional: Create a storage class.
  3. Optional: Create a service in the storage class.
  4. Required: Create a catalog to register the data source.

    For example, if the data source is a CSV file in an S3 Storage, you must create a catalog mapping the S3 bucket folder containing this CSV file. In the following screenshot Data Efficiency is a folder on the S3.

Configure the Lineage in Data Factory Studio

Before you begin: A catalog is configured in Datasets Governance.
  1. Open Data Factory Studio.
  2. From the left panel, select Ingestion.
  3. Select a pipeline (or create a new pipeline), and go to the Lineage tab.
  4. Turn on Activate Lineage.
  5. Go to the Source tab.
  6. In the Source Lineage section, from Source Catalog, select the catalog previously defined in Datasets Governance (see Prepare the Data Source Lineage in Datasets Governance.)

  7. Go to the Run tab and run the pipeline.

If you go back to Datasets Governance, you can see the output created in your catalog.

For each source file, Data Factory Studio registered a dataset in the source catalog.

If the source is an S3 storage, the created datasets have the following name pattern: [bucketname]:[folder1]/[folder2]/.../[file-name].[file-version].[file-extension]

For each pipeline operation, Data Factory Studio registers an Index activity in Datasets Governance, using one or several source datasets and generating the output dataset.

For each output created by the Data Factory Studio pipeline, a dataset can also be generated. If the pipeline task is indexing in an index unit storage, Data Factory Studio registers a dataset in the index unit content catalog.

Note: When Data Factory Studio detects a new file version (for example, a new version of a CSV file in the S3), it creates a new dataset, and invalidates all the previous versions of this dataset.