Copying Object Data

You can configure a pipeline to copy objects from a storage to another. It is useful to schedule the copy of binary objects into an internal DFS Object Storage bucket.

This task shows you how to:

See Also
Main Workflows
Configuring Object Storage Buckets
Exporting and Importing a Pipeline Configuration
Configuring the Data Storage

Add a New Pipeline

Define one pipeline per object type to copy.

Before you begin: You have created an DFS Object Storage bucket storage.
  1. From the left panel, select Storage.
  2. Click Add pipeline.
    The Add new pipeline dialog box opens to let you create a new pipeline or import an existing one (see Exporting and Importing a Pipeline Configuration).
  3. Click Create pipeline.
  4. Enter a Pipeline name.
  5. For Pipeline type, select Copy Object Data.
  6. Click Create.
    The General information page opens.
    Recommendation: Click Save after any editing operation.

Configure the Source

Once the pipeline is created, configure the source, that is the connection to the storage on which Data Factory Studio must upload object files.

  1. Select the Source tab.
  2. Configure the Storage parameters.

    Parameter Description
    Storage

    Defines in which S3 bucket Data Factory Studio fetches source files.

    Folder path prefix

    Defines the folder path where Data Factory Studio must fetch source files.

    Important: Do not forget to add a slash / at the end of the path.

  3. Optional: If lineage with Datasets Governance is activated, you see a Source Lineage section on the page. In that case, you must select a catalog previously defined in Datasets Governance from Source Catalog.

    Important: This is one of the steps to configure the interaction with Datasets Governance. For more info, see Configuring a Lineage with Datasets Governance.

  4. Configure the Scheduling, that is the refresh frequency of the source to fetch.

    By default, Data Factory Studio refreshes data every minute. You can specify longer intervals, like every 10 minutes, every hour, every day, etc. You can also specify the source scheduling to run continuously, which means that each time an S3 fetch request comes to an end, a new one starts automatically.

    You can define your own custom source scheduling using a Quartz Cron Expression. For example:

    • "0 0 13 * * ?" indicates that you want to schedule the processor to run at 1:00 PM every day.
    • "0 20 14 ? * MON-FRI" indicates that you want to schedule the processor to run at 2:20 PM from Monday to Friday.
    • "0 15 10 ? * 6L 2011-2017" indicates that you want to schedule the processor to run at 10:15 AM, on the last Friday of every month, between 2011 and 2017.
    For more information on Quartz format, see https://www.quartz-scheduler.org/documentation/

  5. Click Save.

Configure the Destination

Define in which object storage, the ingestion pipeline must push copied data.

  1. Select the Destination tab.
  2. Enter the Storage where you want to push data.
  3. Select an Object renaming policy to store data in the target object storage.
  4. Click Save.

Run the Pipeline Configuration

Once the configuration is complete, you can apply it to start indexing documents.

  1. Select the Run tab.
    The running status of the indexing displays at the top of the screen.
  2. To start the pipeline indexing, click Run.

    You can stop the indexing at any time and restart it when ready.

    Important: Data Factory Studio blocks the Run action if there is any unsaved modification.