Create and Configure Pipelines
You can create one or more load or export pipelines. When you create a pipeline, you use parameters and set pipeline attributes to configure the pipeline.
The options to create and configure a pipeline follow:
-
Load Pipeline:
-
Export Pipeline:
-
Export incremental results of a query to object store using a date or timestamp column as key for tracking newer data. See Create and Configure a Pipeline for Export with Timestamp Column.
-
Export incremental data of a table to object store using a date or timestamp column as key for tracking newer data. See Create and Configure a Pipeline for Export with Timestamp Column.
-
Export data of a table to object store using a query to select data without a reference to a date or timestamp column (so that the pipeline exports all the data that the query selects for each scheduler run). See Create and Configure a Pipeline to Export Query Results (Without a Timestamp).
-
- Create and Configure a Pipeline for Loading Data
You can create a pipeline to load data from external files in object store to tables in Autonomous Database. - Create and Configure a Pipeline for Export with Timestamp Column
You can create an export pipeline to automatically export time-series data from your Autonomous Database to object store. - Create and Configure a Pipeline to Export Query Results (Without a Timestamp)
You can create an export pipeline to automatically export data from your Autonomous Database to object store. Using this export pipeline option you specify a SQL query that the pipeline runs periodically to export data to object store. You can use this export option to share the latest data from your Autonomous Database to object store for other applications to consume the data.
Parent topic: Using Data Pipelines for Continuous Load and Export
Create and Configure a Pipeline for Loading Data
You can create a pipeline to load data from external files in object store to tables in Autonomous Database.
A load pipeline consumes data placed on object store and loads it to a table in Autonomous Database. When you create a load pipeline, the pipeline runs at regular intervals to consume data placed on object store, when new data files arrive the pipeline loads the new data. You can also use a pipeline to reliably copy files, with resume and retry capabilities, from object store to a table on your database.
With a load pipeline, the pipeline package uses DBMS_CLOUD.COPY_DATA
to load data.
On your Autonomous Database, either use an existing table or create the database table where you are loading data. For example:
CREATE TABLE EMPLOYEE
(name VARCHAR2(128),
age NUMBER,
salary NUMBER);
As an alternative, to set the format for JSON, you could use the following format:
BEGIN
DBMS_CLOUD_PIPELINE.SET_ATTRIBUTE
(
pipeline_name => 'MY_PIPE1',
attribute_name => 'format',
attribute_value => JSON_OBJECT('type' value 'json', 'columnpath' value '["$.NAME", "$.AGE", "$.SALARY"]')
);
END;
/
Parent topic: Create and Configure Pipelines
Create and Configure a Pipeline for Export with Timestamp Column
Using this export pipeline option you specify a table or SQL query and a column with a timestamp that the pipeline uses to keep track of the time of the last upload. You can use an export pipeline to share data for consumption by other applications or to save data to object store.
With an export pipeline, the pipeline package uses DBMS_CLOUD.EXPORT_DATA
to export
data.
An export pipeline exports data from your Autonomous Database to object store. When you create an export pipeline, the pipeline runs at regular intervals and places data on object store.
Parent topic: Create and Configure Pipelines
Create and Configure a Pipeline to Export Query Results (Without a Timestamp)
You can create an export pipeline to automatically export data from your Autonomous Database to object store. Using this export pipeline option you specify a SQL query that the pipeline runs periodically to export data to object store. You can use this export option to share the latest data from your Autonomous Database to object store for other applications to consume the data.
An export pipeline exports data from your Autonomous Database to object store. When you create an export pipeline, the pipeline runs at regular intervals and places data on object store.
Parent topic: Create and Configure Pipelines