Oracle Cloud Infrastructure Documentation

Control Pipelines (Start, Stop, Drop, or Reset a Pipeline)

After you create and test a pipeline, you control a pipeline by starting, stopping, or dropping the pipeline. You can also reset a pipeline.

  • Start a Pipeline
    After you create a pipeline, you can start the pipeline.
  • Stop a Pipeline
    Use STOP_PIPELINE to stop a pipeline. When a pipeline is stopped, no future jobs are scheduled for the pipeline.
  • Drop a Pipeline
    The procedure DROP_PIPELINE drops an existing pipeline.
  • Reset a Pipeline
    Use the reset pipeline operation to clear the record of the pipeline to the initial state.

Parent topic: Using Data Pipelines for Continuous Load and Export

Start a Pipeline

After you create a pipeline, you can start the pipeline.

When a pipeline is started the pipeline runs continuously in a scheduled job. The pipeline's scheduled job repeats, either by default every 15 minutes or at the interval you set with the interval attribute.

  1. Start a pipeline.
    BEGIN
  DBMS_CLOUD_PIPELINE.START_PIPELINE(
      pipeline_name => 'EMPLOYEE_PIPELINE'
  );
END;
/

    By default, a pipeline job begins immediately, as soon as the pipeline is started. To start a pipeline job at a later time, specify a valid future date or timestamp using the start_date parameter.

    See START_PIPELINE Procedure for more information.

  2. Verify that the pipeline is started.

    For example:

    
SELECT pipeline_name, status from USER_CLOUD_PIPELINES
   WHERE pipeline_name = 'EMPLOYEE_PIPELINE';

PIPELINE_NAME            STATUS  
------------------------ ------- 
EMPLOYEE_PIPELINE        STARTED

Stop a Pipeline

Use STOP_PIPELINE to stop a pipeline. When a pipeline is stopped, no future jobs are scheduled for the pipeline.

By default currently running jobs complete when you stop a pipeline. Set the force parameter to TRUE to terminate any running jobs and stop the pipeline immediately.

  1. Stop a pipeline.
    BEGIN
  DBMS_CLOUD_PIPELINE.STOP_PIPELINE(
      pipeline_name => 'EMPLOYEE_PIPELINE'
  );
END;
/

    See STOP_PIPELINE Procedure for more information.

  2. Verify that the pipeline is stopped.
    
SELECT pipeline_name, status from USER_CLOUD_PIPELINES
   WHERE pipeline_name = 'EMPLOYEE_PIPELINE';

PIPELINE_NAME            STATUS  
------------------------ ------- 
EMPLOYEE_PIPELINE        STOPPED

See STOP_PIPELINE Procedure for more information.

Drop a Pipeline

The procedure DROP_PIPELINE drops an existing pipeline.

If a pipeline has been started, it must be stopped before the pipeline can be dropped. See STOP_PIPELINE Procedure for more information.

In order to drop a pipeline that is started, set the force parameter to TRUE to terminate any running jobs and drop the pipeline immediately

  1. Drop a pipeline.
    BEGIN
  DBMS_CLOUD_PIPELINE.DROP_PIPELINE(
      pipeline_name => 'EMPLOYEE_PIPELINE'
  );
END;
/
  2. Verify the pipeline is dropped.
    SELECT pipeline_name, status from USER_CLOUD_PIPELINES
   WHERE pipeline_name = 'EMPLOYEE_PIPELINE';

No rows selected

See DROP_PIPELINE Procedure for more information.

Reset a Pipeline

Use the reset pipeline operation to clear the record of the pipeline to the initial state.
Note

You can optionally use reset pipeline to purge data in the database table associated with a load pipeline or to remove files in object store for an export pipeline. Usually this option is used when you are testing a pipeline during pipeline development.

Reset pipeline operates as follows:

  • Load Pipeline: For a load pipeline, resetting the pipeline clears the record of the files being loaded by the pipeline. When you call either START_PIPELINE or RUN_PIPELINE_ONCE after resetting a load pipeline, the pipeline repeats the data load and includes all the files present in the object store location.

    When purge_data is set to TRUE, the procedure truncates the data in the database table.

  • Export Pipeline: For an export pipeline, resetting the pipeline clears the last tracked data in the database table. When you call either START_PIPELINE or RUN_PIPELINE_ONCE after resetting an export pipeline, the pipeline repeats exporting data from the table or query.

    When purge_data set to TRUE, the procedure deletes existing files in the object store location specified with the location attribute.

To reset a pipeline:

  1. Stop the pipeline you want to reset.

    A data pipeline must be in stopped state to reset it. See STOP_PIPELINE Procedure for more information.

  2. Reset the pipeline.
    BEGIN
     DBMS_CLOUD_PIPELINE.RESET_PIPELINE(
        pipeline_name => 'EMPLOYEE_PIPELINE',
        purge_data => TRUE);
END;
/

    Only use the purge_data parameter with value TRUE if you want to clear data in your database table, for a load pipeline, or clear files in object store for an export pipeline. Usually this option is used when you are testing a pipeline during pipeline development.

    See RESET_PIPELINE Procedure for more information.