Starting a Pipeline Run

Use a Data Science pipeline run to run the code of the pipeline steps according to the workflow defined in a pipeline.

Pipeline creation sets the infrastructure and the actual use case steps, but the pipeline run runs the pipeline with the specified parameters. A pipeline run provisions the specified infrastructure, runs the pipeline steps, and then deprovisions and destroys the used resources when the pipeline ends.

You can view the pipeline steps, runs, and logs if they exist.

1. Use the Console to sign in to a tenancy with the necessary policies.
2. Open the navigation menu and click Analytics & AI. Under Machine Learning, click Data Science.
3. Select the compartment that contains the project with the pipeline.
  
  All projects in the compartment are listed.
4. Click the name of the project.
  
  The project details page opens and lists the notebook sessions.
5. Under Resources, click Pipelines.
6. Select a pipeline to work with.
7. Click Start a pipeline run.
8. (Optional) Select a different compartment for the pipeline.
9. (Optional) You can override the default pipeline configuration that was defined when the pipeline was created by using these options:
  
  Custom environment variable key
  
  The environment variables to control the pipeline.
  
  Value
  
  The value for the custom environment variable key.
  
  You can click Additional custom environment key to specify more variables.
  
  Command line arguments
  
  The command line arguments that you want to use for running the pipeline.
  
  Maximum runtime (in minutes)
  
  The maximum number of minutes that the pipeline can run. The service cancels the pipeline run if its runtime exceeds the specified value. The maximum runtime is 30 days (43,200 minutes). We recommend that you configure a maximum runtime on all pipeline runs to prevent runaway pipeline runs.
10. (Optional) You can override each pipeline step configuration by clicking Edit in each step box, then changing the options. and then click Save. See the description of the fields in step 9.
11. (Optional) You can override the logging configuration. You can turn logging on or off. When logging is on, you can change the log group by clicking Select. Click Select to save your changes.
12. (Optional) Click Show advanced options to set load balancing, use a custom container image, and tags.
13. (Optional) Enter the tag namespace (for a defined tag), key, and value to assign tags to the resource.
  
  To add more than one tag, click Add tag.
  
  Tagging describes the various tags that you can use organize and find resources including cost-tracking tags.
14. Click Start.
  
  The pipeline run is in the Acceptedstate until the run begins, and then it changes to In Progress. When the run finishes, it's either Succeeded or Failed.
  
  The status of each pipeline step is listed. Pipeline steps are in the Waiting state until they run, and then they change to In Progress. When a step finishes, it's either Succeeded or Failed.

These environment variables control the pipeline.

You can use the OCI CLI to start pipeline runs as in this example:

Start a pipeline run with:

oci data-science pipeline-run create \
--display-name <pipeline_run_name> \
--compartment-id <compartment_ocid> \
--project-id <project_ocid> \
--pipeline-id <pipeline_ocid> \
--configuration-override-details file://<optional_pipeline_run_configuration_override_json_file> \
--log-configuration-override-details file://<optional_pipeline_run_logging_configuration_override_json_file>

(Optional) Use this pipeline run configuration override JSON file to override the configurations defined on the parent pipeline:

{
  "pipelineType": "DEFAULT",
  "maximumRuntimeInMinutes": 240,
  "commandLineArguments" : "test-arg",
  "environmentVariables": {
    "SOME_ENV_KEY": "<some_env_value_override>" 
  }
}

(Optional) Use this pipeline run logging configuration override JSON file to override the logging configuration defined on the parent pipeline:
```
{
  "enableLogging": true,
  "enableAutoLogCreation": true,
  "logGroupId": "<log_group_ocid>"
}
```

Use the CreatePipelineRun operation create (start) a pipeline run.

Use the CreatePipelineRun operation create (start) a pipeline run.

Oracle Cloud Infrastructure Documentation

Starting a Pipeline Run