Working with OCI Data Flow Tasks
An Oracle Cloud Infrastructure Data Flow task lets you schedule, run, and monitor an OCI Data Flow application from within Data Integration.
An application that's created in OCI Data Flow consists of a Spark application and version, dependencies, default parameters, and a default runtime resource specification.
Creating and running OCI Data Flow tasks in Data Integration requires relevant permissions and IAM policies to access applications in OCI Data Flow. For details, see Required Policies and Setup.
The following pages describe how you can create, edit, and delete OCI Data Flow tasks in Data Integration:
The following pages describe other management tasks that can be performed after an OCI Data Flow task is created:
Required Policies and Setup
Before you create an OCI Data Flow task, use the following task checklist to ensure that you have the required setup and information you need for using OCI Data Flow tasks in Data Integration.
Task | Requirement |
---|---|
Obtain access to Oracle Cloud Infrastructure Data Flow |
This topic assumes you have already set up what you need to use OCI Data Flow and create applications in OCI Data Flow. Getting Started with OCI Data Flow The OCI Data Flow task you create in Data Integration is associated to an application you create in OCI Data Flow. |
Create an application in OCI Data Flow |
To use an OCI Data Flow task in Data Integration, you must have already created the application in OCI Data Flow for the language you want. |
Obtain the details of an application in OCI Data Flow |
Gather the following details of the application you created in OCI Data Flow:
|
Create policies |
To allow Data Integration to create and access applications in OCI Data Flow:
To allow Data Integration to trigger OCI Data Flow tasks with metastore configured, create two dynamic groups, and create a rule for each dynamic group: Create the following matching rule in
Create the following matching rule in
Then add the following policies:
|
-
The policy statements provided in this topic are examples only. Ensure that you write policies that meet your own requirements.
-
Cross-tenancy policies are required if the resources (such as Object Storage objects and buckets) and the Data Integration workspace are on different tenancies. See Policy Examples and the blog Policies in Oracle Cloud Infrastructure (OCI) Data Integration to identify the policies for your needs.
-
After you add IAM components (for example, dynamic groups and policy statements), don't try to perform the associated tasks immediately. New IAM policies require about five to 10 minutes to take effect.