Oracle Cloud Infrastructure Documentation

Troubleshooting Data Flow

Use troubleshooting information to identify and address common issues that can occur while working with Data Flow.

See also Known Issues.

Resources Exceed Quota Limit

Fix problems where the resources requested exceed to quota limit.

The job doesn't run.

You get a message: The resources you have requested exceed the tenant's Data Flow quota limit.

Cause:

You have exceeded your quota of resources.

Remedy:

  1. In the Oracle Cloud Infrastructure Console, navigate to Governance.
  2. Select Limits, Quotas and Usage.
  3. Click Request a service limit increase, and request more quota for Data Flow.

Spark UI Fails to Load or Returns a 500 Internal Server Error

For in-progress Runs, the Spark UI might fail to load or give a 500 internal server error.

Whilst a Run is in progress, you can't load the Spark UI. It might return a 500 internal server error.

If the Driver is too busy running jobs or there are too many jobs to load, it might take longer for the UI to load and it might time out, resulting in an error.

Use a bigger driver shape

Add more memory and CPU to handle the load by changing to a larger driver shape.

Reduce the retained jobs and stages

By default, spark.ui.retainedjobs and spark.ui.retainedstages are set to 1000. Set the values to a lower number to try to avoid a timeout.

DF-6000: Data Flow Service is Experiencing Capacity Limitation

An application run times out when running with a DF-6000 error.

A run times out with the following state details: DF-6000: Data Flow service is experiencing capacity limitation for <preferred shape family> shape. Please retry in a few minutes, or use a different shape. Please submit a support ticket if you need a limit increase for any shape.

This indicates insufficient compute capacity of the selected shape.

Wait for a few minutes for capacity to become available for the preferred shape, or retry with a different shape.

DF-7001: Your Cloud Tenant has Reached its Data Flow Quota Limit

An application run fails with a DF-7001 error.

A run fails with the following state details: DF-7001: Your cloud tenant has reached its Data Flow quota limit for <preferred shape family> shape. Contact your Administrator to increase the limits then resubmit the run or use a different VM shape.

This indicates that your tenancy has exceeded the quota limit for the preferred shape.

Use a different VM shape, or contact Oracle Support to help increase the quota limits.

DF-7003: The Data Flow Service has Reached its Quota Limit

An application run fails with a DF-7003 error.

A run fails with the following state details: DF-7003: The Data Flow service has reached its quota limit. Please wait for runs to finish or contact Oracle to increase the limits.

This error code indicates that Data Flow has reached its quota limit.

Data Flow might already be working toward increasing the quota limit, but we request you contact Oracle Support to get the current status of when the quota limit is available for use.