Enabling Oracle Cloud Infrastructure Logging Spark Diagnostic Logs

• Console
• CLI
• API

• Follow these steps in the Console to use Oracle Cloud Infrastructure Logging for Spark diagnostic logs in Data Flow.

  Note
  
  Spark Diagnostic Logs (Oracle Cloud Infrastructure Logging Service Logs) can be enabled only at the Data Flow Application level and can't be overridden.

  1. After you have created an Application that uses Oracle Cloud Infrastructure Logging, open the details page for that Application.
  2. Under Resources, click Logs.
  3. Click Diagnostic logs.
  4. Toggle Enable log. The Enable Log panel is displayed.
  5. Select a Compartment.
  6. Select a Log group.
  7. Enter the Log name.
  8. Select the amount of time for Log retention.
  9. Click Enable log.
     Wait for the Status of the Diagnostic logs to be Active. After it's active, any runs using this Application have the diagnostic logs feature turned on for the Spark Driver and Executor logs. The Driver and Executor logs are technically 'Service logs' as they come from an Oracle Cloud Infrastructure native service such as Data Flow.

• Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.

  1. Create a Data Flow Application.

     Request

     oci data-flow application create \
     --compartment-id $COMPARTMENT_ID \
     --display-name "Sample CSV Processing App with OCI Logging Service Logs" \
     --driver-shape "VM.Standard2.1" \
     --executor-shape "VM.Standard2.1" \
     --file-uri "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar" \
     --language "JAVA" \
     --num-executors 1 \
     --spark-version "3.0.2" \
     --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"

     Response

     {
       "data": {
         "archive-uri": "",
         "arguments": null,
         "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample",
         "compartment-id": "<compartment-ocid>",
         "configuration": null,
         "defined-tags": {
           "Dataflow_Resource_Category": {
             "category": "DEV_QA"
           }
         },
         "description": null,
         "display-name": "Sample CSV Processing App with OCI Logging Service Logs",
         "driver-shape": "VM.Standard2.1",
         "execute": null,
         "executor-shape": "VM.Standard2.1",
         "file-uri": "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar",
         "freeform-tags": {},
         "id": "<data-flow-application-ocid>",
         "language": "JAVA",
         "lifecycle-state": "ACTIVE",
         "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/",
         "metastore-id": null,
         "num-executors": 1,
         "owner-principal-id": "<user-ocid>",
         "owner-user-name": "<user-name>",
         "parameters": null,
         "private-endpoint-id": "",
         "spark-version": "3.0.2",
         "time-created": "2022-04-08T19:50:57.795000+00:00",
         "time-updated": "2022-04-08T19:50:57.795000+00:00",
         "warehouse-bucket-uri": null
       }
     }

  2. Create a log group.

     Request

     oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"

     Response

     {
       "opc-work-request-id": "<log-workrequest-ocid>"
     }

  3. Retrieve the log group ID.

     Request

     export LOG_GROUP_WR_ID=<log-workrequest-ocid>
     oci logging work-request get --work-request-id $LOG_GROUP_WR_ID

     Response

     {
       "data": {
         "compartment-id": "<compartment-ocid>",
         "id": "<log-workrequest-ocid>",
         "operation-type": "CREATE_LOG_GROUP",
         "percent-complete": 100.0,
         "resources": [
           {
             "action-type": "CREATED",
             "entity-type": "loggroup",
             "entity-uri": "/logGroups/<log-group-ocid>",
             "identifier": "<log-group-ocid>"
           }
         ],
         "status": "SUCCEEDED",
         "time-accepted": "2022-04-08T19:53:57.065000+00:00",
         "time-finished": "2022-04-08T19:53:57.065000+00:00",
         "time-started": "2022-04-08T19:53:57.065000+00:00"
       }
     }

  4. Enable diagnostic (service) logs on the Data Flow Application.

     Request

     export LOG_GROUP_ID=<log-group-ocid>
     oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json

     config.json

     {
         "compartment-id": "<compartment-ocid>",
         "source": {
             "resource": "<data-flow-application-ocid>",
             "service": "dataflow",
             "source-type": "OCISERVICE",
             "category": "diagnostic"
         }
     }

     Response

     {
       "opc-work-request-id": "<log-workrequest-ocid>"
     }

  5. Retrieve the log ID for the diagnostic (service) logs.

     Request

     export LOG_WR_ID=<log-workrequest-ocid>
     oci logging work-request get --work-request-id $LOG_WR_ID

     Response

     {
       "data": {
         "compartment-id": "<compartment-ocid>",
         "id": "<log-workrequest-ocid>",
         "operation-type": "CREATE_LOG",
         "percent-complete": 100.0,
         "resources": [
           {
             "action-type": "CREATED",
             "entity-type": "log",
             "entity-uri": "/logs/<log-ocid>",
             "identifier": "<log-ocid>"
           }
         ],
         "status": "SUCCEEDED",
         "time-accepted": "2022-04-08T20:12:29.660000+00:00",
         "time-finished": "2022-04-08T20:12:33.356000+00:00",
         "time-started": "2022-04-08T20:12:29.660000+00:00"
       }
     }

• Follow these steps in the API to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.

  1. Create a Data Flow Application.

     Request

     oci data-flow application create \
     --compartment-id $COMPARTMENT_ID \
     --display-name "Sample CSV Processing App with OCI Logging Service Logs" \
     --driver-shape "VM.Standard2.1" \
     --executor-shape "VM.Standard2.1" \
     --file-uri "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar" \
     --language "JAVA" \
     --num-executors 1 \
     --spark-version "3.0.2" \
     --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"

     Response

     {
       "data": {
         "archive-uri": "",
         "arguments": null,
         "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample",
         "compartment-id": "<compartment-ocid>",
         "configuration": null,
         "defined-tags": {
           "Dataflow_Resource_Category": {
             "category": "DEV_QA"
           }
         },
         "description": null,
         "display-name": "Sample CSV Processing App with OCI Logging Service Logs",
         "driver-shape": "VM.Standard2.1",
         "execute": null,
         "executor-shape": "VM.Standard2.1",
         "file-uri": "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar",
         "freeform-tags": {},
         "id": "data-flow-application-ocid>",
         "language": "JAVA",
         "lifecycle-state": "ACTIVE",
         "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/",
         "metastore-id": null,
         "num-executors": 1,
         "owner-principal-id": "<user-ocid>",
         "owner-user-name": "<user-name>",
         "parameters": null,
         "private-endpoint-id": "",
         "spark-version": "3.0.2",
         "time-created": "2022-04-08T19:50:57.795000+00:00",
         "time-updated": "2022-04-08T19:50:57.795000+00:00",
         "warehouse-bucket-uri": null
       }
     }

  2. Create a log group.

     Request

     oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"

     Response

     {
       "opc-work-request-id": "<log-workrequest-ocid"
     }

  3. Retrieve the log group ID.

     Request

     export LOG_GROUP_WR_ID=<log-workrequest-ocid
     oci logging work-request get --work-request-id $LOG_GROUP_WR_ID

     Response

     {
       "data": {
         "compartment-id": "<compartment-ocid>",
         "id": "<log-workrequest-ocid",
         "operation-type": "CREATE_LOG_GROUP",
         "percent-complete": 100.0,
         "resources": [
           {
             "action-type": "CREATED",
             "entity-type": "loggroup",
             "entity-uri": "/logGroups/<log-group-ocid>",
             "identifier": "<log-group-ocid>"
           }
         ],
         "status": "SUCCEEDED",
         "time-accepted": "2022-04-08T19:53:57.065000+00:00",
         "time-finished": "2022-04-08T19:53:57.065000+00:00",
         "time-started": "2022-04-08T19:53:57.065000+00:00"
       }
     }

  4. Enable diagnostic (service) logs on the Data Flow Application.

     Request

     export LOG_GROUP_ID=<log-group-ocid>
     oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json

     config.json

     {
         "compartment-id": "<compartment-ocid>",
         "source": {
             "resource": "data-flow-application-ocid>",
             "service": "dataflow",
             "source-type": "OCISERVICE",
             "category": "diagnostic"
         }
     }

     Response

     {
       "opc-work-request-id": "<log-workrequest-ocid"
     }

  5. Retrieve the log ID for the diagnostic (service) logs.

     Request

     export LOG_WR_ID=<log-workrequest-ocid
     oci logging work-request get --work-request-id $LOG_WR_ID

     Response

     {
       "data": {
         "compartment-id": "<compartment-ocid>",
         "id": "<log-workrequest-ocid",
         "operation-type": "CREATE_LOG",
         "percent-complete": 100.0,
         "resources": [
           {
             "action-type": "CREATED",
             "entity-type": "log",
             "entity-uri": "/logs/<log-ocid>",
             "identifier": "<log-ocid>"
           }
         ],
         "status": "SUCCEEDED",
         "time-accepted": "2022-04-08T20:12:29.660000+00:00",
         "time-finished": "2022-04-08T20:12:33.356000+00:00",
         "time-started": "2022-04-08T20:12:29.660000+00:00"
       }
     }