Enabling Logs in Oracle Cloud Infrastructure Logging
Data Integration lets you use Oracle Cloud Infrastructure Logging to enable logging and retrieve service logs in the Console or from the CLI.
Public logs are available for the following types:
- Execution logs for different task runs including integration, data loader, REST, SQL, and pipeline task runs
- Create, edit, and delete data flow logs
Only administrative users can enable logging. Non-administrative users can only view log contents, and only after administrative users have enabled and set up logging, and given you access to logging groups and content.
Currently, Data Integration service logs are enabled at the workspace resource level using the DIS Workspace Logs
log category. For example, when logging is enabled and active, integration task runs associated with data flows in a specified Data Integration workspace have the logging feature turned on.
Resource logs are grouped in logical containers called log groups in Oracle Cloud Infrastructure Logging. Administrative users must create at least one log group.
Before you start using Oracle Cloud Infrastructure Logging:
-
Get familiar with basic concepts and terminology used in the OCI Logging service. See Logging Overview in the OCI Logging documentation.
-
Create a group to manage access to log groups and log content. See Managing Groups in the OCI IAM with Identity Domains documentation.
-
Add the policy to enable public logging for the Data Integration service.
Note
To restrict the enable permission to specific users, replace
any-user
with a specific user.ALLOW any-user TO {DIS_WORKSPACE_OBJECT_CREATE} in tenancy WHERE ALL {request.service.name = 'dataintegration', request.principal.type = 'service'}
-
Add the policies to allow you to use OCI Logging with the Data Integration service.
allow group <group-name> to manage log-groups in compartment <compartment-name> allow group <group-name> to manage log-content in compartment <compartment-name>
-
Ensure that you have the policies that are required to use OCI Logging. See Required Permissions for Working with Logs and Log Groups in the OCI Logging documentation.
-
In the OCI Logging service, create one or more log groups to organize and manage logs in logical containers. See Creating a Log Group in the OCI Logging documentation.
Enabling Logs in the Console
Follow these steps in the Console to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging. Only administrative users can enable logging.
Currently, Data Integration service logs are enabled at the workspace level. To enable logging, you need to know the workspace and the name of the compartment where the workspace is located.
Enabling Logs in the CLI
Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging for a workspace.
Example of a Data Integration Public Log
The following is an example of a Data Integration service public log in JSON format.
The Data Integration fields in the output are:
Field | Description |
---|---|
|
The level of the log. For example: |
|
The actual log message. |
|
The The Id is the Universal Unique Identifier (UUID) of the object for which the log is generated. For example, for a task run, it's the |
|
The |
|
The workspace OCID. |
{
"datetime": 1660885646612,
"logContent": {
"data": {
"logLevel": "INFO",
"message": "ACTION: INTEGRATION_TASK-TASK-RUN- Submit on TASK b7fba0c4-bf0d-4c29-b49f-b21673cc2791 - SUCCESS",
"objectId": "b7fba0c4-bf0d-4c29-b49f-b21673cc2791",
"opcRequestId": "csidb3d7aef846ab99c53689c81107f0/f11c0c8e208a49e8a1f79176bdf611a6/D4927163D0EAC4B699CD3076A17C5FD0",
"workspaceId": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>"
},
"id": "2d0da51e-e02d-4ba2-b766-6255ed1b1269",
"oracle": {
"compartmentid": "ocid1.tenancy.oc1..<long-string-letters-12345>",
"ingestedtime": "2022-08-19T05:07:55.326Z",
"loggroupid": "ocid1.loggroup.oc1.phx.<long-string-letters-12345>",
"logid": "ocid1.log.oc1.phx.<long-string-letters-12345>",
"tenantid": "ocid1.tenancy.oc1..<long-string-letters-12345>"
},
"source": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>",
"specversion": "1.0",
"time": "2022-08-19T05:07:26.612Z",
"type": "public.log.dis.execute"
}
}
Tips for Searching Data Integration Logs
Use the OCI Logging Search page to search logs.
For information about how to use the Search page, see Logging Search and Viewing and Working with Search Results in the OCI Logging documentation.
Before you perform a task run (execution) log search, copy the relevant task run key in Data Integration by using the Actions menu () as described in Viewing Task Runs.
Use the following tips when searching Data Integration logs:
- Limit results to a narrower or specific date and time range to optimize search time.
- In the results table, logs with the
public.log.dis.execute
type are logs from Data Integration. For better viewing, however, click X to remove the type column so that the data.message column is wider. - If a large log message is truncated in the data.message column (such as a
SYSTEM.ERROR_MESSAGE
), place the cursor over the message to see the full message in a box that pops open. -
For a pipeline with a nested pipeline, search for the root pipeline task run key to retrieve logs for the pipeline and all its child tasks (including the nested pipeline). In such a pipeline execution log, a child task run in the nested pipeline is identified by this format string:
[root pipeline task run key]/[parent pipeline task run key]/[child task run key]
-
To see more detailed logs for a data loader child task run or integration task log run, copy the value of the child task run key and search for it in a separate search.