Ingest Logs from Other OCI Services Using Service Connector
You can analyze the logs to troubleshoot issues, monitor health and performance and observe the operational tasks in Oracle Cloud Infrastructure services by ingesting the logs into Oracle Logging Analytics.
Use the Service Connector to identify your Oracle Cloud Infrastructure service as the source of the logs and Oracle Logging Analytics as the destination. For information on how the Service Connector Hub works, see Service Connector Hub Overview in Oracle Cloud Infrastructure Documentation.
After the service connector is created, an entity is automatically created for processing the logs. To ensure proper log collection, the entity must not be deleted.
In case of Oracle Operator Access Control Logs, the entity is not automatically created. To create an entity, see Create an Entity to Represent Your Log-Emitting Resource.
Topics
Important: Oracle recommends that you use the data ingestion work flow available in Logging Analytics console to quickly ingest logs from other OCI services. Go to Logging Analytics Home or Log Explorer, click Compass, and click Add Data.
- For all types of logs from OCI services except OCI Audit Logs and IDCS Audit Logs, expand the section Monitor OCI resources and click Configure log collection for OCI resources.
- In case of OCI Audit Logs or IDCS Audit Logs, expand the section Security and Compliance and click the logs of your choice. In this work flow, all the required resources like policies, log group, and service connector are automatically created.
Follow the intuitive steps in the work flow to start ingesting logs. As a prerequisite, ensure that you have the required permissions to complete the steps. For a quick walk through of the steps, watch Video: How to Quickly Ingest Logs into Logging Analytics from Other OCI Services in Oracle Cloud Observability and Management Platform.
Alternatively, you can manually set up the log collection by performing the following steps:
Additional Information
-
List of Oracle-defined sources for collecting logs: For the list of Oracle-defined sources to collect logs from Oracle Cloud Infrastructure services, see Oracle-defined Sources and search for sources with title OCI...
-
Types of service logs you can collect: For the types of logs you can collect from the Oracle Cloud Infrastructure services, the parsers, example log content, fields, and JSON path, see OCI Parser Details.
-
Filter logs collected though service connector: The service connector OCID is mapped to the field
Log Origin
. To view the logs flowing from that service connector to Oracle Logging Analytics, filter the logs by the fieldLog Origin
. See Filter Logs by Pinned Attributes and Fields.
Allow Collection of Logs from OCI Logging Service
Based on the type of service logs that you want to ingest, you must create policies to enable Oracle Logging Analytics to get the information about the resources and create an entity for each resource.
After you create the policy, the entity that is created will be auto-associated with all the logs collected from that resource. If the policy is not created, then the logs are still ingested but the entity is not created.
The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements:
allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID>
where all
{request.principal.type = 'serviceconnector',
target.loganalytics-log-group.id = '<Log_Group_OCID>',
request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <userGroup> to MANAGE serviceconnectors in tenancy
allow group <userGroup> to READ logging-family in tenancy
In the above policy statements,
-
Log_Group_Compartment_OCID
: The compartment OCID of the log group in Oracle Logging Analytics where the logs must be stored. -
Log_Group_OCID
: The OCID of the log group in Oracle Logging Analytics where the logs must be stored. -
Service_Connector_Compartment_OCID
: The compartment OCID of the service connector hub.
If you enabled Oracle Logging Analytics using the onboarding UI which is available when you navigate to the service for the first time, then some policies are already created. See Policies Created While Onboarding Logging Analytics.
Policy for Each Type of Service Logs
Oracle Logging Analytics creates
an entity representing the underlying OCI resource when new logs are received through
the service connector. In order to obtain the necessary information from the OCI
resource, you must provide Oracle Logging Analytics with a minimum of read
access to the OCI resource.
For example, in order to read information about a VNIC, you can write one of the
following policies:
Policy statement with the READ PRIVILEGE of the OCI resource:
allow service loganalytics to {VNIC_READ} in compartment <specify_compartment>
OR
Policy statement with the read verb for the OCI RESOURCE:
allow service loganalytics to read vnics in compartment <specify_compartment>
The above policy statements restrict the read
access to a
compartment. To extend the access to the entire tenancy, you can change the policy
statement accordingly.
The following OCI resources are supported in Oracle Logging Analytics for log collection through the service connector. You can either create the policy using read verb for the OCI resource or use the read privilege for the resource as illustrated above.
OCI Resource Description | OCI Resource | Read Privilege |
---|---|---|
Analytics Cloud Instance | analytics-instances |
ANALYTICS_INSTANCE_READ |
API Gateway | api-gateways |
API_GATEWAY_READ |
APM Domain | apm-domains |
APM_DOMAIN_READ |
Container Engine For Kubernetes | clusters |
CLUSTER_READ |
Data Flow (Application) | dataflow-application |
DATAFLOW_APPLICATION_READ |
Data Integration Workspace | dis-workspaces |
DIS_WORKSPACE_READ |
Data Science Jobs | data-science-jobs |
DATA_SCIENCE_JOB_READ |
Data Science Model Deployments | data-science-model-deployments |
DATA_SCIENCE_MODEL_DEPLOYMENT_READ |
DevOps Build Pipeline | devops-build-pipeline |
DEVOPS_BUILD_PIPELINE_READ |
DevOps Build Pipeline Stage | devops-build-pipeline-stage |
DEVOPS_BUILD_PIPELINE_STAGE_READ |
DevOps Build Run | devops-build-run |
DEVOPS_BUILD_RUN_READ |
DevOps Deployment | devops-deployment |
DEVOPS_DEPLOY_DEPLOYMENT_READ |
DevOps Deployment Pipeline | devops-deploy-pipeline |
DEVOPS_DEPLOY_PIPELINE_READ |
DevOps Deployment Stage | devops-deploy-stage |
DEVOPS_DEPLOY_STAGE_READ |
Email Delivery Service | approved-senders |
APPROVED_SENDER_READ |
Events Service | cloudevents-rules |
EVENTRULE_READ |
Functions (FN App) | fn-app |
FN_APP_READ |
Functions (FN Function) | fn-function |
FN_FUNCTION_READ |
GoldenGate Deployment | goldengate-deployments |
GOLDENGATE_DEPLOYMENT_READ |
Instance | instances |
INSTANCE_READ |
IPSec Tunnel | ipsec-connections |
IPSEC_CONNECTION_READ |
Load Balancer | load-balancers |
LOAD_BALANCER_READ |
Media Workflow | media-workflow |
MEDIA_WORKFLOW_READ |
Media Workflow Job | media-workflow-job |
MEDIA_WORKFLOW_JOB_READ |
Network Firewall | network-firewall |
NETWORK_FIREWALL_READ |
Object Storage (Bucket) | buckets |
BUCKET_READ |
OCI Database with PostgreSQL | postgres-db-systems |
POSTGRES_DB_SYSTEM_READ |
OIC Instance | integration-instance |
INTEGRATION_INSTANCE_READ |
Operator Control | operator-control-family |
- |
Service Connector | serviceconnectors |
SERVICE_CONNECTOR_READ |
VCN - VNIC | vnics |
VNIC_READ |
Web Application Firewall | web-app-firewall |
WEB_APP_FIREWALL_READ |
If you enabled Oracle Logging Analytics using the onboarding UI which is available when you navigate to the service for the first time, then some policies are already created. See Policies Created While Onboarding Logging Analytics.
Set Up the Service Connector to Ingest Logs
Before you set up the service connector to ingest logs, ensure that the compartment and log group are identified for the logs that you want to ingest.
In the following example, the steps show you how to collect VCN service logs from Oracle Cloud Infrastructure Logging service:
-
This is a suggestive step to show you how to enable logs in the Oracle Cloud Infrastructure Logging service.
Go to Oracle Cloud Infrastructure Logging service > Go to Logs.
Click Enable Resource Log to enable VCN service logs. The dialog box opens.
- Select the resource compartment.
- Select the service, for example,
Virtual Cloud Network (subnets)
. - Select the resource, for example, the VCN resource.
- Under Configure Log, select the log category, for
example,
Flow Logs
, and the log name. - Under Log Location, select the compartment and log group that Oracle Logging Analytics will refer the logs from.
Click Enable Log.
-
Set up the service connector by specifying the source service of the logs and the target as Oracle Logging Analytics. You can either set it up from the source service that has integrated with Oracle Cloud Infrastructure Service Connector Hub, for example, Oracle Cloud Infrastructure Logging service, or directly from Oracle Cloud Infrastructure Service Connector Hub.
Go to Oracle Cloud Infrastructure Logging service > Go to Service Connectors > Click Create Connector.
Alternatively, go to Oracle Cloud Infrastructure Service Connector Hub service > Click Create Service Connector.
The Create Service Connector page opens.
- Enter a name for the connector and provide a description.
- Select the resource compartment where the connector resource must be created.
- Under Configure Service Connector, specify
Logging
as the Source service, andLogging Analytics
as the Target service. - Under Configure Source Connection, provide the details of
the logs to collect from the service, for example, the VCN service logs.
Select the compartment name, the log group to which the logs belong, and the name of the logs that you had configured in step 1.
You can configure the same service connector to collect more logs. Click Another Log and repeat step 2-d.
Optionally, you can create filters under Configure Task.
Click Create Connector.
After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.
Allow Cross-Tenancy Log Collection from OCI Logging Service
Let Source_Tenant
be the tenant of the source service such
as Oracle Cloud Infrastructure Logging from which logs are collected. Let
Target_Tenant
be the tenant in which the service connector is
created. The service connector is configured with Oracle Logging Analytics as the target for the logs that are collected from the
source service. It is assumed that the service connector hub and Oracle Logging Analytics are available on the
same target tenant.
Set the following policies to configure the log collection from a tenancy that is different from the tenancy the service connector is created in.
Policies To Be Added in the Source Tenant
Here is an example of policy statements which allow any user of the
service connector hub tenancy to have READ
access to the Logging
service:
define tenancy <Target_Tenant> as <Target_Tenant_OCID>
define group <Common_User_Group> as <Common_User_Group_OCID>
admit any-user of tenancy <Target_Tenant> to read logging-family IN TENANCY WHERE ALL {request.principal.type = 'serviceconnector'}
admit group <Common_User_Group> of tenancy <Target_Tenant> to read logging-family IN TENANCY
Ensure to set the policy for the type of service logs that must be collected from the source service. See Allow Collection of Logs from OCI Logging Service.
Policies To Be Added in the Target Tenant
Here is an example of policy statements which allow any user to access
the Logging service through the service connector hub, and the target IAM group
Common_User_Group
to have MANAGE
access to the
service connector hub:
define tenancy <Source_Tenant> as <Source_Tenant_OCID>
endorse any-user to read logging-family IN tenancy <Source_Tenant> WHERE ALL {request.principal.type = 'serviceconnector'}
endorse group <Common_User_Group> to read logging-family IN tenancy <Source_Tenant>
The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. Make sure to manually create the policy that includes the following policy statements:
allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID>
where all
{request.principal.type = 'serviceconnector',
target.loganalytics-log-group.id = '<Log_Group_OCID>',
request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <Common_User_Group> to MANAGE serviceconnectors in tenancy
In the above policy statements,
-
Log_Group_OCID
: The OCID of the Oracle Logging Analytics log group. -
Log_Group_Compartment_OCID
: The OCID of the compartment where the Oracle Logging Analytics log group is located. -
Service_Connector_Compartment_OCID
: The compartment OCID of the service connector. -
Common_User_Group
: The user group that creates the service connector.
Create a Connector Between the Source and Target Tenants
After the required policies are created for the source and target tenants, create a service connector using CLI. The following example CLI command specifies Logging as the source and Oracle Logging Analytics as the target for creating the cross-tenancy service connector:
oci --profile <Target_Profile> sch service-connector create
--display-name XTenancyConnector
--compartment-id <Connector_Compartment_OCID>
--source '{ "kind": "logging", "logSources":
[ { "compartmentId": "<Logging_LogGroup_Compartment_OCID>",
"logGroupId": "<Logging_LogGroup_OCID>" } ] }'
--target '{ "kind": "loggingAnalytics", "logGroupId": "<LoggingAnalytics_LogGroup_OCID>" }'
The above command is formatted for better readability. Remove characters like new line, tab and additional spaces before running it.
In the above CLI command,
-
Target_Profile
: The profile in the .oci/config file that maps to the target tenancy. -
Connector_Compartment_OCID
: The OCID of the compartment where the service connector resource is created. -
Logging_LogGroup_Compartment_OCID
: The OCID of the compartment the Oracle Cloud Logging log group belongs to. This is in the source tenant. -
Logging_LogGroup_OCID
: The OCID of the Oracle Cloud Logging log group. This is in the source tenant. -
LoggingAnalytics_LogGroup_OCID
: The OCID of the Oracle Logging Analytics log group. This is in the target tenant.
For more details about the CLI command, see CLI Command Reference - Create.
After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.