Collect Logs from Your OCI Object Storage Bucket
You can collect log data continuously from Oracle Cloud Infrastructure (OCI) Object Storage. To enable the log collection, create ObjectCollectionRule resource using REST API or CLI. After the successful creation of this resource and having the required IAM policies, the log collection will be initiated.
You can use this method of collecting logs to ingest any type of log stored in an object storage bucket.
You can collect logs from Object Storage bucket in one of the following ways:
-
LIVE: For continuous collection of objects from the time of creation of the rule ObjectCollectionRule. This is the default method.
-
HISTORIC: For onetime collection of objects for a specified time range.
-
HISTORIC_LIVE: For collection of all historic logs in the bucket, and after that, continuous collection of all newly created objects containing logs.
Oracle Logging Analytics uses Events and Streaming services of OCI in conjunction with Object storage to collect and process objects. When you configure a bucket for log collection, Oracle Logging Analytics creates an Events rule to emit event notification for every new object upload to the bucket. The notifications will be delivered to a stream owned by Oracle Logging Analytics.
Note:
-
For proper functioning of log collection from object storage, ensure that the Event rules created by Oracle Logging Analytics are not tampered with.
-
Per bucket, you can have only one ObjectCollectionRule of type LIVE or HISTORIC_LIVE.
-
You can create up to 1000 unique object collection rules per tenancy in a region.
-
The object can be a single raw log file or any archive file (
.zip
,.gz
,.tgz
,.tar
) containing multiple log files. The number of files inside an archive file must be less than 2000, including the directories if any. -
The maximum size of the object (single file or archive file) is 1 GB. The uncompressed size of the object must be less than 10 GB.
Prerequisites: Before enabling log collection using this approach, ensure to:
- Create a new log source or use an Oracle-defined log source that matches your log format. See Create a Source.
- Create a log group or use an existing log group where you will store these logs to control the user access control to the logs and note the log group OCID. See Create Log Groups to Store Your Logs.
- Optionally, if you want to map the logs you are uploading, then create entity or use an existing entity and note the entity OCID. See Create an Entity to Represent Your Log-Emitting Resource.
To stop the collection of objects from the bucket, delete the ObjectCollectionRule rule. This will only delete the associated configuration with the bucket but will have no effect on the already collected log data or your objects in the bucket.
Allow Log Collection from Object Storage
The following IAM policy statements must be included in your policy to provide the permission to the user group for performing the required operations on ObjectCollectionRule:
allow group <group_name> to use loganalytics-object-collection-rule in compartment <object_collection_rule_compartment>
allow group <group_name> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
allow group <group_name> to {LOG_ANALYTICS_SOURCE_READ} in tenancy
allow group <group_name> to {BUCKET_UPDATE, BUCKET_READ} in compartment <object_store_bucket_compartment>
If you are creating IAM policies at Oracle Logging Analytics aggregate resources level, then the following policy statements must be included to use object collection:
allow group <group_name> to use loganalytics-features-family in tenancy
allow group <group_name> to use loganalytics-resources-family in compartment/tenancy
allow group <group_name> to use object-family in compartment <object_store_bucket_compartment>
On the other hand, if you are creating IAM policies at the level of individual resource-types, then the following policy statements are required to use object collection:
allow group <group_name> to use loganalytics-object-collection-rule in compartment <object_collection_rule_compartment>
allow group <group_name> to use loganalytics-log-group in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
allow group <group_name> to read loganalytics-source in tenancy
allow group <group_name> to use object-family in compartment <object_store_bucket_compartment>
group_name
in all the above policy statements refers to the user
group that must be given the required permissions.
When you delete and recreate a bucket, for the existing log collection rule to work, set the flag Emit Objects Events for the bucket after it is recreated.
For log collection to work, along with the above permissions for Creating ObjectCollectionRule, you must also give permission to Oracle Logging Analytics to read the objects from the bucket in your tenancy and manage the cloud event rules in corresponding compartment or tenancy where the object storage bucket is located. The following are the additional IAM policies required:
allow service loganalytics to read buckets in compartment/tenancy
allow service loganalytics to read objects in compartment/tenancy
allow service loganalytics to manage cloudevents-rules in compartment/tenancy
allow service loganalytics to inspect compartments in tenancy
allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags/}
By default, Object Storage disables automatic emission of events at object level. You can either enable emission of events or have required permissions while creating ObjectCollectionRule. To enable event emission, see Managing Objects.
Some of the above policy statements are included in the readily available Oracle-defined policy templates. You may want to consider using the template for your use case. See Oracle-defined Policy Templates for Common Use Cases.
ObjectCollectionRule
Operations
Using REST API or CLI, you can perform operations such as Create
,
Update
, Delete
, List
, and
Get
on the ObjectCollectionRule resource.
For information about using the REST API and signing requests, see REST APIs and Security Credentials.
To communicate with OCI cloud services, create an API Signing Key and register it in your user account in OCI. To generate and register the key and to collect the Tenancy's OCID and user's OCID, see Security Credentials - API Signing Key.
For information about SDKs, see Software Development Kits and Command Line Interface.
Using REST API
Use the following API operations to manage ObjectCollectionRule:
Using CLI
For information about using the CLI, see Command Line Interface (CLI).
For a complete list of flags and options available for CLI commands, see Command Line Reference.
Run the following CLI commands to manage ObjectCollectionRule:
-
Create ObjectCollectionRule:
oci log-analytics object-collection-rule create --from-json <json_file_name> --namespace-name <namespace_name>
For Example:
oci log-analytics object-collection-rule create --from-json file://create.json --namespace-name MyNamespace
In the above example command, the referred sample
create.json
:{ "name": "<Rule_Name>", "compartmentId": "<Compartment_OCID>", "osNamespace": "<Namespace>", "osBucketName": "<Bucket_Name>", "logGroupId": "<Log_Group_OCID>", "logSourceName": "<Log_Source>" }
The following mandatory properties must be provided in the json:
- name: A unique name given to the ObjectCollectionRule. The name must be unique within the tenancy, and cannot be modified.
- compartmentId: The OCID of the compartment in which the ObjectCollectionRule is located.
- osNamespace: The object storage namespace.
- osBucketName: The name of the object storage bucket.
- logGroupId: Logging AnalyticsLogging Analytics log group OCID to associate the processed logs with.
- logSourceName: Name of the Logging Analytics source to use for the processing.
In addition to the mandatory properties, you can also optionally specify the following properties:
- collectionType: The type of collection. Allowed values are
LIVE
,HISTORIC
, andHISTORIC_LIVE
. - charEncoding: Example values
ISO_8859_1
,UTF-16
- definedTags: Defined tags for this resource. Each key is
predefined and scoped to a namespace. For example,
{"foo-namespace": {"bar-key": "value"}}
- description: A string that describes the details of the rule, not more than 400 characters long.
- entityId: OCID of the Logging Analytics entity with which the collected logs will be associated.
- freeformTags: Simple key-value pair that is applied without any
predefined name, type or scope. Exists for cross-compatibility only. For
example,
{"bar-key": "value"}
- isEnabled:
true
by default. - objectNameFilters: For examples, see Perform Selective Object Collection by Applying Filters on Object Names.
- overrides: For examples, see Override ObjectCollectionRule Configuration to Process Specific Objects.
- pollSince: Example value
2019-10-12T07:20:50.52Z
in RFC3339 format. - pollTill: Example value
2019-10-12T07:20:50.52Z
in RFC3339 format.
For more details about the properties for ObjectCollectionRule, see ObjectCollectionRule API Reference.
Sample response of the above example command:
{ "id": "ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID", "name": "My Rule", "compartmentId": "ocid.compartment.oc1..exampleuniqueID", "osNamespace": "MyNamespace", "osBucketName": "MyBucket1", "collectionType": "LIVE", "pollSince": "2020-09-08 14:06:28.028", "logGroupId": "ocid1.loganalyticsloggroup.oc1.. exampleuniqueID", "logSourceName": "MyLogSource", "lifecycleState": "ACTIVE", "timeCreated": "2020-09-08T14:06:28.028Z", "timeUpdated": "2020-09-08T14:06:28.028Z" }
-
Update ObjectCollectionRule: Following is a sample command to update a log source. Similarly, log group, entity, or any other resource can also be updated.
oci log-analytics object-collection-rule update --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID> --log-source-name <Log-Source>
For Example:
oci log-analytics object-collection-rule update --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID --log-source-name MyLogSource
-
Delete ObjectCollectionRule :
oci log-analytics object-collection-rule delete --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID>
For Example:
oci log-analytics object-collection-rule delete --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID
-
List ObjectCollectionRule:
oci log-analytics object-collection-rule list --namespace-name <Namespace> --compartment-id <compartment-OCID>
For Example:
oci log-analytics object-collection-rule list --namespace-name “My Namespace” --compartment-id ocid.compartment.oc1..exampleuniqueID
-
Get ObjectCollectionRule:
oci log-analytics object-collection-rule get --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID>
For Example:
oci log-analytics object-collection-rule get --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID
-
Add Overrides to ObjectCollectionRule:
If the ObjectCollectionRule already exists, create an override json, for example
update_override.json
, with the override conditions that you want to add to ObjectCollectionRule:{ "overrides":{"items":[{"matchType":"contains","matchValue":"audit","propertyName":"logSourceName","propertyValue":"myLOGANAuditSource"}]} }
Now update the ObjectCollectionRule to include the override:
oci log-analytics object-collection-rule update --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID> --from-json file://path-to/file
For Example:
oci log-analytics object-collection-rule update --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID --from-json file:///scratch/update_override.json
For more details about overrides in ObjectCollectionRule, see Override ObjectCollectionRule Configuration to Process Specific Objects.
-
Remove Overrides from ObjectCollectionRule:
If you want to remove all the override conditions from the ObjectCollectionRule, then create a json, for example
remove_overrides.json
, that details the override property as follows:{ "overrides":{"items":[]} }
Now update the ObjectCollectionRule:
oci log-analytics object-collection-rule update --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID> --from-json file://path-to/file
For Example:
oci log-analytics object-collection-rule update --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID --from-json file:///scratch/remove_overrides.json
For more details about overrides in ObjectCollectionRule, see Override ObjectCollectionRule Configuration to Process Specific Objects.
-
Add objectNameFilters to ObjectCollectionRule:
If the ObjectCollectionRule already exists, create a filters json, for example
filters.json
, with the filters on object names that you want to add to ObjectCollectionRule:{ "objectNameFilters":["a/*","*audit*"] }
Note
You can update an existing ObjectCollectionRule with ObjectNameFilters only if it is of the typeLIVE
orHISTORIC_LIVE
. TypeHISTORIC
is not supported for this operation.Now update the ObjectCollectionRule to include the filters:
oci log-analytics object-collection-rule update --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID> --from-json file://path-to/file
For Example:
oci log-analytics object-collection-rule update --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID --from-json file:///scratch/filters.json
For more details about filters in ObjectCollectionRule, see Perform Selective Object Collection by Applying Filters on Object Names.
Allow Cross-Tenancy Log Collection from Object Storage
Set the following policies to configure the object collection rule for collecting logs from a bucket in a guest tenant.
Let Guest_Tenant
be the guest tenant and
Guest_Compartment
the compartment in that guest tenant which has
the object storage buckets from which the logs must be collected. Let
Host_Tenant
be the tenant which is subscribed to Oracle Logging Analytics.
For additional information about writing policies that let your tenancy access Object Storage resources in other tenancies, see Accessing Object Storage Resources Across Tenancies in Oracle Cloud Infrastructure Documentation.
Policies To Be Added in the Guest Tenant
Here is an example of policy statements which allow the IAM group
Host_User_Group
from the host tenant
Host_Tenant
and the Oracle Logging Analytics service to access the objects in the guest compartment
Guest_Compartment
in the guest tenancy:
define group <Host_User_Group> as <Host_User_Group_OCID>
define tenancy <Host_Tenant> as <Host_Tenant_OCID>
admit group <Host_User_Group> of tenancy <Host_Tenant> to use buckets in compartment <Guest_Compartment>
allow service loganalytics to read buckets in compartment <Guest_Compartment>
allow service loganalytics to read objects in compartment <Guest_Compartment>
allow service loganalytics to manage cloudevents-rules in compartment <Guest_Compartment>
allow service loganalytics to inspect compartments in tenancy
allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags /}
Policies To Be Added in the Host Tenant
Here is an example of policy statements which allow the host IAM group
Host_User_Group
to have USE
access to the
buckets in the guest tenancy Guest_Tenant
:
define tenancy <Guest_Tenant> as <Guest_Tenant_OCID>
endorse group <Host_User_Group> to use buckets in tenancy <Guest_Tenant>
allow group <Host_User_Group> to use loganalytics-object-collection-rule in compartment <Rule_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <LogGroup_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_SOURCE_READ} in tenancy
Optionally, define this policy if ObjectCollectionRule has associated entities:
allow group <Host_User_Group> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <Entity_Compartment>
In the above policies,
-
Rule_Compartment
: The compartment in which ObjectCollectionRule must be created. -
LogGroup_Compartment
: The compartment of Oracle Logging Analytics log group in which the logs must be stored. -
Entity_Compartment
: The compartment of Oracle Logging Analytics entity.
After the required policies are created, you can create
ObjectCollectionRule to collect the logs from guest tenancy Object
Storage. Provide the namespace osNamespace
and bucket name
osBucketName
of the guest tenant in the JSON file, as shown in
the following example:
{
"name": "<My_Rule>",
"compartmentId": "<Compartment_OCID>",
"osNamespace": "<Guest_Tenant_Namespace>", // Namespace of the guest tenant
"osBucketName": "<Guest_Tenant__Bucket1>", // Bucket in the guest tenant object store namespace
"logGroupId": "<Log_Group_OCID>",
"logSourceName": "<My_Log_Source>"
}
For details about creating ObjectCollectionRule, see ObjectCollectionRule Operations.
Override ObjectCollectionRule Configuration to Process Specific Objects
When you want to process specific objects inside a bucket by using a configuration that's different from the one defined for ObjectCollectionRule, you can use the override feature. The override feature facilitates to provide configuration properties based on object names matching specific patterns or prefixes or directories inside the bucket.
-
The properties for which you can specify overrides:
logSourceName
,charEncoding
,entityId
,timezone
-
The match type that you can use in the override:
contains
-
The match value is always applied on the object name
If you are creating the ObjectCollectionRule for the first time, then one of the following examples of the create.json can be used to build the json for your use case:
- Sample
create.json with overrides to process object names containing
audit
with logSourceNamemyLOGANAuditSource
:{ "name":"testRule1", "compartmentId":"ocid1.compartment.oc1..aaa...", "osNamespace":"myObjectStorageNameSpace", "osBucketName":"myObjectStorageBucket", "logGroupId":"ocid1.loganalyticsloggroup.oc1..aaa...", "logSourceName":"myLOGANSourceName", "overrides":{"items":[{"matchType":"contains","matchValue":"audit","propertyName":"logSourceName","propertyValue":"myLOGANAuditSource"}]} }
Using the above rule, the objects containing
audit
are processed with logSourceNamemyLOGANAuditSource
while all other objects in the bucket are processed with the logSourceNamemyLOGANSourceName
. - Sample create.json with overrides to process object names
containing
audit
with logSourceNamemyLOGANAuditSource
and those object names containingdir1/
with charEncodingUTF-16
:{ "name":"testRule2", "compartmentId":"ocid1.compartment.oc1..aaa...", "osNamespace":"myObjectStorageNameSpace", "osBucketName":"myObjectStorageBucket", "logGroupId":"ocid1.loganalyticsloggroup.oc1..aaa...", "logSourceName":"myLOGANSourceName", "overrides":{"items":[{"matchType":"contains","matchValue":"audit","propertyName":"logSourceName","propertyValue":"myLOGANAuditSource"}, {"matchType":"contains","matchValue":"dir1/","propertyName":"charEncoding","propertyValue":"UTF-16"}]} }
After creating the configuration json, specify the file path when you create the ObjectCollectionRule. See section Create ObjectCollectionRule in ObjectCollectionRule Operations.
If you have already created the ObjectCollectionRule, then create a json with the details of the override and update the ObjectCollectionRule to include it. See section Add Overrides to ObjectCollectionRule in ObjectCollectionRule Operations.
If you want to remove all the override conditions, then create a json as specified in the section Remove Overrides from ObjectCollectionRule and update the ObjectCollectionRule. See ObjectCollectionRule Operations.
Perform Selective Object Collection by Applying Filters on Object Names
Use the Selective Objects Collection feature to collect only a subset of objects in a given object storage bucket. The feature is supported with filters on object names. When the filters are applied, only the objects matching the filters are collected for processing.
The matchType
property only supports exact match along with the
wildcard *
. Following are some of the examples of filters using the
wildcard:
- Filter objectName* specifies objects with the prefix objectName.
- Filter *objectName specifies objects with the suffix objectName.
- Filter *objectName* specifies objects that contain the text objectName.
For more information about filters, see Matching Events with Filters.
If you are creating the ObjectCollectionRule for the first time, then one of the following examples of the create.json can be used to build the json for your use case:
- Sample create.json with objectNameFilters to process object names that
have a prefix
a/
and contain the textaudit
:{ "name":"testRule1", "compartmentId":"ocid1.compartment.oc1..aaa...", "osNamespace":"myObjectStorageNameSpace", "osBucketName":"myObjectStorageBucket", "logGroupId":"ocid1.loganalyticsloggroup.oc1..aaa...", "logSourceName":"myLOGANSourceName", "objectNameFilters":["a/*","*audit*"] }
- If required, you can also use the override feature to specify different
configuration for each of the filters. For the above example, you can additionally
specify that all the objects containing the text
audit
must use the sourcemyLOGANAuditSource
:{ "name":"testRule2", "compartmentId":"ocid1.compartment.oc1..aaa...", "osNamespace":"myObjectStorageNameSpace", "osBucketName":"myObjectStorageBucket", "logGroupId":"ocid1.loganalyticsloggroup.oc1..aaa...", "logSourceName":"myLOGANSourceName", "objectNameFilters":["a/*","*audit*"], "overrides":{"items":[{"matchType":"contains","matchValue":"audit","propertyName":"logSourceName","propertyValue":"myLOGANAuditSource"}]} }
For more information about overrides, see Override ObjectCollectionRule Configuration to Process Specific Objects.
After creating the configuration json, specify the file path when you create the ObjectCollectionRule. See section Create ObjectCollectionRule in ObjectCollectionRule Operations.
If you have already created the ObjectCollectionRule, then create a json with the details of the filters and update the ObjectCollectionRule to include it. See section Add objectNameFilters to ObjectCollectionRule in ObjectCollectionRule Operations.