Ingesting Data into an OpenSearch Cluster with Fluent Bit
Ingest log data into an OpenSearch cluster with Fluent Bit.
Fluent Bit is a lightweight logging and metrics processor and forwarder.
Prerequisites
Complete the following tasks before proceeding with the steps described in this topic:
-
Create an OpenSearch cluster and make note of the connection information. See Creating an OpenSearch Cluster and Getting an OpenSearch Cluster's Details.
-
Create and connect to a compute instance from your local machine. The instance must be in the same VCN as the OpenSearch cluster. See Create a VM Instance (Optional) and Connecting to an Instance.
Install and Configure Fluent Bit
Download Fluent Bit and install on the VM instance. To install the latest version on Linux-based instances, see Linux Packages. For other operating systems, see Supported Platforms.
Run the applicable command for the VM instance's operating system to install the Fluent Bit agent. For example, run the following command to install the Fluent Bit agent on a Linux instance:
sudo rpm -i <fluent-bit-rpm file>
Update the
OUTPUT
section of the Fluent Bit configuration file,/fluent-bit/fluent-bit.conf
to direct the output plugin to the OpenSearch cluster. For more information about this section, see Configuration File - Output.The following is sample code for this section of the configuration file. Replace the applicable text with the details for your cluster. Specify the OpenSearch cluster's OCID for
host
, see Getting an OpenSearch Cluster's Details. Forhttp_User
andhttp_Passwd
, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.name - es host - <cluster_ID> port - 9200 tls - on tls.verify - on index - <index_name> http_User - <cluster_user> http_Passwd - <cluster_password> Suppress_Type_Name - On Replace_Dots - On Retry_Limit - 6 Buffer_Size - 400MB
Update the
INPUT
section of the Fluent Bit configuration file to specify the source for the log data and how it should be ingested. For more information about this section, see Configuration File - Input.For example:
Name tail Tag file_log Path /etc/fluent-bit/project/file.log Buffer_Max_Size 600m Multiline On Parser_Firstline multiline Buffer_Chunk_Size 2m Skip_Long_Lines Off Skip_Empty_Lines Off Mem_Buf_Limit 700m DB /etc/fluent-bit/db/file.db DB.Sync normal Refresh_Interval 30 Rotate_Wait 20 Exit_On_Eof Off Ignore_Older 5m storage.type filesystem
Test Data Ingestion with Fluent Bit
This section walks through the steps to test log data ingestion with Fluent Bit based on the configuration you specified in Install and Configure Fluent Bit. Perform these steps while you are connected to the VM instance. The sample commands included are for Linux-based operating systems, for other operating systems, use the applicable commands for that operating system.
Run the following commands to install Java and Python:
sudo yum install python3
sudo yum install java
- Run the following command to create the directory for the log data:
mkdir /etc/fluent-bit/project
Run the following command to start Fluent Bit:
sudo fluent-bit -c /etc/fluent-bit/fluent-bit.confb
Create the python script file
generate_log.py
in/etc/fluent-bit/project
and copy the following code into the file:Import logging # Configure logging to write to a file logging.basicConfig(filename='file.log', level=logging.DEBUG) # Generate some log messages logging.debug('This is a debug message.') logging.info('This is an informational message.') logging.warning('This is a warning message.') logging.error('This is an error message.') logging.critical('This is a critical message.') # Example of logging additional details using formatting name = 'John' age = 30 logging.info('User %s, age %d, logged in.', name, age) # Example of logging an exception stack trace try: result = 10 / 0 except Exception as e: logging.exception('An error occurred: %s', str(e))
Use the following command to run the script created in the previous step:
python3 /etc/fluent-bit/project/generate_log.py
Running this script creates a file
file.log
and adds the specified logging messages to the file.
After you run the python script, you can verify that the log data was ingested into
the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check
for the index you specified in the OUTPUT
section of the Fluent Bit
configuration file. See Task 6: Connect to OpenSearch Dashboards
and Quickstart guide for OpenSearch
Dashboards.