Ingesting Data into an OpenSearch Cluster with Fluent Bit

Ingest log data into an OpenSearch cluster with Fluent Bit.

Fluent Bit is a lightweight logging and metrics processor and forwarder.

Prerequisites

Complete the following tasks before proceeding with the steps described in this topic:

Install and Configure Fluent Bit

  1. Download Fluent Bit and install on the VM instance. To install the latest version on Linux-based instances, see Linux Packages. For other operating systems, see Supported Platforms.

  2. Run the applicable command for the VM instance's operating system to install the Fluent Bit agent. For example, run the following command to install the Fluent Bit agent on a Linux instance:

    sudo rpm -i <fluent-bit-rpm file>
  3. Update the OUTPUT section of the Fluent Bit configuration file, /fluent-bit/fluent-bit.conf to direct the output plugin to the OpenSearch cluster. For more information about this section, see Configuration File - Output.

    The following is sample code for this section of the configuration file. Replace the applicable text with the details for your cluster. Specify the OpenSearch cluster's OCID for host, see Getting an OpenSearch Cluster's Details. For http_User and http_Passwd, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.

    name - es
    host - <cluster_ID>
    port - 9200
    tls - on
    tls.verify - on
    index - <index_name>
    http_User - <cluster_user>
    http_Passwd - <cluster_password>
    Suppress_Type_Name - On
    Replace_Dots - On
    Retry_Limit - 6
    Buffer_Size - 400MB
  4. Update the INPUT section of the Fluent Bit configuration file to specify the source for the log data and how it should be ingested. For more information about this section, see Configuration File - Input.

    For example:

    Name  tail
    Tag   file_log
    Path  /etc/fluent-bit/project/file.log
    Buffer_Max_Size  600m
    Multiline  On
    Parser_Firstline  multiline
    Buffer_Chunk_Size  2m
    Skip_Long_Lines  Off
    Skip_Empty_Lines  Off
    Mem_Buf_Limit  700m
    DB  /etc/fluent-bit/db/file.db
    DB.Sync  normal
    Refresh_Interval  30
    Rotate_Wait  20
    Exit_On_Eof  Off
    Ignore_Older  5m
    storage.type  filesystem

Test Data Ingestion with Fluent Bit

This section walks through the steps to test log data ingestion with Fluent Bit based on the configuration you specified in Install and Configure Fluent Bit. Perform these steps while you are connected to the VM instance. The sample commands included are for Linux-based operating systems, for other operating systems, use the applicable commands for that operating system.

  1. Run the following commands to install Java and Python:

    sudo yum install python3
    sudo yum install java
  2. Run the following command to create the directory for the log data:
    mkdir /etc/fluent-bit/project
  3. Run the following command to start Fluent Bit:

    sudo fluent-bit -c /etc/fluent-bit/fluent-bit.confb
  4. Create the python script file generate_log.py in /etc/fluent-bit/project and copy the following code into the file:

    Import logging
     
    # Configure logging to write to a file
     
    logging.basicConfig(filename='file.log', level=logging.DEBUG)
     
    # Generate some log messages
     
    logging.debug('This is a debug message.')
     
    logging.info('This is an informational message.')
     
    logging.warning('This is a warning message.')
     
    logging.error('This is an error message.')
     
    logging.critical('This is a critical message.')
     
    # Example of logging additional details using formatting
     
    name = 'John'
     
    age = 30
     
    logging.info('User %s, age %d, logged in.', name, age)
     
    # Example of logging an exception stack trace
     
    try:
     
        result = 10 / 0
     
    except Exception as e:
     
     logging.exception('An error occurred: %s', str(e))
  5. Use the following command to run the script created in the previous step:

    python3 /etc/fluent-bit/project/generate_log.py

    Running this script creates a file file.log and adds the specified logging messages to the file.

After you run the python script, you can verify that the log data was ingested into the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check for the index you specified in the OUTPUT section of the Fluent Bit configuration file. See Task 6: Connect to OpenSearch Dashboards and Quickstart guide for OpenSearch Dashboards.