Configure Management Agent to Collect Metrics using Prometheus Node Exporter
To be able to have the Management Agent to scrape the Prometheus metrics, a configuration file requires to be created.
The configuration file needs to be placed under the agent's detection directory
which location is
/opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/
Once the agent detects the existence of the configuration file in the detection direction, the scraping of Prometheus metrics take place.
It's important to use a naming convention for the configuration
files: The base name of the file should be indicative of the metrics
scraped, and the extension of the file should be
.properties
.
Each line of the file is a name/value pair with the format:
propertyName=propertyValue
.
The presence of a file in the detection directory indicates the current settings for the monitoring and scraping. You need to keep that file in the directory, unchanged and as long as the monitoring is needed.
- Create a Configuration File
- Create Policies to Post Metrics to OCI Monitoring
- Configuration File Properties
Create a Configuration File
You need to create a configuration file to scrape all metrics exposed by
Node Exporter to poc_prometheus
namespace,
specifying nodeName
as a dimension on every metric
datapoint.
$ cat /opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/nodeVM1-ol7.properties
# property file scraping all metrics of the node_exporter that has been set up on 10.20.30.40
url=http://10.20.30.40:9100/metrics
namespace=poc_prometheus
nodeName=nodeVM1-ol7
metricDimensions=nodeName
allowMetrics=*
compartmentId=ocidl.compartment.ocl..aaaaaaaa...kovd6xlq
- The
url
uses IP address:10.20.30.40
. You need to replace it with the IP address or the resolvable hostname where the Node Exporter is running. - The
compartmentId
uses OCID:ocidl.compartment.ocl..aaaaaaaa...kovd6xlq
. You need to replace it with the OCID of the compartment where the Prometheus metrics will get posted.
The lines starting with the hashtag symbol (#) are considered comments and they are ignored.
After the above configuration file is created and placed
under the detection directory:
/opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/
,
the Management Agent will process it in few minutes and the metrics
will start flowing to the OCI Monitoring service.
Create Policies to Post Metrics to OCI Monitoring
-
Confirm a dynamic group was created.
Before you create policies, you first need to confirm that a required dynamic group has been created.
For example, confirm that a Management Agent dynamic group, namedManagement-Agent-Dynamic-Group
, has been created with the following:ALL {resource.type='managementagent', resource.compartment.id='<AGENT_COMPARTMENT_OCID>'}
In this example, the
<AGENT_COMPARTMENT_OCID>
is the compartment OCID of the Agent. -
Create an IAM policy for the dynamic group to allow the Management Agent service to post the metrics collected to the OCI Monitoring service in the compartment that you have selected using the
compartmentId
property from the previous configuration file created.allow dynamic-group Management-Agent-Dynamic-Group to use metrics in compartment <PROMETHEUS_METRIC_COMPARTMENT_NAME> where target.metrics.namespace = 'poc_prometheus'
-
The
Management-Agent-Dynamic-Group
is the name of the dynamic group created. -
The
<PROMETHEUS_METRIC_COMPARTMENT_NAME>
is the name of the compartment which OCID was provided using thecompartmentId
property when creating the configuration file. See Create a Configuration File.This is the compartment where the Prometheus metrics will get posted.
-
The
poc_prometheus
is the namespace and it must match thenamespace
value provided when creating the configuration file.
-
You need to restart the agent to apply the policy changes to the agent.
If the policies are not created or they are not set
up properly, you will receive HTTP 404
NotAuthorized
error messages in the Agent
logs.
Configuration File Properties
This section lists the mandatory and optional properties available to collect metrics using Prometheus exporter.
Mandatory Properties
The following properties are required when creating the configuration file:
Name | Units/Format | Description |
---|---|---|
url | String | The url through which the Prometheus Exporter publishes its metrics. (http only) |
namespace | String | The OCI monitoring namespace to which scraped metrics should be uploaded. |
compartmentId | String | The OCID of the compartment to which the scraped metrics should be uploaded. |
Optional Properties:
The following optional properties can be used in the configuration file:
Name | Units/Format | Description |
---|---|---|
allowMetrics | String - comma separated metric name list. | The complete set of desired scraped metrics. Use this property to limit the set of metrics uploaded if required. |
connection-timeout | Number in milliseconds. | The timeout for connecting to the Prometheus Exporter's endpoint. |
metricDimensions | String - comma separated property name list. | The names of other user-supplied properties expressed as fixed values to be used as dimensions for every uploaded datapoint. |
proxy-url | String - URL. | The URL of the network proxy that provides access to the Prometheus Exporter's endpoint (URL required property). |
read-data-limit | Number in kilobytes. | The limit on the data being sent, not to exceed the agent's fixed limit of 400 (KB). |
read-timeout | Number in milliseconds. | The timeout for reading the response from the Prometheus Exporter's endpoint. |
resourceGroup | String. It includes only alphanumeric characters, periods (.), underscores (_), hyphens (-), and dollar signs ($). | The OCI Monitoring resource group.
If it's specified, the resource group will be assigned to each of the emitted metrics. |
scheduleMins | Scraping interval in minutes | If it's specified, the scraping should occur at the specified interval in minutes.
The minimum value is 1 minute. If it is not specified, the default value is 5 minutes. Note: Lowering the collection interval can have an impact on the CPU. Example: |