Configure OpenTelemetry and Other Tracers
Application Performance Monitoring supports OpenTelemetry and open-source tracing tools such as Jaeger and Zipkin.
Application Performance Monitoring (APM) can ingest OpenTelemetry (OTLP) spans and metrics, as well as Zipkin and Jaeger spans.
APM also supports context propagation between the APM agents and open-source tracers.
For trace sampling decisions, which ensure that traces and spans are sampled to keep the trace data small, but representative, APM complies with any sampling decision made by the open-source tracers if the root span is generated by the open-source tracer. If a trace involves the APM Browser Agent, every trace which originates in the Browser Agent is sampled.
- Configure context propagation to get the trace context information propagated in the appropriate format, and add it to one or more HTTP request headers.
- Configure the open-source tracers to report spans in the accepted format to Application Performance Monitoring (OpenTelemetry Protocol and JSON-encoded Zipkin v2 are supported).
- Configure the open-source tracers to connect to and send trace data to the APM collector URL.
The following sections explain how to configure open-source data sources to send data to APM, configure context propagation if applicable, and more:
Configure OpenTelemetry Data Sources
Here's information on configuring the OpenTelemetry (OTEL) data sources (such as instrumentation libraries and OpenTelemetry Collector) to upload monitoring data (metrics and traces) to Application Performance Monitoring.
Use the OTEL-Compatible Endpoint
To use OpenTelemetry to upload data to Application Performance Monitoring, specify the following as the OTEL data upload endpoint:
Depending on the data source type, the OTEL data upload endpoint can be specified in a configuration file, environment variable, code or even the user interface.
-
Set the APM data upload endpoint to upload traces to Application Performance Monitoring.
Traces can be authenticated with a public or private data key.
- For traces authenticated with public data key, add the
following:
https://<dataUploadEndpoint>/20200101/opentelemetry/public/v1/traces
- For traces authenticated with private data key, add the
following:
https://<dataUploadEndpoint>/20200101/opentelemetry/private/v1/traces
For example, if the traces are authenticated with public data key, the APM data upload endpoint looks like the following:
https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/public/v1/traces
- For traces authenticated with public data key, add the
following:
-
Add the following APM data upload endpoint to upload metrics to Application Performance Monitoring:
https://<dataUploadEndpoint>/20200101/opentelemetry/
v1/metrics
Metrics are always authenticated with private data key.
The above APM data upload endpoints provide the OpenTelemetry standard
to report the data. These calls do not accept any query parameters. Data key are
supplied as a header like the following: “Authorization”: “dataKey
aaaabbbbccc”
- Some OpenTelemetry clients automatically add the
/v1/traces
or/v1/metrics
suffix while others don't. If the suffix is not added, it must be specified as part of the URL in the configuration. APM supports JSON and Protocol Buffers (protobuf) encoded data. They should be explicitly specified in theContent-Type
header, but this is usually done automatically by the OTEL library/collector. - Span links are not supported.
Examples
Node.js Example
bold
:const opentelemetry = require("@opentelemetry/sdk-node");
const {
getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const {
OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-http");
const sdk = new opentelemetry.NodeSDK({
traceExporter: new OTLPTraceExporter({
url: "https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/private/v1/traces",
headers: {"Authorization": "dataKey AAAAABBBBBCCCC"},
}),
instrumentations: [getNodeAutoInstrumentations()]
});
sdk.start()
(base) [ssirajud@ssirajud node_with_opentelemetry]$
Java Agent Example
bold
:export OTEL_TRACES_EXPORTER=otlp
export OTEL_SERVICE_NAME=my-service
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=dataKey 1111aaaabbbccc"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/private/v1/traces/
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/v1/metrics
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="authorization=dataKey 1111aaaabbbccc"
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
Configure Jaeger Data Sources
Here's information on the changes that must be made to the code of a Java application that uses the Jaeger client for distributed tracing, and use it to upload traces to Application Performance Monitoring.
To use the Jaeger tracer to upload traces from a Java application to Application Performance Monitoring, you must configure the Jaeger Java client for Zipkin compatibility (report in Zipkin v2 format) and add the Application Performance Monitoring collector URL. To do so, you must make the following changes to your service's code:
- Along with the required Jaeger client dependencies, add the following
dependencies to the
pom.xml
file.<dependency> <groupId>io.zipkin.reporter2</groupId> <artifactId>zipkin-sender-urlconnection</artifactId> <version>2.15.0</version> </dependency> <dependency> <groupId>io.jaegertracing</groupId> <artifactId>jaeger-zipkin</artifactId> <version>1.3.2</version> </dependency>
- Use the
ZipkinV2Reporter
to adapt a Zipkin 2 reporter to the Jaeger reporter interface and convert Jaeger spans to the JSON-encoded Zipkin v2 format. Ensure that you've specified the Application Performance Monitoring collector URL to upload spans to Application Performance Monitoring, and the name of your service or application.import io.jaegertracing.Configuration; import io.jaegertracing.zipkin.ZipkinV2Reporter; import zipkin2.reporter.AsyncReporter; import zipkin2.reporter.urlconnection.URLConnectionSender; /* .... */ ZipkinV2Reporter reporter = new ZipkinV2Reporter(AsyncReporter.create(URLConnectionSender.create("<Application Performance Monitoring collector URL>"))); JaegerTracer tracer = Configuration .fromEnv("<service-name>") .getTracerBuilder() .withReporter(reporter) .withSampler(new ConstSampler(true)) .build(); /* .... */
- Register
B3TextMapCodec
to use B3 propagation, which is a specification for theb3
header. These headers are used for trace context propagation across service boundaries.import io.jaegertracing.Configuration; import io.jaegertracing.internal.propagation.B3TextMapCodec; import io.opentracing.propagation.Format; /* .... */ B3TextMapCodec b3Codec = new B3TextMapCodec.Builder().build(); JaegerTracer tracer = Configuration .fromEnv("<service-name>") .getTracerBuilder() .registerInjector(Format.Builtin.HTTP_HEADERS, b3Codec) .registerExtractor(Format.Builtin.HTTP_HEADERS, b3Codec) .build(); /* .... */
Note that if the Jaeger tracer is created using environment configuration (
Configuration.fromEnv()
) , setting theJAEGER_PROPAGATION
environment variable tob3c
sets the Zipkin (B3) context propagation mode without code changes.
Configure Zipkin Data Sources
Here's information on the changes that must be made to the code of a Node.js application that uses the Zipkin client for distributed tracing, and use it to upload traces to Application Performance Monitoring.
To do so, make the following changes to the Zipkin tracer initialization code:
- Ensure that you've made the necessary changes to report spans in the
JSON-encoded Zipkin v2 format.
For information about the Zipkin v2 format, see Zipkin 2.
- Add the Application Performance Monitoring collector URL, to upload spans to Application Performance Monitoring, and the name of your service or application.
- Disable the Zipkin "span joining" feature, if it's enabled. OpenTracing specification does not allow Span ID sharing between different spans within the same trace and this feature must be disabled.
Node.js Example
Here's a sample of the Zipkin tracer initialization code, and the changes
detailed in the procedure are in bold
:
const {
Tracer,
BatchRecorder,
jsonEncoder: {JSON_V2}
} = require('zipkin');
const CLSContext = require('zipkin-context-cls');
const {HttpLogger} = require('zipkin-transport-http');
// Setup the tracer
const tracer = new Tracer({
ctxImpl: new CLSContext('zipkin'), // implicit in-process context
recorder: new BatchRecorder({
logger: new HttpLogger({
endpoint: '<Application Performance Monitoring collector URL>', //Span collection endpoint URL setting
jsonEncoder: JSON_V2 //Span format and encoding setting
})
}), // batched http recorder
localServiceName: '<service-name>', // name of the application/service
supportsJoin: false //Span join disable setting
});
APM Collector URL Format
The Application Performance Monitoring collector URL is required when configuring open-source tracers to communicate with Application Performance Monitoring service. This topic provides details about the APM Collector URL format.
The Application Performance Monitoring collector URL has the following format:
<dataUploadEndpoint>/<API version>/observations/<observationType>?dataFormat=<dataFormat>&dataFormatVersion=<dataFormatVersion>&dataKey=<dataKey>
Obtain the <dataUploadEndpoint>
value from the APM domain
details page.
Parameter | Description | Available Value |
---|---|---|
<dataUploadEndpoint> |
This is the same value as the "dataUploadEndpoint" value from APM Domain. User can obtain this value from the APM Domain details page. | String value.
For example: https://aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com |
<API version> | API version number. |
|
<observationType> | The type of the observation to retrieve. |
|
<dataFormat> | The observation's data format. |
|
<dataFormatVersion> | The observation's data-format version.
Zipkin V2, otlp-metric and otlp-span formats are supported. |
|
<dataKey> | The observation's data key.
You can obtain this value from the APM Domain details page. |
String value.
For example: 1111OLQSUZ5Q7IGN |
Zipkin Example
public-span
, data format
zipkin
, data format version
2
and data key
11115Q7IGN
:https://aaaabbbb.example.us-ashburn-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=11115Q7IGN
In some cases, the Zipkin configured URL may need to be split into the base URL and the relative path. For instance, applications that use Spring Sleuth require the above URL to be set as like the following:
spring.zipkin.baseUrl = https://aaaabbbb.example.us-ashburn-1.oci.oraclecloud.com
spring.zipkin.apiPath = /20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=11115Q7IGN
OpenTelemetry Example
private-span
, data format
otlp-span
, data format version
1
and data key
11112222UZ5
:https://aaaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/observations/private-span?dataFormat=otlp-span&dataFormatVersion=1&dataKey=11112222UZ5
OpenTelemetry Considerations
- The
observationType
can bepublic-span
,private-span
ormetric
. - The
dataFormat
parameter should beotlp-metric
orotlp-span
depending on the data. - The
dataFormatVersion
parameter should be1
.
APM and Open Source Interoperability
Configure context propagation to get the trace context information propagated in the appropriate format, and add it to one or more HTTP request headers. The trace context enables the correlation of spans that come from different microservices, but belong to the same transaction (trace) and are identified by a common Trace ID.
APM Agent/Tracer
The APM Java Agent/Tracer can be configured to work with other OpenTracing tracers (such as Jaeger and Zipkin) in order to form a complete trace.
When a HTTP call occurs, the APM Java Agent/Tracer will attempt to add HTTP headers to the call which can then be read on the other end to help connect spans together.
You can change this format via the
com.oracle.apm.agent.tracer.propagation.type
property,
which can be found in the AgentConfig.properties
file of the agent's
config directory.
You can utilize the following propagation types by setting the property to the name:
- B3 : This is used by default. Zipkin's default header format. For details, see https://github.com/openzipkin/b3-propagation.
Note
The APM Java Agent supports only the B3 multi-header case (X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled). Currently it does not support the B3 single header (b3). - Jaeger : Jaeger's default header format. For details, see https://www.jaegertracing.io/docs/1.21/client-libraries/#propagation-format.
- W3C : W3C's Trace Context header format. For details, see https://www.w3.org/TR/trace-context/.
Set the appropriate propagation type in order to have your various tracers being able to work together to form a complete trace.
OpenTelemetry, Zipkin & Jaeger
-
OpenTelemetry: OpenTelemetry supports W3C and B3 context propagation. For details, see Context Propagators Distribution.
-
Zipkin: APM uses the Zipkin format (B3) to propagate trace context so the Zipkin tracer does not require additional configuration.
-
Jaeger: You can either configure Jaeger to consume and propagate trace context in the Zipkin format (B3), or configure the APM Java Agent to use Jaeger propagation context by setting the
com.oracle.apm.tracer.propagation.type
property.