Configure OpenTelemetry and Other Tracers

Application Performance Monitoring supports OpenTelemetry and open-source tracing tools such as Jaeger and Zipkin.

Application Performance Monitoring (APM) can ingest OpenTelemetry (OTLP) spans and metrics, as well as Zipkin and Jaeger spans.

APM also supports context propagation between the APM agents and open-source tracers.

For trace sampling decisions, which ensure that traces and spans are sampled to keep the trace data small, but representative, APM complies with any sampling decision made by the open-source tracers if the root span is generated by the open-source tracer. If a trace involves the APM Browser Agent, every trace which originates in the Browser Agent is sampled.

If you have an existing distributed tracing system setup and want to switch to Application Performance Monitoring, you must perform the following tasks to ensure that the data collected using open-source tracers flows into Application Performance Monitoring:
  • Configure context propagation to get the trace context information propagated in the appropriate format, and add it to one or more HTTP request headers.
  • Configure the open-source tracers to report spans in the accepted format to Application Performance Monitoring (OpenTelemetry Protocol and JSON-encoded Zipkin v2 are supported).
  • Configure the open-source tracers to connect to and send trace data to the APM collector URL.

The following sections explain how to configure open-source data sources to send data to APM, configure context propagation if applicable, and more:

Configure OpenTelemetry Data Sources

Here's information on configuring the OpenTelemetry (OTEL) data sources (such as instrumentation libraries and OpenTelemetry Collector) to upload monitoring data (metrics and traces) to Application Performance Monitoring.

Use the OTEL-Compatible Endpoint

To use OpenTelemetry to upload data to Application Performance Monitoring, specify the following as the OTEL data upload endpoint:

Note

Depending on the data source type, the OTEL data upload endpoint can be specified in a configuration file, environment variable, code or even the user interface.
  1. Set the APM data upload endpoint to upload traces to Application Performance Monitoring.

    Traces can be authenticated with a public or private data key.

    • For traces authenticated with public data key, add the following:

      https://<dataUploadEndpoint>/20200101/opentelemetry/public/v1/traces

    • For traces authenticated with private data key, add the following:

      https://<dataUploadEndpoint>/20200101/opentelemetry/private/v1/traces

    For example, if the traces are authenticated with public data key, the APM data upload endpoint looks like the following:

    https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/public/v1/traces

  2. Add the following APM data upload endpoint to upload metrics to Application Performance Monitoring:

    https://<dataUploadEndpoint>/20200101/opentelemetry/v1/metrics

    Metrics are always authenticated with private data key.

The above APM data upload endpoints provide the OpenTelemetry standard to report the data. These calls do not accept any query parameters. Data key are supplied as a header like the following: “Authorization”: “dataKey aaaabbbbccc”

Note

  • Some OpenTelemetry clients automatically add the /v1/traces or /v1/metrics suffix while others don't. If the suffix is not added, it must be specified as part of the URL in the configuration. APM supports JSON and Protocol Buffers (protobuf) encoded data. They should be explicitly specified in the Content-Type header, but this is usually done automatically by the OTEL library/collector.
  • Span links are not supported.

Examples

Node.js Example

Here's a sample of the OpenTelemetry code for Node.js. See the changes for the APM data upload endpoint in bold:
const opentelemetry = require("@opentelemetry/sdk-node");
const {
 getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const {
 OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-http");

  const sdk = new opentelemetry.NodeSDK({
traceExporter: new OTLPTraceExporter({
  url: "https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/private/v1/traces",
  headers: {"Authorization": "dataKey AAAAABBBBBCCCC"},
 }),

  instrumentations: [getNodeAutoInstrumentations()]
});

 sdk.start()
(base) [ssirajud@ssirajud node_with_opentelemetry]$

Java Agent Example

Here's a sample of the OpenTelemetry code for Java agent. See the changes for the APM data upload endpoint in bold:
export OTEL_TRACES_EXPORTER=otlp
export OTEL_SERVICE_NAME=my-service
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=dataKey 1111aaaabbbccc"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/private/v1/traces/
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://aaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/opentelemetry/v1/metrics
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="authorization=dataKey 1111aaaabbbccc"
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf

Configure Jaeger Data Sources

Here's information on the changes that must be made to the code of a Java application that uses the Jaeger client for distributed tracing, and use it to upload traces to Application Performance Monitoring.

To use the Jaeger tracer to upload traces from a Java application to Application Performance Monitoring, you must configure the Jaeger Java client for Zipkin compatibility (report in Zipkin v2 format) and add the Application Performance Monitoring collector URL. To do so, you must make the following changes to your service's code:

  1. Along with the required Jaeger client dependencies, add the following dependencies to the pom.xml file.
    <dependency>
        <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-sender-urlconnection</artifactId>
        <version>2.15.0</version>
    </dependency>
    <dependency>
        <groupId>io.jaegertracing</groupId>
        <artifactId>jaeger-zipkin</artifactId>
        <version>1.3.2</version>
    </dependency>
  2. Use the ZipkinV2Reporter to adapt a Zipkin 2 reporter to the Jaeger reporter interface and convert Jaeger spans to the JSON-encoded Zipkin v2 format. Ensure that you've specified the Application Performance Monitoring collector URL to upload spans to Application Performance Monitoring, and the name of your service or application.
    import io.jaegertracing.Configuration;
    import io.jaegertracing.zipkin.ZipkinV2Reporter;
    import zipkin2.reporter.AsyncReporter;
    import zipkin2.reporter.urlconnection.URLConnectionSender;
     
    /* .... */
     
    ZipkinV2Reporter reporter = new ZipkinV2Reporter(AsyncReporter.create(URLConnectionSender.create("<Application Performance Monitoring collector URL>")));
    JaegerTracer tracer = Configuration
        .fromEnv("<service-name>")
        .getTracerBuilder()
        .withReporter(reporter)
        .withSampler(new ConstSampler(true))
        .build();
     
    /* .... */
  3. Register B3TextMapCodec to use B3 propagation, which is a specification for the b3 header. These headers are used for trace context propagation across service boundaries.
    import io.jaegertracing.Configuration;
    import io.jaegertracing.internal.propagation.B3TextMapCodec;
    import io.opentracing.propagation.Format;
     
    /* .... */
     
    B3TextMapCodec b3Codec = new B3TextMapCodec.Builder().build();       
    JaegerTracer tracer = Configuration
        .fromEnv("<service-name>")
        .getTracerBuilder()
        .registerInjector(Format.Builtin.HTTP_HEADERS, b3Codec)
        .registerExtractor(Format.Builtin.HTTP_HEADERS, b3Codec)
        .build();
     
    /* .... */

    Note that if the Jaeger tracer is created using environment configuration (Configuration.fromEnv()) , setting the JAEGER_PROPAGATION environment variable to b3c sets the Zipkin (B3) context propagation mode without code changes.

Configure Zipkin Data Sources

Here's information on the changes that must be made to the code of a Node.js application that uses the Zipkin client for distributed tracing, and use it to upload traces to Application Performance Monitoring.

To do so, make the following changes to the Zipkin tracer initialization code:

  1. Ensure that you've made the necessary changes to report spans in the JSON-encoded Zipkin v2 format.

    For information about the Zipkin v2 format, see Zipkin 2.

  2. Add the Application Performance Monitoring collector URL, to upload spans to Application Performance Monitoring, and the name of your service or application.
  3. Disable the Zipkin "span joining" feature, if it's enabled. OpenTracing specification does not allow Span ID sharing between different spans within the same trace and this feature must be disabled.

Node.js Example

Here's a sample of the Zipkin tracer initialization code, and the changes detailed in the procedure are in bold:

const {
  Tracer,
  BatchRecorder,
  jsonEncoder: {JSON_V2}
} = require('zipkin');
 
const CLSContext = require('zipkin-context-cls');
const {HttpLogger} = require('zipkin-transport-http');
 
// Setup the tracer
const tracer = new Tracer({
  ctxImpl: new CLSContext('zipkin'), // implicit in-process context
  recorder: new BatchRecorder({
    logger: new HttpLogger({
      endpoint: '<Application Performance Monitoring collector URL>', //Span collection endpoint URL setting
      jsonEncoder: JSON_V2 //Span format and encoding setting
    })
  }), // batched http recorder
  localServiceName: '<service-name>', // name of the application/service
  supportsJoin: false //Span join disable setting
});

APM Collector URL Format

The Application Performance Monitoring collector URL is required when configuring open-source tracers to communicate with Application Performance Monitoring service. This topic provides details about the APM Collector URL format.

The Application Performance Monitoring collector URL has the following format:

<dataUploadEndpoint>/<API version>/observations/<observationType>?dataFormat=<dataFormat>&dataFormatVersion=<dataFormatVersion>&dataKey=<dataKey>

Obtain the <dataUploadEndpoint> value from the APM domain details page.

Parameter Description Available Value

<dataUploadEndpoint>

This is the same value as the "dataUploadEndpoint" value from APM Domain. User can obtain this value from the APM Domain details page. String value.

For example: https://aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com

<API version> API version number.

20200101

<observationType> The type of the observation to retrieve.

public-span when sending spans with a public data key.

private-span when sending spans with a private data key.

metric when sending metrics.

<dataFormat> The observation's data format.

zipkin

otlp-metric

otlp-span

<dataFormatVersion> The observation's data-format version.

Zipkin V2, otlp-metric and otlp-span formats are supported.

1 for otlp-metric and otlp-span.

2 for Zipkinv V2.

<dataKey> The observation's data key.

You can obtain this value from the APM Domain details page.

String value.

For example: 1111OLQSUZ5Q7IGN

Zipkin Example

The following APM collector URL uses observation type public-span, data format zipkin, data format version 2 and data key 11115Q7IGN:
https://aaaabbbb.example.us-ashburn-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=11115Q7IGN
Note

In some cases, the Zipkin configured URL may need to be split into the base URL and the relative path. For instance, applications that use Spring Sleuth require the above URL to be set as like the following:
spring.zipkin.baseUrl =  https://aaaabbbb.example.us-ashburn-1.oci.oraclecloud.com
spring.zipkin.apiPath = /20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=11115Q7IGN

OpenTelemetry Example

The following APM collector URL uses observation type private-span, data format otlp-span, data format version 1 and data key 11112222UZ5:
https://aaaabbbb.example.us-phoenix-1.oci.oraclecloud.com/20200101/observations/private-span?dataFormat=otlp-span&dataFormatVersion=1&dataKey=11112222UZ5

OpenTelemetry Considerations

  • The observationType can be public-span, private-span or metric.
  • The dataFormat parameter should be otlp-metric or otlp-span depending on the data.
  • The dataFormatVersion parameter should be 1.

APM and Open Source Interoperability

Configure context propagation to get the trace context information propagated in the appropriate format, and add it to one or more HTTP request headers. The trace context enables the correlation of spans that come from different microservices, but belong to the same transaction (trace) and are identified by a common Trace ID.

APM Agent/Tracer

The APM Java Agent/Tracer can be configured to work with other OpenTracing tracers (such as Jaeger and Zipkin) in order to form a complete trace.

When a HTTP call occurs, the APM Java Agent/Tracer will attempt to add HTTP headers to the call which can then be read on the other end to help connect spans together.

You can change this format via the com.oracle.apm.agent.tracer.propagation.type property, which can be found in the AgentConfig.properties file of the agent's config directory.

You can utilize the following propagation types by setting the property to the name:

Set the appropriate propagation type in order to have your various tracers being able to work together to form a complete trace.

OpenTelemetry, Zipkin & Jaeger

  • OpenTelemetry: OpenTelemetry supports W3C and B3 context propagation. For details, see Context Propagators Distribution.

  • Zipkin: APM uses the Zipkin format (B3) to propagate trace context so the Zipkin tracer does not require additional configuration.

  • Jaeger: You can either configure Jaeger to consume and propagate trace context in the Zipkin format (B3), or configure the APM Java Agent to use Jaeger propagation context by setting the com.oracle.apm.tracer.propagation.type property.