Configure Thread Snapshots in APM Tracer

The thread snapshots feature allows thread stack snapshots collection which can be used for deep dive analysis.

The thread snapshots allows collection of traces like thread stack snapshots, thread memory overhead, thread cpu overhead and gc impact on trace. It collects the thread stack snapshots and stack related data like state and locks, at a specific interval.

Users can enable thread stack snapshots collection for taking stack dump of spans. It provides user code stack visibility.

Thread snapshots collection default time interval is 250 ms.

APM Tracer 1.4 version or higher supports thread snapshots and it's disabled by default.

APM Tracer and APM Agent support thread snapshots. For details about configuring thread snapshots in APM Agent, see Configure Thread Snapshots in APM Agent.

Enable Thread Snapshots in APM Tracer

Thread Snapshot is integrated with ApmTracer. It will get initialized as part of ApmTracer object creation.

If you are using Helidon, skip to Thread Snapshots in Helidon.

The below example shows many configuration methods including the one for thread snapshots feature: withCollectDeepDiveTraceSnapshot with value (true). Thread snapshots will get called from the build function of ApmTracer:


ApmTracer tracer = (ApmTracer) new ApmTracer.Builder(AGENT_TRACER_NAME,serviceName)
                      .withProperty(PROP_NAME_HYBRID_TRACER_STARTER, PROP_VALUE_HYBRID_TRACER_STARTER)                      
                      .withMicrosecondAccurateTimestamp(microsecondAccurate)                      
                      .withObject(agentCommon)                       
                      .withTraceListener(traceListener)                      
                      .withCollectThreadId(collectThreadId)                      
                      .withCollectThreadName(collectThreadName)                      
                      .withCollectDeepDiveTraceSnapshot(true)
                      .build();

For information about how to instantiate APM Tracer, see Instantiate APM Tracer.

Thread Snapshots Configuration Methods

The following configuration methods are available:

Method Description Unit Default Value Supported Values
withMaxPercentTraceExportToCollectorPerMinute Maximum percentage of per minute incoming traces to export them to collector. Percentage 100

Minimum:1

Maximum: 100

withMaxTraceExportToCollectorPerMinute Maximum number of top traces per minute to export them to collector. Number 200

Minimum:1

Maximum: 1500

withSnapshotCollectionIntervalInMS Interval at which thread snapshot data will be collected. Milliseconds 250

Minimum:250

Maximum: 300000

withMaxSnapshotsPerTrace Maximum number of snapshots allowed per trace. Number 16

Minimum: 1

Maximum: 16

withMaxStackTraceDepthPerSnapshot Maximum stack trace frames collected for any thread. Number 256

Minimum: 1

Maximum: 256

withLogLevel Print deep dive logs, which are equal or above to this property value. String INFO

DEBUG, INFO, WARN, SEVERE

withMaxProcessCpuUsagePercent Maximum process CPU usage. It's recommended to maintain overall process CPU usage under this threshold. Number 90

Minimum: 1

Maximum: 90

withDeepDiveCpuUsagePercent Minimum threshold to maintain overall deepdive CPU usage. Percentage 3

Minimum: 1

Maximum: 5

  Maximum threshold to maintain overall deepdive CPU usage. Percentage 5

Minimum: 5

Maximum: 20

withTraceLifeInMS Minimum threshold for a trace. Milliseconds 50

Minimum: 50

Maximum: 60000

  Maximum threshold for a trace. Milliseconds 60000

Minimum: 60000

Maximum: 600000

For example:
DeepDiveConfiguration deepDiveConfiguration = DeepDiveConfigurationBuilder.builder()
                                              .withMaxPercentTraceExportToCollectorPerMinute(100) 
                                              .withMaxTraceExportToCollectorPerMinute(200) 
                                              .withSnapshotCollectionIntervalInMS(250)     
                                              .withMaxSnapshotsPerTrace(16)               
                                              .withMaxStackTraceDepthPerSnapshot(256)                   
                                              .withLogLevel("INFO")                             
                                              .withMaxProcessCpuUsagePercent(90)                              
                                              .withDeepDiveCpuUsagePercent(Range.RangeBuilder.builder()
                                              .withMinThreshold(3).withMaxThreshold(5).build()) 
                                              .withTraceLifeInMS(Range.RangeBuilder.builder()
                                              .withMinThreshold(50).withMaxThreshold(60000).build()) 
                                              .build();

ApmTracer tracer = (ApmTracer) new ApmTracer.Builder(AGENT_TRACER_NAME, serviceName)
                            .withObject(deepDiveConfiguration)
                            .withCollectDeepDiveTraceSnapshot(true)
                            .build();

Thread Snapshots in Helidon

Thread Snapshots is supported in Helidon. It will get initialized as part of the ApmTracer object creation inside Helidon.

To enable thread snapshots, update the YAML file of Helidon and add the collect-deepdive-trace-snapshots property and set it to true.

For example:

tracing:
    name: "Helidon APM Tracer"
    service: "helidonse-http"
    data-upload-endpoint:<endpoint-url>
    private-data-key: <private-data-key>
    collect-metrics:true
    collect-resources: true
    collect-deepdive-trace-snapshots:true

For information about using APM Tracer in Helidon, see Use APM Tracer in Helidon.

Configuration Properties for Helidon

To configure thread snapshots in Helidon, update the YAML file of Helidon using the following properties:

Property Name Description Unit Default Value Supported Values
com.oracle.apm.agent.deepdive.maxPercentTraceExportToCollectorPerMinute Maximum percentage of per minute incoming traces to export them to collector. Percentage 100

Minimum:1

Maximum: 100

com.oracle.apm.agent.deepdive.maxTraceExportToCollectorPerMinute Maximum number of top traces per minute to export them to collector. Number 200

Minimum:1

Maximum: 1500

com.oracle.apm.agent.deepdive.snapshotCollectionIntervalInMS Interval at which thread snapshot data will be collected. Milliseconds 250

Minimum:250

Maximum: 300000

com.oracle.apm.agent.deepdive.maxSnapshotsPerTrace Maximum number of snapshots allowed per trace. Number 16

Minimum: 1

Maximum: 16

com.oracle.apm.agent.deepdive.maxStackTraceDepthPerSnapshot Maximum stack trace frames collected for any thread. Number 256

Minimum: 1

Maximum: 256

com.oracle.apm.agent.deepdive.logLevel Print deep dive logs, which are equal or above to this property value. String INFO

DEBUG, INFO, WARN, SEVERE

com.oracle.apm.agent.deepdive.maxProcessCpuUsagePercent Maximum process CPU usage. It's recommended to maintain overall process CPU usage under this threshold. Number 90

Minimum: 1

Maximum: 90

com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.minThreshold Minimum threshold to maintain overall deepdive CPU usage. Percentage 3

Minimum: 1

Maximum: 5

com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.maxThreshold Maximum threshold to maintain overall deepdive CPU usage. Percentage 5

Minimum: 5

Maximum: 20

com.oracle.apm.agent.deepdive.traceLifeInMS.minThreshold Minimum threshold for a trace. Milliseconds 50

Minimum: 50

Maximum: 60000

com.oracle.apm.agent.deepdive.traceLifeInMS.maxThreshold Maximum threshold for a trace. Milliseconds 60_000

Minimum: 60000

Maximum: 600000

For example:
deepDiveProperties:
 com.oracle.apm.agent.deepdive.maxPercentTraceExportToCollectorPerMinute: 100
 com.oracle.apm.agent.deepdive.maxTraceExportToCollectorPerMinute: 200
 com.oracle.apm.agent.deepdive.snapshotCollectionIntervalInMS: 250
 com.oracle.apm.agent.deepdive.maxSnapshotsPerTrace: 16
 com.oracle.apm.agent.deepdive.maxStackTraceDepthPerSnapshot: 256
 com.oracle.apm.agent.deepdive.logLevel: INFO
 com.oracle.apm.agent.deepdive.maxProcessCpuUsagePercent: 90
 com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.minThreshold: 3
 com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.maxThreshold: 5
 com.oracle.apm.agent.deepdive.traceLifeInMS.minThreshold: 50
 com.oracle.apm.agent.deepdive.traceLifeInMS.maxThreshold: 60000

Verify Thread Snapshots

To verify that thread snapshots feature has been configured properly and data is being uploaded to Application Performance Monitoring, use the Snapshot view option from View Trace or Span Details.