Oracle Cloud Infrastructure Documentation

Application Performance Monitoring Metrics

Overview

Application Performance Monitoring data sources such as APM Java agents and APM Browser agents collect and upload key application metrics to the Oracle Cloud Infrastructure Monitoring service. For information on Application Performance Monitoring data sources, see Application Performance Monitoring Data Sources.

Here are the Monitoring service metric namespaces related to Application Performance Monitoring:

  • oracle_apm_rum: Metrics related to Real User Monitoring (RUM), specifically session metrics for all configured web applications as well as metrics for all monitored user activities (page activities, AJAX calls, and script errors).
  • oracle_apm_synthetics: Metrics related to Synthetic Monitoring, specifically the monitor's availability, the number of connections established, the number of DNS lookups, and so on.
  • oracle_apm_monitoring: All other Application Performance Monitoring metrics, such as the metrics stemming from application servers and custom metrics.

For information on the Monitoring service and its features, see Monitoring Overview.

Prerequisites

IAM policies: To monitor resources, you must be given the required type of access in a policy written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool. The policy must give you access to the monitoring services as well as the resources being monitored. If you try to perform an action and get a message that you don’t have permission or are unauthorized, confirm with your administrator the type of access you've been granted and which compartment you should work in. For more information on user authorizations for monitoring, see the Authentication and Authorization section for the related service: Monitoring or Notifications.

Span Enrichment

About Span Enrichment

  • Provide users with the ability to enrich spans with custom attributes or change, combine or remove data from existing attributes.
  • Enrichment operations are defined through span rules which contain one or more actions to be applied in a particular order on a given span.
  • There are two types of span rules: Custom and Oracle-provided.
  • Span enrichment is configured at APM Domain level.
Span Enrichment Examples:
  • Removal of Personal Identifiable Information (PII) from spans.
  • Combine data from multiple attributes into a single one.
  • Extract data from an attribute. For example, a partial path from a longer URL.
  • Apply mathematical functions on data. For example, convert a unit from seconds to milliseconds.
  • Map an attribute value to a new value. For example, map the value '1' to the color 'blue', value '2' to the color 'red'.
  • Distribute numeric values into buckets. For example, a temperature value range that is mapped into 'cold', 'warm' and 'hot' buckets.
Topics:

Span Enrichment Usage

Span Enrichment is configured through Span Rules which operate on a span and are used to modify the span's attributes. The Span Rules toolbox provides a variety of actions that can be performed as part of a rule.

Span Rules are managed within groups. A group is a collection of rules that are applied in a specified order. Each rule can apply one or more actions on a span and the output of each rule can be used as input to follow-up rules. Rule groups require a Span Filter that is used to select the spans on which to apply the rules.

Multiple groups can be defined. This is useful when you need to maintain different types of rules for different types of Spans as it creates logical entities with functionality designed for a specific purpose. This reduces the overall complexity of your Rule Set.

If multiple groups have Span Filters set up that match the same span, then all of them will apply their rules to that span. The order in which this is performed is determined by the alphabetical order of the group names. Keep this in mind when you name or rename your groups. Oracle recommends prefixing the names with a number to avoid the group names getting sorted alphabetically. For example, "1. Remove Location Data", "2. Convert Length Units".

Keep in mind that when multiple groups are operating on the same span, this means that each group will be operating on a version of the span that may have been modified by a previous group in the chain. This can be useful, as you can write rules in follow-up groups that rely on modifications created by previous groups. On the other hand, it can also cause follow-up rules to fail if an attribute that a rule relies on, is removed or modified by a previous group.

Note

The above is only important when a span matches more than one group.

If your groups have clearly disjoint filters, then a span will always be modified by a single group and the above considerations become irrelevant.

Attributes in Span Events

Spans may contain a list of events which are also referred to as log messages. These are typically used to store information about important moments during the execution of the span. For example, if a span describes a database query, then events could be generated to mark the moment in time when the connection was open, then when the query request was actually submitted and when the response was received.

An event is identified by a name, timestamp and a set of attributes (key-value pairs) that provide further context for the recorded information.

All Span Rule actions described below, which require an attribute as input or output, can also be applied on the attributes of individual events. Since the event attributes are independent of regular span attributes, you will need to use the following event reference syntax to identify the proper event attribute:
event['NameOfEvent']:'NameOfAttribute'.
Note

Event attributes cannot be used as Span Filters.

Create and Manage Span Rules

You can create and manage Span Rules depending on your reporting needs.

To create Span Rules for a given APM domain, do the following:

  • Go to the APM Domains page and click on the desired APM domain.

    The Domain Details page is displayed.

  • Go to Resources, located at the bottom left, and click Span Enrichment.

    The Span Enrichment pane is displayed.

  • Click on Create group to create a new span rule group.

    The Create span rule group pane is displayed.
    • Enter the Name of the new group.
    • Select a Span Filter to use with this group.
    • Optionally, provide a Description for the new group.
    • Click Next to proceed to Rule actions pane to start adding span rules to the new group depending on the span rule type: Custom span rule or Oracle provided span rule.
    • Enter the Rule name.
    • Optionally, select a Rule span filter to further narrow down the scope of the rule's operation.
      Note

      Span Filters are created separately. For details, see Span Filters.
    • Optionally, provide a Rule description.
    • Select a span rule type.

      • If you select Oracle Provided span rule, you are offered a list of rules to choose from.

        The rules have already been created by Oracle in order to assist the users. The list includes the following:

        • Adds the client IP address to a span.

          It adds the client IP address to a span as an attribute named ClientIP.

        • EBS suite.

          It provides the E-Business Suite template.

        • OpenTelemetry to APM naming conversion.

          It converts span attribute names from the OpenTelemetry naming convention to the APM naming convention. For information about using OpenTelemetry with APM, see Configure OpenTelemetry Data Sources .

        Click on a rule from the list to add it to the group.

        Note

        You cannot configure any actions since they are managed internally by the rule itself. Also, you cannot see what the actions are for an Oracle provided rule.
      • If you select to create a Custom span rule, you need to define one or more actions for it. (A rule always contains at least one action).
        Before you can choose an Action, you have to select an Action target and Action type.
        • The Action target specifies the part of the span on which the action operates. There are two targets available:
          • Entity: Target the entire span.
          • Attribute: Target a single attribute.
        • The Action type specifies the type. It is mainly used to group actions logically together. For example, string or numeric.
        • Now select the appropriate action from the Available Actions drop-down.

          Since actions have various contexts, the window will change accordingly with each action type. For more details about actions, see Working with Actions.

        For more information about the available actions for Action target and Action type, see Available Actions.

        • Use the Add Action button to add more actions to the rule.

        • Use the action item menu at the top right of each Action to move it up or down in the rule ordering, or delete an action.

Working with Actions

Actions that have a target of Attribute always require you to provide the name of the attribute whose value will be used as input to the action. The configuration window for each action refers to it as Input Attribute.

Some actions also require you to provide the name of an attribute in which to store the result of the operation. This is referred to as the Output Attribute. If the attribute exists on the span, any value it has at that time will be overwritten by the result of the action. If it does not yet exist, it will be added to the span with the new value. You can use this mechanism to replace the value of any attribute on the span (even the input attribute) with the result of the action's operation.

Using this, you can feed the results of any action to a follow-up action (or even a rule within the same rule group).

You can also use it to store values into temporary attributes that can be used by follow up rules or actions, but without the need to keep them as part of the final result. Just make sure to remove the temporary attributes as a last step using the Remove Attribute action.

Note

A Block action rejects the span and you cannot use it later on, even from another Rule Group.

Available Actions

Action Target Action Type Available Actions
Entity

Passthrough

Actions that control whether the span in its entirety is passed through or blocked.

Blocked spans are rejected from the processing pipeline. Rejected spans will no longer be processed by APM, will not be stored anywhere, and will not be available for viewing later on in other APM tools or dashboards.

Block: Blocks this span unconditionally. Any follow up actions and rules will not be applied to it and the span will no longer be available in APM OCI tools and products.

Sample: Samples (accepts) spans based on a sample percentage. For example, a sample percentage of 25% will (randomly) accept 25% of all spans, and will block the rest. Spans blocked here are treated the same as spans that were blocked by the Block action.

Attribute

String

Actions that allow various operations on the value of attribute which is a string. For example, changing the case, replacing a value or extracting a portion of the string.

Append: Appends a static string to the value of an attribute using an optional separator string.

Concatenate: Concatenates two attribute values. An optional separator string can be provided on which the values will be joined. The result is returned as a new attribute.

Extract: Extracts a portion of an attribute's value based on a regular expression. The result is returned as a new attribute.

The regular expression must contain a capturing group operator which identifies the portion of the string to capture. If the regular expression doesn't match, or there is no capture group, the action will produce no output. You can optionally provide a fallback value, which will be used as the output of the action in case the extract itself produced no result.

Lowercase: Converts the value of an attribute to lower case.

Search & Replace: Replaces one or more portions of an attribute's value using a regular expression. The amended string is returned as a new attribute.

The regular expression may contain one capture group. Without a capture group, the full content of what the expression matches will be replaced. When a capture group is present, only the content that is captured by the group will be replaced. The optional match occurrence can be used if only a specific occurrence of a match must be replaced. If this is not set, or set to 0, all matched occurrences will be replaced. This is the default behavior.

Trim: Trims any leading and trailing spaces from an attribute's value.

Uppercase: Converts the value of an attribute to upper case.

Attribute

Numeric

Actions that allow various mathematical operations on the attribute values (provided they are numbers). Numeric actions support both actual numbers and numbers represented as strings in the attribute value, that is the following two are equivalent:
  • "amount" : 2.5
  • "amount" : "2.5"
Numeric actions can take as input either a string representing the name of an attribute whose value to use in the operation, or a number which will be used directly in the computation.
All numeric actions have an option where you can apply rounding to the output value. The rounding can be one of the following:
  • Round: Rounds the number up or down to the nearest whole number.
  • Floor: Rounds the number down to the nearest whole number.
  • Ceil: Rounds the number up to the nearest whole number.
  • No rounding: Default option.

Absolute: Returns the absolute value of the input value.

Add: Sums two values.

Bucketize: Maps a numeric attribute value into bucket names. If the attribute value falls within the range of a bucket, the bucket name will be returned. If no bucket matches, a fallback value will be returned.

Divide: Divides the first by the second value. If the second value is 0, the result will be 0.

Maximum: Returns the maximum of the two values.

Minimum: Returns the minimum of the two values.

Multiply: Multiplies the two values.

Negate: Negates (changes the sign) of the value.

Subtract: Subtracts the second from the first attribute.

Attribute

Generic

Generic actions provide attribute level operations, such as removing an attribute from the span.

Map attribute value: Takes the value of an attribute and maps it to a new value.

Remove attribute: Removes the attribute from the span. If the attribute is an event name reference, event['EventName'], then the entire span event will be removed.

Rename attribute: Renames the attribute.

Set attribute: Sets an attribute to a static value. If the attribute doesn't exist, it is added, otherwise it is assigned the new value, overwriting the existing one.

Examples of Action Configuration

This section provides examples of the configuration steps required for some individual actions.
  • Bucketize Example:

    This action uses a mini-editor to set up a number of buckets and their threshold values. You can use the control on the right to create, update and remove buckets.

    Buckets are automatically ordered based on their threshold values. A fallback value is used when a value is too large for the last bucket. The following image shows an example with five buckets and a fallback value:

    Bucketize Action from Numeric Action Type

  • Map Example:

    This action uses a mini-editor to set up map of input / output value pairs. Both values can be numbers or strings. Use the control on the right to create, update and/or remove a mapping.

    The Export button allows you to download any mappings to a file (in CSV format) to your computer.

    The Import button allows you to populate the mappings from a CSV file on your computer. This is particularly useful if you have a large number of mappings.

    Map Action editor from Generic Action Type

Span Enrichment Rules Validation

Span enrichment configurations can be complex to write and it can sometimes be difficult to know beforehand if a rule or rule group will work as expected. To help with this, the Span Enrichment Configuration Editor provides an option to test a rule or rule group on a sample span before saving it.

To use the test option, do the following:
  • Select a group, and then click on Edit to open the Edit span rule group panel. Another way is to create a new group and the Edit panel get displayed.
  • Click through to the Configure rule actions step.
  • Click Test to test the entire rule group. Alternatively, to test an individual rule, select the rule and click Test rule.
    Note

    Test rule is not available when the rule group contains only one rule.
  • The Test span rule group panel is displayed.

    The rules to test will be shown with a checkbox next to them. You can toggle those as desired.

  • Enter a sample span in the Span sample text box.
  • Click Test span to perform a test of the selected rules on the given span.

Figure 8-1 Span Enrichment Rule Validation

Span Enrichment Rule Validation

The test run will report whether the span is matching the configured span filters or not.

If the filters match, then the rules will be applied to the span and the resulting differences (if any) will be displayed in the table below. Any attributes that were added as a result of the rule will be highlighted in green. Attributes that were removed will be highlighted in red. Modified attributes will be highlighted in yellow.

Trace Explorer: To get accurate samples of spans from your traffic, click Trace Explorer located at the top right to open Trace Explorer and see the actual spans that have been processed. You can select a span, open the context menu on the right (the three vertical dots) and click Copy root span data.

A popup is displayed with the contents of the span. You can copy the span data and use it during testing.

Another option is to paste your own spans or enter them manually, as long as they conform to the span format as reported by Trace Explorer.

Span Property Aliases

Note

Few of the actual span properties are referred to by a different name when used in Span Enrichment or Span Filters features. Other than the visual difference, this has no impact on how you setup your rules and filters. For example, you should always use the alias when referring to the given property.

Here are the properties that currently have an alias:

Span Property Span Enrichment Alias Remarks
td-micros SpanDuration td-micros is in microseconds, while SpanDuration is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
name OperationName The name of the span represents the operation performed
ts-micros StartTime ts-micros is in microseconds, while StartTime is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
id SpanId  
parent-id ParentId  
trace-id TraceId  

Available Metrics: oracle_apm_rum

The RUM metrics are available after an APM Browser agent is configured and uploads user experience-related data. You don't need to enable monitoring on the APM domain to get these metrics.

RUM metrics include the following dimensions:

  • Generic Dimensions
    • ApdexLevel: The level of user satisfaction. For example, Satisfied or Frustrated.
    • ApmrumType: The type of monitored activity. For example, Page, AJAX call or Script Error.
    • ApmrumPageUpdateType: The subtype of the ApmrumType dimension, which is related to user activity. For example, Page Load, Full Update or Click.
  • User Session-Based Dimensions
    • BrowserName: The name of the web browser derived from the user agent. For example, Chrome.
    • DeviceType: The type of device used by the user. For example, Personal Computer.
    • OsFamily: The Operating System family derived from the user agent. For example, Linux.
    • ResourceId: The OCID of the APM domain.
    • Type: The type of metric. For example, Gauge or Counter.
    • WebApplicationName: The name of the web application as specified in instrumentation. If no value is specified for this dimension, then "Default WebApp" is assigned.
Note

When building metric queries for the oracle_apm_rum namespace, you must include the general dimension, MetricGroupName = "No Group" to ensure correctness of data. Similarly, the WebApplicationName dimension with the value All Web Applications or the name of the required web application must be added. For information on adding dimensions, see Build Application Performance Monitoring Metric Queries.

The following table lists RUM metrics:

Metric Description Unit
ActiveSessionsPerMin

The number of active sessions per minute (in the context of web applications).

–
AjaxCalls

The number of AJAX calls observed.

–
AjaxDownloadTime

The time taken to download a response.

Milliseconds
AjaxErrors

The number of AJAX errors observed.

–
AjaxFirstByteTime

The time taken to receive the first byte of an AJAX call response after the request is sent to the server. This is mainly impacted by network latency and server response time.

Milliseconds
AjaxInitTime

The time taken to initialize an AJAX request inside the browser, from the creation of fetch/xhr till the request is sent.

Milliseconds
AjaxResponseTime

The time taken to process the AJAX call response, which is the sum of AjaxInitTime, AjaxFirstByteTime, and AjaxDownloadTime.

Milliseconds
ApdexScore

The Apdex value between 0 and 1.

–
ConnectCount

The number of connections established.

–
ConnectTime

The average time taken to establish server connections.

Milliseconds
DNSLookups

The number of DNS lookups.

–
DNSTime

The average time taken to perform DNS lookups.

Milliseconds
FrustratedPageViews

The number of page views with a 'frustrating' performance.

–
PageClicks

The number of clicks on a page.

–
PageDownloadTime

The time from the browser sending a request for the page URL till the full content is received.

Milliseconds
PageFirstByteTime

The time from the browser sending the request for the page URL till the first byte of response from the server is sent for the page URL. This is mainly impacted by server response time and network latency.

Milliseconds
PageInitTime

The time from the browser receiving the navigation trigger till the request for the page URL is started, which includes connection setup time. This time is calculated for Page Loads and Page Updates.

Milliseconds
PageInteractiveTime

The time taken for the page to become interactive.

Milliseconds
PageRenderTime

The time from the browser receiving full HTML till the content is rendered.

Milliseconds
PageResponseTime

The time taken from the beginning of navigation till the loading of script is complete. This is the sum of PageInitTime, PageFirstByteTime, PageDownloadTime, and PageRenderTime.

Milliseconds
PageViews

The number of page views or page updates.

–
RedirectTime

The average time spent in handling HTTP redirects.

Milliseconds
SatisfiedPageViews

The number of page views with a 'satisfying' performance.

–
ScriptErrors

The number of JavaScript errors observed.

–
SSLTime

The average time taken to establish secure server connections (as part of ConnectTime).

Milliseconds
ToleratingPageViews

The number of page views with a 'tolerable' performance.

–

Available Metrics: oracle_apm_synthetics

The Synthetic Monitoring metrics are emitted as soon as the monitor is created, although there may be a delay of a few minutes before the metrics are displayed initially. You don't need to enable monitoring on the APM domain to get these metrics.

Synthetic Monitoring metrics include the following dimensions:

  • CustomMarker: Indicates the custom target name using which an operation is added to a script.
  • DNSConfigType: Indicates the DNS configuration type. The supported values are DNS_SERVER_CONFIG, DNS_TRACE_CONFIG and DNSSEC_CONFIG.
  • ErrorCategory: The category of the error, if an error occurs when the script is executed.
  • Genre: Indicates that the metric is a Synthetic Monitoring metric. The default value is Synthetics.
  • Host: The host of the network timing metrics. The network timing metrics such as SSLTime and DNSTime are averaged by this host value.
  • IsAPMAgentMonitored: Indicates if the application is also being monitored by another Application Performance Monitoring agent.
  • IsDnsDataAvailable: Indicates if DNS data is available.
  • IsHarAvailable: Indicates if the HAR .zip file is available.
  • IsLogAvailable: Indicates if the log .zip file is available.
  • IsNetworkDataAvailable: Indicates if the network .zip file is available.
  • IsRetryExecution: Indicates if monitor execution has been tried more than once.
  • IsScreenshotAvailable: Indicates if screenshots are available.
  • MaintenanceWindowActive: Indicates if a maintenance window is currently active (1 for active, 0 for inactive).
  • MonitorId: The OCID of the monitor created in Synthetic Monitoring.
  • MonitorName: The name assigned to the monitor.
  • MonitorType: The type of monitor.
  • RequestType: The type of request. The values are CSS, Image, JavaScript, AJAX, or Others.
  • ResourceId: The OCID of the APM domain in which the monitor is created.
  • SqlState: The database state function. It returns a 5-character data type of CHARACTER with a default value of '00000' .
  • Target: The base URL specified in the uploaded script for the Scripted Browser or Scripted REST monitor type or specified in the user interface when creating a Browser or REST monitor type.
  • UserAgent: The type of agent used. For the Browser and Scripted Browser monitor types, Chrome is the default value. For the REST and Scripted REST monitor types, the respective monitor type is the default value.
  • VantagePoint: The vantage point on which the monitor is running.
  • VantagePointDisplayName: The display name of the vantage point on which the monitor is running.

The following table lists the common Synthetic Monitoring metrics for all monitor types:

Metric Name Description Unit
Availability

The availability of the monitor.

1 for success, 0 for failure
AverageBytesPerRequest

The average number of bytes loaded per request for this request type.

–
AverageLoadTimePerRequest

The average time taken to load the resources of this request type.

Milliseconds
CloseTime

Time taken to close the database connection.

Milliseconds
CloudWalletExpiry

Number of days remaining from the current day for the expiration of the cloud wallet. Only available if the connection type is Cloud Wallet.

Days
ConnectCount

The number of connections established.

–
ConnectTime

The average time taken to establish server connections.

For SQL monitor, it's the time taken to create the database connection.

For FTP monitor, it's the time taken to create FTP connection.

Milliseconds
CustomMetric

Records the custom markers added to the scripts.

Milliseconds
DNSLookups

The number of DNS lookups.

–
DNSTime

The average time taken to perform DNS lookups.

Milliseconds
ExecutionTime

Time taken to execute SQL query.

Milliseconds
Failure

Indicates if the monitor execution failed.

0 for success, 1 for failure
FetchedRowCount

Total number of rows fetched.

 -
FinalQueryTime The resolution time of the very last query in the trace. Applicable to DNS Trace monitor type. Milliseconds
HTTP4xxFailureCount

The total number of 4xx request failures during monitor execution.

–
HTTP5xxFailureCount

The total number of 5xx request failures during monitor execution.

–
Latency

The average of the round-trip packet time.

Milliseconds
LatencyDeviation

The standard deviation of latency.

Milliseconds
LoginTime

The time taken for authentication to the FTP server. Applicable to FTP monitors.

Milliseconds
MonitorExecutionTime

The total time taken to execute the monitor.

Milliseconds
MonitorLoadTime

The total time taken to execute the monitor, excluding the pauses given in the script.

Milliseconds
OperationTime

The time taken for the selected FTP operation (download/upload/list).

Milliseconds
PacketLossPercentage

The percentage of probe packets lost.

Percentage
PrepareTime

The time taken to create PreparedStatement object containing the pre-compiled SQL statement.

Milliseconds
RequestCount

The total number of requests for this request type.

–
ResolutionTime

The time it takes to query a specific name server for the given domain. Applicable to DNS Server monitor type.

 Milliseconds
SSLTime

The average time taken to establish secure server connections (as part of ConnectTime).

Milliseconds
Success

Indicates if the monitor execution was successful.

1 for success, 0 for failure
TotalQueries The number of queries used to produce the trace. Applicable to DNS Trace monitor type. -
TotalRequestFailures

The total number of requests that failed to receive a response.

–
TotalSteps

The total number of clicks or enter keys that are captured during monitor execution.

–
TotalTime

The total time of ConnectTime, LoginTime and OperationTime. Applicable to FTP monitor.

Milliseconds

The following table lists the Synthetic Monitoring metric for the REST monitor type:

Metric Name Description Unit
CertificateExpiry

The number of days left before the certificate expires.

Days

Available Metrics: oracle_apm_monitoring for APM Java Agent

The generic Application Performance Monitoring Java agent metrics include default and custom (user-defined) metrics uploaded by the APM Java agent in the oracle_apm_monitoring namespace. You don't need to enable monitoring on the APM domain to get these metrics.

This section provides information on the default Application Performance Monitoring metrics. For information on how to create custom metrics, see Custom Metrics.

Dimensions

Default Application Performance Monitoring metrics include the following dimensions:

  • Application Server Dimensions
    • Appserver: The flag to indicate if an application server has been discovered. When the application server type is java-jmx or java-no-jmx, then this dimension is set to false.
    • AppserverDisplayPort: The port number of the application server. The default is the lowest HTTPS port, if available, or the lowest HTTP port.
    • AppserverDomainName: The domain name of the application server.
    • AppserverEngine: The name of the application server (J2EE container) used. AppserverEngine is set if using only Spring Boot or Dropwizard. For example: Apache Tomcat 8.5.32, Jetty 9.4.11.v20180605 or Undertow 1.4.25.Final.
    • AppserverId: The hash of significant application server resources, which indicates if there is resource change in the application server over time.
    • AppserverName: The name of the application server or application framework.
    • AppserverPorts: The port numbers of the application server. Usually, there is only one port number for the Oracle WebLogic server and multiple for the Apache Tomcat server. In case of multiple port numbers, the values are comma separated.
    • AppserverServerName: The name of the application server instance.
    • AppserverType: The type of application server. For example, weblogic or tomcat.
    • AppserverVersion: The version of the application server.
  • Generic Dimensions
    • ApmVersion: The version of the Application Performance Monitoring agent.
    • DisplayName: The display name of the application server.
    • ServiceInstanceId: The observer ID of the agent.
    • ServiceName: The name you've assigned to the service.
    • OraPackagedApp: The Oracle Packaged Application being used. For example: E-Business Suite (EBS), JD Edwards (JDE), Business Intelligence (OBIEE), Peoplesoft (PSFT) or Siebel CRM (SIEBEL).
    • CreatedBy: Source of the trace/span creation. For example: oracle-apm-java-agent or oracle-apm-java-tracer.
  • Host Dimensions
    • HostAddress: The IPv4 address resolved from the host name.
    • HostAddresses: The list of IPv4 addresses in the network interfaces of the host.
    • Hostname: The name of the host.
    • HostnameCanonical: The canonical name (FQDN) of the host.
    • Hostnames: The list of host names in the network interfaces of the host.
  • Java Virtual Machine (JVM) Dimensions
    • GCCollectors: The garbage collection strategies used for minor and major garbage collections by JVM. For example: G1 young generation or G1 old generation.
    • ProcessId: The process ID of the application server.
    • VmName: The name of the virtual machine.
    • VmVendor: The vendor of the virtual machine.
    • VmVersion: The version of the virtual machine.
    • WorkingDirectory: The working directory of the application server.
  • Kubernetes Dimensions
    • KubernetesNamespace: The namespace in the Kubernetes cluster the pod is running in.
    • KubernetesNodeName: The name of the Kubernetes node the pod is running in.
    • KubernetesPodAnnotations: The annotations you've assigned to the pod, if any.
    • KubernetesPodLabels: The labels you've assigned to the pod, if any.
    • KubernetesPodName: The name of the pod (container) in the Kubernetes cluster.
  • Oracle Cloud Infrastructure Dimensions
    • OciAvailabilityDomain: The Oracle Cloud Infrastructure availability domain the compute instance is running in.
    • OciCompartmentId: The OCID of the compartment in which the compute instance resides.
    • OciComputeShape: The shape of the compute instance.
    • OciDisplayName: The display name of the compute instance or Kubernetes node.
    • OciFaultDomain: The name of the Oracle Cloud Infrastructure fault domain the compute instance is running in.
    • OciInstanceId: The OCID of the compute instance.
    • OciRegion: The Oracle Cloud Infrastructure region that contains the availability domain the compute instance is running in.
  • Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Dimensions
    • OkeClusterId: The OCID of the OKE cluster.
    • OkeClusterLabel: The cluster label, which are the last 11 characters of the OCID of the OKE cluster.
    • OkeCompartmentName: The Oracle Cloud Infrastructure compartment where the OKE cluster resides.
    • OkeKubernetesVersion: The Kubernetes version.
    • OkeNodePoolId: The OCID of the OKE node pool.
    • OkeNodepoolLabel: The node pool label, which are the last 11 characters of the OCID of the OKE node pool.
    • OkeTenancyId: The OCID of the OKE tenancy.
  • Oracle E-Business Suite Dimensions
    • EbsAkRegionAppId: The application ID of a region.
    • EbsAkRegionCode: The ID of a region.
    • EbsClassName: The full package name of the page that is rendered.
    • EbsFunctionId: The ID of the function that is ran, as defined in the Oracle E-Business Suite Functions form.
    • EbsOAFunc: The function name that is passed in the URL.
    • EbsOAHP: The parameter used to change the menu context to the new Home Page and selected function.
    • EbsOAPB: The function name that represents the product branding text.
    • EbsOASF: The function that is selected in the current menu context.
    • EbsRegionAppId: The application ID of a region.
    • EbsRegionClass: The full package name of the region that is rendered.
    • EbsRegionCode: The ID of a region.
    • EbsRespAppId: This represents the ID of the application.
    • EbsRespId: This represent the ID of the Responsibility
  • Operating System Dimensions
    • OsAvailableProcessors: The number of processors available to the JVM.
    • OsName: The name of the Operating System.
    • OsVersion: The version of the Operating System.
  • Siebel Resource Dimensions
    • SiebelResourceType: The category of the Siebel resource.
    • SiebelResourceName: The name of the Siebel resource.
    • SiebelResourceOperation:The operation invoked on the Siebel resource.

Metrics

The following table lists the default Application Performance Monitoring metrics in the oracle_apm_monitoring namespace:

Metric Description Unit
GcAfterOldCommitted

The heap committed after old generation garbage collection.

Bytes
GcAfterOldUsed

The heap used after old generation garbage collection.

Bytes
GcAfterYoungCommitted

The heap committed after young generation garbage collection.

Bytes
GcAfterYoungUsed

The heap used after young generation garbage collection.

Bytes
GcOldTotalCount

The number of old generation garbage collections since the JVM started.

–
GcOldTotalTime

The total time used for old generation garbage collection since the JVM started.

Milliseconds
GcYoungTotalCount

The number of young generation garbage collections since the JVM started.

–
GcYoungTotalTime

The total time used for young generation garbage collection since the JVM started.

Milliseconds
HeapCommitted

The current system memory allocated for heap usage.

Bytes
HeapUsed

The current heap memory usage. This value is less than or equal to the HeapCommitted value.

Bytes
NonHeapCommitted

The current system memory allocated for non-heap usage.

Bytes
NonHeapUsed

The current non-heap memory usage. This value is less than or equal to the NonHeapCommitted value.

Bytes
ProcessCpuLoad

The current CPU load of the JVM process.

–
ProcessCpuTime

The total CPU time of the JVM process.

Nanoseconds
SystemCpuLoad

The current CPU load of the system.

–
SystemPhysicalMemoryFree

The free memory remaining in the system.

Bytes
SystemSwapFree

The free swap disk space remaining in the system.

Bytes
SystemVirtualMemoryCommitted

The committed virtual memory of the JVM process.

Bytes
ThreadBlockedCount

The number of threads in "BLOCKED" state in the JVM.

Note: BLOCKED is the state in which a thread is blocked waiting for a monitor lock. For more information, see Thread States: BLOCKED in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification.

–
ThreadCount

The total number of threads in the JVM.

–
ThreadDeadlockCount

The number of deadlocked threads in the JVM.

Note: DEADLOCK occurs when two or more threads form a cyclic dependency with each other.

–
ThreadTimedWaitingCount

The number of threads in "TIMED_WAITING" state in the JVM.

Note: TIMED_WAITING is the state in which a thread is waiting for another thread to perform a particular action for a specified waiting time. For more information, see Thread State: TIMED_WAITING in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification.

–
ThreadWaitingCount

The number of threads in "WAITING" state in the JVM.

Note: WAITING is the state in which a thread is waiting indefinitely for another thread to perform a particular action. For more information, see Thread State: WAITING in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification.

–
TotalTraceCount

The total number of traces evaluated.

 –
TotalSpanCount

The total number of spans evaluated.

 –
TotalTraceSampledCount

The total number of traces sampled.

This value will increment at the same pace as TotalTraceCount metric if there's no sampling configured or enabled.

–
TotalSpanSampledCount

The total number of spans sampled.

This value will increment at the same pace as TotalSpanCount metric if there's no sampling configured or enabled.

–
Uptime

The time that has elapsed since the JVM started.

Milliseconds
WeblogicJDBCActiveConnectionsAverageCount The average number of active connections in this instance of the data source. –
WeblogicJDBCActiveConnectionsCurrentCount The number of connections currently in use by applications. –
WeblogicJDBCActiveConnectionsHighCount The highest number of active database connections in this instance of the data source since the data source was instantiated. –
WeblogicJMSConnectionsCurrentCount The current number of connections to this WebLogic Server. –
WeblogicJMSConnectionsHighCount The peak number of connections to this WebLogic Server since the last reset. –
WeblogicJMSConnectionsTotalCount The total number of connections made to this WebLogic Server since the last reset. –
WeblogicJMSDestinationsBytesCurrentCount The current number of bytes stored in the destination. Bytes
WeblogicJMSDestinationsBytesHighCount The peak number of bytes stored in the destination since the last reset. Bytes
WeblogicJMSDestinationsBytesReceivedCount The number of bytes received in this destination since the last reset. Bytes
WeblogicJMSDestinationsMessagesHighCount The peak number of messages in the destination since the last reset.  
WeblogicJMSDestinationsMessagesPendingCount The number of pending messages in the destination.  
WeblogicJMSDestinationsMessagesReceivedCount The number of messages received in this destination since the last reset.  
WeblogicJMSServersCurrentCount The current number of JMS servers that are deployed on this WebLogic Server instance. –
WeblogicJMSServersDestinationsCurrentCount The current number of destinations for this JMS server.  
WeblogicJMSServersDestinationsHighCount The peak number of destinations on this JMS server since the last reset.  
WeblogicJMSServersDestinationsTotalCount The number of destinations instantiated on this JMS server since the last reset.  
WeblogicJMSServersHighCount The peak number of JMS servers that were deployed on this WebLogic Server instance since the server was started. –
WeblogicJMSServersTotalCount The number of JMS servers that were deployed on this WebLogic Server instance since the server was started. –
WeblogicJTAActiveTransactionsTotalCount The number of active transactions on the server.

–
WeblogicJTATransactionAbandonedTotalCount The total number of transactions that were committed. –
WeblogicJTATransactionCommittedTotalCount The total number of transactions committed since the server was started.

–
WeblogicJTATransactionHeuristicsTotalCount The number of transactions that completed with a heuristic status since the server was started.

–
WeblogicJTATransactionRolledBackAppTotalCount The number of transactions that were rolled back due to an application error.

–
WeblogicJTATransactionRolledBackResourceTotalCount The number of transactions that were rolled back due to a resource error. –
WeblogicJTATransactionRolledBackSystemTotalCount The number of transactions that were rolled back due to an internal system error. –
WeblogicJTATransactionRolledBackTimeoutTotalCount The number of transactions that were rolled back due to a timeout expiration. –
WeblogicJVMHeapFreeCurrent The current amount of memory that is available in the JVM heap. Bytes
WeblogicJVMHeapFreePercent The percentage of the JVM heap that is free. Percentage
WeblogicJVMHeapSizeCurrent The current size of the JVM heap. Bytes
WeblogicJVMHeapSizeMax The maximum size of the JVM heap. Bytes
WeblogicServerOpenSocketsCurrentCount The current number of sockets registered for socket muxing on this server.

–
WeblogicServerState
The current state of the server as an integer. It can be:
  • 0: Shutdown
  • 1: Starting
  • 2: Running

–
WeblogicServerHealthState
The current health state of the server as an integer. It can be:
  • 0: Ok
  • 1: Warn
  • 2: Critical
  • 3: Failed
  • 4: Overloaded

For details, see the 5 health values defined in Class HealthState from the Weblogic documentation.

-
WeblogicThreadPoolCompletedRequestCount

The number of completed requests in the priority queue.

–
WeblogicThreadPoolExecuteThreadIdleCount The number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.

–
WeblogicThreadPoolExecuteThreadTotalCount The total number of threads in the pool.

–
WeblogicThreadPoolHoggingThreadCount The threads that are currently being held by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.

–
WeblogicThreadPoolPendingUserRequestCount

The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.

–
WeblogicThreadPoolQueueLength The number of pending requests in the priority queue. This is the total of internal system requests and user requests.

–
WeblogicThreadPoolStandbyCount The number of threads in the standby pool. Threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.

–
WeblogicThreadPoolStuckCount The number of stuck threads in the thread pool.

–
WeblogicThreadPoolThroughput The mean number of requests completed per second. Request per second
WeblogicWebAppOpenSessionsCurrentCount The count of the current total number of open sessions in this module. –

Available Metrics: oracle_apm_monitoring for APM Dotnet Agent

The Application Performance Monitoring Dotnet agent metrics include default metrics uploaded by the APM Dotnet agent in the oracle_apm_monitoring metric namespace with oracle_apm_dotnet_agent resource group. You don't need to enable monitoring on the APM domain to get these metrics available.

Dimensions

The APM Dotnet agent metrics include the following dimensions:

  • Generic Dimensions
    • ApmVersion: The version of the Application Performance Monitoring Dotnet agent. For example: 1.0.0.
    • CreatedBy: The agent type. In this case, it's oracle-apm-dotnet-agent.
    • ServiceName: The name you have assigned to the service.
  • Host Dimensions
    • Hostname: The name of the host where the APM Dotnet agent is running.
  • CLR Runtime Dimensions
    • ProcessName: : The process name where the APM Dotnet agent is running. For example: w3wp.exe.
  • OpenTelemetry Dimensions
    • Telemetry-auto-version: The OpenTelemetry version of the APM Dotnet agent. For example: 0.6.0.
    • Telemetry-sdk-language: The OpenTelemetry SDK language. For example: dotnet.
    • Telemetry-sdk-name: The OpenTelemetry SDK name. For example: opentelemetry.
    • Telemetry-sdk-version: The OpenTelemetry SDK version. For example: 1.4.0.687.

Metrics

Available Metrics: oracle_apm_agent for APM Java Agent

The Application Performance Monitoring Java agent emits agent health related metrics to oracle_apm_agent namespace. You don't need to enable monitoring on the APM domain to get these metrics.

This section provides information on the Application Performance Monitoring agent health metrics. For information on how to create custom metrics, see Custom Metrics.

Dimensions

Default Application Performance Monitoring metrics include the following dimensions:

  • Application Server Dimensions
    • Appserver: The flag to indicate if an application server has been discovered. When the application server type is java-jmx or java-no-jmx, then this dimension is set to false.
    • AppserverDisplayPort: The port number of the application server. The default is the lowest HTTPS port, if available, or the lowest HTTP port.
    • AppserverDomainName: The domain name of the application server.
    • AppserverId: The hash of significant application server resources, which indicates if there is resource change in the application server over time.
    • AppserverName: The name of the application server.
    • AppserverPorts: The port numbers of the application server. Usually, there is only one port number for the Oracle WebLogic server and multiple for the Apache Tomcat server. In case of multiple port numbers, the values are comma separated.
    • AppserverServerName: The name of the application server instance.
    • AppserverType: The type of application server. For example, weblogic or tomcat.
    • AppserverVersion: The version of the application server software.
  • Generic Dimensions
    • ApmVersion: The version of the Application Performance Monitoring agent.
    • DisplayName: The display name of the application server.
    • ServiceInstanceId: The observer ID of the agent.
    • ServiceName: The name you've assigned to the service.
  • Host Dimensions
    • HostAddress: The IPv4 address resolved from the host name.
    • HostAddresses: The list of IPv4 addresses in the network interfaces of the host.
    • Hostname: The name of the host.
    • HostnameCanonical: The canonical name (FQDN) of the host.
    • Hostnames: The list of host names in the network interfaces of the host.
  • Java Virtual Machine (JVM) Dimensions
    • ProcessId: The process ID of the application server.
    • VmName: The name of the virtual machine.
    • VmVendor: The vendor of the virtual machine.
    • VmVersion: The version of the virtual machine.
    • WorkingDirectory: The working directory of the application server.
  • Kubernetes Dimensions
    • KubernetesNamespace: The namespace in the Kubernetes cluster the pod is running in.
    • KubernetesNodeName: The name of the Kubernetes node the pod is running in.
    • KubernetesPodAnnotations: The annotations you've assigned to the pod, if any.
    • KubernetesPodLabels: The labels you've assigned to the pod, if any.
    • KubernetesPodName: The name of the pod (container) in the Kubernetes cluster.
  • Oracle Cloud Infrastructure Dimensions
    • OciAvailabilityDomain: The Oracle Cloud Infrastructure availability domain the compute instance is running in.
    • OciCompartmentId: The OCID of the compartment in which the compute instance resides.
    • OciComputeShape: The shape of the compute instance.
    • OciDisplayName: The display name of the compute instance or Kubernetes node.
    • OciFaultDomain: The name of the Oracle Cloud Infrastructure fault domain the compute instance is running in.
    • OciInstanceId: The OCID of the compute instance.
    • OciRegion: The Oracle Cloud Infrastructure region that contains the availability domain the compute instance is running in.
  • Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Dimensions
    • OkeClusterId: The OCID of the OKE cluster.
    • OkeClusterLabel: The cluster label, which are the last 11 characters of the OCID of the OKE cluster.
    • OkeCompartmentName: The Oracle Cloud Infrastructure compartment where the OKE cluster resides.
    • OkeKubernetesVersion: The Kubernetes version.
    • OkeNodePoolId: The OCID of the OKE node pool.
    • OkeNodepoolLabel: The node pool label, which are the last 11 characters of the OCID of the OKE node pool.
    • OkeTenancyId: The OCID of the OKE tenancy.
  • Operating System Dimensions
    • OsAvailableProcessors: The number of processors available to the JVM.
    • OsName: The name of the Operating System.
    • OsVersion: The version of the Operating System.
  • Agent Health Dimensions
    • Probe: The name of the probe.
    • DataType: The type of data. For example: span or metric.
    • Circuit Breaker Dimensions
      • PerformanceFactorName: The name of the performance factor.
      • PerformanceFactorType: The type of the performance factor.
    • Errors Dimensions
      • Cause: The cause of upload error.
      • Type: The type of the error observed. For example: severe or warning.

Metrics

The following table lists the default Application Performance Monitoring metrics in the oracle_apm_agent namespace:
Note

Starting APM Java agent version 1.12, DataQueueSize metric is no longer available. Use DataQueueUsageMaxPercent metric instead.
Metric Description Unit
BrowserAgentInjectionCount The number of times the APM browser agent is injected. –
CircuitBreakerActivations Indicates activation of a circuit breaker performance factor.

Note: This metric is emitted only when a performance factor is activated.

-
CircuitBreakerProbeSuspensions Indicates suspension of a probe due to circuit breaker.

Note: This metric is emitted only when a probe is suspended.

-
DataRejectCount The number of data items rejected due to queue being full. -
DataQueueUsageMaxPercent Indicates the maximum size of the data queue with respect to its capacity.

This metric is available starting with APM Java agent version 1.12.

 Percentage
DataUploaded The size of data uploaded. Bytes
DataUploadTime The time spent in uploading data. Miliseconds
ErrorCount The number of errors observed in the APM agent. -
SpanStartCount The number of spans started by a probe. -
SpanFinishCount The number of spans finished by a probe. -
SpanTimedOutCount The number of spans that timed out. -
TotalSpanCount The total number of spans evaluated. -
TotalSpanSampledCount The total number of spans sampled.

This value will increment at the same pace as the TotalSpanCount metric if there's no sampling configured or enabled.

-
TotalTraceCount The total number of traces evaluated. -
TotalTraceSampledCount The total number of traces sampled.

This value will increment at the same pace as the TotalTraceCount metric if there's no sampling configured or enabled.

-
UploadErrorCount The number of upload errors observed by the APM agent. -

Available Metrics: oci_apm

When spans or metrics are ingested by Application Performance Monitoring, metrics reflecting status of ingest are automatically emitted to the oci_apm namespace.

This section provides information on the Application Performance Monitoring metrics.

Dimensions

The metrics include the following dimensions:

  • Generic Dimension
    • OracleApmType: Indicates the type of metric such as counter or gauge.
    • ResourceId: The OCID of the APM domain for which data is ingested.
  • PayloadRejections Dimensions
    • ObservationType: The type of data that was rejected, such as metric, private-span, public-span.
    • RejectionCause: The reason for rejecting the data such as MISSING_DATA_KEY, MISSING_DATA_FORMAT, INVALID_OBSERVATION_TYPE, OVERSIZED_PAYLOAD, PAYLOAD_THROTTLED, INTERNAL_SERVER_ERROR, INVALID_DATA_KEY and more.

Metrics

The following table lists the default Application Performance Monitoring metrics in the oci_apm namespace:

Metric Description Unit
PayloadRejections The number of observations of different types that cannot be processed due to reasons details in the RejectionCause dimension. –
SpanIngestions The number of spans ingested during the specified period of time. The apmDomainId dimension can be used to report span counts for specific domains. -
SyntheticMonitorRuns The number of monitor runs executed during the specified period of time. The apmDomainId dimension can be used to report on monitor executions for specific domains. -

Using the Console

Build Application Performance Monitoring Metric Queries

You can build Application Performance Monitoring metric queries in Metrics Explorer and monitor your applications by various dimensions. For example, you can build a query using the oracle_apm_rum namespace, ScriptErrors metric, WebApplicationName dimension to monitor the script errors in your application for a specified period of time. You can also add another dimension, BrowserName, to determine if the script errors are browser-related.

  1. Sign in to the Oracle Cloud Infrastructure console.
  2. Open the navigation menu, click Observability & Management. Under Monitoring, click Metrics Explorer.

    The Metrics Explorer page displays an empty chart with fields to create a query.

  3. Select a time period using the Start time and End time or Quick Selects fields on the top of the page.
  4. Scroll down and fill in the following fields for your query:
    • Compartment: Ensure that the compartment in which your APM domain resides is selected.
    • Metrics namespace: Select one of the following Application Performance Monitoring namespaces:
      • oracle_apm_rum
      • oracle_apm_synthetics
      • oracle_apm_monitoring
    • Metrics name: Select a metric within the selected Metric Namespace.
    • Interval: Select an aggregation window.
    • Statistic: Select an aggregation function.
    • Metric dimensions: Select a dimension in the Dimension name field and a value for the specified dimension in the Dimension value field to filter the metric data. Optionally, you can add an additional dimension.
      Note

      In addition to the dimensions you want to select, all the metric queries for the oracle_apm_rum namespace must include the following dimensions to ensure correctness of data:
      • MetricGroupName = "No Group"
      • WebApplicationName = All Web Applications or WebApplicationName = <name of the individual web application>
  5. Click Update Chart.

The chart will be updated to display the metrics that have been requested.

For more information on Metrics Explorer and how to build a metric query, see Building Metric Queries.

Create an Alarm for Application Performance Monitoring Metrics

You can create an alarm using the Oracle Cloud Infrastructure Monitoring service to be notified if an Application Performance Monitoring metric breaches the specified threshold. For example, if you've built a query to monitor the JavaScript errors in browsers, you can create an alarm to be notified if the errors for any browser go over 5%.

After you create an alarm for an Application Performance Monitoring metric, you can go to the Application Performance Monitoring Home page and monitor firing alarms that require your attention in the Alarms widget and click an alarm to navigate to the Alarms Definition page.

For information on how to create a threshold alarm for Application Performance Monitoring metrics, see Using the Console to Create an Alarm.

Using the API

Use the following APIs for monitoring: