Publishing Custom Metrics

A custom metric is a metric that you design to collect and analyze data.

See the following:

Prerequisites for Publishing Custom Metrics

  • Add the following required policies, replacing variables for your environment, to create namespace and provide permission to the user group to push the metric to the Monitoring service.

    • Allow group <group-name> to use metrics in tenancy where target.metrics.namespace='<namespace-name>'
    • Allow group <group-name> to read metrics in tenancy
    • Allow group <group-name> to inspect metrics in tenancy
    • Allow group <group-name> to manage metrics in tenancy where target.metrics.namespace='<namespace-name>'
  • Generate an API Key . Big Data Service uses the user principal auth to communicate with OCI Monitoring service through REST API or SDK.

  • Ambari user credentials for fetching required metrics using the Ambari API.

Publishing Custom Metrics Example

  1. Construct the required payload for monitoring service API, and push the metric payload to the monitoring service using user credentials created as part of the prerequisites.
  2. See the following sample code to fetch metrics using Ambari APIs / Ambari-Metric-Collector. See Fetching Required Metrics.
    // Pass all the required user creds (Generated in pre-requisite step1) to AuthProvider.
     
    AuthenticationDetailsProvider provider =
                    SimpleAuthenticationDetailsProvider.builder()
                            .tenantId(TENANCY_ID)
                            .userId(USER_ID)
                            .fingerprint(FINGERPRINT)
                            .passPhrase(PASS_PHRASE)
                            .privateKeySupplier(
                                    () -> {
                                        try {
                                            return new FileInputStream(PRIVATE_KEY_PATH);
                                        }
                                        catch (IOException ie) {
                                            throw new RuntimeException(
                                                    String.format("Unable to find private key file %s", PRIVATE_KEY_PATH), ie);
                                        }
                                    })
                            .build();
     
            MonitoringClient monitoringClient = new MonitoringClient(provider);
            monitoringClient.setRegion(REGION);
     
    // Set the monitoring service end point : https://telemetry-xyz.com
     
            monitoringClient.setEndpoint(MONITORING_CLIENT_ENDPOINT);
            
    // Fetch metrics using Ambari API. Pass all the required userInfo to get Metrics from Ambari metric API. 
       Here Url can be any service metrics url : https://<Ambari>:<port>/api/v1/clusters/ClusterName/services/HDFS/components/NAMENODE. 
       This API returns json as response. User can extract required metrics example metrics/jvm/memHeapCommittedM,metrics/jvm/memHeapUsedM; 
       convert it to any Object. Ex: Here it is  Map<url,Map<MetricName, MetricData>>
     
            Map<String, Map<String, MetricData>> metrics = getMetrics(url, sslContext, userInfo);
     
    // Iterate over the metric data, construct the payload and push to monitoring service
     
            List<MetricDataDetails> metricDataDetailsList = new ArrayList<>();
            for (Map.Entry<String, Map<String, MetricData>> entry : metrics.entrySet()) {
                String url = entry.getKey();
                Map<String, MetricData> metricMap = entry.getValue();
                for (Map.Entry<String, MetricData> metricMapEntry : metricMap.entrySet()) {
                    HashMap<String, String> dimensions = Maps.newHashMap();
                    String metricName = metricMapEntry.getKey();
                    MetricData md = metricMapEntry.getValue();
                   // These dimensions can be used to filter metrics in OCI metric monitoring console
                    dimensions.put("resourceId", md.getResourceId());
                    dimensions.put("nodeHostName", md.getHostname());
                    dimensions.put("metricType", md.getMetrictype()); // Can be Cluster/Host/Component/Host-Component.
                    dimensions.put("clusterName", md.getClusterName());
                    dimensions.put("serviceName", md.getServiceName()); // Ex: HDFS
                    dimensions.put("componentName", md.getComponentName()); // Ex: DATANODE,NAMENODE
                    metricDataDetailsList.add(MetricDataDetails.builder()
                            .namespace(METRIC_NAMESPACE) // This is the custom namespace where user wants to push the metrics
                            .compartmentId(COMPARTMENT_ID) // Compartment Id in tenancy.
    //                        .resourceGroup("resourceGroup1") // optional, replace with your resource group
                            .name(metricName) // Name of the metric
                            .dimensions(dimensions) // Dimensions of metric like resourceId, hostname etc.,
                            .datapoints(Arrays.asList(Datapoint.builder().timestamp(md.getTimestamp())
                                    .value(Double.valueOf(md.getValue())).count(1).build())) // data points like value, timestamp, 
                                       count of the metric.
                            .build());
                }
            }
     
    // Create MetricData request.
            PostMetricDataRequest postMetricDataRequest =
                    PostMetricDataRequest.builder()
                            .postMetricDataDetails(
                                    PostMetricDataDetails.builder()
                                            .metricData(metricDataDetailsList)
                                            .build())
                            .build();
     
    // Push the metrics to monitoring service using client.
     
            PostMetricDataResponse response = monitoringClient.postMetricData(postMetricDataRequest);
        }
     
    // url is Ex:https://<Ambari>:<port>/api/v1/clusters/ClusterName/services/HDFS/components/NAMENODE. sslcontext, UserInfo objects to 
       communicate with Ambari server API.
       private String getMetrics(String url, SSLContext sslContext, UserInfo ambariServerInfo)
                throws IOException
        {
            URL obj = new URL(url);
            HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
     
            con.setSSLSocketFactory(sslContext.getSocketFactory());
            con.setRequestMethod("GET");
            con.setRequestProperty("Authorization", "Basic " + encodeCredentials(ambariServerInfo.getUsername(), ambariServerInfo.getPassword()));
     
            int responseCode = con.getResponseCode();
            if (responseCode == HttpURLConnection.HTTP_OK) {
                BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
                String inputLine;
                StringBuffer response = new StringBuffer();
     
                while ((inputLine = in.readLine()) != null) {
                    response.append(inputLine);
                }
                in.close();
     
                return response.toString();
            }
            else {
                throw new IOException("HTTP GET request failed with error code: " + responseCode);
            }
        }   
  3. View the metrics from the OCI Monitoring Console. See Viewing Published Metrics.
    Note

    After updating the required policy statements mentioned in the Prerequisites, the custom new namespace will get created automatically when you push the metrics to the Monitoring service.