Troubleshooting Oracle Cloud Agent

Verify that the Instance Can Access the Instance Metadata Service Endpoint

These steps verify whether the instance can access the Instance Metadata Service endpoint.

1. Connect to the instance and run one of the following commands, depending on you operating system.
   Oracle Linux and Ubuntu

   curl -v -H 'Authorization: Bearer Oracle' http://169.254.169.254/opc/v2/instance/

   If Oracle Cloud Agent is running, a message similar to the following displays:

   * About to connect() to 169.254.169.254 port 80 (#0)
   *   Trying 169.254.169.254...
   * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
   > GET /opc/v2/instance/ HTTP/1.1
   > User-Agent: curl/7.29.0
   > Host: 169.254.169.254
   > Accept: */*
   > Authorization: Bearer Oracle
   >
   < HTTP/1.1 200 OK
   < Server: server
   < Date: Wed, 24 Mar 2021 20:52:38 GMT
   < Content-Type: application/json
   < Content-Length: 1800
   < Last-Modified: Wed, 03 Mar 2021 01:43:50 GMT
   < Connection: keep-alive
   < ETag: "603ee9d6-708"
   < Accept-Ranges: bytes
   <
   {
     "availabilityDomain" : "uybn:<availability_domain>",
     "faultDomain" : "<fault_domain>",
     "compartmentId" : "ocid1.compartment.oc1..<unique_ID>",
     "displayName" : "<instance_name>",
     "hostname" : "<host_name>",
     "id" : "<unique_ID>",
     "image" : "ocid1.image.oc1.iad.<unique_ID>",
     "metadata" : {
       "ssh_authorized_keys" : ""
     },
     "region" : "<region_key>",
     "canonicalRegionName" : "<region_name>",
     "ociAdName" : "<availability_domain>",
     "regionInfo" : {
       "realmKey" : "<realm>",
       "realmDomainComponent" : "oraclecloud.com",
       "regionKey" : "<region_key>",
       "regionIdentifier" : "<region>"
     },
     "shape" : "<shape>",
     "state" : "Running",
     "timeCreated" : 1614637343723,
     "agentConfig" : {
       "monitoringDisabled" : false,
       "managementDisabled" : false,
       "allPluginsDisabled" : false,
       "pluginsConfig" : [ {
         "name" : "OS Management Service Agent",
         "desiredState" : "ENABLED"
       }, {
         "name" : "Custom Logs Monitoring",
         "desiredState" : "ENABLED"
       }, {
         "name" : "Compute Instance Run Command",
         "desiredState" : "ENABLED"
       }, {
         "name" : "Compute Instance Monitoring",
         "desiredState" : "ENABLED"
       } ]
     },
     "freeformTags" : {
       "keep" : "keep"
     }
   * Connection #0 to host 169.254.169.254 left intact
   

   Windows Server

   Run the command in Windows PowerShell as an administrator.

   Invoke-WebRequest -Headers @{'Authorization'='Bearer Oracle'} http://169.254.169.254/opc/v2/instance/
   

   If Oracle Cloud Agent is running, a message similar to the following displays:

   StatusCode        : 200
   StatusDescription : OK
   Content           : {
                         "availabilityDomain" : "<availability_domain>",
                         "faultDomain" : "<fault_domain>",
                         "compartmentId" :
                       "ocid1.tenancy.region1..<unique_ID>",
                         "displayNam...
   RawContent        : HTTP/1.1 200 OK
                       Connection: keep-alive
                       Accept-Ranges: bytes
                       Content-Length: 1197
                       Content-Type: application/json
                       Date: Wed, 24 Mar 2021 21:07:42 GMT
                       ETag: "<unique_ID>"
                       Last-Modified: Wed, 24 M...
   Forms             : {}
   Headers           : {[Connection, keep-alive], [Accept-Ranges, bytes], [Content-Length, 1197], [Content-Type,
                       application/json]...}
   Images            : {}
   InputFields       : {}
   Links             : {}
   ParsedHtml        : mshtml.HTMLDocumentClass
   RawContentLength  : 1197
   

2. If you get a successful response without proxy errors, check for clock skew errors. If proxy server errors occur, check your proxy server settings.

Check for Clock Skew Errors

Sometimes, the clock on an instance is not synchronized with the NTP service. Clock skew can cause TLS negotiations to fail, preventing the instance from connecting to Oracle services. Follow these steps to check for clock skew errors.

1. Connect to the instance and run one of the following commands to generate the monitoring.log file.

   Linux

   sudo tail -15 /var/log/oracle-cloud-agent/plugins/gomon/monitoring.log

   Windows Server 2019, Windows Server 2022

   Run the command in Windows PowerShell as an administrator.

   Get-Content -tail 15 C:\Windows\ServiceProfiles\OCA\AppData\Local\OracleCloudAgent\plugins\gomon\monitoring.log

   Windows Server earlier than 2019

   Run the command in Windows PowerShell as an administrator.

   Get-Content -tail 15 C:\Users\OCA\AppData\Local\OracleCloudAgent\plugins\gomon\monitoring.log

   If there is a clock skew error, a message similar to the following displays:

   failed to call: Service error:NotAuthenticated. Date 'Tue, 09 Mar 2021 06:39:35 UTC' is not within allowed clock skew.
   Current 'Tue, 09 Mar 2021 06:45:45 UTC', valid datetime range: ['Tue, 09 Mar 2021 06:40:45 UTC', 'Tue, 09 Mar 2021 06:50:46 UTC'].
   http status code: 401. Opc request id: <unique_id>

2. If a clock skew error occurs, configure the Oracle Cloud Infrastructure NTP service for your instance. If no clock skew error occurs, verify that gateways are configured correctly.
3. If you configured the NTP service in the previous step, after you complete the configuration, run one of the following commands to restart Oracle Cloud Agent:
   Oracle Linux 6.x

   sudo initctl restart oracle-cloud-agent

   Oracle Linux 7.x and later versions

   sudo systemctl restart oracle-cloud-agent

   Ubuntu

   sudo snap restart oracle-cloud-agent

   Windows Server

   Run the command in Windows PowerShell as an administrator.

   net stop OCA
   net start OCA

4. Generate the monitoring.log file again.

   If Oracle Cloud Agent is running correctly, a successful response is 200 OK. In the monitoring.log, look for a message similar to the following:

   2021/03/18 03:12:44.391381 t2.go:139: Sent metrics status: 200; took: 387ms; with opc-request-id:<unique_ID>;
   2021/03/18 03:13:44.006391 instancemetadata_client.go:64: fetched metadata from http://169.254.169.254/opc/v2/instance/ , status 200 OK
   2021/03/18 03:13:44.730102 t2.go:139: Sent metrics status: 200; took: 723ms; with opc-request-id:<unique_ID>;
   2021/03/18 03:14:44.324046 t2.go:139: Sent metrics status: 200; took: 320ms; with opc-request-id:<unique_ID>;

Verify Permissions for Windows Domain Joined Instances

If you have a Windows instance that is joined to a domain, verify that the virtual account is granted the Log on as a service user right in the local Group Policy. To set permissions, follow the steps for enabling service log on through a local group policy in Microsoft's Enable Service Logon guide. For Log on as a service, add the user NT SERVICE\ALL SERVICES or the specific user.

Verify that Gateways are Configured Correctly

For Oracle Cloud Agent to communicate with Oracle services, gateways in subnets must be configured correctly. Follow these steps to verify and correct your configuration.

1. Configure the internet gateway, NAT gateway, or service gateway for the subnet in your VCN.
2. After you follow the configuration steps, restart the services using the commands in the Verify that the Instance Can Access the Instance Metadata Service Endpoint section. After you restart the services, check the monitoring.log file for successful requests to Oracle services.

Change Proxy Server Settings

Sometimes, local proxy servers prevent Oracle Cloud Agent from communicating with any services. Each proxy server is different.

Often, setting the http_proxy, https_proxy, and no_proxy environment variables for the oracle-cloud-agent and oracle-cloud-agent-updater services on the proxy client instances resolves proxy issues. After you set these environment variables, in the proxy server access.log file (or equivalent, depending on your system), verify that you see requests from the proxy client to services that Oracle Cloud Agent accesses.