• Troubleshoot: Please uninstall the agent and remove the service file before installing the new agent!
• Troubleshoot: Java is not a 64-bit JVM! Please set path of a 64-bit JVM in the environment variable JAVA_HOME or Java not found please set your preferred path in JAVA_HOME.
• Troubleshoot: Agent Installation failed with message: useradd: Can't get unique GID (no more available GIDs.
• Troubleshoot: useradd: cannot create directory /usr/share/mgmt_agent
• Troubleshoot: Windows: The system cannot find the path specified. Agent install failed.
• Troubleshoot: Management Agent status is "Not Available" in Console after the initial installation
• Troubleshoot: After configuration, the Management Agent is not visible in console or through the API
• Troubleshoot: Prometheus or Kubernetes metrics monitored using Management Agent are not available
• Troubleshoot: Agent runs into OutOfMemoryException
• Troubleshoot: IP address being displayed in host column when Management Agent installed on Windows host
• Troubleshoot: Management Agent installation fails on SELinux when using external volume
• Troubleshoot: Management Agent installation fails on Red Hat Enterprise Linux 9.x

Troubleshoot: Please uninstall the agent and remove the service file before installing the new agent!

Cause: There's an agent already installed on your host. A previous deinstall process did not remove the agent service file successfully.

Troubleshoot: Java is not a 64-bit JVM! Please set path of a 64-bit JVM in the environment variable JAVA_HOME or Java not found please set your preferred path in JAVA_HOME.

Cause: The JAVA_HOME environment variable is not set or it's not pointing to a 64 bit JDK location.

Action: Set JAVA_HOME environment variable to the right JDK version and retry installing the agent. Currently, only 64 bit JDK is supported.

Troubleshoot: Agent Installation failed with message: useradd: Can't get unique GID (no more available GIDs)

Cause: The installation script cannot add a user and group during the management agent installation process because the available group ids on your Linux system are already in use.

Executing install
    Unpacking software zip
    Copying files to destination dir (/opt/oracle/mgmt_agent) 
useradd: Can't get unique GID (no more available GIDs) 
useradd: can't create group 
Agent installation failed, please check log file

Action: Consult with the system administrator before proceeding with the following:

1. Edit the /etc/login.defs file. You require sudo privileges to edit the file.

   Look for the following entries:

   SYS_GID_MIN               nnnn
   SYS_GID_MAX               mmmm
   SYS_UID_MIN               pppp
   SYS_UID_MAX               qqqq

   Where nnnn and pppp are the minimum value and mmmm and qqqq are the maximum value.

   If the above entries don't exist in the file, add them.

2. Update the value of SYS_GID_MAX entry based on the system administrator's recommendation, and save the file.

3. Remove the failed agent installation by running: sudo rpm -e oracle.mgmt_agent.

4. Logout of the shell followed by login.

5. Retry the agent installation.

Troubleshoot: useradd: cannot create directory /usr/share/mgmt_agent

During the Management Agent installation, the mgmt_agent user is created with the default home directory location under /usr/share/mgmt_agent.

Cause: There's not enough file permissions under /usr/share or the file system is read-only.

Possible Actions:

• Set file permissions to give mgmt_agent user access to the default user home directory location: /usr/share.

• Set a different home directory location using the USER_HOME_DIR_ROOT environment variable if you want to use a different location.

  Set the USER_HOME_DIR_ROOT environment variable with the path that you prefer to use as a home directory for mgmt_agent user, and ensure the management agent user has the right file permissions on that preferred directory.

Troubleshoot: Windows: The system cannot find the path specified. Agent install failed.

ERRORLEVEL=9009

Possible Cause: Environment variables have not been set properly due to spaces in the directory/folder name.

Windows environments allow to use spaces within a directory/folder name which causes an issue with the Management Agent installation since quotes are added to the name automatically by Windows. For example, there's a directory/folder named: Program Files. In this case Windows auto-inserts quotes since there's a space within the folder name, and it will now say: "Program Files".

Extra quotes can cause an issue since Management Agent installer does not allow quotes for environment variables like JAVA_HOME and AGENT_INSTALL_BASEDIR.

Action:

Troubleshoot: Management Agent status is "Not Available" in Console after the initial installation

Possible Cause 1: Incorrect system timestamp

Action: Verify the system time of the agent's host, and then you can correct the time if needed.

Possible Cause 2: If you use the input.rsp response file for the Management Agent, you must define the tags for your Management Agent compartment.

If the tags are not defined, you may see an error like this:

Attempts:
<--> Endpoint:       management-agent.us-ashburn-1.oci.oraclecloud.com
     opc-request-id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXJ8
     StartTime:      2024-09-18 03:45:12,662 GMT
     Status:         400 Bad Request
     Headers:        Strict-Transport-Security=max-age=31536000; includeSubDomains;
                     Connection=close
                     Content-Length=63
                     opc-request-id=XXXXXXXXXXXXXXXXXXXXXXXX..................B25ADA8E
                     Date=Wed, 18 Sep 2024 03:45:12 GMT
                     Content-Type=application/json
     ErrorBody:      
{
  "code" : "InvalidParameter",
  "message" : "Invalid tags"
}

Action: To define the tags specific for your environment, in the input.rsp response file, add the following parameters and specify the key-value pairs for your environment. For more information, see Create a Response file .

DefinedTags = [{"namespace1":{"<key1>":"<value1>"}},
      {"namespace2":{"<key2>":"<value2>"}}]

Troubleshoot: After configuration, the Management Agent is not visible in console or through the API

Possible Cause: If after you configure the management agent or the management gateway agent the agent does not display in the Oracle Cloud console or through the API, the correct policies may not be set up for the user or the user group.

Action: Verify the user or the user group has the required policies configured for the management agent or gateway agent. To setup polices, see Create policies for user group.

Troubleshoot: Prometheus or Kubernetes metrics monitored using Management Agent are not available

Troubleshoot: Agent runs into OutOfMemoryException

Possible Cause: The agent might run out of heap memory if it is not tuned properly to support the load that has been assigned to it.

Action: Update the heap memory settings for the Management Agent.

Troubleshoot: IP address being displayed in host column when Management Agent installed on Windows host

Problem: Management Agent is installed on a Windows host and the Management Agent console displays the IP address of the Windows host in the UI page instead of displaying fully qualified domain name (hostname) of the Windows host.

Management Agent installation should be successful and on the Agent UI page FQDN of the Windows host would be displayed in the host column

Troubleshoot: Management Agent installation fails on SELinux when using external volume

systemctl start mgmt_agent
Job for mgmt_agent.service failed because the control process exited with error code.
See "systemctl status mgmt_agent.service"and "journalctl -xeu mgmt_agent.service" for details.

journalctl -xeu mgmt_agent.service
...
Dec 08 15:48:19 ol9-arm systemd[1261408]: mgmt_agent.service: Failed to execute /dir1/oracle/managementagent/agent_inst/bin/agentcore: Permission denied
Dec 08 15:48:19ol9-arm systemd[1261408]: mgmt_agent.service: Failed at step EXEC spawning /dir1/oracle/managementagent/agent_inst/bin/agentcore: Permission denied

$ ausearch -ts recent -m avc -i
...
type=AVC msg=audit(12/08/202315:49:26.991:51338) : avc:  denied  { read open } for  pid=1261576comm=(gentcore) path=/dir1/oracle/managementagent/agent_inst/bin/agentcore dev="dm-0"ino=915154scontext=system_u:system_r:init_t:s0           tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

All the above error messages indicate that your SELinux does not allow you to execute commands in the chosen folder.

Action: Contact your system administrator and create the required policies that allow installing and running the Management Agent.

Troubleshoot: Management Agent installation fails on Red Hat Enterprise Linux 9.x

The Management Agent installation fails and the following error message may display: mgmt_agent service creation failed. Reason: Detected Linux .

Additionally, the install failure log messages may confirm the error and indicate the set up attempts use an incorrect service manager to install the agent.

Cause: Red Hat removed the chkconfig package in the Red Hat Enterprise Linux (RHEL) 9 distribution, for more details see the Red Hat Knowledge base.

Action:

Solution 1 - Install the chkconfig package

1. Install the missing package by executing the following command:

   $ dnf install chkconfig

2. Validate the package exists in the environment by executing the following command:

   $ rpm -qa | grep chkconfig

3. Install the Management Agent again.

Solution 2 - Without Installing the chkconfig package

If installing the chkconfig package is not an option as described in the above Solution 1 section, then complete the following steps as an alternative solution to install the Management Agent software.

$ sudo /bin/bash

$ export DIST_LINUX_FAMILY_OVERRIDE="Red Hat"

# RPM install
$ rpm -ivh <rpm_file_name.rpm>

# ZIP install
$ ./installer.sh <full_path_of_response_file>

• Error:… specifies multiple packages
• Error:… scriptlet failed with exit code

Error:… specifies multiple packages

Cause: The rpm registry has multiple packages with that name.

rpm -e oracle.mgmt_agent --allmatches

Error:… scriptlet failed with exit code

Cause: The rpm could not stop the running agent or failed to remove the agent service file from the system.

• Agent is in 'Not Available' state
• Management Agent setup failed with fork/exec oracle.polaris.oca.main: permission denied
• Management Agent authentication failure due to clock skew, a different time on the compute instance compared to the time on the server

Agent is in Not Available state and agent log file reports "Invalid tags"

The Management Agents page shows the Agent in 'Not available' state and the mgmt_agent.log file (located under <Agent_Inst>/logs directory) reports the following message:

ErrorBody:{"code" : "InvalidParameter","message" : "Invalid tags: Resource creation failed because the resource requires tag value(s). Add a value to the each of the following tag definition(s): \nGLOBAL.ComponentType, GLOBAL.ApplicationName,

Cause:

This issue can happen when the compartment requires mandatory tags for every resource and the resource creation request does not include the tags, then the activation request would fail with the message:"Invalid tags: Resource creation failed because the resource requires tag value(s)" and the agent status is shown as 'Not Available'.

Action:

• Management Agents

  If you have a standalone Management Agent, it must be uninstalled.

  If the Management Agent was installed using an RPM or a ZIP file, it must be uninstalled and reinstalled by providing a response file using the DefinedTags parameter as described in the Review Agent Parameters section.

• Management Agents on Compute Instances
  If the Management Agent is enabled through the OCI Console using the OCA plugin, then there is no response file since it's not used for compute instances. In this case, do the following:

  1. Login to the instance where the Management Agent is deployed and sudo as oracle-cloud-agent user using command:

     sudo -u oracle-cloud-agent sh

  2. Create agent.definedtags file under/var/lib/oracle-cloud-agent/plugins/oci-managementagent/polaris/agent_inst/config/security/resource/ directory.

  3. Add defined tags needed for the resource to be created in agent.definedtags file.

     For example, you can add the following:

     [{"GLOBAL":{"ComponentType":"<value>"}}, {"GLOBAL":{"ApplicationName":"<value>"}}]

  4. Restart oracle-cloud-agent using the command:

     sudo systemctl restart oracle-cloud-agent

Management Agent setup failed with fork/exec oracle.polaris.oca.main: permission denied

Users may encounter this error resulting in failure to install or start the Management Agent.

The error message shown in the Plugin view of compute instance for the Management Agent Plugin looks similar to the following:

workflow.go:23: [ERROR] step [*core.SetupImageStep] execution failed with [setup image failed with [fork/exec 230821.1905/bin/oracle.polaris.oca.main: permission denied]]
mgmtagent_image.go:139: [ERROR] bootstrap workflow failed with error setup image failed with [fork/exec 230821.1905/bin/oracle.polaris.oca.main: permission denied]
agent.go:74: [ERROR] failed to start agent during bootstrap with [setup image failed with [fork/exec 230821.1905/bin/oracle.polaris.oca.main: permission denied]]

Possible Cause:

This issue may happen when a compute instance disallows fork/execute operations from the /tmp directory by mounting the tmpfs with the noexec flag.

$ mount | grep tmpfs
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,inode64)

The output should say does have the noexec flag.

Action:

1. Stop Oracle Cloud Agent.

    sudo systemctl stop oracle-cloud-agent

2. Add the following setting to the file: /etc/oracle-cloud-agent/plugins/oci-managementagent/config.yml

   overrideTmpDir: true

3. Start Oracle Cloud Agent.

   $ sudo systemctl start oracle-cloud-agent

Management Agent authentication failure due to clock skew, a different time on the compute instance compared to the time on the server

Cause: If there is a clock skew of more than 5 minutes between the Compute Instance where the agent is running and Oracle Cloud Infrastructure Identity service, then the requests will be rejected with a HTTP 401.

On the OCI Compute Instance, go to the Oracle Cloud Agent tab, the Management Agent displays an error in the Message column:

rpc error: code = Unavailable desc = connection error: desc = "transport: error while
      dialing: dial unix /var/lib/oracle-cloud-agent/tmp/plugin1825606937: connect: connection
      refused"

Or in the logs you may find the following error:

2022-12-09 07:41:22,144 [SysExecutor.0 (Resource Principal Token Refresher)-47] WARN  - #-#  invocation access log [request-id-prefix: K9YBE4AY] #-#
Service: OCI
Method:  GET
Path:    /20200202/managementAgents/ocid1.managementagent.....rvf6i3ba/resourcePrincipalToken
Headers:opc-rpt-request-token=********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************         
        date=Fri, 09 Dec 2022 07:41:19 GMT         
        host=management-agent.ap-tokyo-1.oci.oraclecloud.com         
        Authorization=******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************         
        opc-request-id=K9YBE4AYQFIMP2J6HGQ2JUKU1IJPFYVO         
        User-Agent=Jersey/2.34 (Apache HttpClient 4.5.13)Attempts:
<--> Endpoint:       management-agent.ap-tokyo-1.oci.oraclecloud.com     
     opc-request-id: K9YBE4AYQFIMP2J6HGQ2JUKU1IJPFYVO     
     StartTime:      2022-12-09 07:41:19,976 GMT     
     Status:         401 Unauthorized     
     Headers:        X-Content-Type-Options=nosniff                     
                     Content-Length=187                         
                     opc-request-id=K9YBE4AYQFIMP2J6HGQ2JUKU1IJPFYVO/E4356B68C6C541BAD867E46760316D35/4118B130EE46A8E25F90DC91AB7F12D7                     
                     Date=Fri, 09 Dec 2022 07:41:21 GMT                     
                     Content-Type=application/json     
      ErrorBody:
{  
    "code" : "NotAuthenticated",  
    "message" : "Unable to authenticate the request for ocid1.managementagent.oc1.ap-tokyo-1.amaaaa...6frjnrbvqrvf6i3ba"}

Action

Fix the clock skew and restart. If the agent has been down for days because of this error, then you must clean up thedonotrestart file before restarting the agent.

Additionally, Oracle recommends to set up the OS date time to auto-sync with NTP servers to avoid downtime in the future. If additional services are running on the machine it's best practice to restart the machine after the time change so the services can reset with the new time.

To correct the OS date time where the agent is running and then restart the agent you can follow these steps:

1. To stop the agent run the following command:

   sudo systemctl stop oracle-cloud-agent

2. Correct the date and time.
3. Run the following command to delete the configure.donotrestart file.

   sudo rm /var/lib/oracle-cloud-agent/plugins/oci-managementagent/polaris/agent_inst/config/configure.donotrestart

4. Start the agent.

   sudo systemctl start oracle-cloud-agent

• Troubleshoot: Remove Management Gateway
• Troubleshoot: Configure Management Gateway
• Troubleshoot: Management Gateway installation fails on Red Hat Enterprise Linux 9.x
• Troubleshoot: Management Gateway Installation Fails With Error: Certificates could not be created and the Identity logs report: Authentication failed: DATE_OUTSIDE_CLOCK_SKEW
• Troubleshoot: When installing or configuring Management Gateway, Timed Out Error

Troubleshoot: Remove Management Gateway

Cause: In some cases, it may be necessary to remove an existing Management Gateway installation, in order to reinstall it.

Troubleshoot: Configure Management Gateway

Cause: In some cases, the hostname might not be resolved in the installation environment which might cause the installation to fail with the following error message:

Troubleshoot: "Could not resolve hostname <hostname value> in the installation environment. Resolve the hostname or provide the GatewayCertCommonName in the response file and rerun the gateway setup script."

Action:

• Check and resolve the hostname of the environment to get the fully qualified doamin name (FQDN) value after running the command: hostname -f
• Optionally a user can provide a custom fully qualified domain name for the gateway configuration via seeding the GatewayCertCommonName property in input response file. See Response File Parmaters
• Re-run the configure gateway script again.

  sudo /opt/oracle/mgmt_agent/agent_inst/bin/setupGateway.sh opts=<user_home_directory>/gateway.rsp

Cause: In some cases, the Management Gateway installation might fail with the following error message due to the absence of policies in OCI or because of resource limit issues in the tenancy. If you see the following error, follow the steps below.

Troubleshoot: "Failed to start Management Gateway as certificates could not be created, initialized or retrieved in OCI. Please check the logs for more details."

Action:

• Open the log file in the Management Gateway installation directory, for example: /opt/oracle/mgmt_agent/plugins/GatewayProxy/statedir/log/mgmt_gateway.log
• If the log file contains any of the following 404 error codes, then choose one of the following options to resolve the issue:

  2023-07-25 15:38:06.694/CEST [pool-3-thread-1] INFO  com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateUtility - Response String {  "code" : "NotAuthorizedOrNotFound",  "message" : "Authorization failed or requested resource not found."}
  2023-07-25 15:38:06.696/CEST [pool-3-thread-1] ERROR com.oracle.mgmtagent.proxy.ProxyServer - Error while initializing and loading certificate bundlescom.oracle.mgmtagent.proxy.exception.CertificateFailureException: The response status is 404 after multiple retries at com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateUtility.executeRequest(CertificateUtility.java:293) ~

  • Recommended option: Use the Management Gateway Quick Start Marketplace Application to automatically create the dynamic groups, policies, and manage the certificates required to install Management Gateway.
  • Manually add and confirm the correct dynamic groups and policies required for installing the Management Gateway are added to the specific compartment within the tenancy where you want to install the Management Gateway. For more information, see Perform Prerequisites for Deploying Management Gateway.

• If the log file contains any of the following 400 error codes, then review the following options to resolve the issue:

  2023-09-20 18:51:32.772/GMT [pool-3-thread-1] INFO  com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateCreationUtil - Create Vault Service Url invoked https://kms.us-ashburn-1.oraclecloud.com/20180608/vaults
  2023-09-20 18:51:33.400/GMT [pool-3-thread-1] INFO  com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateUtility - Received response code 400
  2023-09-20 18:51:33.400/GMT [pool-3-thread-1] INFO  com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateUtility - Header name opc-request-id , value /5704D03441842D3818B824B2D6B2712E/1D1FED893474FDA900188E24F3DEE59B
  2023-09-20 18:51:33.401/GMT [pool-3-thread-1] INFO  com.oracle.mgmtagent.proxy.oci.certificate.util.CertificateUtility - Response String {  "code" : "LimitExceeded",  "message" : "The limit for this tenancy has been exceeded."}

  • Check the limit for the Default Vault Count resource for the Key Management Service in OCI console. You can raise a request to increase the resource limits. For more information, see Managing Keys and Managing Vaults.
  • You can set up certificates manually, for details see Perform Prerequisites for Deploying Management Gateway and go to the Manual Certificate Management section.
    Note
    
    When you create the Issued by internal CA certificates, the Certificate Profile must be either TLS Server or TLS Client and only the RSA signing algorithms are supported.

• If there are any other failures related to the Vault or the Key service API's in the logs, then you can raise a request and reach out to the oci_kms team by providing the response body and opc-request-id.
• If there are any other failures related to Certificate Authorities or Certificate service API's in the logs, then raise a request and reach out to oci_certificates team by providing the response body and opc-request-id.

Troubleshoot: Management Gateway installation fails on Red Hat Enterprise Linux 9.x

Additionally, the install failure log messages may confirm the error and indicate the set up attempts use an incorrect service manager to install the gateway.

Cause: Red Hat removed the chkconfig package in the Red Hat Enterprise Linux (RHEL) 9 distribution, for more details see the Red Hat Knowledge base.

Action:

1. Install the missing package by executing the following command:

   $ dnf install chkconfig

2. Validate the package exists in the environment by executing the following command:

   $ rpm -qa | grep chkconfig

3. Install the Management Gateway again.

If installing the chkconfig package is not an option as described in the Solution 1 section above, then complete the following steps as an alternative solution to install the Management Gateway software.

$ sudo /bin/bash

$ export DIST_LINUX_FAMILY_OVERRIDE="Red Hat"

# RPM install
$ rpm -ivh <rpm_file_name.rpm>

# ZIP install
$ ./installer.sh <full_path_of_response_file>

Troubleshoot: Management Gateway Installation Fails With Error: Certificates could not be created and the Identity logs report: Authentication failed: DATE_OUTSIDE_CLOCK_SKEW

Cause:

Identity logs report authentication failed: DATE_OUTSIDE_CLOCK_SKEW

# /opt/oracle/mgmt_agent/agent_inst/bin/setupGateway.sh opts=<PATH>/gateway_agent.rsp/opt/oracle/mgmt_agent/agent_inst/bin/setupAgent.sh
    opts=<PATH>/gateway_agent.rsp

Executing configure

Parsing input response file
Validating install key
Generating communication wallet
Generating security artifactsRegistering Management Gateway
Found service plugin(s):[GatewayProxy]

Starting gateway...
Gateway started successfully
Starting plugin deployment for: [GatewayProxy]
Deploying service plugin(s)...Done.
GatewayProxy : Successfully deployed external plugin

Gateway setup completed and the gateway is running.
In the future gateway can be started by directly running: sudo systemctl start mgmt_gateway

Please make sure that you delete <PATH>/gateway_agent.rsp or store it in secure location.

Creating gateway system properties file
Creating properties fileCreating or validating certificates
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Waiting for Management Gateway to create or validate certificates...
Failed to start Management Gateway as certificates could not be created, initialized or retrieved in OCI. Please check the logs for more details.
Management Gateway stopped

Action:

On the host where Management Gateway is installed, ensure the host time is correct and then install Management Gateway.

Troubleshoot: When installing or configuring Management Gateway, Timed Out Error

If you verify the OCI Console displays the Management Gateway as active, but the metrics are not populating.

/opt/oracle/mgmt_agent/agent_inst/bin/setupGateway.sh opts=<user_home_directory>/gateway.rsp
Starting gateway...
Gateway started successfully

Starting plugin deployment for: [GatewayProxy]
Deploying service plugin(s)...............Timed out.

Agent is unable to check if it deployed requested service plugin(s) successfully or not. 
Please check back later on the console.

Cause: A longer than expected time to complete the Management Gateway setup task may result in a network communication issue and may cause the Management Gateway to time out.