Database backups can fail during the RMAN configuration stage or during a running RMAN backup job. RMAN configuration tasks include validating object store connectivity, backup module installation, and RMAN configuration changes. The log files you examine depend on which stage the failure occurs.

1. Log on to the host as the root user.

2. Check the applicable log file:

   • If the failure occurred during RMAN configuration, navigate to the /var/opt/oracle/log/<database_name>/bkup/ directory and check the bkup.log file.
   • If the failure occurred during the backup job, navigate to the /var/opt/oracle/log/<database_name>/obkup/ directory and check the obkup.log file.

1. From a command prompt, check the status of the agent:

   systemctl status dbcsagent.service

2. If the agent is in the stop/waiting state, try to restart the agent:

   systemctl start dbcsagent.service

3. Check the status of the agent again to confirm that it has the stop/running status:

   systemctl status dbcsagent.service

1. Create a Swift user in your tenancy. See Working with Auth Tokens.
2. With the user you created in the previous step, use the following command to verify the host can access the object store.

   curl -v -X HEAD -u <user_ID>:'<auth_token>' https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<object_storage_namespace>

   See Object Storage FAQ for the correct region to use. See Understanding Object Storage Namespaces for information about your Object Storage namespace.
3. If you cannot connect to the object store, refer to Prerequisites for Backups on Exadata Cloud Service topic for information on configuring object store connectivity.

If an interactive command such as oraenv, or any command that might return an error or warning message, was added to the .bash_profile file for the grid or oracle user, Database service operations like automatic backups can be interrupted and fail to complete. Check the .bash_profile file for these commands, and remove them.

Backup operations require space in the /u01 directory on the host file system. Use the df -h command on the host to check the space available for backups. If the file system has insufficient space, you can remove old log or trace files to free up space.

Your system might not have the required version of the backup module (opc_installer.jar). See Unable to use Managed Backups in your DB System for details about this known issue. To fix the problem, you can follow the procedure in that section or simply update your DB system and database with the latest bundle patch.

Customizing the site profile file ( $ORACLE_HOME/sqlplus/admin/glogin.sql ) can cause managed backups to fail in Oracle Cloud Infrastructure. In particular, interactive commands can lead to backup failures. Oracle recommends that you not modify this file for databases hosted in Oracle Cloud Infrastructure.

The database must be active and running (ideally on all nodes) while the backup is in progress.

When you provision a new database, the archiving mode is set to ARCHIVELOG by default. This is the required archiving mode for backup operations. Check the archiving mode setting for the database and change it to ARCHIVELOG, if applicable.

Instance <instance_identifier> is running on node *<node_identifier>. Instance status: Stuck Archiver

Refer to ORA-00257:Archiver Error (Doc ID 2014425.1) for information on resolving a stuck archiver process.

Instance <instance_identifier> is running on node *<node_identifier>. Instance status: Open

If the instance status does not change after you resolve the underlying issue with the device or resource being full or unavailable, try restarting the database using the srvctl command to update the status of the database in the clusterware.

Editing certain RMAN configuration parameters can lead to backup failures in Oracle Cloud Infrastructure. To check your RMAN configuration, use the show all command at the RMAN command line prompt.

See the following list of parameters for details about RMAN the configuration settings that should not be altered for databases in Oracle Cloud Infrastructure.

CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;

CONFIGURE CONTROLFILE AUTOBACKUP ON;

CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 5 BACKUP TYPE TO COMPRESSED BACKUPSET;

CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/var/opt/oracle/dbaas_acfs/<db_name>/opc/libopc.so, ENV=(OPC_PFILE=/var/opt/oracle/dbaas_acfs/<db_name>/opc/opc<db_name>.ora)';

CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';

CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;

CONFIGURE ENCRYPTION FOR DATABASE ON;

RMAN backups fail when an object store wallet file is lost. The wallet file is necessary to enable connectivity to the object store.

Database backups fail if the TDE wallet is not in the proper state. The following scenarios can cause this problem:

If the database was started using SQL*Plus, and the ORACLE_UNQNAME environment variable was not set, the wallet is not opened correctly.

srvctl start database -d <db_unique_name>

In a multitenant environment for Oracle Database versions that support PDB-level keystore, each PDB has its own master encryption key. For Oracle 18c databases, this encryption key is stored in a single keystore used by all containers. (Oracle Database 19c does not support a keystore at the PDB level.) After you create or plug in a new PDB, you must create and activate a master encryption key for it. If you do not do so, the STATUS column in the v$encryption_wallet view shows the value OPEN_NO_MASTER_KEY.

To check the master encryption key status and create a master key, do the following:

1. Review the the STATUS column in the v$encryption_wallet view, as shown in the following example:

   
   SQL> alter session set container=pdb2;
   Session altered.
   
   SQL> select WRL_TYPE,WRL_PARAMETER,STATUS,WALLET_TYPE from v$encryption_wallet;
   
   WRL_TYPE   WRL_PARAMETER                                   STATUS             WALLET_TYPE
   ---------- ----------------------------------------------- ------------------ -----------
   FILE       /var/opt/oracle/dbaas_acfs/testdb30/tde_wallet/ OPEN_NO_MASTER_KEY AUTOLOGIN

2. Confirm that the PDB is in READ WRITE open mode and is not restricted, as shown in the following example:

   
   SQL> show pdbs
   
   CON_ID CON_NAME     OPEN MODE              RESTRICTED
   ------ ------------ ---------------------- ---------------
   2      PDB$SEED     READ ONLY              NO
   3      PDB1         READ WRITE             NO
   4      PDB2         READ WRITE             NO

   The PDB cannot be open in restricted mode (the RESTRICTED column must show NO). If the PDB is currently in restricted mode, review the information in the PDB_PLUG_IN_VIOLATIONS view and resolve the issue before continuing. For more information on the PDB_PLUG_IN_VIOLATIONS view and the restricted status, review the Oracle Multitenant Administrator’s Guide on pluggable database for your Oracle Database version.

3. Create and activate a master encryption key for the PDB:

   • Set the container to the PDB:

     ALTER SESSION SET CONTAINER = <pdb>;

   • Create and activate a master encryption key in the PDB by executing the following command:

     ADMINISTER KEY MANAGEMENT SET KEY USING TAG '<tag>' 
     FORCE KEYSTORE IDENTIFIED BY <keystore-password> WITH BACKUP USING '<backup_identifier>';

   Note the following:

   • The USING TAG clause is optional and can be used to associate a tag with the new master encryption key.

   • The WITH BACKUP clause is optional and can be used to create a backup of the keystore before the new master encryption key is created.

   You can also use the dbaascli commands dbaascli tde status and dbaascli tde rotate masterkey to investigate and manage your keys.

4. Confirm that the status of the wallet has changed from OPEN_NO_MASTER_KEY to OPEN by querying the v$encryption_wallet view as shown in step 1.

Configuration parameters related to the TDE wallet can cause backups to fail.

# ls -altr /var/opt/oracle/dbaas_acfs/<database_name>/tde_wallet/ewallet.p12
total 76
-rw------ 1 oracle oinstall 5467 Oct 1 20:17 ewallet.p12

The TDE wallet file should have file permissions with the octal value "600" (-rw-------), and the owner of this file should be a part of the oinstall operating system group.

# ls -altr /var/opt/oracle/dbaas_acfs/<database_name>/tde_wallet/cwallet.sso
total 76
-rw------ 1 oracle oinstall 5512 Oct 1 20:18 cwallet.sso

The auto login wallet file should have file permissions with the octal value "600" (-rw-------), and the owner of this file should be a part of the oinstall operating system group.

On each compute node associated with the Data Guard configuration, gather log files pertaining to the problem you experienced.

• Enablement stage log files (such as those documenting the Create Standby Database operation) and the logs for the corresponding primary or standby system.
• Enablement job ID logfiles. For example: 23.
• Locations of enablement log files by enablement stage and Exadata system (primary or standby).
• Database name logfiles (db_name or db_unique_name, depending on the file path).

Data Guard Deployer (DGdeployer) is the process that performs the configuration. When configuring the primary database, it creates the /var/opt/oracle/log/<dbname>/dgdeployer/dgdeployer.log file.

This log should contain the root cause of a failure to configure the primary database.

• The primary log from the dbaasapi command-line utility is: /var/opt/oracle/log/dbaasapi/db/dg/<job_ID>.log. Look for entries that contain dg_api.
• One standby log from the dbaasapi command-line utility is: /var/opt/oracle/log/dbaasapi/db/dg/<job_ID>.log. In this log, look for entries that contain dg_api.
• The other standby log is: /var/opt/oracle/log/<dbname>/dgcc/dgcc.log. This log is the Data Guard configuration log.

• The Oracle Cloud Deployment Engine (ODCE) creates the /var/opt/oracle/log/<dbname>/ocde/ocde.log file. This log should contain the cause of a failure to create the standby database.
• The dbaasapi command line utility creates the var/opt/oracle/log/dbaasapi/db/dg/<job_ID>.log file. Look for entries that contain dg_api.
• The Data Guard configuration log file is /var/opt/oracle/log/<dbname>/dgcc/dgcc.log.

• DGdeployer is the process that performs the configuration. It creates the following /var/opt/oracle/log/<dbname>/dgdeployer/dgdeployer.log file. This log should contain the root cause of a failure to configure the standby database.
• The dbaasapi command-line utility creates the /var/opt/oracle/log/dbaasapi/db/dg/<job_ID>.log file. Look for entries that contain dg_api.
• The Data Guard configuration log is /var/opt/oracle/log/<dbname>/dgcc/dgcc.log.

DGdeployer is the process that performs the configuration. While configuring Data Guard, it creates the /var/opt/oracle/log/<dbname>/dgdeployer/dgdeployer.log file. This log should contain the root cause of a failure to configure the primary database.

On each node of the primary and standby sites, gather log files for the related database name (db_name).

• Database alert log: /u02/app/oracle/diag/rdbms/<dbname>/<dbinstance>/trace/alert_<dbinstance>.log
• Data Guard Broker log: /u02/app/oracle/diag/rdbms/<dbname>/<dbinstance>/trace/drc<dbinstance>.log
• Cloud tooling log file for Data Guard: /var/opt/oracle/log/<dbname>/odg/odg.log

The password entered for enabling Data Guard didn't match the primary admin password for the SYS user. This error occurs during the Validate Primary stage of enablement.

The database may not be running. This error occurs during the Validate Primary stage of enablement. Check with srvctl and sql on the host to verify that the database is up and running on all nodes.

The primary database could not be configured. Invalid Data Guard commands or failed listener reconfiguration can cause this error.

The TDE wallet could not be created. The Oracle Transparent Database Encryption (TDE) keystore (wallet) files could not be prepared for transportation to the standby site. This error occurs during the create TDE Wallet stage of enablement. Either of the following items can cause failure at this stage:

• The TDE wallet files could not be accessed
• The enablement commands could not create an archive containing the wallet files

Troubleshooting procedure:

1. Ensure that the cluster is accessible. To check the status of a cluster, run the following command:

   crsctl check cluster -all

2. If the cluster is down, run the following command to restart it:

   crsctl start crs -wait

3. If this error occurs when the cluster is accessible, check the logs for create TDE Wallet (enablement stage) to determine cause and resolution for the error.

The archive containing the TDE wallet was likely not transmitted to the standby site. Retrying usually solves the problem.

• The primary and standby sites may not be able to communicate with each other to configure the standby database. These errors occur during the configure standby database stage of enablement. In this stage, configurations are performed on the standby database, including the rman duplicate of the primary database. To resolve this issue:
  1. Verify the connectivity status for the primary and standby sites.
  2. Ensure that the host can communicate from port 1521 to all ports. Check the network setup, including Network Security Groups (NSGs), Network Security Lists, and the remote VCN peering setup (if applicable). The best way to test communication between the host and other nodes is to access the databases using SQL*PLUS from the primary to standby and from the standby to the primary.
• The SCAN VIPs or listeners may not be running. Use the test above to help identify the issue.

Possible causes:

• SCAN VIPs or listeners may not be running. You can confirm this issue by using the following commands on any cluster node.

  • [grid@exa1-****** ~]$ srvctl status
                scan

  • [grid@exa1-****** ~]$ srvctl status
                  scan_listener

• Databases may not be reachable. You can confirm this issue by attempting to connect using an existing Oracle Net alias.

Troubleshooting procedure:

1. As the oracle OS user, check for the existence of an Oracle Net alias for the container database (CDB). Look for an alias in $ORACLE_HOME/network/admin/<dbname>/tnsnames.ora.

   The following example shows an entry for a container database named db12c:

   cat $ORACLE_HOME/network/admin/db12c/tnsnames.ora 
   DB12C = (DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = exa1-*****-scan.********.******.******.com)(PORT = 1521)) 
   (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = db12c.********.******.******.com) 
   (FAILOVER_MODE = (TYPE = select) (METHOD = basic))))

2. Verify that you can use the alias to connect to the database. For example, as sysdba, enter the following command:

   sqlplus sys@db12c

A possible cause for this error is that the Oracle Database sys or system user passwords for the database and the TDE wallet may not be the same. To compare the passwords:

1. Connect to the database as the sys user and check the TDE status in

   V$ENCRYPTION_WALLET

   .
2. Connect to the database as the system user and check the TDE status in

   V$ENCRYPTION_WALLET

   .
3. Update the applicable passwords to match. Log on to the system host as opc and run the following commands:
   1. To change the SYS password:

      sudo dbaascli database changepassword --dbname <database_name>

   2. To change the TDE wallet password:

      sudo dbaascli tde changepassword --dbname <database_name>

For possible causes and resolutions to TDE wallet issues, see TDE Wallet and Backup Failures .

When the switchover, failover, and reinstate commands are run, multiple error messages may occur. Refer to the Oracle Database documentation for these error messages.

Note

Oracle recommends using the Data Guard broker command line interface (dgmgrl) to validate the configurations.

1. As the Oracle User, connect to the primary or standby database with dgmgrl and verify the configuration and the database:

   dgmgrl sys/<pwd>@<database>
   DGMGRL> VALIDATE CONFIGURATION VERBOSE
   DGMGRL> VALIDATE DATABASE VERBOSE <PRIMARY>
   DGMGRL> VALIDATE DATABASE VERBOSE <STANDBY>

2. Consult the Oracle Database documentation to check for the respective error message. For example:
   • ORA-16766: Redo apply is stopped.
   • ORA-16853: Apply lag has exceeded specified threshold.
   • ORA-16664: Unable to receive the result from a member (under the standby database).
   • ORA-12541: TNS: no listener (under the primary database)

For cause and resolution, review the errors in Database Error Messages.

DBAASCLI Logs