Troubleshooting the Terraform Provider
Troubleshoot common OCI Terraform provider issues.
Start with Troubleshooting Basics and then see the following guidance:
Troubleshooting Basics
When troubleshooting or getting support for the OCI Terraform provider, it's often useful to first check the status of the OCI services, the version of Terraform and the provider, and enable and collect verbose logging.
Checking service status and verbose log output can help you determine whether an issue is related to the Terraform provider or the OCI service the provider is using.
Refer to the list of common issues after you start with the basics.
Checking OCI Service Status and Outages
To check on the latest status and whether there are any outages in OCI, see OCI Status.
Checking the Terraform and OCI Terraform Provider Versions
To verify the version of Terraform and the OCI Terraform provider, initialize Terraform from a directory with your configurations and then run the -version
command. For example:
terraform init
terraform -version
The versions are displayed:
Terraform v0.12.20
+ provider registry.terraform.io/hashicorp/oci v3.95.0
Newer versions of the OCI Terraform provider include the version of the provider in error messages.
The OCI Terraform provider documentation reflects the latest version. You can view documentation for earlier provider versions by visiting the HashiCorp reference and selecting a specific version. You can also download and install a specific version of the provider.
Verbose Logging for OCI Terraform Provider
To get verbose console output when the provider is running, precede your Terraform command with the TF_LOG
and OCI_GO_SDK_DEBUG
flags. For example:
TF_LOG=DEBUG OCI_GO_SDK_DEBUG=v terraform plan
The TF_LOG
level and OCI_GO_SDK_DEBUG
flags can also be set as environment variables.
Automatic Retries
While applying, refreshing, or destroying a plan, Terraform may encounter some
intermittent OCI errors (such as 429 or 500 errors) that could succeed on retry. By
default, the OCI Terraform provider automatically retries such operations for up to 10
minutes. The following fields can be specified in the provider
block to
further configure the retry behavior:
disable_auto_retries
- Disable automatic retries for retriable errors.retry_duration_seconds
- The minimum duration (in seconds) to retry a resource operation in response to HTTP 429 and HTTP 500 errors. The actual retry duration may be slightly longer due to jittering of retry operations. This value is ignored if thedisable_auto_retries
field is set totrue
.
Concurrency Control Using Retry Backoff and Jitter
To alleviate contention between parallel operations against OCI services; the OCI Terraform provider schedules retry attempts using quadratic backoff and full jitter. Quadratic backoff increases the maximum interval between subsequent retry attempts, while full jitter randomly selects a retry interval within the backoff range.
For example, the wait time between the first and second retry attempts is chosen randomly between 1 and 8 seconds. The wait time between the second and third retry attempts is chosen randomly between 1 and 18 seconds. Regardless of the number of retry attempts, the retry interval time is capped after the 12th attempt at 288 seconds.
Note that the retry_duration_seconds
field only affects retry duration
in response to HTTP 429 and 500 errors; as these errors are more likely to result in
success after a long retry duration. Other HTTP errors (such as 400, 401, 403, 404, and
409) are unlikely to succeed on retry. The retry_duration_seconds
field
does not affect the retry behavior for such errors.
Common Issues
Resources Are Destroyed When Applying Changes
Existing OCI resources might be destroyed and
re-created when Terraform configurations attempt to update a resource property that is
not updatable. Terraform warns you when changes will destroy a resource. Always run
terraform plan
before applying changes to see what resources will
be affected. See Destructive Changes for more
information.
Dependent Resource Cannot Be Updated
Sometimes, an update to an OCI resource should force the resource to be destroyed and re-created, but another resource's dependency on the resource disallows the operation.
For example, you might want to make an update to an
oci_core_instance_configuration
resource, but an instance pool uses
that instance configuration. The instance configuration cannot be deleted because the
instance pool references it in a required argument.
To work around this type of behavior, you can use the lifecycle
and
create_before_destroy
meta-arguments in the resource block.
In this example, Terraform creates a second
oci_core_instance_configuration
resource that includes your
updates, then assigns the new instance configuration to the related instance pool.
Finally, Terraform destroys the original instance configuration. For example:
resource "oci_core_instance_configuration" "test_instance_configuration" {
...
lifecycle {
create_before_destroy = true
}
}
See The lifecycle
Meta-Argument
and Destructive Changes
for more information.
Resource Cannot Be Destroyed or Updated
You can prevent an OCI resource from being
destroyed by including the lifecycle
and prevent_destroy =
true
meta-arguments in the resource block of your Terraform configuration
file. The following configuration, for example, results in an Object Storage bucket that cannot be destroyed:
resource "oci_objectstorage_bucket" "test_bucket" {
#Required
compartment_id = var.tenancy
name = "test"
namespace = "exampleNamespace"
lifecycle {
prevent_destroy = true
}
}
This meta-argument prevents the use of terraform destroy
. Because
certain configuration updates require resources to be destroyed before they can be
applied, this setting can make some updates impossible to apply as well. In this
example, name
is a property that cannot be updated without destroying
and re-creating the resource. Therefore, you cannot update the name of the bucket
without removing or changing the lifecycle
meta-argument.
See The lifecycle
Meta-Argument
for more information.
Resource Argument Cannot Be Unset
Many Oracle Cloud Infrastructure resources managed by the OCI Terraform provider accept configuration arguments that are optional. Once set, whether during resource creation or a subsequent update, these arguments cannot be unset by passing an empty string or removing the argument from the configuration. Attempts to unset these arguments are ignored by Terraform.
The full reference of the OCI Terraform provider's supported resources and data sources contains usage, argument, and attribute details. The full reference is available at docs.oracle.com and Terraform Registry.
Data sources and resources are grouped by service within the reference.
Referencing Triggers in Lifecycle Meta-arguments
Terraform v0.14 and later might require that you replace global variables in your
configuration files with a combination of local variables and triggers. To reference a
trigger in lifecycle
and ignore_changes
meta-arguments
and avoid executing the configuration on subsequent Terraform apply operations,
reference the trigger as follows:
resource "null_resource" "exampleB" {
depends_on = [null_resource.exampleA]
triggers = {
os_user = var.os_user
}
lifecycle {
ignore_changes = [
triggers["os_user"]
]
}
Cannot Delete Compartment
By default, the Terraform provider does not delete a compartment when using the
destroy
command.
You must set the enable_delete
argument to true
for the
provider to attempt to delete the compartment. For example:
resource "oci_identity_compartment" "test_compartment" {
compartment_id = var.compartment_id
description = var.compartment_description
name = var.compartment_name
enable_delete = true
}
To destroy a compartment, the compartment must also be empty. Use the
depends_on
argument to ensure that any hidden dependencies are defined. See
Resources for more information."Operation Timeout" Error
If the Terraform CLI returns an error message like the following:
Error: Operation Timeout
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: timeout while waiting for state to become 'SUCCEEDED, FAILED, CANCELED' (last state: 'IN_PROGRESS', timeout: 15m)
Then the specified OCI service is indicating that the resource has not yet reached the expected state after polling for some time.
You may need to increase the operation timeout for your resource to continue polling for longer. See Operation Timeouts for details on how to do this.
"Unexpected LifeCycle state" Error
If the Terraform CLI returns an error message like the following:
Error: Unexpected LifeCycle state
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: During deletion, Terraform expected the resource to reach state: TERMINATED, but the service reported unexpected state: RUNNING.
Resource OCID: exampleuniqueID
Suggestion: Please retry or contact support for help with service: <service>
Then the specified OCI service encountered an unknown error. Retry, or contact support regarding that service.
Terraform CLI Issues
This section contains information dealing with the installation and configuration of the Terraform CLI.
"No such file" Error After Upgrading the OCI Terraform Provider
If the Terraform CLI returns an error message like the following:
Error asking for user input: 1 error(s) occurred:
* provider.oci: dial unix /var/folders/6r/8fk5dmbj4_z3sl0mc_y_fhjw0000gn/T/plugin811254328|netrpc: connect: no such file or directory
You are likely using a version of the OCI Terraform provider that is not compatible with the Terraform binary you have installed. For OCI Provider versions v3.x.x and above, a minimum Terraform version of v.0.10.1 is required.
"TCP...i/o timeout" Message When Connecting Via Proxy
If the Terraform CLI returns an error message like the following:
* provider.oci: ... dial tcp 134.70.16.0:443: i/o timeout
Then you may not have properly configured your proxy settings. The OCI Terraform provider
supports http_proxy
, https_proxy
and
no_proxy
variables where the inclusion or exclusion lists can be
defined as follows:
export http_proxy=http://www.your-proxy.com:80/
export https_proxy=http://www.your-proxy.com:80/
export no_proxy=localhost,127.0.0.1
"x509: certificate signed by unknown authority" Error Message
If the Terraform CLI returns an error message like the following:
Error: Get https://iaas.<region>.oraclecloud.com/20160918/services: x509: certificate signed by unknown authority
on ../modules/network/modules/main/main.tf line 3...
Ensure that Terraform is using trusted TLS certificates and the certificate chain is valid. See Terraform runs failing with "x509: certificate signed by unknown authority" error for more information. If using MacOS Catalina, refer to the MacOS section of the document for more specifics on resolving certificate issues.
"Outdated GPG key...unable to verify new provider releases" Error Message
If the Terraform CLI returns an error message like the following:
Warning: registry.terraform.io:
This version of Terraform has an outdated GPG key and is unable to verify new provider releases.
Please upgrade Terraform to at least <version> to receive new provider updates.
For details see: https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512
This message means that the Terraform registry is omitting the Terraform provider versions signed by a new GPG key. The Terraform CLI will install the last version of the OCI Terraform provider that it can successfully verify, which might not be the latest version.
To remove this message and ensure you can use the latest version of the OCI Terraform provider, upgrade the Terraform CLI to the latest maintenance release available for the major Terraform version you are using. For example, if you are using Terraform v0.12.21, upgrade to the lastest available version of v0.12.
Terraform Provider Issues
This section contains information related to the installation and configuration of the OCI Terraform provider.
"NotAuthenticated" Error When Using Modules
If you are using modules and the Terraform CLI returns an error message like the following:
Error: 401-NotAuthenticated, The required information to complete authentication was not provided or was incorrect.
Verify that each module declares its own provider requirements. For more information, see Providers Within Modules.
"NotAuthenticated" Error When Using Terraform
Symptom: The Terraform CLI returns an error message such as the following:
Error: 401-NotAuthenticated
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: The required information to complete authentication was not provided or was incorrect.
OPC request ID: exampleuniqueID
Suggestion: Please retry or contact support for help with service: <service>
Possible causes: Incorrect configuration.
Resolution: Verify the following.
- Correctly configured attributes:
user_ocid
tenancy_ocid
fingerprint
private_key_path
- verify that it's pointing to a private key, and not the corresponding public key.
- The public key corresponding to
private_key_path
was added to the user account specified asuser_ocid
. - The public and private key pairs use the correct format. For details, and for steps to generate keys, see Required Keys and OCIDs.
- The user account is part of a group with the appropriate permissions to perform the actions in the plan you're executing.
- The tenancy is subscribed to the region targeted in the plan.
- If you're using modules, verify that each module declares its own provider requirements. For more information, see Providers Within Modules.
"Can not create client, bad configuration: did not find a proper configuration for tenancy" Message When Using Aliases
If the Terraform CLI returns a message like this, it might indicate an issue with your environment:
Error: can not create client, bad configuration: did not find a proper configuration for tenancy
If your provider configuration includes an alias, your resources should explicitly
specify the provider alias using provider = "oci.alias_name"
. If a
resource does not use the alias to specify the provider, Terraform creates a
default provider to use with such resources. The default provider loads configuration
values from environment variables or the ~/.oci/config
file. These
values may differ from those used by your aliased provider and cause the configuration
error.
Either remove the alias in your provider configuration, or ensure that every resource
specifies the provider by the proper alias. Read more about using alias
in the official Terraform documentation, and see
Configuring the Provider for more information about how
Terraform uses environment variables and the OCI config file.
"Field cannot be set" Error Message
If the Terraform CLI returns an error message like the following:
* Error: "field_name": this field cannot be set
You are likely using an older version of the OCI Terraform provider and the field you are trying to set was released in a later version. Use the following command to check your Terraform provider version.
terraform -version
The OCI Terraform provider documentation reflects the latest version.
"Could not get info about the first DbHome in the dbSystem" Error Message When Importing db_home
If the oci_database_db_system
being imported is missing a primary
db_home
, an empty placeholder for db_home
is set
in the Terraform state file. To keep configurations consistent with the imported state,
add an empty placeholder for db_home
to your configuration. For
example:
# Add this placeholder into your oci_database_db_system configuration to indicate that the primary db home is empty.
db_home {
database {
admin_password = ""
}
}
The full reference of the OCI Terraform provider's supported resources and data sources contains usage, argument, and attribute details. The full reference is available at docs.oracle.com and Terraform Registry.
Data sources and resources are grouped by service within the reference.
"Failed to query available provider packages" Error Message When Running Resource Discovery
If the Terraform CLI returns an error message like the following when using resource discovery:
Failed to query available provider packages
Could not retrieve the list of available versions for provider hashicorp/oci:
the previously-selected version is no longer available
Then you can ensure that Terraform uses an existing local provider binary by specifying
its location using the provider_bin_path
environment variable. For
example:
export provider_bin_path=/Users/user/go/bin/
Terraform attempts to download the latest version of the OCI Terraform provider when you use resource discovery.
Default Tags Deleted on Apply
Sometimes, the OCI Terraform provider can
unexpectedly delete existing tag defaults from a resource when running terraform apply
.
This issue affects the Oracle-Tags
automatic tag defaults in
particular.
To work around this issue, you can add the ignore_defined_tags
attribute
to your provider block.
The ignore_defined_tags
attribute allows you to list out the keys of the
defined tags that Terraform will ignore as part of plan or apply. The
ignore_defined_tags
attribute can only be specified at the provider
level, and has a maximum allowed size of 100. The tags provided in this attribute are
ignored for all the resources in that Terraform file.
In the following example, "Oracle-Tags.CreatedOn" and "Oracle-Tags.CreatedBy" are the
keys in the defined_tags
map associated with a remote resource:
provider "oci" {
ignore_defined_tags = ["Oracle-Tags.CreatedBy", "Oracle-Tags.CreatedOn"]
}
For more information, see Provider Definitions and refer to the related GitHub issue.
Service API Errors
Because the Terraform provider interacts with OCI services on your behalf, many error messages surfaced by the Terraform provider come directly from OCI services. The API Errors reference lists common errors returned by all services.
Service error messages returned by the OCI Terraform provider include the following information:
- Error - the HTTP status code and error code (see API Errors)
- Provider version - the version of the OCI Terraform provider used to make the request
- Service - the OCI service responding with the error
- Error message - details regarding the error returned by the service
- OPC request ID - the request ID
- Suggestion - suggested next steps
For example:
Error: <http_code>-<api_error_code>
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: <error_message>
OPC request ID: exampleuniqueID
Suggestion: <next_steps>
This section details a few of the more commonly returned service errors.
"400-InvalidParameter" Error
If the Terraform CLI returns an error message like the following:
Error: 400-InvalidParameter
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: <error_message>
OPC request ID: exampleuniqueID
Suggestion: Please update the parameter(s) in the Terraform config as per error message: <error_message>
Update the parameter specified in the error message in the Terrform configuration for the resource.
The full reference of the OCI Terraform provider's supported resources and data sources contains usage, argument, and attribute details. The full reference is available at docs.oracle.com and Terraform Registry.
Data sources and resources are grouped by service within the reference.
"400-LimitExceeded" Error
While using Terraform, you might encounter errors indicating that you have reached or exceeded the service limits for a resource. For example:
Error: 400-LimitExceeded
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: Fulfilling this request exceeds the Oracle-defined limit for this tenancy for this resource type.
OPC request ID: exampleuniqueID
Suggestion: Request a service limit increase for this resource <service>
To understand more about OCI service limits and how to request a limit increase, see Service Limits
"404-NotAuthorized" Error
If the Terraform CLI returns an error message like the following:
Error: 404-NotAuthorizedOrNotFound
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: Authorization failed or requested resource not found.
OPC request ID: exampleuniqueID
Suggestion: Either the resource has been deleted or service <service> need policy to access this resource. Policy reference: <link>
Verify the user account is part of a group with the appropriate permissions to perform the actions in the plan you are executing. Refer to the policy reference for your service for more information.
"500-InternalError" Error
If the Terraform CLI returns an error message like the following:
Error: 500-InternalError
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: Internal error occurred
OPC request ID: exampleuniqueID
Suggestion: The service for this resource encountered an error. Please contact support for help with service <service>
The service responded to the request from the Terraform provider with an internal error. If you contact support for this issue, reference the information in the message.