Common Load Balancer Errors
Learn about common load balancer errors associated with the load balancers.
Common load balancer errors include, series 500 and series 400 errors, health check errors, client errors, and SSL errors. The subsequent topics in this section describe these common errors and detail troubleshooting procedures for each, where applicable.
Server Errors (500-599)
504
Error messages:
-
lbStatusCode: "504"
-
backendStatusCode: ""
Oracle Cloud Infrastructure log category: Access log
Symptoms:
The client fails with a 504
error.
Possible causes:
The load balancer is not able to establish connections with any of the backends, even though the health check is marking the backends as available.
Possible solutions:
Configure the health check correctly.
Troubleshooting documentation: Editing a Load Balancer's Health Check Policies
502, 502
-
lbStatusCode: "502"
-
backendStatusCode: "502"
Oracle Cloud Infrastructure log category: Access log and error log
-
The client fails with a
502 Bad Gateway
error. -
The backend health check succeeds.
-
The backend returns a
502
error.
-
An application on the backend is returning a
502
error. -
The backend is configured incorrectly.
-
The backend is likely another reverse proxy or load balancer.
Possible solutions:
Examine the backend application logs to determine why a 502
error is returned.
Troubleshooting documentation: HTTP 502 Bad Gateway Errors and Testing TCP and HTTP Backend Servers.
502
-
lbStatusCode: "502"
-
backendStatusCode: ""
-
No healthy backends available in associated backend set
Oracle Cloud Infrastructure log category: Access log and error log
-
The client fails with a
502 Bad Gateway
error. -
The backend health check fails.
-
No traffic observed to a specific backend or all backends.
-
A backend application is not responding to the health check with the expected response.
-
If no error occurs from the backend, then a TCP health check is configured.
-
A single backend or all backends are configured in drain mode.
-
Determine why TCP health check is failing.
-
Convert to HTTP health check.
-
Change the drain mode to false (undrain) for a given backend or all backends.
Troubleshooting documentation: HTTP 502 Bad Gateway Errors and Testing TCP and HTTP Backend Servers.
Session Persistence Issue
Persistence selected backend ip_address which failed and no_fallback is selected
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
Session persistence is failing.
-
Backend set is configured with session persistence and the expected backend is not available because the connection failed or timed out.
-
Fallback option is disabled.
-
Determine why backend application is not reachable.
-
Enable fallback option in case the selected server is unavailable.
Troubleshooting documentation: Fallback
For all other 5nn errors, the most likely causes are issues with the backend server.
Client Errors (400-499)
400
-
lbStatusCode: "400"
-
backendStatusCode: ""
-
400 bad request header or cookie too large
Oracle Cloud Infrastructure log category: Access log
-
The load balancer returns a status code
400
. -
The backend does not return a status code.
Possible causes:
The client is sending a request that exceeds the configured buffer size.
Possible solutions:
Increase the HTTP request header size on the load balancer. By default, the size limit is 8 KB but raising it to 64 KB resolves the issue.
Troubleshooting documentation: HTTP Header Rules
404, 404
-
lbStatusCode: "404"
-
backendStatusCode: "404"
Oracle Cloud Infrastructure log category: Access log
-
The load balancer returns a
404
status code. -
The backend returns a
404
status code.
Possible causes:
The expected page does not exist on the backend.
-
Create the missing page.
-
Configure the client to call the correct page.
403, 403
-
lbStatusCode: "403"
-
backendStatusCode: "403"
Oracle Cloud Infrastructure log category: Access log
-
The load balancer returns a
403
status code. -
The backend returns a
403
status code.
-
Expected page does not have sufficient permission on the backend.
-
Expected authentication token is missing or not being forwarded.
Possible solutions:
-
Create missing permissions on backend.
-
Adjust client configuration to ensure that tokens are sent properly.
-
Ensure that all tokens being sent are arriving at the backend.
-
If the header is missing:
-
Adjust header size on the load balancer or the client.
-
Allow headers with special characters.
-
Troubleshooting documentation: HTTP Header Rules
Health Check Errors
No Healthy Backends
No healthy backends available in associated backendSet
Oracle Cloud Infrastructure log category: Error log
Symptoms:
The client fails with a 502 Bad Gateway
error.
-
No backends in the backend set.
-
No backends responding to health check.
-
Determine why backends aren't responding to health check.
-
Check and adjust any health check settings, including status code, regular expressions, interval timeout, port, and protocol.
Troubleshooting documentation: Editing a Load Balancer's Health Check Policies
Status Code Issues
Backend health status failure reason: Status code mismatch
Oracle Cloud Infrastructure category: Backend Health Status
Status code mismatch
Oracle Cloud Infrastructure log category: Error log
-
The backend fails the health check.
-
The client fails with a
502 Bad Gateway
error. -
invalid statusCode
appears in the error logs.
-
The backend is responding with an incorrect response code.
-
The backend health check fails because of response code mismatch.
-
The health check failures are because of an unexpected status code in the regular expression body.
-
Determine why the backend is sending the incorrect response code.
-
Adjust the path or status code of the health check to match the backend.
Troubleshooting documentation: Editing a Load Balancer's Health Check Policies
Response Match Failed
Backend Health Status Failure Reason: Regular expression mismatch
Oracle Cloud Infrastructure category: Backend Health Status
"response match result: failed"
Oracle Cloud Infrastructure log category: Error log
-
The backend fails the health check.
-
The client fails with a
502 Bad Gateway
error. -
"response match result: failed"
appears in the error logs.
Possible causes:
The backend health check fails because of regular expression mismatch, incorrect value returned, or incorrect value provided to the health check.
-
Determine why the backend is sending the incorrect body.
-
Adjust the path or regular expression pattern of the health check to match the backend.
Troubleshooting documentation: Editing a Load Balancer's Health Check Policies
Unreachable Host
Backend Health Status Failure Reason: Connection failed
Oracle Cloud Infrastructure category: Backend Health Status
"errno":"EHOSTUNREACH","syscall":"connect"
"ECONNREFUSED","errno":"ECONNREFUSED"
Oracle Cloud Infrastructure log category: Error log
-
The backend fails the health check.
-
The client fails with a
502 Bad Gateway
error. -
"EHOSTUNREACH"
appears in error logs.
-
The backend health check fails because of an unreachable host.
-
The backend health check fails because of a connection reset.
-
An application or firewall is actively refusing the connection.
-
Check the local instance firewall to confirm that traffic is being allowed.
-
Check the local instance to confirm that the application is running.
-
Check the network security group and security lists to confirm that traffic is allowed.
Troubleshooting documentation: Access and Security
Health Status Issues
"healthStatus":"Unhealthy to Healthy"
"healthStatus":"Healthy to Unhealthy"
Oracle Cloud Infrastructure log category: Error log
-
The client behaves as expected but fails periodically.
-
The backend switches between passing and failing the health check.
-
"Unhealthy to Healthy"
or"Healthy to Unhealthy"
appears in error logs.
-
An unhealthy backend becomes healthy.
-
If the health status of the backend changes often, it can indicate a chronic problem.
-
Ensure that the instance is not changing health status abnormally.
-
Check application logs on the backend server for any application-specific issues.
Connection Issues
Backend Health Status Failure Reason: Timed out
Oracle Cloud Infrastructure category: Backend Health Status
"msg":"connect timed out","elapsed":3000}
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
The backend is periodically or chronically failing health checks.
-
"connect timed out"
appears in the error logs.
-
The backend server is not responding to health checks in the expected time period.
-
Slow upstream dependency including, database, application service or API, or slow storage services, such as Oracle Cloud Infrastructure File Storage service, Elastic Block Store, or Object Storage.
-
Perform a local test to the backend to eliminate the load balancer as a cause.
-
Check the performance of all upstream dependencies.
-
Check application logs on the backend server for any dependencies reporting any sort of timeout.
Troubleshooting documentation: Testing TCP and HTTP Backend Servers.
SSL Errors
SSL Virtual Listener Issues
Not all SSL virtual listeners on port 443 have the same set of SSL protocols defined
Symptoms:
You cannot create backends for an existing load balancer nor can you add new servers to the backend created previously within the same load balancer.
Possible causes:
Mismatch of transport layer security (TLS) versions.
Possible solutions:
Match TLS versions on the listeners.
Troubleshooting documentation: SSL Certificates for Load Balancers
SSL Handshake Issues
(SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshake error
Oracle Cloud Infrastructure log category: Client log
Symptoms:
The client experiences SSL handshake failures in Load Balancer metrics (see Load Balancer Metrics).
Possible causes:
The backend is not configured to accept SSL.
-
Confirm that the backend certificate matches the certificate authority that is provided.
-
Ensure that all certificates in the chain are provided in the correct order in the Certificate field.
-
Ensure that you provide the correct certificate depth.
Troubleshooting documentation: SSL Certificates for Load Balancers
Backend SSL Handshake Issues
Peer backend_ip_address closed connection in SSL handshake
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).
-
The backend is not configured to accept SSL.
-
The backend certificate is invalid.
-
Confirm that the backend certificate matches the certificate authority that is provided.
-
Ensure that all certificates in the chain are provided in the correct order in the Certificate field.
-
Ensure that you provide the correct certificate depth.
Troubleshooting documentation: SSL Certificates for Load Balancers
SSL Certificate Issues
Error:
Client backend_ip_address
has SSL certificate verify error.
Oracle Cloud Infrastructure log category: Error log
Symptoms:
The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).
-
The client certificate is invalid.
-
The client certificate is not trusted.
-
Invalid peer certification verify depth.
-
Ensure that the client certificate is valid.
-
Remove Peer Cert Verify feature on the listener.
Troubleshooting documentation: Key Pair Mismatch and Private Key Consistency.
Client SSL Certificate Issues
Client backend_ip_address sent no required SSL certificate
Oracle Cloud Infrastructure log category: Error log
-
The client experiences a
400 Response
error. -
no required SSL certificate
appears in error logs.
Possible causes:
The client is not sending a client certificate.
-
Update the client to send the correct client certificate.
-
Remove Peer Cert Verify feature on the listener.
-
Adjust the certificate verification depth.
Troubleshooting documentation: Configuring Peer Certificate Verification.
SSL Error Causes Backend Health Check Failure
"code":"EPROTO","errno":"EPROTO"
Oracle Cloud Infrastructure log category: Error log
Symptoms:
The backend health check fails because of the SSL error.
Possible causes:
The backend is configured to accept SSL but the health check protocol selected does not match that of the backend.
Possible solutions:
Confirm that you are using non-TLS health check on a backend that has TLS enabled.
Troubleshooting documentation: Editing a Load Balancer's Health Check Policies
SSL Host Name Verification Fails
SSL host name verification failed for host_name
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
Error message contains
SSL host name verification failed
.
Possible causes:
Host name provided does not match what is expected.
-
Configure client to use the expected host name.
-
Configure certificate to match the host name sent by the client.
Troubleshooting documentation: SSL Certificates for Load Balancers
Client-Side Errors
Client Access Denied
Error:
Access for client_ip_address
denied by HTTP ACL rule.
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
The backend doesn't pass health check.
-
forbidden by HTTP ACL rule
appears in the error log.
Possible causes:
Access control rule set is enabled bud doesn't include the source IP address.
Possible solutions:
Check and apply respective rule set to include the source IP address.
Troubleshooting documentation: Access Control Rules
Client Timeout Issue
Error:
Client client_name
timed out
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a
502 Bad Gateway
error. -
The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).
Possible causes:
The client terminated the connection sooner than the configured timeout for the load balancer.
-
Configure client timeout to match expected application configuration.
-
Determine why the backend server didn't respond in the configured amount of time.
Troubleshooting documentation: Testing TCP and HTTP Backend Servers.
Client Connection Closed Abruptly
Error:
Connection to address
was abruptly closed by
Oracle Cloud Infrastructure log category: Error log
Symptoms:
The client fails with a 502 Bad Gateway error.
Possible causes:
The listener has an Max listener connection rule and an IP tried to make more connections to the listener than allowed by the rule.
Possible solutions
-
Increase the allowed number of connections an IP can make to the listener.
-
Reduce the number of connections the IP is making to the listener.
Troubleshooting documentation: Max Listener Connection Rules.
Backend Server Errors
Backend Server Connection Issue
Error:
Backend ip_address
abruptly closes connection.
Oracle Cloud Infrastructure log category: Error log
-
The client fails with a 502 Bad Gateway error.
-
The client reports IO error in load balancer metrics.
-
The backend server set uses HTTPS and the cipher suites or TLS versions aren't compatible.
Possible causes:
-
The backend server connection timeout is configured incorrectly, with a lower timeout value than the load balancer.
-
The backend server or its containing backend set has
maxConnections
set and the number of connections to the backend server has reached the specified limit.
Possible solutions:
-
Determine why the backend server application is timing out.
-
If the backend server timeout value needs to be adjusted, then adjust it to be greater than the load balancer timeout value.
-
Add more backend servers to handle the load.
-
Increase the
maxConnections
setting.
No Healthy Backend Servers
Error:
No healthy backends available in associated backendSet
Oracle Cloud Infrastructure log category: Error log
Symptoms:
The client fails with a 502 Bad Gateway error.
Possible causes:
-
No backend servers in the backend set.
-
No backend servers responding to health check.
-
All health backend servers in the backend set have reached their
maxConnections
limit.
Possible solutions:
-
Determine why backends aren't responding to health check.
-
Check and adjust any health check settings, including status code, regular expressions, interval timeout, port, and protocol.
-
Check if backend servers have a
maxConnections
limit set. If so, add more backend servers to handle the load or increase themaxConnections
limit
Troubleshooting documentation: Testing TCP and HTTP Backend Servers.