Oracle Cloud Infrastructure Documentation

Health Check

Use health checks with Service Mesh.

In Kubernetes, health checks are defined using Liveness, Readiness, and Startup Probes. A kubelet runs the probes. For example, in an HTTP or HTTPS probe, the kubelet makes HTTP calls to the given address mentioned in the probe configured. Generally, the address of the application container running inside the pod is used.

Figure 1. Health Checks Part I
A high level diagram of a Kubernetes application performing a liveness probe health check without Service Mesh. Only HTTP protocol is supported. The app connects to a container.

In Service Mesh, the operator alters the HTTP probes to the proxy container which performs proxy health checks in addition to application health checks.

Figure 2. Health Checks with Service Mesh Part II
A high level diagram of a Kubernetes application performing a liveness probe health check with Service Mesh. Only HTTP protocol is supported. The app connects to a Service Mesh. However, when the Service Mesh proxy intercepts the check and performs the check on another application container.

Currently, Service Mesh only supports HTTP connections from the kubelet to the proxy. If the original probe is HTTPS or TCP, Service Mesh rewrites the probe to use HTTP from the kubelet to the proxy container. Based on the original probe settings, the proxy supports HTTP and HTTPS health checks from the proxy to the destination endpoint. If other types of probes are found (for example, tcpSocket, grpc) they are ignored silently. The proxy continues to perform proxy related health checks and any application-related health checks.

Health Check Examples

Example: Configure HTTP Liveness and Startup Probes

ratings-v1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  namespace: bookinfo-demo
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      namespace: bookinfo-demo
      labels:
        app: ratings
        version: v1
      annotations:
        servicemesh.oraclecloud.com/proxy-log-level: debug
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
        - name: ratings
          image: iad.ocir.io/idwul7vntmtk/istio/examples-bookinfo-ratings-v1:1.16.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          securityContext:
            runAsUser: 1000
         livenessProbe:
           initialDelaySeconds: 2
           periodSeconds: 5
           httpGet:
             path: /health
             port: 9080
         startupProbe:
           httpGet:
             path: /health
             port: 9080
           failureThreshold: 30
           periodSeconds: 5

Health check probes are defined at spec:template:spec:livenessProbe and spec:template:spec:startupProbe. Make sure to define the ports used in this spec for health check endpoints in the virtual deployment listener as well.

The host name is not defined for the probes in the httpHeaders field or host field or under the httpGet field. Therefore, the request defaults to the IP address of the pod where the ratings-v1 container is running. The kubelet tries to reach http://podipaddress:9080/health during startup. Because of the startupProbe configured here, the application has a maximum of 150 (30*5) seconds to be ready. If the startup probe never succeeds, the container is killed after 150 seconds subject to the pod's restartPolicy.

Since startupProbe is defined, livenessProbe stays disabled during startup until the http://podipaddress:9080/health endpoint returns a 2xx response. Post a successful startup and the liveness probe becomes active and keeps checking for a 2xx response every 5 seconds from the http://podipaddress:9080/health endpoint after an initial delay of 2 seconds. If the health check fails, the kubelet kills and restarts the container.

In a meshified scenario, the request flows from kubelet to Proxy container to Application container. Both the probes use the HTTP scheme by default as no scheme is defined.

Example: Configure an HTTPS livenessProbe for the ratings Container

ratings-v1.yaml (partial)
livenessProbe:
  initialDelaySeconds: 2
  periodSeconds: 5
  httpGet:
    host: ratings # Defaults to podIpaddress, Can be defined in httpHeaders field as well
    path: /health # Defaults to /
    port: 9080
    scheme: https # Defaults to http if not defined
    httpHeaders:
      - name: Accept
        value: application/json

With this configuration, the kubelet tries to reach the https://ratings:9080/health endpoint. As described in the health check diagram, mesh proxy intercepts this request and changes the scheme to HTTP. Further, the proxy initiates an HTTPS request from proxy to destination endpoint. In short, kubelet to Proxy container is HTTP. Proxy container to ratings container is HTTPS. If this probe succeeds, the pod is marked as live. A similar rewrite of the probe scheme happens if the request is a TCP probe, kubelet to Proxy container is HTTP.

Example: Configure a GRPC livenessProbe for the ratings Container

ratings-v1.yaml (partial)
livenessProbe:
  grpc:
    service: ratings
    port: 6174
  initialDelaySeconds: 10

Probes using the GRPC protocol or `exec` aren't modified by Service Mesh. Therefore, these probes may not work as expected when the virtual service uses a STRICT TLS mode after you meshify an application. Currently, Service Mesh only supports HTTP connections from the kubelet to the proxy.