Oracle Cloud Infrastructure Documentation

Run Active Health Checks on Kubernetes

Install the Cluster Health Checks (CHC) Helm chart on an Oracle Kubernetes Engine (OKE) cluster or another supported Kubernetes cluster, then run a preflight NCCL test on selected GPU nodes.

Note

OKE GPU host repair requires OKE’s repair controller and managed add-ons. Installing CHC on a Kubernetes cluster that you manage outside OKE does not enable these repair capabilities. For CHC installation requirements, see the documentation included with your CHC release. For OKE repair requirements and configuration, see Repairing GPU Worker Nodes with OKE and Repairing GPU Hosts

Install the CHC Helm Chart

For a new OKE cluster, use the CHC option in a supported OCI HPC stack. For an existing cluster, the CHC-only onboarding path provisions or reuses the required OCI resources without creating a new cluster.

Helm installs the Kubernetes components. A Helm-only installation still requires the Object Storage bucket, Diagnosis Store configuration, worker identity permissions, and image access, as described in your downloaded CHC install (see the QuickStart README.md file). Templates can require privileged containers, host paths, and GPU/RDMA device access. Review them against your cluster security policy.

Use a released chart version qualified for your stack. Keep the chart version and its compatible test images together. The newest standalone Dr. HPC tag isn't necessarily the image qualified with that chart.

Start from the values file generated by onboarding, as described in your downloaded CHC install (see the QuickStart README.md file). For the OCA-passive model, retain the following settings during initial installation:

activeTests:
  enabled: false
activeTestSuites: []
passive:
  enabled: false
  labeler:
    enabled: false
  daemonset:
    enabled: false
cn-backend-nw-monitoring:
  enabled: false

These are overrides, not a complete installation values file. The file must also contain the result-backend and image configuration, as described in your downloaded CHC install (see the QuickStart README.md file). Backend network monitoring is a separate optional component and isn't required for these examples.

  1. If Terraform already installed CHC, don't perform a second initial installation. For the manual Helm path: 
    export CHC_NAMESPACE=cluster-health-checks
export CHC_CHART_VERSION='<qualified-chart-version>'
export CHC_CHART=oci://iad.ocir.io/hpc/oci-dr-hpc-v2/chc-helm/cluster-health-checks

helm upgrade --install cluster-health-checks "$CHC_CHART" \
 --version "$CHC_CHART_VERSION" \
 --namespace "$CHC_NAMESPACE" --create-namespace \
 -f customer-chc-values.yaml --wait --timeout 20m
  2. For this Object Storage and Diagnosis Store onboarding path, verify that installation completed and the Diagnosis Store ID is populated: 
    kubectl -n "$CHC_NAMESPACE" get pods,jobs
kubectl -n "$CHC_NAMESPACE" get configmap cluster-healthchecks-objectstorage \
  -o jsonpath='{.data.DIAGNOSIS_STORE_ID}{"\n"}'
  3. Then enable the active-test controller and templates while preserving the installation values: 
    helm upgrade cluster-health-checks "$CHC_CHART" \
 --version "$CHC_CHART_VERSION" \
 --namespace "$CHC_NAMESPACE" --reuse-values \
 --set activeTests.enabled=true --wait --timeout 20m

kubectl -n "$CHC_NAMESPACE" get deployments,pods
kubectl get crd chcactivetestsuites.clusterhealthchecks.oraclecloud.com
  4. For multi-node tests implemented as MPIJob resources, install the compatible MPI Operator, as described in your downloaded CHC install (see the QuickStart README.md file). Native Kubernetes scheduling is the standard path in this example; Kueue isn't a prerequisite.

Additional Documentation Resources

For all installation and usage instructions, including how to run a custom test suite, see:

  • "Running RDMA (Remote Direct Memory Access) GPU Workloads on OKE," available in your downloaded CHC install (see the README.md in the QuickStart documentation).

  • docs/README.md, available in your downloaded CHC install (see the QuickStart documentation).

Run a Preflight NCCL Test

This example runs a onetime NVIDIA Collective Communications Library (NCCL) smoke test on two selected, compatible GPU nodes. Confirm that they're ready and have capacity reserved for the test. Use a chart/controller release that supports the nodeCount interface.

  1. Label only the intended nodes, using a label key owned by your organization: 
    kubectl get nodes -o wide
kubectl label node <node-a> <node-b> example.com/chc-preflight=selected
  2. Save the following as chc-nccl-preflight.yaml. Replace <supported-shape> with the nodes' exact instance shape. Change the namespace if your CHC installation uses a different one. 
    apiVersion: clusterhealthchecks.oraclecloud.com/v1alpha1
kind: ClusterHealthChecksActiveTestSuite
metadata:
  name: chc-nccl-preflight
  namespace: cluster-health-checks
spec:
  runOnce: true
  scheduler: native
  nodeSelector:
    example.com/chc-preflight: selected
    node.kubernetes.io/instance-type: <supported-shape>
    nvidia.com/gpu: "true"
  nodeCount: 2
  testNames:
    - nccl_multi_node_smoke

    Verify that the GPU label in the selector matches your installation. CHC sorts the matching Ready nodes by Kubernetes node name and selects the first two. Label only your intended test nodes. This interface uses the first selected node as both launcher and worker, and the other as a worker. It doesn't require a third GPU node solely for the launcher.

  3. Apply the suite and inspect the generated workload: 
    kubectl apply -f chc-nccl-preflight.yaml
kubectl -n "$CHC_NAMESPACE" get chcactivetestsuites chc-nccl-preflight -o yaml
kubectl -n "$CHC_NAMESPACE" get mpijobs,pods \
 -l cluster-health-checks.oraclecloud.com/suite=chc-nccl-preflight -o wide
  4. Confirm the expected nodes participated. For this smoke test, look for MPIJob Succeeded=True, a passing suite lastRunResult, and launcher logs showing PASS for the all_reduce and alltoall checks. Also inspect the uploaded or local result JSON, whose results.localhost.final field records the diagnostic outcome. A completed Kubernetes workload alone isn't proof of a passing health check.
  5. Record the object keys and recommendations returned by the run. Use a new suite name for a new one-time test. Remove the temporary selection label when it's no longer needed. Use RCCL/ROCm templates for AMD nodes rather than the NVIDIA NCCL example.