Cluster Health Checks
Cluster Health Checks (CHC) combine passive monitoring, active validation, and supported recovery integrations to help you assess GPU cluster health.
In an Oracle Cloud Infrastructure (OCI) AI cluster of multiple accelerator nodes, your AI workload health and reliability depends on coordination between the host hardware, firmware, OS, GPU drivers, network drivers, and software stack. Interruptions can disrupt your workload and require you to spend time investigating, running tests, debugging, and resolving issues. OCI can help your AI cluster reliability with Cluster Health Checks for GPU, driver, and network issues, and integrations for auto-repair and remediation.
CHC provides health checks and recommended actions to help you identify unhealthy nodes and validate a cluster before starting a workload. You can use the components that fit your operating model:
| Component | What it does | How you use it |
|---|---|---|
| Passive health checks | Examine host configuration, device state, counters, and logs without intentionally loading the GPUs or RDMA network. | Run through Oracle Cloud Agent (OCA) on supported GPU instances. You don't need any Slurm or Kubernetes add-on for this baseline. |
| Active health checks | Exercise GPUs, data transfer, and multi-node communication to find faults or performance issues under load. | Install the CHC add-on for Slurm or the CHC Helm chart for Kubernetes, or use the integration in a supported OCI HPC stack. Select and submit tests through your scheduler. |
| OKE GPU host repair | Act on supported health signals according to a repair policy you configure. | Enable the OKE GPU health monitoring integration and select the nodes and repair controls for your cluster. See Repairing GPU Worker Nodes with OKE and Repairing GPU Hosts. |
You can use passive checks alone, add active validation with CHC, or enable OKE GPU host repair automation with its required health-signal integration.
CHC uses Dr. HPC v2 as its diagnostic engine. The checks and thresholds available to a node depend on its GPU shape, software configuration, and installed release. CHC complements your existing cluster and HPC Monitoring plugin metrics and logging tools. CHC can run on demand, or opportunistically on idle GPU nodes. CHC doesn't schedule your application workloads, guarantee future job success, or replace OCI hardware maintenance workflows.