Run Active Health Checks on Slurm

Use the Cluster Health Checks (CHC) add-on for Slurm to install the diagnostic tools, prepare selected GPU nodes, run active tests, and inspect results.

The CHC add-on provides the oci-chc command on the Slurm controller. It submits tests to GPU workers, where the Dr. HPC container runs the diagnostics.

Note

Installing the native oci-dr-hpc-v2 package doesn't install the Slurm add-on. If your OCI HPC stack already includes the add-on, run oci-chc --version and review its configuration before installing or updating it.

Before You Begin

Have your cluster administrator verify these prerequisites:

You can connect to the controller and allocate the intended GPU nodes through Slurm. Your controller account has passwordless SSH access to those nodes.
The controller has Bash, curl, jq, Python 3, the OCI CLI, and Slurm CLI tools. GPU workers have compatible drivers, Enroot, and Pyxis.
A writable directory is shared at the same absolute path on the controller and workers. The following examples use /config.
The controller and workers have the IAM permissions needed to configure and use the results bucket and Diagnosis Store.
Network access permits downloading ORAS and the CHC bundle and images, and accessing the required OCI services.

Active tests consume GPU and network resources. Select nodes and a test window appropriate for your workloads.

Install and Configure the Add-on

Connect to the controller and select the shared directory. Run the remaining commands in the same Bash session on the controller:
```
export CHC_SHARED=/config
export PATH="$HOME/.local/bin:$HOME/bin:$PATH"
```
Replace /config if your cluster uses another shared directory. Creating a local directory alone doesn't make it shared.
Install or verify ORAS. ORAS downloads the CHC bundle from the container registry. Install ORAS 1.2.0 or later using the ORAS installation instructions, then verify it:
```
oras version
```

Download, verify, and install the latest stable CHC bundle.

(
  set -euo pipefail

  REPO=iad.ocir.io/hpc/cluster-health-checks
  VERSION=$(oras repo tags "$REPO" \
    | sed -n 's/^chc-addon-\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\)$/\1/p' \
    | sort -V | tail -1)

  [ -n "$VERSION" ] || {
    echo 'No stable CHC release found' >&2
    exit 1
  }

  echo "Installing CHC $VERSION"
  cd "$(mktemp -d /tmp/chc-download.XXXXXX)"
  oras pull "$REPO:chc-addon-$VERSION"
  sha256sum -c "chc-addon-$VERSION.tar.gz.sha256"
  tar xzf "chc-addon-$VERSION.tar.gz"

  bash "chc-addon-$VERSION/scripts/install.sh" \
    --shared-dir "$CHC_SHARED"
)

Confirm that checksum verification reports OK and installation prints release_installed and command_installed. Then check the installed version:

oci-chc --version

Installation places versioned releases under $CHC_SHARED/chc-addon/releases/ and installs the command at ~/.local/bin/oci-chc.

Prepare result storage, the diagnostic image, and node readiness.
```
oci-chc setup
```
Setup allocates an available GPU node, verifies shared storage, discovers or creates the results bucket and Diagnosis Store, prepares a compatible Dr. HPC image, and runs readiness checks.

To select a particular node, replace GPU-001 with its Slurm node name:
```
oci-chc setup --node GPU-001
```
Confirm that the output includes setup=PASS. Record the selected node and partition, and review any warnings. A successful setup confirms readiness to run tests; it isn't a GPU health-test result.

Run an Active Test

Use the node selected during setup. For an NVIDIA node with a compatible image:

oci-chc test dcgm -w GPU-001

For an AMD node with a compatible ROCm image:

oci-chc test rvs -w GPU-001

Replace GPU-001 with your node name. If setup required a particular partition or reservation, supply those options to the test command as well.

Check the Results

Check both slurm_state and result. A Slurm job reaching COMPLETED doesn't establish that the diagnostic passed. Confirm that the requested test completed and reported result=PASS.

Use the run_dir printed by the command to locate its artifacts:

RUN_DIR='<printed-run_dir>'
less "$RUN_DIR/run.log"

If run.log is absent, inspect slurm-<job_id>.out in that directory. Review the files identified by result_json and, when available, remediation_json. Check upload_status and diagnosis_status separately from the diagnostic result.

Additional Documentation Resources

The installed add-on includes references matched to its release:

docs/active-testing-guide.md: Test execution and advanced options.
docs/support-matrix.md: Test and image compatibility.
docs/iam.md: Required permissions.
docs/upgrade-rollback-uninstall.md: Updating, rolling back, and removing the add-on.

Find these files under $CHC_SHARED/chc-addon/current/. List the available test commands with: