Skip to main content
Version: main 🚧

Monitor GPU nodes with NVSentinel

NVSentinel is NVIDIA's open source, Kubernetes-native service for GPU fault handling. It detects hardware and software faults on GPU nodes through DCGM, system logs, and cloud provider maintenance events. It can respond by cordoning nodes, draining workloads, and triggering node reboots or terminations.

When the Argo CD integration is enabled, vCluster Platform ships an ArgoCDApplicationTemplate for NVSentinel alongside the other GPU monitoring templates. You reference the template from an ArgoCDApplication to deploy NVSentinel into a cluster that runs GPU nodes, typically a tenant cluster with private nodes. NVSentinel metrics then flow through the Fleet Observability pipeline like any other workload metrics and appear on the bundled GPU dashboards.

Prerequisites​

  • The Argo CD integration is enabled.
  • An Argo CD connector is registered on the target cluster. See Connect to Argo CD.
  • The NVIDIA GPU Operator is deployed in the same cluster, per the deployment order.
  • cert-manager is deployed, if any component you enable needs internal TLS. See When you need cert-manager.
  • A default StorageClass exists, or one is named in the template parameters, if any component you enable needs the MongoDB store.
  • Prometheus Operator CRDs exist, if you enable the PodMonitor.

What the template deploys​

The template deploys the upstream nvsentinel Helm chart from oci://ghcr.io/nvidia/nvsentinel. In an ArgoCDApplication, reference it through spec.templateRef.name: nvsentinel. The Platform UI lists it as NVSentinel.

NVSentinel is composed of independent components that you enable per deployment:

ComponentDefaultWhat it does
GPU Health MonitortrueWatches GPU hardware through DCGM for thermal issues, ECC errors, and XID events.
Syslog Health MonitortrueScans system logs on each node for fault patterns.
NIC Health MonitorfalseWatches network interface health.
CSP Health MonitorfalsePulls maintenance events from cloud provider APIs.
Fault QuarantinefalseCordons and taints nodes with detected faults.
Node DrainerfalseEvicts workloads from quarantined nodes.
Fault RemediationfalseCreates maintenance resources to recover failing nodes after drain completes.
JanitorfalseExecutes the node reboots and terminations that Fault Remediation requests.
Health Events AnalyzerfalseAnalyzes patterns across health events and produces recommended actions.
MongoDB StorefalsePersistent event storage; the coordination layer for the remediation components.

With the defaults, NVSentinel only observes health events and metrics. Nothing acts on nodes. The remediation pipeline (Fault Quarantine, Node Drainer, Fault Remediation, Janitor) is opt-in.

Component dependencies the template resolves for you​

Some components can't start without others. The template enables the dependencies automatically, so the deployed application can differ from the toggles you set:

  • MongoDB store turns on whenever Fault Quarantine, Node Drainer, Fault Remediation, or Health Events Analyzer is enabled, even if you leave Enable MongoDB Store off. Those components coordinate through MongoDB change streams and crash without it.
  • Janitor turns on whenever Fault Remediation is enabled, because Fault Remediation can't reboot or terminate nodes without it.
  • The Janitor's default provider is kind, which only simulates reboots and terminations. Set the Janitor Provider parameter to your cloud provider (or generic for bare metal) before relying on real remediation. See Janitor provider parameters.
note

Node Drainer is the exception. It needs Fault Quarantine to have anything to drain, but the template doesn't turn on Fault Quarantine for you the way it does for the dependencies above. Set both enableFaultQuarantine and enableNodeDrainer to "true" yourself, as the deploy example does. Enabling Node Drainer alone deploys a drainer with no quarantined nodes to evict from.

When you need cert-manager​

With the observe-only defaults, NVSentinel runs without cert-manager. It becomes required when you enable any of the following:

  • The internal MongoDB store. TLS/mTLS is enabled by default, and cert-manager creates the server, client, and CA certificates. Fault Quarantine, Node Drainer, and Health Events Analyzer turn the MongoDB store on automatically, so they indirectly require cert-manager too.
  • The Janitor or Fault Remediation. The Janitor uses TLS certificates for its validating webhook and its provider gRPC endpoint.
  • Chart features that aren't exposed as template parameters, such as preflight checks (the mutating admission webhook needs TLS, and cert-manager is the default provider) and internal PostgreSQL with the chart's TLS configuration.

Deploy NVSentinel​

Deploy NVSentinel into the cluster whose GPU nodes it should watch. For a tenant cluster with private GPU nodes, target the tenant cluster itself.

  1. Select your project from the projects dropdown at the top of the left navigation bar.

  2. Navigate to Tenant Clusters and open the tenant cluster with the GPU nodes to monitor.

  3. Select the ArgoCD Apps tab and click .

  4. In the Basics section, enter a Display Name such as NVSentinel.

  5. Select Deploy to tenant cluster to deploy into the tenant cluster itself.

  6. From the Argo CD Application Template dropdown, select the NVSentinel template.

  7. Fill the Template Parameters. Keep Dry Run enabled for the first rollout, and enable the components you need. The DCGM defaults match a standard GPU Operator install.

  8. Click .

Dry run​

The template defaults to dryRun: "true". As with any dry run, NVSentinel runs the full detection and decision pipeline but skips changes to node state or workloads. Dry run still writes real node annotations and labels, and still increments the metrics that feed the bundled GPU dashboard.

When a fault is detected in dry-run mode, NVSentinel:

  • Logs the intended action, for example Cordoning node.
  • Writes its own annotations and the dgxc.nvidia.com/nvsentinel-state label on the node to track the quarantine decision.
  • Increments quarantine metrics such as fault_quarantine_current_quarantined_nodes and fault_quarantine_cordons_applied_total, which feed the Quarantined Nodes panel on the bundled GPU dashboard.

During a dry run, NVSentinel doesn't:

  • Cordon the node. kubectl get node <name> -o jsonpath='{.spec.unschedulable}' stays empty and the node keeps accepting workloads.
  • Evict workloads.
  • Reboot or terminate nodes.

Because the metrics increment but nodes are not cordoned or evicted during a dry run, both the node object and the dashboard can look like a genuine quarantine happened.

Use dry run to validate detection end to end. Inject or wait for a health event, confirm the logs, node annotations, and metrics, and then set dryRun: "false" to let NVSentinel cordon, drain, and remediate for real.

Before turning dry run off, make sure the Janitor provider is set to a real provider, since the default kind provider only simulates reboots and terminations.

Uninstall and cleanup​

Deleting the NVSentinel ArgoCDApplication prunes the workloads, but two artifacts are retained by design and stay in the cluster until you remove them:

  • MongoDB PVCs. The three datadir-mongodb-* PVCs (10Gi each by default, 30Gi total) survive uninstall so health-event history isn't destroyed by accident.
  • Namespace. The destination namespace (nvsentinel) stays Active with zero pods. Deleting it also removes the namespaced cert-manager resources (Certificates and Issuers) that NVSentinel created.
Clean up retained NVSentinel artifacts
# MongoDB data (destroys health-event history)
kubectl delete pvc -n nvsentinel datadir-mongodb-0 datadir-mongodb-1 datadir-mongodb-2

kubectl delete namespace nvsentinel

Neither carries blocking finalizers, so deletion completes cleanly.

warning

Deleting the MongoDB PVCs permanently deletes NVSentinel's health-event history.

To remove the rest of the GPU stack, including the retained cert-manager CRDs, see Tear down the GPU stack.

Parameters​

The template groups parameters by component. All boolean parameters are passed as strings ("true" / "false").

Core parameters​

ParameterDefaultDescription
destinationNamespacenvsentinelNamespace NVSentinel deploys into.
projectdefaultArgo CD project that owns the application.
versionv1.13.0NVSentinel Helm chart version.
repoURLoci://ghcr.io/nvidia/nvsentinelChart repository. Change to use a mirror or private registry.
releaseNamenvsentinelHelm release name.
chartnvsentinelChart name pulled from the repository.
imageTagv1.13.0NVSentinel image tag.
dryRuntrueLog and record intended actions without cordoning, draining, or remediating. See Dry run.

DCGM parameters​

ParameterDefaultDescription
dcgmEndpointnvidia-dcgm.gpu-operator.svcDCGM service the GPU Health Monitor reads from. The default matches a standard GPU Operator install.
dcgmPort5555DCGM service port.

Component toggles​

ParameterDefaultDescription
enableGpuHealthMonitortrueGPU hardware health through DCGM.
enableSyslogHealthMonitortrueSystem log fault patterns.
enableNicHealthMonitorfalseNetwork interface health.
enableCspHealthMonitorfalseCloud provider maintenance events.
enableFaultQuarantinefalseCordon and taint faulty nodes. Turns on the MongoDB store automatically.
enableNodeDrainerfalseEvict workloads from quarantined nodes. Turns on the MongoDB store automatically, but not Fault Quarantine. Enable that yourself too.
enableFaultRemediationfalseCreate maintenance resources to recover failing nodes. Turns on the MongoDB store and the Janitor automatically.
enableJanitorfalseExecute reboots and terminations. Turns on automatically with Fault Remediation.
enableHealthEventsAnalyzerfalseAnalyze patterns across health events. Turns on the MongoDB store automatically.

MongoDB store parameters​

ParameterDefaultDescription
enableMongodbStorefalseEnable the MongoDB datastore explicitly. Turns on automatically with the components listed above.
mongodbStorageClass(empty)StorageClass for the MongoDB PVCs. Empty uses the cluster default.
mongodbStorageSize10GiVolume size per MongoDB replica. The chart runs three replicas, so the default provisions 30Gi total.

Node labeler parameters​

ParameterDefaultDescription
assumeDriverInstalledfalseEnable when NVIDIA drivers are preinstalled on the host rather than deployed by GPU Operator driver pods (driver.enabled=false). The labeler then marks all GPU nodes as having the driver installed instead of watching for driver pods. Without it there are no driver pods to detect, so the syslog and metadata DaemonSets never start on those nodes.

Metrics and operations parameters​

ParameterDefaultDescription
enablePodMonitorfalseCreate a PodMonitor for NVSentinel metrics. Requires Prometheus Operator CRDs.
podMonitorInterval30sPodMonitor scrape interval.
enableNetworkPolicytrueRestrict ingress to metrics ports. Disable if it blocks a same-namespace webhook such as cert-manager.
enableTracingfalseEnable OpenTelemetry tracing.
tracingEndpoint(empty)OTLP gRPC endpoint (host:port) for trace ingestion. Required when tracing is enabled.
enableAuditLoggingfalseEnable audit logging.

Janitor provider parameters​

The Janitor talks to your infrastructure provider to reboot and terminate nodes. Pick the provider with janitorProvider and fill the matching provider section. The kind and kwok providers only simulate actions and are meant for test clusters.

ParameterDefaultDescription
janitorProviderkindOne of kind or kwok (both simulated), aws, gcp, azure, oci, nebius, or generic (bare metal or on-prem).
janitorServiceAccountAnnotationKey(empty)Annotation key on the Janitor ServiceAccount for cloud identity, for example eks.amazonaws.com/role-arn or iam.gke.io/gcp-service-account. Set together with the value.
janitorServiceAccountAnnotationValue(empty)Value for the annotation key above, for example the IAM role ARN or GCP service account email.

Select the tab for your Janitor provider:

ParameterDefaultDescription
janitorAwsRegion(empty)AWS region of the EKS cluster (for example us-east-1). The Janitor uses this one region for all EC2 calls, so every node must live in it. Multiple availability zones within the region are fine.
janitorAwsAccountId(empty)AWS account ID (12 digits), required for IRSA.
janitorAwsIamRoleName(empty)IAM role name for the Janitor ServiceAccount. The role needs ec2:RebootInstances and ec2:TerminateInstances.