Skip to main content
Version: main 🚧

Deploy GPU monitoring with Argo CD

When the Argo CD integration is enabled, vCluster Platform ships a set of ArgoCDApplicationTemplate resources for GPU monitoring. Together they deploy the NVIDIA GPU stack into a cluster that runs GPU nodes, typically a tenant cluster with private nodes. The GPU Operator provides drivers and telemetry, the Fleet Intelligence Agent provides node-level GPU health reporting, and NVSentinel provides fault detection and remediation. A fourth template, cert-manager, supplies the internal TLS certificates some of those components need.

This page summarizes the templates, the order to deploy them in, and the metric behaviors to know before you build queries on top of them. For the NVSentinel deploy guide and parameter reference, see Monitor GPU nodes with NVSentinel.

Bundled GPU templates​

In an ArgoCDApplication, reference each template by its resource name through spec.templateRef.name. The Platform UI lists them by display name. For the general template and application model, see Deploy applications.

Template (Platform UI)templateRef.nameDeploysDefault namespace
cert-managercert-managercert-manager, for the internal TLS certificates NVSentinel components needcert-manager
NVIDIA GPU Operatornvidia-gpu-operatorNVIDIA drivers, device plugin, DCGM, and dcgm-exportergpu-operator
Fleet Intelligence Agentfleet-intelligence-agentNVIDIA's gpud-based node agent that reports GPU health and DCGM metricsfleet-intelligence
NVSentinelnvsentinelGPU fault detection, quarantine, and remediationnvsentinel

Deployment order​

The applications depend on each other at runtime, so deploy them in this order:

  1. cert-manager, if you plan to enable NVSentinel components that need internal TLS, such as the MongoDB store or the Janitor. See the NVSentinel prerequisites for exactly which components require it.
  2. NVIDIA GPU Operator. Everything else builds on it. It installs the drivers and device plugin, and runs the DCGM service that NVSentinel reads GPU health from, plus dcgm-exporter for GPU metrics.
  3. Fleet Intelligence Agent, optionally, for node-level GPU health reporting. The bundled GPU dashboard reads some panels from its metrics.
  4. NVSentinel, to detect and remediate GPU faults. See Monitor GPU nodes with NVSentinel.

GPU metrics and attribution​

When the GPU templates are deployed, two separate exporters report metrics for the same GPUs, and by default those metrics don't show which pod used a GPU. Both behaviors are intentional. Keep them in mind when you write your own queries or bill tenants for their GPU usage based on the bundled dashboards.

Two DCGM metric families​

The same physical GPU is reported by two exporters when the full GPU stack is deployed:

SourceDeployed byMetric familyExample
dcgm-exporterNVIDIA GPU Operator templateUPPERCASE DCGM_FI_DEV_*DCGM_FI_DEV_GPU_UTIL
gpudFleet Intelligence Agent templatelowercase dcgm_fi_dev_*dcgm_fi_dev_gpu_util

The bundled GPU dashboard mixes both families. Some panels read the dcgm-exporter series and others read the Fleet Intelligence Agent series.

Pick one metric family per query

Aggregating across both families counts every GPU twice, for example with a regular expression matcher that is case-insensitive or that spans both spellings. When you write your own PromQL, pick one family per query.

Per-pod GPU attribution is off by default​

The NVIDIA GPU Operator template ships dcgm-exporter with dcgmExporterEnablePodLabels: "false" and dcgmExporterEnablePodUID: "false". GPU metrics then carry node, GPU, UUID, and modelName labels but no pod or namespace label, so GPU usage can't be attributed to a workload:

DCGM_FI_DEV_GPU_UTIL{gpu="0", UUID="GPU-9d9d49d7-...", modelName="Tesla T4", Hostname="ip-192-168-75-24"} 0

To attribute GPU usage to workloads and bill tenants for it, set both parameters to "true" on the NVIDIA GPU Operator application and let it sync.

Benign crashloop on startup

With pod labels enabled, dcgm-exporter can restart a few times right after rollout while its ServiceAccount token is projected, logging Failed to get in-cluster config, pod labels will not be available. It heals on its own within a few restarts. No action is needed.

The cluster-collector template scrapes GPU metrics automatically​

If you deploy edge collectors with the cluster-collector template, described in Deploy collectors with Argo CD, it also scrapes the GPU stack. No extra collector configuration is needed. Alongside its per-node DaemonSet, the template deploys a cluster-singleton collector preconfigured with scrape jobs for:

  • The Fleet Intelligence Agent, in the fleet-intelligence namespace.
  • The NVIDIA GPU Operator's dcgm-exporter, in the gpu-operator namespace.
  • NVSentinel's GPU Health Monitor, in the nvsentinel namespace.

These namespaces are hardcoded in the collector template. They aren't derived from the GPU templates' destinationNamespace parameters, so this only works when you keep those parameters at their defaults, shown in Bundled GPU templates above. If you deploy a GPU template into a non-default namespace, its metrics stop reaching Fleet Observability through this collector. If you skip a component, such as the optional Fleet Intelligence Agent, its scrape job simply finds no targets.

Tear down the GPU stack​

Deleting the ArgoCDApplication resources prunes the workloads, but some artifacts are retained by design and stay in the cluster until you remove them. These are NVSentinel's MongoDB PVCs, the cert-manager CRDs (keepCRDs: true), and the destination namespaces, which stay Active with zero pods.

Target the tenant cluster's kubectl context for every command below. CRDs and namespaces are cluster-wide, and running these against the wrong cluster, such as your control plane cluster, can remove cert-manager or a namespace that other workloads there still depend on.

First remove NVSentinel's retained data and namespace, described in Uninstall and clean up NVSentinel. Then remove the remaining GPU namespaces:

Clean up GPU stack namespaces
TENANT_CLUSTER_CONTEXT=my-vcluster
kubectl --context "$TENANT_CLUSTER_CONTEXT" delete namespace gpu-operator fleet-intelligence

Remove cert-manager entirely (optional)​

CRDs are cluster-wide type definitions, so they can't be scoped to the GPU templates. The only choice is to keep them or remove them for the whole cluster. Run this step only when cert-manager was installed solely for the GPU stack and nothing else in the cluster uses it. Deleting a CRD cascade-deletes every resource of that type in every namespace.

Remove cert-manager CRDs and namespace
TENANT_CLUSTER_CONTEXT=my-vcluster

# Review what exists first
kubectl --context "$TENANT_CLUSTER_CONTEXT" get crd -o name | grep -E '\.cert-manager\.io$'

kubectl --context "$TENANT_CLUSTER_CONTEXT" delete crd \
  certificates.cert-manager.io \
  certificaterequests.cert-manager.io \
  challenges.acme.cert-manager.io \
  clusterissuers.cert-manager.io \
  issuers.cert-manager.io \
  orders.acme.cert-manager.io

kubectl --context "$TENANT_CLUSTER_CONTEXT" delete namespace cert-manager

None of these carry blocking finalizers, so deletion completes cleanly.