banshee/hardware-collection/hardware/gpu/nvidia/ssh/gpus

NVIDIA GPUs, by SSH

Returns, for each NVIDIA GPU of a Linux machine, read by nvidia-smi, its utilization, its memory used, its temperature, its power draw, whether its hardware slows its clocks down, and its uncorrected ECC errors on a GPU that has ECC. Five watches are offered, each one set on the probe. No sudo: nvidia-smi reads all this as any user.

Versions

Generic: the fields asked for are those nvidia-smi --query-gpu has served for years, on every driver that knows the slowdown field under one of its two names. The field of the hardware slowdown is named clocks_throttle_reasons.hw_slowdown by the older drivers and clocks_event_reasons.hw_slowdown by the later ones: the collector asks the driver which it knows first. One collector serves every driver, so it stays at the common level.

How it works

The collector runs two commands:

nvidia-smi --help-query-gpu: the fields the driver knows; it tells which name of the slowdown field to ask for;
nvidia-smi --query-gpu=index,uuid,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw,<slowdown field>,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits.

0, GPU-5e4c0d3a-9f6b-4c1e-8a7d-2b3c4d5e6f70, NVIDIA A100-SXM4-40GB, 37, 10240, 40960, 54, 212.41, Not Active, 0

A GPU is named by its UUID (GPU-…), which follows the card when it changes slot or when the driver numbers the GPUs otherwise at a boot; its index (the order of the PCI bus) and its model are outputs. The index is shorter to read but changes when a card is added or removed: a missing watch on it would follow the wrong card.
utilization is the share of time a kernel ran on the GPU over the last sample, in %; memory-used its memory in use, in MiB, and memory-used-percent the same against its total; temperature its core temperature in °C; power its power draw in W.
slowdown reads active when the hardware cuts the clocks (by a factor of two or more: the GPU too hot, or a power brake from the system), not-active otherwise. The software power cap and the idle clocks, which are normal, are not counted.
ecc-errors counts the uncorrected ECC errors since the driver was loaded; one is enough to make a computation wrong, and the GPU must be reset.
A field the GPU does not serve reads [N/A] or [Not Supported] (ECC on a consumer GPU, the power draw on some): it is not returned.
nvidia-smi missing, a driver not loaded, or a query refused, stops the collector misconfigured with what it said; so does a machine where it lists no GPU.
Nothing is written on the target: both commands only read.

Prerequisites

An SSH account for monitoring, without write rights.
The NVIDIA driver installed and loaded, with nvidia-smi in the account's path (/usr/bin/nvidia-smi on most distributions).
The fingerprint of the server's host key, set in protocol:ssh:fingerprint (not needed when the Security settings trust the first SSH fingerprint presented).

Outputs

Name	Label	Unit	Per instance
`index`	Index	—	yes, per GPU
`model`	Model	—	yes, per GPU
`utilization`	Utilization	%	yes, per GPU
`memory-used`	Used memory	MiB	yes, per GPU
`memory-used-percent`	Used memory	%	yes, per GPU
`temperature`	Temperature	°C	yes, per GPU
`power`	Power draw	W	yes, per GPU
`slowdown`	Hardware slowdown	—	yes, per GPU
`ecc-errors`	Uncorrected ECC errors	—	yes, per GPU

Watches

Name	Title	Kind	Defaults	Checked by default
`hot`	High temperature	above a threshold, on `temperature`	warning above 80 °C, critical above 90 °C, for 3 runs	yes
`slowdown`	Hardware slowdown	state to avoid, on `slowdown`	warning on `active`, for 3 runs	yes
`ecc-errors`	Uncorrected ECC errors	above a threshold, on `ecc-errors`	critical above 0	yes
`memory-full`	GPU memory full	above a threshold, on `memory-used-percent`	warning above 95 %, for 3 runs	no
`missing`	GPU missing	expected instances, on `model`	no GPU named: it raises nothing until names are set on the probe	no

A GPU at 100 % of utilization is a GPU at work: no watch is offered on it by default. A framework that takes all the GPU's memory at start is normal too, hence memory-full unchecked.
A card that falls off the bus (the driver's Xid 79) disappears from the list: only missing sees it, once its UUID is set.

Limits

Linux only, where nvidia-smi is run through the shell of the SSH account; the Windows nvidia-smi.exe is not read.
A model name with a comma would cut its line wrongly: that line is left out.
The temperature thresholds suit most data-center GPUs; the slowdown temperature of each model is given by nvidia-smi -q -d TEMPERATURE, and the thresholds are set lower than it on the probe.
Not checked against a GPU: the fields, their order and the words Active, Not Active, [N/A] are those of NVIDIA's documentation of nvidia-smi and of its NVML reference; that each driver still accepts clocks_throttle_reasons.hw_slowdown when it does not list clocks_event_reasons.hw_slowdown is not verified.