disks

SMART health of the disks, by SSH

Returns, for each SATA, SAS or NVMe disk of a Linux machine, named by its serial number, its SMART health, its path, its temperature, its reallocated, pending and uncorrectable sectors, its power-on time and, for an SSD or an NVMe, its remaining life. A spun-down disk is left asleep, and keeps the health and sectors last read from it, with their age, and warns when that age passes a week. Nine watches are offered, each one set on the probe. It needs sudo, restricted to the commands below.

Versions

Generic: the JSON of smartctl -j is the same from smartmontools 7.0 on (Debian 11 and later, Ubuntu 20.04 and later, RHEL 8 and later, Fedora, Alpine), so it stays at the common level. A smartctl older than 7.0 knows no -j: the collector stops misconfigured and says so.

How it works

The collector runs, through sudo -n (never a password prompt: a refused sudo stops at once):

1. /usr/sbin/smartctl --scan-open -j: the disks smartctl can open;
2. /usr/sbin/smartctl -j -n standby -a /dev/<disk>, once per disk named sdX, sdXY, nvmeN or nvmeNN, the names the sudoers line allows; the disk's name goes as a separate, quoted argument. -n standby makes smartctl stop without waking a disk spun down (standby or sleep).

• A disk is named by its serial number, which follows it when it changes slot or when the kernel renames it at a boot; device returns its path (/dev/sda). A disk that serves no serial number (some USB bridges), or that has never been seen awake, is named by the kernel's name (sda).
• The health is passed or failed, the overall verdict smartctl reads from the disk: an ATA attribute at or under its threshold, an NVMe critical warning, a SCSI failure prediction all read failed. It is unknown when the disk serves no SMART, or could not be opened (pulled out since the scan, a USB bridge that does not pass SMART through); only health and device are then returned.
• power reads active or standby. A disk asleep tells nothing, its serial number included: the last reading of the disk awake stands for it. Its serial number names it; its health, reallocated, pending, uncorrectable and remaining-life are those read then, and raise as they did: a failure never goes out because the disk sleeps. read-age says, in hours, how old that reading is: 0 for a disk read at this run. The temperature and the power-on time of a disk asleep are not returned. A disk asleep that has never been seen awake reads standby in health, which raises nothing.
• The last reading of each disk is kept in the probe's cache, by path, rewritten at every run for the disks present: it lasts as long as the probe runs, and is lost seven days after its last run, or when its cache is cleared.
• The sectors, by kind of disk: on an ATA disk, the raw values of attributes 5 (reallocated), 197 (pending) and 198 (uncorrectable); on an NVMe, the media errors (uncorrectable), the data it could not read back; on a SAS disk, the grown defect list (reallocated) and the read errors it could not correct (uncorrectable). A count the disk does not serve is not returned.
• The remaining life, in %: 100 minus the percentage used of an NVMe or of a SAS SSD, never under 0; on an ATA SSD, the normalized value of the attribute that gives it, under the names smartmontools gives them (Wear_Leveling_Count for Samsung, SSD_Life_Left for Kingston and SandForce, Media_Wearout_Indicator for Intel, Percent_Lifetime_Remain for Crucial and Micron). A hard disk returns none.
• Nothing is written on the target: smartctl -a, -n standby and --scan-open only read. The sudoers line below allows no other option.

Prerequisites

• An SSH account for monitoring, without write rights (monitoring below; put the account's name in its place).
• smartmontools 7.0 or later, and sudo: apt install smartmontools sudo (Debian, Ubuntu), dnf install smartmontools sudo (Fedora, RHEL), apk add smartmontools sudo (Alpine, which ships without sudo).
• The sudoers lines, in /etc/sudoers.d/banshee-smart, owned by root, mode 0440 (chmod 0440 /etc/sudoers.d/banshee-smart), checked with visudo -cf /etc/sudoers.d/banshee-smart before anything else. They name smartctl where the distribution puts it.

Debian, Ubuntu, Alpine, RHEL, and Fedora up to 41 (/usr/sbin/smartctl):

Cmnd_Alias BANSHEE_SMART = /usr/sbin/smartctl --scan-open -j, /usr/sbin/smartctl -j -n standby -a /dev/sd?, /usr/sbin/smartctl -j -n standby -a /dev/sd??, /usr/sbin/smartctl -j -n standby -a /dev/nvme?, /usr/sbin/smartctl -j -n standby -a /dev/nvme??
monitoring ALL=(root) NOPASSWD: BANSHEE_SMART

Fedora 42 and later (/usr/bin/smartctl; /usr/sbin is a link to /usr/bin, so the collector's /usr/sbin/smartctl is the same file, which sudo matches):

Cmnd_Alias BANSHEE_SMART = /usr/bin/smartctl --scan-open -j, /usr/bin/smartctl -j -n standby -a /dev/sd?, /usr/bin/smartctl -j -n standby -a /dev/sd??, /usr/bin/smartctl -j -n standby -a /dev/nvme?, /usr/bin/smartctl -j -n standby -a /dev/nvme??
monitoring ALL=(root) NOPASSWD: BANSHEE_SMART

The account may run, as root and without a password, smartctl by its absolute path, with these options in this order, and nothing else: no other option of smartctl (-s, -t, --set, which change a disk's settings or start a self-test), no other command. Only the disk's name varies, and only by ?, one character each: in sudoers a * matches across the arguments (/dev/* would let -j -a /dev/sda -s off through), a ? cannot hold an option. The risk left: ? matches any one character, so /dev/sd?? also lets through a partition (/dev/sda1) or a one-letter extra argument, which smartctl only reads or refuses.

• The fingerprint of the server's host key, set in protocol:ssh:fingerprint (not needed when the Security settings trust the first SSH fingerprint presented).

Outputs

Watches

• reallocated and uncorrectable raise on a rise, not on a count: a disk that has carried a few reallocated sectors for years stays green, a new one raises for a week. A pending sector is a sector the disk could not read and waits to rewrite: it raises as long as there is one.
• stale warns when the reading that stands for a disk asleep is more than a week old: its health and sectors then say what the disk was a week ago, not what it is. A week lets a disk sleep a night or a few days, as an archive or a backup disk does, without raising. A disk woken once a week sits on the edge: raise the threshold on the probe (192 h, eight days).
• The temperature thresholds are those of a hard disk; an NVMe often runs warmer, and its thresholds are raised on the probe.

Limits

• A sudo that refuses (no sudoers line, a line that does not match, a password asked), a missing smartctl, or a smartctl older than 7.0, stops the collector misconfigured, with what sudo or smartctl said. So does a machine with no disk the sudoers line allows.
• Only disks named sdX, sdXY, nvmeN and nvmeNN are read: the names of another system (FreeBSD's ada0, da0) are left out.
• Hardware RAID is not this collector's: the disks behind a RAID controller (/dev/bus/0 -d megaraid,N, cciss, areca) are hidden from the system and left out; the controller's own tools or its management card see them.
• While a disk sleeps, its health and sectors are those of its last reading, read-age hours old: a new failure shows only when it wakes, and stale warns once the reading is more than a week old. A disk swapped while asleep is shown with the reading of the disk it replaced until it is read awake.
• The readings kept fit in the probe's cache, 16,384 characters, about a hundred disks: beyond, the collector stops misconfigured.
• A disk seen through two paths (multipath) is read once, by the first path that names it.
• Whether -n standby leaves an NVMe asleep depends on the version of smartmontools; an NVMe is read as if awake when it does not.
• One SSH command per disk: a machine with many disks takes as many round trips.
• The raw value of attribute 5, 197 or 198 is read as a count, as most makers write it; a maker that packs other figures in it would show a wrong count.
• Not checked against a disk: the fields are those smartmontools documents for its JSON output, and a sleeping disk is recognised by the words smartctl writes for it (STANDBY mode, SLEEP mode); the sudoers lines are not checked against a sudo.