disks

SMART health of the disks, by SSH

Returns, for each SATA, SAS or NVMe disk of a Linux machine, named by its serial number, its SMART health, its path, its temperature, its reallocated, pending and uncorrectable sectors, its power-on time and, for an SSD or an NVMe, its remaining life. A spun-down disk is left asleep, and keeps the health and sectors last read from it, with their age. Eight watches are offered, each one set on the probe. It needs sudo, restricted to the commands below.

Versions

Generic: the JSON of smartctl -j is the same from smartmontools 7.0 on (Debian 11 and later, Ubuntu 20.04 and later, RHEL 8 and later, Fedora, Alpine), so it stays at the common level. A smartctl older than 7.0 knows no -j: the collector stops misconfigured and says so.

How it works

The collector runs, through sudo -n (never a password prompt: a refused sudo stops at once):

1. /usr/sbin/smartctl --scan-open -j: the disks smartctl can open;
2. /usr/sbin/smartctl -j -n standby -a /dev/<disk>, once per disk named sdX, sdXY, nvmeN or nvmeNN, the names the sudoers line allows; the disk's name goes as a separate, quoted argument. -n standby makes smartctl stop without waking a disk spun down (standby or sleep).

• A disk is named by its serial number, which follows it when it changes slot or when the kernel renames it at a boot; device returns its path (/dev/sda). A disk that serves no serial number (some USB bridges), or that has never been seen awake, is named by the kernel's name (sda).
• The health is passed or failed, the overall verdict smartctl reads from the disk: an ATA attribute at or under its threshold, an NVMe critical warning, a SCSI failure prediction all read failed. It is unknown when the disk serves no SMART, or could not be opened (pulled out since the scan, a USB bridge that does not pass SMART through); only health and device are then returned.
• power reads active or standby. A disk asleep tells nothing, its serial number included: the last reading of the disk awake stands for it. Its serial number names it; its health, reallocated, pending, uncorrectable and remaining-life are those read then, and raise as they did: a failure never goes out because the disk sleeps. read-age says, in hours, how old that reading is: 0 for a disk read at this run. The temperature and the power-on time of a disk asleep are not returned. A disk asleep that has never been seen awake reads standby in health, which raises nothing.
• The last reading of each disk is kept in the probe's cache, by path, rewritten at every run for the disks present: it lasts as long as the probe runs, and is lost seven days after its last run, or when its cache is cleared.
• The sectors, by kind of disk: on an ATA disk, the raw values of attributes 5 (reallocated), 197 (pending) and 198 (uncorrectable); on an NVMe, the media errors (uncorrectable), the data it could not read back; on a SAS disk, the grown defect list (reallocated) and the read errors it could not correct (uncorrectable). A count the disk does not serve is not returned.
• The remaining life, in %: 100 minus the percentage used of an NVMe or of a SAS SSD, never under 0; on an ATA SSD, the normalized value of the attribute that gives it, under the names smartmontools gives them (Wear_Leveling_Count for Samsung, SSD_Life_Left for Kingston and SandForce, Media_Wearout_Indicator for Intel, Percent_Lifetime_Remain for Crucial and Micron). A hard disk returns none.
• Nothing is written on the target: smartctl -a, -n standby and --scan-open only read. The sudoers line below allows no other option.

Prerequisites

• An SSH account for monitoring, without write rights (monitoring below; put the account's name in its place).
• smartmontools 7.0 or later, and sudo: apt install smartmontools sudo (Debian, Ubuntu), dnf install smartmontools sudo (Fedora, RHEL), apk add smartmontools sudo (Alpine, which ships without sudo).
• The sudoers lines, in /etc/sudoers.d/banshee-smart, owned by root, mode 0440 (chmod 0440 /etc/sudoers.d/banshee-smart), checked with visudo -cf /etc/sudoers.d/banshee-smart before anything else. They name smartctl where the distribution puts it.

Debian, Ubuntu, Alpine, RHEL, and Fedora up to 41 (/usr/sbin/smartctl):

Cmnd_Alias BANSHEE_SMART = /usr/sbin/smartctl --scan-open -j, /usr/sbin/smartctl -j -n standby -a /dev/sd?, /usr/sbin/smartctl -j -n standby -a /dev/sd??, /usr/sbin/smartctl -j -n standby -a /dev/nvme?, /usr/sbin/smartctl -j -n standby -a /dev/nvme??
monitoring ALL=(root) NOPASSWD: BANSHEE_SMART

Fedora 42 and later (/usr/bin/smartctl; /usr/sbin is a link to /usr/bin, so the collector's /usr/sbin/smartctl is the same file, which sudo matches):

Cmnd_Alias BANSHEE_SMART = /usr/bin/smartctl --scan-open -j, /usr/bin/smartctl -j -n standby -a /dev/sd?, /usr/bin/smartctl -j -n standby -a /dev/sd??, /usr/bin/smartctl -j -n standby -a /dev/nvme?, /usr/bin/smartctl -j -n standby -a /dev/nvme??
monitoring ALL=(root) NOPASSWD: BANSHEE_SMART

The account may run, as root and without a password, smartctl by its absolute path, with these options in this order, and nothing else: no other option of smartctl (-s, -t, --set, which change a disk's settings or start a self-test), no other command. Only the disk's name varies, and only by ?, one character each: in sudoers a * matches across the arguments (/dev/* would let -j -a /dev/sda -s off through), a ? cannot hold an option. The risk left: ? matches any one character, so /dev/sd?? also lets through a partition (/dev/sda1) or a one-letter extra argument, which smartctl only reads or refuses.

• The fingerprint of the server's host key, set in protocol:ssh:fingerprint (not needed when the Security settings trust the first SSH fingerprint presented).

Outputs

Watches

• reallocated and uncorrectable raise on a rise, not on a count: a disk that has carried a few reallocated sectors for years stays green, a new one raises for a week. A pending sector is a sector the disk could not read and waits to rewrite: it raises as long as there is one.
• The temperature thresholds are those of a hard disk; an NVMe often runs warmer, and its thresholds are raised on the probe.

Limits

• A sudo that refuses (no sudoers line, a line that does not match, a password asked), a missing smartctl, or a smartctl older than 7.0, stops the collector misconfigured, with what sudo or smartctl said. So does a machine with no disk the sudoers line allows.
• Only disks named sdX, sdXY, nvmeN and nvmeNN are read: the names of another system (FreeBSD's ada0, da0) are left out.
• Hardware RAID is not this collector's: the disks behind a RAID controller (/dev/bus/0 -d megaraid,N, cciss, areca) are hidden from the system and left out; the controller's own tools or its management card see them.
• While a disk sleeps, its health and sectors are those of its last reading, read-age hours old: a new failure shows only when it wakes. A disk swapped while asleep is shown with the reading of the disk it replaced until it is read awake.
• The readings kept fit in the probe's cache, 16,384 characters, about a hundred disks: beyond, the collector stops misconfigured.
• A disk seen through two paths (multipath) is read once, by the first path that names it.
• Whether -n standby leaves an NVMe asleep depends on the version of smartmontools; an NVMe is read as if awake when it does not.
• One SSH command per disk: a machine with many disks takes as many round trips.
• The raw value of attribute 5, 197 or 198 is read as a count, as most makers write it; a maker that packs other figures in it would show a wrong count.
• Not checked against a disk: the fields are those smartmontools documents for its JSON output, and a sleeping disk is recognised by the words smartctl writes for it (STANDBY mode, SLEEP mode); the sudoers lines are not checked against a sudo.