banshee/containers-collection/containers/docker/ssh/resources

Docker container resources, by SSH

Returns, for each running container of a Docker host, the processor and memory it uses and the number of its processes, read by docker stats --no-stream. Three watches are offered, each one set on the probe. It runs docker through sudo, restricted to the one command below, or straight when the monitoring account is in the docker group.

Versions

Generic: docker stats --format '{{json .}}' and the fields read (Name, CPUPerc, MemPerc, PIDs) are those of every Docker Engine still supported. It stays at the common level.

How it works

One command, which only reads, once, and returns:

sudo -n /usr/bin/docker stats --no-stream --format '{{json .}}'

(without sudo -n when the variable docker:sudo is off). It writes one JSON object per running container:

{"Name": "web", "CPUPerc": "0.03%", "MemPerc": "0.24%", "MemUsage": "2.352MiB / 982.5MiB", "PIDs": "2", ...}

A container is named by its name. Only the running containers are read: a stopped one is the collector of the containers'.
The processor in use is counted as Docker counts it, 100 % being one processor: a container busy on four processors reads 400 %.
The memory in use, the page cache left out, is a share of the limit Docker gives the container, the figure MemUsage writes after its slash; Docker writes 0 % when it gives no limit.
A figure Docker cannot give (--: a sample it could not read, the memory and processes of a Windows container) is not returned.
A Docker daemon that does not answer stops the collector critical; sudo refusing the command, docker missing, or an account outside the docker group, stop it misconfigured, with what was said.

Prerequisites

An SSH account for monitoring, without write rights (monitoring below; put the account's name in its place).
The Docker command at /usr/bin/docker.
One of the two: - a sudoers line, the default (docker:sudo on), in /etc/sudoers.d/banshee-docker, owned by root, mode 0440, checked with visudo -cf /etc/sudoers.d/banshee-docker:

monitoring ALL=(root) NOPASSWD: /usr/bin/docker stats --no-stream --format {{json .}}

The account may run, as root and without a password, this one command and nothing else. The lines of all the Docker collectors together are given in the documentation of the collector of the containers. - the docker group (usermod -aG docker monitoring), with docker:sudo off. Simpler, but the docker group is worth root: whoever may talk to the Docker daemon may start a container that mounts the host's disk. The sudoers line is the choice this documentation recommends.

The fingerprint of the server's host key, set in protocol:ssh:fingerprint (not needed when the Security settings trust the first SSH fingerprint presented).

Variables

Name	Label	Says	Default
`docker:sudo`	Through sudo	runs `docker` through `sudo -n`; off when the account is in the `docker` group	on

docker:sudo is the collector's own variable: it is set on the probe, not on a folder or a host. On a host whose account is in the docker group, it is turned off on each Docker probe; duplicating a probe already set carries it to the next host.

Outputs

Name	Label	Unit	Per instance
`cpu`	Processor in use	%	yes, per container
`memory`	Memory in use	%	yes, per container
`processes`	Processes	—	yes, per container

Watches

Name	Title	Kind	Defaults	Checked by default
`memory`	Memory near its limit	above a threshold, on `memory`	warning above 90 %, critical above 95 %, for 3 runs	yes
`cpu`	Processor busy	above a threshold, on `cpu`	none: 100 % is one processor, and the threshold depends on how many a container may use	no
`processes`	Many processes	above a threshold, on `processes`	none	no

A container that reaches its memory limit is killed by the kernel and restarted, or stopped: memory warns before.

Limits

Docker waits for a reading of each container before it answers, up to two seconds, after which that container's figures are not given: the run lasts that much longer.
A container without a memory limit of its own is measured against the limit Docker reports for it, the figure after the slash of MemUsage (docker stats shows it): read it there before setting the thresholds.
Sources: the fields of docker stats --format are those of the Docker documentation (docker container stats); that 100 % is one processor, the memory against the limit, and -- for a figure not given, are read from the Docker command's source (cli/command/container/stats_helpers.go, formatter_stats.go), its documentation saying only "the percentage of the host's CPU". Not run against a Docker host, nor the sudoers line against a sudo.