banshee/containers-collection/containers/docker/ssh/compose

Docker Compose services, by SSH

Returns, for each service of each Docker Compose project of a Docker host, whether all its containers run, their health, and how many of them run. Three watches are offered, each one set on the probe. It runs docker through sudo, restricted to the one command below (the same as the collector of the containers), or straight when the monitoring account is in the docker group.

Versions

Generic: the projects and services are read from the labels Compose puts on every container it creates (com.docker.compose.project, com.docker.compose.service, com.docker.compose.oneoff), the same for Compose v1 (docker-compose) and Compose v2 (docker compose). The Compose plugin is not needed on the host, nor the projects' files: only docker ps. It stays at the common level.

How it works

One command, which only reads:

sudo -n /usr/bin/docker ps --all --format '{{json .}}'

(without sudo -n when the variable docker:sudo is off). It writes one JSON object per container, its labels written key=value and separated by commas:

{"Names": "myapp-web-1", "State": "running", "Status": "Up 2 hours (healthy)", "Labels": "com.docker.compose.project=myapp,com.docker.compose.service=web,...", ...}

A service is named <project>/<service> (myapp/web): the selection of the probe keeps one project (myapp/*), or leaves out a service that runs once and stops on purpose (myapp/migrate).
The containers of docker compose run (com.docker.compose.oneoff=True) are left out, as are the containers Compose did not create.
The state of a service is running when all its containers run; otherwise the Docker state of the first that does not (exited, restarting, paused, created, dead).
Its health is the worst of its containers': unhealthy, then starting, then healthy; none when none has a health check.
containers counts its containers, running those that run: a service scaled to three replicas with one stopped reads 3 and 2, and is not running.
A Docker daemon that does not answer stops the collector critical; sudo refusing the command, docker missing, or an account outside the docker group, stop it misconfigured, with what was said.

Prerequisites

An SSH account for monitoring, without write rights (monitoring below; put the account's name in its place).
The Docker command at /usr/bin/docker.
One of the two: - a sudoers line, the default (docker:sudo on), in /etc/sudoers.d/banshee-docker, owned by root, mode 0440, checked with visudo -cf /etc/sudoers.d/banshee-docker. It is the line of the collector of the containers: one line serves both.

monitoring ALL=(root) NOPASSWD: /usr/bin/docker ps --all --format {{json .}}

The account may run, as root and without a password, this one command and nothing else. The lines of all the Docker collectors together are given in the documentation of the collector of the containers. - the docker group (usermod -aG docker monitoring), with docker:sudo off. Simpler, but the docker group is worth root: whoever may talk to the Docker daemon may start a container that mounts the host's disk. The sudoers line is the choice this documentation recommends.

The fingerprint of the server's host key, set in protocol:ssh:fingerprint (not needed when the Security settings trust the first SSH fingerprint presented).

Variables

Name	Label	Says	Default
`docker:sudo`	Through sudo	runs `docker` through `sudo -n`; off when the account is in the `docker` group	on

docker:sudo is the collector's own variable: it is set on the probe, not on a folder or a host. On a host whose account is in the docker group, it is turned off on each Docker probe; duplicating a probe already set carries it to the next host.

Outputs

Name	Label	Unit	Per instance
`state`	State	—	yes, per service
`health`	Health	—	yes, per service
`containers`	Containers	—	yes, per service
`running`	Containers running	—	yes, per service

Watches

Name	Title	Kind	Defaults	Checked by default
`not-running`	Service not running	expected value, on `state`	critical when the state is anything but `running`	yes
`unhealthy`	Service unhealthy	state to avoid, on `health`	critical on `unhealthy`	yes
`missing`	Service missing	expected instances, on `state`	no service named: it raises nothing until names are set on the probe	no

A service whose containers have all been removed (docker compose down, docker compose rm) is no longer listed: missing, with <project>/<service> names set on the probe, sees it go.

Limits

The services are those that have containers: a service written in the Compose file but never created is not seen, the files of the projects not being read.
A service that runs once and stops on purpose reads exited, and raises not-running until the selection leaves it out.
Two projects of the same name, in two folders, share their containers under one name, as Compose itself mixes them.
Sources: the labels and the value True of a one-off container are those of the Compose source (pkg/api/labels.go); the fields of docker ps --format those of the Docker documentation, and the labels written key=value, sorted and joined by commas, as the Docker command's source writes them (cli/command/formatter/container.go); the health in the status, as the Docker Engine's source writes it. Not run against a Docker host, nor the sudoers line against a sudo.