banshee/containers-collection/containers/kubernetes/https/nodes

Kubernetes nodes, by its API

Returns, for each node of a Kubernetes cluster, whether it is ready, whether the kubelet reports a lack of memory, of disk or of process ids, and whether new pods may be scheduled on it, read from the API server with the token of a read-only service account. Six watches are offered, each one set on the probe.

Versions

Generic: GET /api/v1/nodes and the node conditions read (Ready, MemoryPressure, DiskPressure, PIDPressure) are those of the core API, stable in every Kubernetes still supported, managed clusters (EKS, AKS, GKE) and distributions (k3s, RKE2, OpenShift) included. It stays at the common level.

How it works

GET https://<host:address>:6443/api/v1/nodes?limit=500, the token in the header Authorization: Bearer <token>, page by page: the API cuts a long list and gives the token of the next page under metadata.continue, which the collector follows to the last.

{"kind": "NodeList", "metadata": {"continue": ""}, "items": [{"metadata": {"name": "worker-1"}, "spec": {"unschedulable": true}, "status": {"conditions": [{"type": "Ready", "status": "True"}, {"type": "MemoryPressure", "status": "False"}, ...]}}]}

A node is named as the cluster names it.
ready, memory-pressure, disk-pressure and pid-pressure are the status of the condition of that name: True, False, or Unknown when the node has stopped reporting. A condition the node does not give reads Unknown.
schedulable reads no for a node cordoned (kubectl cordon, a drain under way), yes otherwise.
No node at all stops the collector misconfigured. A code 401 (the token refused) or 403 (the account may not list the nodes) stops it misconfigured; any other code but 200 stops it unreachable.

The token comes from protocol:http:token, a secret variable: it only goes in the header. The answer is not secret.

Prerequisites

The address of the API server as the host's address, reachable from the poller in HTTPS on port 6443, or the port set in the variable kubernetes:port (443 for most managed clusters). Its certificate is issued by the cluster's own authority: trust it by that authority (/etc/kubernetes/pki/ca.crt on a kubeadm control plane, certificate-authority-data in a kubeconfig) under protocol:tls:authorities, by its pinned fingerprint in protocol:http:fingerprint, or by the first fingerprint trusted in the tab "Security". The name the certificate carries is the one to give as the host's address, or in protocol:tls:server-name.
A service account, allowed to list the nodes and nothing else, and its token in protocol:http:token. Applied with kubectl apply -f:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: banshee
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: banshee-nodes
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: banshee-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: banshee-nodes
subjects:
  - kind: ServiceAccount
    name: banshee
    namespace: monitoring
---
apiVersion: v1
kind: Secret
metadata:
  name: banshee-token
  namespace: monitoring
  annotations:
    kubernetes.io/service-account.name: banshee
type: kubernetes.io/service-account-token

The namespace monitoring must exist (kubectl create namespace monitoring). The token, long-lived, is then read once: kubectl -n monitoring get secret banshee-token -o jsonpath='{.data.token}' | base64 -d. The verb list alone reads the list, nothing else: neither one object by its name, nor a watch, nor any change. The nodes are not in a namespace: a ClusterRole is needed. With the other Kubernetes collectors, their rules go in the same ClusterRole, under one binding.

Variables

Name	Label	Says	Default
`kubernetes:port`	API port	the port of the API server	`6443`

Outputs

Name	Label	Unit	Per instance
`ready`	Ready	—	yes, per node
`memory-pressure`	Memory pressure	—	yes, per node
`disk-pressure`	Disk pressure	—	yes, per node
`pid-pressure`	Process id pressure	—	yes, per node
`schedulable`	Schedulable	—	yes, per node

Watches

Name	Title	Kind	Defaults	Checked by default
`not-ready`	Node not ready	expected value, on `ready`	critical when it is anything but `True`	yes
`memory-pressure`	Memory pressure	state to avoid, on `memory-pressure`	warning on `True`	yes
`disk-pressure`	Disk pressure	state to avoid, on `disk-pressure`	warning on `True`	yes
`pid-pressure`	Process id pressure	state to avoid, on `pid-pressure`	warning on `True`	yes
`cordoned`	Node cordoned	state to avoid, on `schedulable`	warning on `no`	no
`missing`	Node missing	expected instances, on `ready`	no node named: it raises nothing until names are set on the probe	no

Under a pressure, the kubelet evicts pods from the node and refuses new ones.

Limits

The probe reads the cluster through the API server it is set on: that server down, the collector stops unreachable, which a probe per control-plane node, or the address of their load balancer, avoids.
A node removed from the cluster is no longer listed: only missing, with its name set on the probe, sees it go.
The address is written into the URL as is: an IPv6 address must be given as a name.
Sources: the conditions and their values, and the cordon written in the node's spec (unschedulable), are those of the Kubernetes documentation ("Node Status"); the pages of a list, metadata.continue, those of the API reference (ListMeta); the token and its Secret those of "Configure Service Accounts for Pods". Not run against a cluster.