pods

Kubernetes pods, by its API

Returns, for each pod of a Kubernetes cluster, or of one namespace, its phase, why a container of it waits to run (a crash loop, an image that does not pull), and how many times its containers have restarted, read from the API server with the token of a read-only service account. Four watches are offered, each one set on the probe.

Versions

Generic: GET /api/v1/pods and the fields read (status.phase, containerStatuses, initContainerStatuses) are those of the core API, stable in every Kubernetes still supported, managed clusters and distributions included. It stays at the common level.

How it works

GET https://<host:address>:6443/api/v1/pods?limit=500&fieldSelector=status.phase%21%3DSucceeded, or /api/v1/namespaces/<namespace>/pods when the variable kubernetes:namespace names one, the token in the header Authorization: Bearer <token>, page by page, following metadata.continue to the last page.

{"items": [{"metadata": {"namespace": "shop", "name": "web-7d4b9c-x2kq"}, "status": {"phase": "Running", "containerStatuses": [{"name": "web", "restartCount": 4, "state": {"waiting": {"reason": "CrashLoopBackOff"}}}]}}], "metadata": {"continue": ""}}

• A pod is named <namespace>/<pod> (shop/web-7d4b9c-x2kq): the selection of the probe keeps or leaves out a namespace (kube-system/*) by its pattern.
• The pods that have finished their work (Succeeded, the pods of a job done) are not read: the API leaves them out (fieldSelector).
• The phase is Kubernetes's: Pending, Running, Failed, or Unknown, which the API reference marks as no longer set (the pods of a node lost are seen by the collector of the nodes).
• waiting is the reason a container of the pod, or one of its init containers, waits to run (CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, ContainerCreating...), a crash loop told before any other; none when no container waits.
• restarts adds up the restarts of the pod's containers, its init containers left out.
• A code 401 (the token refused) or 403 (the account may not list the pods) stops the collector misconfigured; any other code but 200 stops it unreachable. No pod at all is an answer: a namespace may be empty.

The token comes from protocol:http:token, a secret variable: it only goes in the header. The answer is not secret.

Prerequisites

• The address of the API server, its port and its certificate, as for the collector of the nodes: the host's address, kubernetes:port (6443, often 443 for a managed cluster), the cluster's authority under protocol:tls:authorities or a pinned fingerprint.
• A service account allowed to list the pods and nothing else, and its long-lived token in protocol:http:token. The service account and its token Secret are those of the documentation of the collector of the nodes; the rule:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: banshee-pods
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: banshee-pods
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: banshee-pods
subjects:
  - kind: ServiceAccount
    name: banshee
    namespace: monitoring

For one namespace only, set in kubernetes:namespace, a Role and a RoleBinding in that namespace, the same rule, in place of the ClusterRole and its binding: the account then reads that namespace and no other. The verb list reads the list only, never a pod's logs, never an exec into it, never a Secret.

Variables

Outputs

Watches

• A pod waits in Pending while it is scheduled and its images pull: pending raises only when it lasts. A pod that no node can take (resources short, a selector no node matches) stays there.
• Failed is the phase of a pod whose containers have all stopped and one in error, such as the pod of a job that failed; it raises until the pod is deleted.
• A pod is new at each rollout, under a new name: a crash loop that comes back with a new pod raises again, under its new name.

Limits

• The pods of a large cluster make a large answer: one page of 500 pods at a time, every run. A probe per namespace (kubernetes:namespace) reads less.
• Whether a pod is ready (its readiness probe) is not read: a workload short of ready replicas is the collector of the workloads'.
• The address is written into the URL as is: an IPv6 address must be given as a name.
• Sources: the route, the phases, containerStatuses, initContainerStatuses and state.waiting.reason are those of the Kubernetes API reference (Pod, core/v1), restartCount that of its Go types (k8s.io/api/core/v1), the field selector that of the documentation ("Field Selectors"), CrashLoopBackOff that of "Pod Lifecycle". The other reasons a container waits are those the kubelet writes, and others may come. Not run against a cluster.