events

Kubernetes warning events, by its API

Returns, for each reason of the warning events a Kubernetes cluster, or one namespace, keeps (BackOff, FailedScheduling, FailedMount, Unhealthy...), how many times it was reported, read from the API server with the token of a read-only service account. One watch is offered, not checked by default: a cluster in good health reports warnings too.

Versions

Generic: GET /api/v1/events with the field selector type=Warning is served by the core API of every Kubernetes still supported. It stays at the common level.

How it works

GET https://<host:address>:6443/api/v1/events?limit=500&fieldSelector=type%3DWarning, or /api/v1/namespaces/<namespace>/events when the variable kubernetes:namespace names one, the token in the header Authorization: Bearer <token>, page by page, following metadata.continue to the last page.

{"items": [{"type": "Warning", "reason": "BackOff", "count": 12, "involvedObject": {...}, "message": "..."}], "metadata": {"continue": ""}}

• A reason is named as Kubernetes writes it. An event without a reason counts under unknown.
• count adds up, for each reason, how many times each of its events occurred: series.count when the event is a series, its count otherwise; an event that writes neither counts once.
• The API server keeps an event one hour by default (its option --event-ttl): the counts are those of that window, and fall as the events expire.
• A code 401 (the token refused) or 403 (the account may not list the events) stops the collector misconfigured; any other code but 200 stops it unreachable. No warning at all is an answer.

The token comes from protocol:http:token, a secret variable: it only goes in the header. The answer is not secret.

Prerequisites

• The address of the API server, its port and its certificate, as for the collector of the nodes.
• A service account allowed to list the events and nothing else, and its long-lived token in protocol:http:token. The service account and its token Secret are those of the documentation of the collector of the nodes; the rule:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: banshee-events
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: banshee-events
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: banshee-events
subjects:
  - kind: ServiceAccount
    name: banshee
    namespace: monitoring

For one namespace only, set in kubernetes:namespace, a Role and a RoleBinding in that namespace, the same rule, in place of the ClusterRole and its binding.

Variables

Outputs

Watches

• Checked, it warns on any warning the cluster keeps; the selection of the probe keeps the reasons that matter (only: FailedMount, FailedScheduling), or leaves out the noisy ones.

Limits

• The message and the object of an event are not returned: the reason says what, kubectl get events says where.
• A cluster whose API server keeps events longer, or shorter, than an hour counts over that window.
• The address is written into the URL as is: an IPv6 address must be given as a name.
• Sources: type, reason, count and series.count are those of the Kubernetes API reference (Event, core/v1), the field selector type that of the documentation ("Field Selectors"), and the hour of --event-ttl the default the kube-apiserver reference gives. Not run against a cluster.