Alerts

The platform watches your cluster resources and notifies you when one of them needs attention. You have nothing to install and nothing to subscribe to: alerting is on for every cluster of your organization, with sensible defaults you can adjust per resource.

What is watched

Alert	Fires when	Checked
Volume usage	A Persistent Volume Claim has less than 8% free space left	Every hour

This list grows with the platform. Alerts that concern the platform itself (nodes, storage backend, control plane) are handled by the operations team and never reach you.

What you receive

Each alert reaches you two ways:

• an entry in the Notifications panel of your dashboard, opened from the bell in the header, with a link to the resource concerned,
• an email to every Owner, Admin and Operator of the organization that owns the cluster. Developers do not receive alert emails.

The message names the resource, its namespace and the measured value, for example volume data-postgres-0 in acme-prod has 6% space left.

An alert that stays active does not email you every hour: for a given resource, at most one email per 24 hours goes out. The Notifications panel keeps the entry up to date in the meantime.

Note

An alert is scoped to the organization that owns the namespace. Nobody outside your organization receives it.

Configuring an alert on a resource

The configuration lives on the resource itself, so it follows your manifests and your GitOps repository rather than a setting stored elsewhere. There are two levels, and they are carried differently.

Turning detection off

A label stops the alert at the source: the resource is excluded before anything is evaluated, so there is no notification, no email, and no entry anywhere.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scratch-data
  labels:
    alert.h8l.io/disabled: "true"

kubectl label pvc scratch-data alert.h8l.io/disabled=true    # off
kubectl label pvc scratch-data alert.h8l.io/disabled-        # back on

Use it for volumes meant to run close to full, where an alert is noise rather than a signal.

Adjusting the notification

Annotations tune what happens when an alert does fire. Each one overrides a default; the ones you leave out keep theirs.

Annotation	Effect	Default
alert.h8l.io/email: "false"	keep the entry in the Notifications panel, send no email	emails are sent
alert.h8l.io/recipients	who is emailed: roles (owner, admin, operator) and/or addresses, comma separated	owner,admin,operator
alert.h8l.io/silence	how long before the same alert emails again, ISO 8601 duration	PT24H (24 hours)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgres
  annotations:
    alert.h8l.io/recipients: "owner,storage-oncall@acme.com"
    alert.h8l.io/silence: "PT6H"

Four things worth knowing:

• Values are always strings. "true", not true, and "PT6H" in quotes. Kubernetes stores label and annotation values as text, and an unquoted value is rejected or silently converted.
• Labels and annotations are not interchangeable here. disabled is a label because the monitoring engine selects on it; the other three are annotations, read when the alert fires. Putting one in the other’s place has no effect.
• A value we cannot read is ignored, and the default applies. A misspelled duration never costs you the alert.
• Changes take effect at the next check, hourly for volume usage, not immediately.

Alerts

Volume usage

Fires when a Persistent Volume Claim has less than 8% free space left, checked every hour. Reaching 100% usually means the application writing to the volume starts failing, so the alert is there to leave you time to resize the volume or clean it up.

You can watch the same signal yourself before it ever fires, from Grafana:

kubelet_volume_stats_available_bytes{namespace="acme-prod"}
  / kubelet_volume_stats_capacity_bytes{namespace="acme-prod"}

Use it to build your own alert with a different threshold in your own Prometheus instance, for example a first warning at 20%.

Resources

• Volumes, to resize a volume that is genuinely running out of space
• Prometheus, to define your own alerting rules
• Labels and Selectors and Annotations