h8lio

Prometheus

Tutorial to install a Prometheus instance used to scrap yours pod’s metrics.

Requirements

  • A Grafana instance (see Grafana Helm chart or search “grafana” on KubeApps)
  • A deployed service (Kubernetes Deployment & Service) providing a metrics endpoint
  • kubectl installed and configured to your Kubernetes clusters

Install

To install Prometheus, we are using the Prometheus Operator integrated to Kubernetes.

RBAC

Creates the Service Account to grant to the Prometheus instance the permissions to scrap the metrics from the Kubernetes Services endpoints:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  # set the namespace where Prometheus will be installed
  namespace: <namespace>

Associate the “prometheus” service account to the “oidc:developer” Role (you could create your own Role)

kubectl edit -n <namespace> oidc:<organissation>:developer

Adds the “prometheus” service account as a subject:

subjects:
  ...
  - kind: ServiceAccount
    name: prometheus

if you plan to scrap services in multiple namespaces, edit the RoleBinding “oidc:<organissation>:developer” in every namespaces is required.

Prometheus

Example of Prometheus manifest to create your own instance:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
  # set the namespace
  namespace: <namespace>
spec:
  enableAdminAPI: false
  portName: web
  replicas: 1
  # service account
  serviceAccountName: prometheus
  # adjust resources to your needs
  resources:
    limits:
      cpu: 1
      memory: 800Mi
    requests:
      cpu: 250m
      memory: 400Mi
  # scrapping configuration
  scrapeInterval: 30s
  evaluationInterval: 30s
  retention: 7d
  # service monitor selector
  serviceMonitorSelector:
    matchLabels:
      release: my-prometheus
  # rule selector
  ruleSelector:
    matchLabels:
      release: my-prometheus
  # security context
  securityContext:
    fsGroup: 65534
    runAsUser: 65534
  # storage
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            # adjust to your needs
            storage: 10Gi
        # specifies the storage class used to create
        # the persistency
        storageClassName: eu-west-fr-gra-block-hdd-ec-ext4

See the Prometheus CRD for more details

One the Prometheus manifest is applied to your namespace, a prometheus pod will be created.

To connect to your Prometheus instance locally:

kubectl -n <namespace> port-forward svc/prometheus-operated  9090:9090

Then go to loclhost:9090

Service Monitoring

Once your Prometheus instance is up and running, you will be able to deploy ServiceMonitor to select and scrap the Kubernetes Services providing metrics.

Example of ServiceMonitor used to scrap MariaDB metrics (using mysqld-exporter sidecar container):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    # your prometheus instance name
    release: my-prometheus
  name: mariadb
  # mariadb namespace
  namespace: <namespace>
spec:
  # name of the Prometheus job
  jobLabel: mariadb
  endpoints:
    - interval: 15s
      # MariaDB service "metrics" endpoint
      port: metrics
  # MariaDB service selector
  selector:
    matchLabels:
      app.kubernetes.io/component: primary
      app.kubernetes.io/instance: mariadb

See the ServiceMonitor CRD for more details

Once your ServiceMonitor has been deployed, you should see the target and its endpoints on the Prometheus console (restart may be required).

If the target is up, you should be able to do your first PromQL queries on the scrapped metrics.

Rules

You may want to create rules to be used by Grafana Alerting.

Example of PrometheusRule for the above MariaDB Service monitor:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    # your prometheus instance name
    release: my-prometheus
  name: mariadb
  # mariadb namespace
  namespace: <namespace>
spec:
  groups:
    # rules group name
    - name: mariadb
      rules:
        # check if the MariaDB ServiceMonitor job is down
        - alert: MariaDB-Down
          annotations:
            message: MariaDB instance {{ $labels.instance }} is down
            summary: MariaDB instance is down
          expr: absent(up{job="mariadb"} == 1)
          for: 5m
          labels:
            service: mariadb
            severity: warning
        # check if the MariaDB has more than 100 active connections
        # using PromQL
        - alert: HighMariaDBConnections
          annotations:
            description: >-
              MariaDB has more than 100 active connections for more than 5
              minutes.
            summary: High number of MariaDB connections
          expr: mysql_global_status_threads_connected > 100
          for: 5m
          labels:
            severity: warning

See the PrometheusRule CRD for more details

Next Steps

  • Add the Prometheus instance to Grafana as datasource
  • Import or create a Grafana dashboard
  • Configure the Grafana Alerting
January 9, 2026
Loki
Sealed Secrets