Loki

Tutorial to collect your namespace’s pod logs with a Grafana Alloy Deployment, reading them through the Kubernetes API, and ship them to Grafana Loki.

Note

This guide assumes a Spring Boot service, but the collection model is application-agnostic: any container that writes to stdout/stderr is picked up automatically. Adapt it to your needs.

Requirements

• A Grafana instance (see the Grafana Helm chart)
• A deployed service (Kubernetes Deployment) that logs to stdout/stderr
• kubectl and helm installed and configured for your Kubernetes cluster
• (optional) a namespace dedicated to your cross-namespace tooling (example: mycorp-monitoring). It can host both Loki and Grafana.

Loki

Install Loki with its Helm chart. For a small, single-tenant customer setup, run Loki in monolithic mode (the SingleBinary deployment, equivalent to -target=all): all Loki components run in one process, which keeps operations simple and scales fine to a few hundred GB/day. See the deployment modes guide for when to graduate to the scalable or distributed topologies.

Note

The OSS Loki Helm chart moved to the grafana-community/helm-charts repository in March 2026. If you have an existing grafana/loki release, review the community repo migration guide before upgrading.

Configure

Create a loki-values.yaml. The two key choices are the deployment mode and the storage backend. Below is a minimal sketch; consult the chart values.yaml for the exhaustive and current schema before applying.

deploymentMode: SingleBinary
loki:
  # Single-tenant setup: disable multi-tenancy so pushes/queries need no tenant header.
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
singleBinary:
  replicas: 1
# Disable the scalable-mode component groups in monolithic mode.
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

For production-grade durability, point object_store at your S3-compatible bucket (Ceph RGW on h8lio) rather than filesystem. The chart docs cover the exact storage block.

Installation

# OSS chart, community repository (March 2026 onward)
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm search repo grafana-community/loki
helm install loki grafana-community/loki -f loki-values.yaml --namespace [NAMESPACE]

(optional) Expose

Deploy Traefik routes to reach your Loki instance from outside the cluster. Most setups keep Loki internal (queried by an in-cluster Grafana) and skip this.

• routes.yaml

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: loki
spec:
  entryPoints:
    - http
  routes:
    - kind: Rule
      match: Host(`loki.mydomain.com`)
      middlewares:
        - name: https-redirect
          namespace: traefik
      services:
        - name: loki
          port: 3100
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: loki-tls
spec:
  entryPoints:
    - https
  routes:
    - kind: Rule
      match: Host(`loki.mydomain.com`)
      services:
        - name: loki
          port: 3100
  tls:
    certResolver: default

Collect logs with Grafana Alloy

Caution

Promtail reached End-Of-Life on 2026-03-02. It receives no further development. Grafana Alloy is its successor and the recommended collector going forward. If you are migrating an existing Promtail config, the alloy convert --source-format=promtail command translates it for you.

On a managed h8lio cluster you operate within Kubernetes namespaces, not whole nodes, so log collection stays namespace-scoped: run a single Grafana Alloy Deployment that reads your pods’ logs through the Kubernetes API with the loki.source.kubernetes component and pushes them to Loki. A node-level DaemonSet is deliberately not used here: tailing node log files requires cluster-wide privileges (and scheduling a pod on every node of shared infrastructure) that a namespace tenant does not have on a managed cluster.

The recommended layout mirrors Prometheus: host Loki, Alloy, and Grafana in a dedicated monitoring namespace (an h8lio cluster such as acme-monitoring) and collect logs from your organization’s other namespaces (acme-prod, acme-staging, …). Alloy reaches each namespace through the API with a namespace-scoped Role granted in that namespace, so no cluster-wide access is ever needed. The simplest single-namespace case (collect only your own namespace) is shown first; the organization-wide variant follows.

RBAC: a namespace-scoped Role

Alloy needs read access to pods and their logs in your namespace only. A Role (not a ClusterRole) is sufficient, and it grants no node access.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: alloy-logs
  namespace: [NAMESPACE]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: alloy-logs
  namespace: [NAMESPACE]
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alloy-logs
  namespace: [NAMESPACE]
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: alloy-logs
subjects:
  - kind: ServiceAccount
    name: alloy-logs
    namespace: [NAMESPACE]

Organization-wide collection

To collect from your other namespaces, keep the ServiceAccount in the monitoring namespace (acme-monitoring) and apply the same Role plus a RoleBinding in each namespace you collect from (acme-prod, acme-staging, …). Each RoleBinding references the single ServiceAccount in acme-monitoring as its subject (just like the Prometheus scraping RBAC). A namespace tenant cannot use a ClusterRole, so cross-namespace access is granted one explicit RoleBinding at a time.

Alloy configuration

This pipeline discovers the pods in your namespace, tails their logs via the API, and forwards them to Loki. The field names below are the current discovery.kubernetes, loki.source.kubernetes, and loki.write component arguments; check those pages for the exhaustive schema.

discovery.kubernetes "pods" {
  role = "pod"
  namespaces {
    own_namespace = true
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

For the organization-wide layout, replace own_namespace = true with an explicit list of the namespaces you granted the Role in (Alloy only lists pods where its ServiceAccount is allowed):

discovery.kubernetes "pods" {
  role = "pod"
  namespaces {
    names = ["acme-prod", "acme-staging", "acme-monitoring"]
  }
}

Deploy Alloy

Install Alloy with its Helm chart, set the controller type to Deployment (not DaemonSet), and bind it to the ServiceAccount above. Pass the configuration through the chart’s alloy.configMap values (or mount it as a ConfigMap). A plain Kubernetes Deployment that runs the grafana/alloy image with the config mounted works just as well.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install alloy-logs grafana/alloy \
  -f alloy-values.yaml --namespace [NAMESPACE]

In alloy-values.yaml, set controller.type: deployment, controller.replicas: 1, and serviceAccount.create: false with serviceAccount.name: alloy-logs. See the chart docs for the current values schema rather than copying keys that may drift.

Once running, every pod in your namespace that writes to stdout/stderr appears in Loki automatically. There is nothing to add to your application’s Deployment.

The labels you get

The Kubernetes API source attaches a small, fixed set of labels. Everything else stays inside the log line, to be parsed at query time. Knowing this set is what makes the difference between a query that works and one that silently returns nothing.

Label	Value
namespace	the pod’s namespace
pod	the pod name, which changes at every rollout
container	the container inside the pod, useful when a pod runs more than one
service_name	populated by Loki from the pod’s identifying labels (on a Deployment labelled the conventional way, app.kubernetes.io/name)
detected_level	a Loki heuristic applied to the text of the line, not a value your application emitted

Two practical consequences.

Query by service_name, not by pod. A pod name carries a replica hash and changes at every deployment, so a dashboard built on it stops matching the moment you ship. service_name is the stable handle for “this service, across rollouts”. Check the value you actually get before hardcoding it, since Loki considers several conventional labels to derive it.

Treat detected_level with caution. It guesses the severity from the text of the line. Measured on a live namespace, it classified the overwhelming majority of lines as unknown and labelled none as error, while hundreds of genuine ERROR lines were present in the same window. If your application emits structured JSON (see below), filter on the field it actually produces, | json | log_level="ERROR", rather than on the heuristic. An alert built on detected_level can stay silent through a real incident.

Alternative: a sidecar for file-based logs

If an application genuinely cannot log to stdout/stderr (for example a legacy app that only writes rotating log files), add a per-pod Grafana Alloy sidecar that tails a shared emptyDir volume with the loki.source.file component, forwarding to the same loki.write endpoint. The sidecar must be Alloy, not Promtail, which is EOL. Prefer fixing the application to log to the console; the sidecar is a last resort.

Application logging best practice

Log to stdout/stderr (a console appender), ideally as structured JSON, so the Kubernetes API source picks logs up with no extra wiring and Loki/Grafana can parse fields directly. Do not write to a shared log file for a collector to tail; that pattern is obsolete.

For a Spring Boot service (3.4 and later), structured JSON requires no extra dependency and no logback-spring.xml. A single property switches the console appender to a JSON format:

logging:
  structured:
    format:
      console: ecs

ecs (Elastic Common Schema) is a good default; gelf and logstash ship alongside it, and a custom format can be registered. See Structured Logging in the Spring Boot reference for the full set of options.

Two things follow from this being a plain property rather than an XML file baked into the image. It can be steered per environment from a ConfigMap or an env var, with no rebuild. And every MDC entry is emitted as a top-level JSON field, so a correlation id placed in the MDC becomes directly queryable in Loki (| json | correlation_id="..."), with no parsing rule on the collector side.

Older guides recommend adding logstash-logback-encoder to the classpath and writing a logback-spring.xml. That still works, but on a current Spring Boot it buys a dependency and an XML file for something the framework now does natively.

If you do not need JSON, the Spring Boot default console pattern already writes to stdout and is collected as-is. In either case, no logging.file.name and no emptyDir volume are required.

Grafana

h8lio provisions a shared Grafana at https://monitoring.h8l.io with read-only dashboards scoped to your organization, but it does not expose your Loki logs. To explore your logs and build your own dashboards, run your own Grafana in the monitoring namespace (the same place that hosts Prometheus for your metrics) and add Loki as a datasource.

DataSource

Configure a Loki datasource on http://loki:3100 (if Grafana runs in the same namespace as Loki). To wire an external Grafana instance, use the public route you exposed above. This step is unchanged from earlier Loki setups.

Dashboard

Here is a link to a Loki Dashboard you can import into your Grafana instance. You can also import a dashboard from the Grafana Marketplace.

It is built on the labels listed above (namespace and service_name), so it works with the Alloy Deployment described on this page without any editing. Its severity panels read | json | log_level, which assumes the structured JSON logging recommended above. If your application logs plain text, those panels stay empty: replace log_level with a line filter of your own (|~ "(?i)error"), or with detected_level while keeping its limits in mind.

Explore the logs

Once the Alloy Deployment is running in your namespace and your services are producing logs, open your Grafana instance, go to Explore, select the Loki datasource, and query with a label selector such as {namespace="acme-prod", service_name="checkout"}. Logs appear within a few seconds of being emitted.

From there, refine inside the line rather than with more labels: | json exposes the fields of a structured log, so {namespace="acme-prod"} | json | log_level="ERROR" gives every error of the namespace whatever the service, and | json | correlation_id="abc123" follows one request across services.

Note

You can build LogQL-based alerts on critical log levels across services and be notified through your usual Grafana alerting channels.