Signal Path: From metric to post-mortem

Monitoring is not observability

Monitoring answers questions you already knew you would ask: is the container up, did CPU cross 80%, does the endpoint respond? Observability lets you answer questions you never anticipated: why did only some /checkout requests take 8 seconds between 14:02 and 14:07?

The raw material is the data your system emits — telemetry — and the three classic signals are metrics, logs and traces. Each answers a different question, and the real value shows up when you connect them.

The four golden signals

Latency (how long requests take), traffic (how much work arrives), errors (how many fail) and saturation (how close to the limit you are). Those four numbers answer in ten seconds whether a service is healthy.

Two derived methods organise the work: RED (Rate, Errors, Duration) looks at the service from the user's side; USE (Utilization, Saturation, Errors) looks at the resource — CPU, memory, disk, database connections. A good dashboard uses both, in that order: impact first, cause second.

Percentiles, not averages

If 99 requests take 100 ms and one takes 10 s, the average lies and that user is gone. So you measure p50 (typical experience), p95 (the one used in agreements) and p99 (the long tail, where timeouts and retries live).

Three metric types cover almost everything

A counter only goes up and counts events: requests, errors, messages processed. A gauge goes up and down and measures instantaneous state: open connections, queue depth, memory. A histogram spreads observations across buckets and is what lets you compute latency percentiles server-side.

The minimum instrumentation for any HTTP service is two metrics: a request counter with route, method and status labels, and a duration histogram with the same labels.

Prometheus on Docker Compose

services:
  prometheus:
    image: prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports: ["9090:9090"]

The scrape config points at your service by internal network name: in Compose (and in a Coolify or Portainer stack) containers reach each other by service name, so targets: ['api:3000'] is enough. Never expose /metrics to the internet unprotected.

The three queries you will always use

# traffic (req/s)
sum(rate(http_requests_total[5m]))

# error rate (share of 5xx)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# p95 latency
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Order matters more than the panels

An operational dashboard reads top to bottom like a triage. Layer 1: impact — availability, error rate, latency, error budget. Layer 2: RED — traffic, errors and duration per route. Layer 3: saturation — CPU, memory, replicas, restarts, connection pool. Layers 4 and 5 (from module 6 on): logs and traces for the window you are looking at.

Grafana goes in the compose too

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports: ["3001:3000"]

The provisioning volume is what separates a toy from something operable: datasources and dashboards defined as files, versioned in git, reproducible on another VPS. A dashboard hand-built in the UI and never exported is lost the day the volume breaks.

Variables: one board, many services

Instead of cloning the dashboard per service, define a $service variable with label_values(http_requests_total, service) and use it in every query. The same board serves your API, your worker and the WhatsApp bot.

Three acronyms that are not synonyms

The SLI is what you measure (availability = 99.92%). The SLO is the internal target you set (99.9% over 30 days). The SLA is a contractual commitment with money or credits attached. Almost no service needs an SLA; almost every one should have an SLO.

The error budget

If the SLO is 99.9%, you are allowed to fail 0.1%. Over 30 days that is 43 minutes and 12 seconds of unavailability. That budget is a decision tool: while budget remains you can ship fast and take risks; once you have burned it, the priority becomes stabilising.

SLO 99.9%   → 43m 12s / 30 days
SLO 99.95%  → 21m 36s / 30 days
SLO 99.99%  →  4m 19s / 30 days

Burn rate: how fast you are spending

Burn rate is how much faster than allowed you are consuming the budget: error_ratio / (1 − SLO). A burn rate of 1× consumes the budget in exactly 30 days. 14.4× consumes it in a little over two days — that is worth waking someone. A sustained 3× is not urgent tonight, but it destroys the month.

Recording rules: compute once

groups:
  - name: slo
    interval: 30s
    rules:
      - record: job:http_error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m]))
      - record: job:http_availability:ratio30d
        expr: 1 - (
          sum(rate(http_requests_total{status=~"5.."}[30d]))
          / sum(rate(http_requests_total[30d])))

The 3 AM test

Before creating an alert, answer this: if it fires at 3 AM, does someone have to get up and act now? If the answer is no, it is not a page: it is a ticket, or plain noise. Alert fatigue is the fastest way to get a team to stop looking at their phone.

Multi-window burn-rate alerts

Google's standard recipe uses two windows per severity, to detect fast without firing on a thirty-second spike: the long window confirms the problem is real, the short one confirms it is still happening.

groups:
  - name: slo-alerts
    rules:
      - alert: ErrorBudgetFastBurn
        expr: |
          job:http_error_ratio:rate1h > (14.4 * 0.001)
          and job:http_error_ratio:rate5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Fast error-budget burn"
          runbook: "https://git.your-domain/runbooks/availability.md"

      - alert: ErrorBudgetSlowBurn
        expr: |
          job:http_error_ratio:rate6h > (6 * 0.001)
          and job:http_error_ratio:rate30m > (6 * 0.001)
        for: 15m
        labels: { severity: ticket }

Alertmanager decides who hears about it

Prometheus detects the condition; Alertmanager does the rest: it groups related alerts so you do not send twenty messages, deduplicates, silences during maintenance and inhibits child alerts once the parent has fired. Routing is by label: severity: page goes to the channel that wakes people, severity: ticket to the one read during business hours.

Every alert needs a runbook

The runbook annotation is not decoration: it is the link the on-call person opens half asleep. If the alert has no runbook, it is not finished.

A log is an event, not a sentence

Free-text log: Error processing user order. Structured log:

{"ts":"2026-09-08T14:03:11Z","level":"error","service":"api",
 "route":"/orders/:id","status":500,"duration_ms":831,
 "trace_id":"b2fe643bc4d341a1f7076f265910e649","order_id":"o_8812",
 "msg":"payment gateway timeout"}

The second can be filtered, grouped and counted. The minimum fields that must be there: timestamp, level, service, route, status, duration_ms and trace_id. That last field is what will connect this log to its trace in module 8.

Loki is not Elasticsearch

Loki does not index log content: it indexes only the stream labels and stores the rest compressed. That is why it is cheap to run on a VPS, and why labels must be few and low-cardinality: service, env, level. Nothing more.

Collection on Docker

  loki:
    image: grafana/loki:latest
    command: -config.file=/etc/loki/local-config.yaml
    volumes: [ "loki-data:/loki" ]

  alloy:
    image: grafana/alloy:latest
    volumes:
      - ./alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: run /etc/alloy/config.alloy

Alloy discovers containers through the Docker socket, applies labels from container names and labels, and pushes to Loki. A simpler alternative if your VPS is already tidy: the Docker logging driver pointing straight at Loki.

LogQL in three moves

# every error from the service
{service="api", level="error"}

# only 5xx, parsing the JSON
{service="api"} | json | status >= 500

# error rate per route, last 5 minutes
sum by (route) (
  rate({service="api"} | json | status >= 500 [5m]))

Trace, span and context

A trace is the full path of a request through your system. Each measurable step is a span: the inbound HTTP request, the Postgres query, the GET to the payments API, the job that got queued. Spans form a tree, and that tree shows immediately where the time went:

POST /orders                    2.41s
├── auth.verify                  38ms
├── db.query orders_insert      142ms
├── http POST payments.api      2.14s   ←
└── queue.publish order.created  21ms

The conclusion is not "the app is slow": it is "the payment gateway is eating 89% of the time". Those are two completely different conversations.

OpenTelemetry: instrument once

OTel is the open standard that keeps you from being locked to a vendor: you instrument with its SDK and then decide where the data goes — Tempo today, Datadog or Grafana Cloud tomorrow, without touching app code.

Auto-instrumentation covers the obvious parts (HTTP server, HTTP client, database driver, Redis) with very little code. Manual spans go where your expensive logic lives: the heavy computation, the LLM call, the PDF render.

The Collector as a buffer

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel/config.yaml"]
    volumes: [ "./otel/config.yaml:/etc/otel/config.yaml:ro" ]
    ports: ["4318:4318"]   # OTLP/HTTP

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo.yaml:ro
      - tempo-data:/var/tempo

The app speaks OTLP to the Collector, and the Collector decides what to do: drop health-check spans, sample, add environment attributes, and export to Tempo. Switching backend later means changing an exporter, not redeploying the application.

TraceQL to find the needle

{ resource.service.name = "api" && duration > 1s }
{ span.http.status_code >= 500 }
{ name = "http POST payments.api" && duration > 2s }

One common key: trace_id

The three signals you have built so far live in different stores. What connects them is an identifier that travels with the request: the trace_id. If your histogram attaches it as an exemplar, your log writes it as a field and your trace has it by definition, you can walk the whole path without writing a single query by hand.

Exemplars: from metric to trace

An exemplar is a concrete point attached to a histogram bucket, carrying the trace_id of a real request that landed there. On the latency chart it shows up as a small dot on the curve: you click it and the trace that produced that value opens.

# Prometheus needs the feature enabled
command:
  - --enable-feature=exemplar-storage

# and the scrape in OpenMetrics format
scrape_configs:
  - job_name: api
    static_configs: [{ targets: ["api:3000"] }]

On the app side, the Prometheus library must record the observation with the active trace's exemplar. In Node with prom-client it goes as the third argument to observe(); in Python, via exemplar=.

Derived fields: from log to trace

In the Loki datasource config you define a derived field that detects the trace_id in the JSON and turns it into a button that opens Tempo:

derivedFields:
  - name: TraceID
    matcherRegex: '"trace_id":"(\w+)"'
    url: '${__value.raw}'
    datasourceUid: tempo

And in the Tempo datasource you enable the reverse path (Trace to logs): from a span, jump to the logs for that same trace_id within the span's time window. The round trip is closed.

The full path

Burn-rate alert
      ↓  (runbook link)
Dashboard: p95 at 12s since 14:02
      ↓  (click the exemplar)
Trace: POST /orders — 11.8s in payments.api
      ↓  (trace to logs)
Log: "payment gateway timeout" order_id=o_8812
      ↓
Root cause found — 4 minutes since the alert

Four stages and three clocks

An incident is detected, acknowledged, mitigated and resolved — and each leg has its metric. MTTD (detect): first error to alert. MTTA (acknowledge): alert to someone picking it up. MTTR (restore): start to normal service. If you do not measure the three separately, you do not know which to improve: a 20-minute MTTD is not fixed by more on-call people, it is fixed by better alerts.

Severity: define it before you need it

SEV1  Service down or data loss.
      All users. People get woken. Comms immediately.
SEV2  Serious degradation: SLO at risk, important
      functionality broken. Handled now, in or out of hours.
SEV3  Limited impact or a workaround exists.
      Ticket, resolved during business hours.

The criterion is always user impact, never how much code needs touching. A CSS typo in checkout that blocks purchases is SEV1; a downed reporting worker may be SEV3.

Anatomy of a runbook that works

A runbook is for someone opening their phone at 3 AM, not for whoever wrote the system. Five sections:

1. Symptom — which alert fired and what the user is seeing.
2. Verification — the exact, copyable queries, with dashboard links.
3. Mitigation — concrete commands, in order, least to most destructive.
4. Escalation — who to call, and from which minute.
5. Post-incident — what to save before it is lost: trace IDs, screenshots, the time window.

Blameless post-mortems, for real

"Blameless" is not politeness: it is the only thing that makes people tell you what actually happened. If whoever pressed the button knows they will be named, next time the timeline will have holes exactly where the useful information was. The right question is never who made the mistake, but what made that action look reasonable at the time.

The six sections

1. Summary       — 3 lines: what happened, who was affected, how long.
2. Impact        — users, failed requests, error budget consumed,
                   money if applicable.
3. Timeline      — with exact times: first error, alert, ack,
                   mitigation, resolution. MTTD/MTTA/MTTR come from here.
4. Cause         — the technical one and the contributing one (why the
                   system allowed it to happen).
5. What worked   — yes, this section stays. The alert that fired correctly
                   and the runbook that helped are results too.
6. Action items  — each with an owner, a date and a ticket. Without an
                   owner it is not an action item, it is a comment.

GameDay: breaking things on purpose, in office hours

A GameDay is a planned incident: you pick a day, tell the team, inject real failures and check whether your observability catches them. It is the only honest way to know whether the stack works — because on the day of the real incident you do not want to discover the alert was never configured properly.

Hypothesis:  "If the payment provider starts returning 500s,
              the fast-burn alert fires within 5 minutes and
              the runbook leads us to the cause within 10."
Injection:   a proxy returning 500 on 30% of calls
Observation: did it fire? how fast? was the runbook enough?
Learning:    what was missing — a span, a panel, a runbook step

Three failures to start with

A slow or broken external dependency, resource saturation (CPU at the limit or connection pool exhausted), and losing a container (kill it and see whether the system recovers on its own). Those three alone will find at least one hole in your instrumentation.