Stop Grepping Logs: Build a Modern Observability Stack

TL;DR

Logs without context are noise. Metrics without alerts are decoration. Traces without correlation are useless.

• The Stack: Prometheus (Metrics), Grafana (Visualization), Loki (Log Aggregation), Tempo (Distributed Tracing), and Alertmanager (Intelligent Alerting).
• The Verdict: A properly built observability stack cuts your Mean Time To Detection (MTTD) from hours to minutes. It is the difference between finding the bug and the bug finding your customers.

The "It Works on My Machine" Incident

It's Monday morning. Users are reporting intermittent 500 errors. Your backend team checks the application logs. Nothing. The database team checks RDS metrics. Everything looks normal. The network team checks the load balancer. All healthy.

Three hours later, a junior engineer discovers that a single Kubernetes pod has been OOMKilled (Out of Memory) and restarting every 90 seconds. The pod logs were lost on each restart because nobody configured persistent log shipping.

If your team is debugging production incidents by SSHing into servers and running 'grep,' you are operating at the speed of 2010.

Three hours of combined engineering time wasted because the infrastructure couldn't answer a simple question: "What broke, when, and why?"

The Three Pillars of Observability

Monitoring tells you that something is wrong. Observability tells you why.

Modern observability is built on three pillars:

Most companies implement one or two of these pillars. The magic happens when you connect all three.

Deep Dive 1: Metrics with Prometheus

Prometheus scrapes numerical time-series data from your applications and infrastructure. It answers questions like: "What is the current CPU usage? How many HTTP 500 errors per second? What is the p99 latency?"

The critical insight most teams miss: you must instrument your application, not just your infrastructure.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-service
spec:
  selector:
    matchLabels:
      app: payment-service
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

Your application should expose a /metrics endpoint using a Prometheus client library. This gives you business-level metrics like:

• payment_transactions_total (How many payments processed?)
• payment_duration_seconds (How long does a payment take?)
• payment_failures_total (How many payments are failing?)

Infrastructure metrics tell you the server is on fire. Application metrics tell you the business is on fire.

Deep Dive 2: Logs with Loki (Not Elasticsearch)

For years, the default logging stack was the EFK stack (Elasticsearch, Fluentd, Kibana). It works, but Elasticsearch is a resource-hungry monster that requires dedicated engineers just to keep it alive.

Grafana Loki is the modern alternative. Instead of indexing the full text of every log line (like Elasticsearch), Loki only indexes the metadata labels (like namespace, pod, container) and keeps the log text compressed in S3. That makes it far cheaper to store and much simpler to run.

loki:
  auth_enabled: false
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  storage:
    type: s3
    bucketNames:
      chunks: loki-logs-production
      ruler: loki-logs-production
      admin: loki-logs-production
    s3:
      region: eu-west-1
  limits_config:
    retention_period: 30d
  # retention_period does nothing until the compactor is allowed to delete
  compactor:
    retention_enabled: true
    delete_request_store: s3

With Loki and a log agent deployed (Grafana Alloy is the current one), every log line from every pod in your Kubernetes cluster is shipped, labeled, and queryable from Grafana. When an alert fires, you click directly from the Prometheus metric to the exact log lines that occurred at the same timestamp. No more kubectl logs. No more guessing.

Deep Dive 3: The "Golden Signals" Alert Strategy

Most alerting configurations are broken because they alert on symptoms instead of signals. Getting an alert every time CPU crosses 70% is useless noise that trains your team to ignore alerts entirely.

Google's SRE handbook defines four Golden Signals that you should alert on:

1. Latency — How long requests take (alert on p99 > threshold)
2. Traffic — How many requests per second (alert on sudden drops)
3. Errors — The rate of failed requests (alert on error rate > 1%)
4. Saturation — How "full" a resource is (alert on memory > 90%)

groups:
  - name: golden-signals
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
          > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate exceeds 1% for 5 minutes"
 
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency exceeds 2 seconds"

With Golden Signals, your team only gets paged when users are actually impacted. No more alert fatigue. No more "the CPU alert fired again, just ignore it."

Run It Yourself: The Whole Stack in One Command

Reading about three pillars is one thing; clicking from a latency spike to the log line that caused it is another. I packaged a working version of this stack as linux-monitoring-stack on GitHub: nine containers (Nginx, a Flask app, Prometheus, Node Exporter, cAdvisor, Grafana, Alertmanager, Loki and Promtail) behind one command.

git clone https://github.com/MedArkidDev-wq/linux-monitoring-stack.git
cd linux-monitoring-stack
docker compose up -d
 
# Generate traffic, slow requests and errors to watch the dashboards move
for i in {1..50}; do curl -s http://localhost/ > /dev/null; done
for i in {1..10}; do curl -s http://localhost/slow > /dev/null; done
for i in {1..10}; do curl -s http://localhost/error > /dev/null; done

Grafana is on localhost:3000, Prometheus on localhost:9090 and Alertmanager on localhost:9093. Import community dashboards 1860 (Node Exporter), 13659 (Docker containers) and 12708 (Nginx) for instant host and container views.

Application metrics, not just host metrics

The sample app instruments every request with a counter, a histogram and a gauge. This is the part most teams skip:

REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "endpoint", "status"],
)
 
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    ["method", "endpoint"],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0],
)
 
ACTIVE_REQUESTS = Gauge("http_active_requests", "Currently active HTTP requests")
 
 
@app.after_request
def after_request(response):
    latency = time.time() - request.start_time
    ACTIVE_REQUESTS.dec()
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.path,
        status=response.status_code,
    ).inc()
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.path,
    ).observe(latency)
    return response

One thing to change before you copy this into a real service: the endpoint label uses request.path. That is fine for a demo with five routes, but on an API with paths like /users/1234 every user ID becomes a new time series (the cardinality problem described below). Use the route template instead, request.url_rule.rule in Flask, so /users/<id> stays one series.

Logs labeled by container, automatically

The repo uses Promtail, which Grafana has replaced with Grafana Alloy: Promtail reached end of life in March 2026. The labeling idea is identical, and alloy convert --source-format=promtail turns the config below into an Alloy config. For a new setup, start with Alloy.

Promtail discovers containers through the Docker socket and attaches low-cardinality labels, so in Grafana you filter by {service="flask-app"} instead of grepping:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ["__meta_docker_container_name"]
        regex: "/(.*)"
        target_label: "container"
      - source_labels: ["__meta_docker_container_log_stream"]
        target_label: "logstream"
      - source_labels:
          ["__meta_docker_container_label_com_docker_compose_service"]
        target_label: "service"

Practice an incident before a real one

./scripts/simulate_incident.sh kills the app container, waits for Prometheus to notice, checks that Alertmanager fired ContainerDown, restarts the container and writes a timestamped post-mortem to docs/. Running it once is a cheap way to find out whether your alerts actually reach anyone.

The Operational Reality (What Breaks)

Building observability is an ongoing investment, not a one-time project:

• Cardinality Explosion: If you add a user_id label to a Prometheus metric, you will create millions of unique time series and crash your Prometheus server. Labels must be low-cardinality (e.g., status_code, method, service).
• Log Volume Costs: If your application logs every HTTP request body in production, your Loki S3 bucket will cost thousands per month. Log judiciously—log errors in full detail, log successful requests as summaries.
• Dashboard Rot: Grafana dashboards that nobody looks at are technical debt. Every dashboard must be tied to a runbook. If there's no action to take, delete the dashboard.

The Payoff

When observability is done right, incident response transforms completely.

Instead of a 3-hour investigation involving 5 engineers, a single on-call engineer opens Grafana, sees the error rate spike on the golden signals dashboard, clicks through to the correlated Loki logs, identifies the failing microservice, and rolls it back using ArgoCD—all within 5 minutes.

That is the difference between a lost afternoon and a 5-minute blip.

If your debugging strategy is "SSH in and grep," start small: one alert that reaches a human before a customer does beats a dashboard nobody opens.

Frequently Asked Questions