Cloud Observability: Prometheus, Grafana, and OpenTelemetry

Observability is the ability to infer the internal health and performance of distributed systems based on external telemetry outputs: Metrics, Logs, and Traces (M.E.L.T.). This guide covers Prometheus time-series scraping, PromQL queries, Alertmanager routing, Grafana dashboard architecture, and OpenTelemetry (OTel) distributed tracing.

⚡ Quick Dive

The 3 Pillars of Observability

Pillar What It Measures Ideal Tooling Primary Use Case
Metrics Numeric time-series values over intervals Prometheus, VictoriaMetrics, Datadog Real-time alerting & trend detection
Logs Discrete event text records with timestamps Loki, Fluentd, Vector, Elasticsearch Root-cause error debugging
Traces Request journey and latency across microservices Jaeger, Tempo, OpenTelemetry Identifying distributed latency bottlenecks

Essential PromQL Cheat Sheet

# 1. HTTP Request Rate per second over 5-minute window
sum(rate(http_requests_total[5m])) by (status_code)

# 2. HTTP Error Rate Percentage (5xx errors)
(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100

# 3. 99th Percentile Latency (p99) from Histogram
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 4. Memory Utilization Percentage per Node
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

📖 Extended Guide

1. Prometheus Monitoring Architecture

Prometheus uses a Pull-Based time-series data model:

┌──────────────────┐               ┌─────────────────────────────────┐
│ Target Services  │ ◄── Pull ──── │       Prometheus Server         │
│ (/metrics)       │   (Scrapes)   │ ┌──────────────┬──────────────┐ │
└──────────────────┘               │ │ TSDB Storage │ PromQL Engine│ │
                                   │ └──────┬───────┴──────┬───────┘ │
                                   └────────┼──────────────┼─────────┘
                                            │              │
                                            ▼              ▼
                                     [ Alertmanager ]  [ Grafana ]
                                            │
                                            ▼
                                     [ PagerDuty / Slack ]
  • Exporters: Standalone daemons exposing metrics in Prometheus text format (e.g. node-exporter for hardware, kube-state-metrics for Kubernetes objects).
  • TSDB (Time Series Database): Highly compressed block-based storage on disk.

2. The 4 Golden Signals of SRE / Monitoring

When designing production Grafana dashboards, structure alerts around the Google SRE Golden Signals:

  1. Latency: The time taken to service a request (differentiating successful requests from failed requests).
  2. Traffic: Demand placed on system (HTTP requests/sec, network bandwidth, concurrent transactions).
  3. Errors: The rate of requests failing explicitly (HTTP 500s) or implicitly (wrong content returned).
  4. Saturation: How full the resource is (CPU core usage, memory headroom, queue depths, file descriptor consumption).

3. OpenTelemetry (OTel) Distributed Tracing

OpenTelemetry provides a vendor-neutral standard for collecting and exporting telemetry data.

[ User Request ] ──► [ Frontend Service (Span A) ]
                           │ (Injects W3C Traceparent Header)
                           ▼
                     [ API Gateway (Span B) ]
                           │
                           ▼
                     [ Payment Service (Span C) ] ──► [ OTel Collector ] ──► [ Jaeger / Tempo ]
  • Trace: Represents the complete end-to-end journey of a single user request through a distributed microservices network.
  • Span: An individual unit of work within a trace (with start time, end time, and contextual attributes/tags).
⌂ All Guides