Chaos Engineering, Fault Injection, and GameDay Simulations

Chaos Engineering is the discipline of experimenting on a software system in production or staging to build confidence in the system's capability to withstand turbulent conditions. This guide covers principles of chaos, blast-radius mitigation, automated fault injection using Chaos Mesh, and organizing company-wide GameDays.

⚡ Quick Dive

The 4 Phases of a Chaos Experiment

1. Formulate Hypothesis ──► "If Redis Primary crashes, Sentinel fails over within 3s with zero 500s."
2. Measure Steady State ──► Monitor baseline HTTP 200 rate (99.98%) and p99 latency (45ms).
3. Inject Fault ──────────► Simulate kill -9 on Redis Primary pod.
4. Verify & Learn ────────► Measure whether hypothesis held true; create RCA fix if violated.

Common Fault Injection Types

Fault Category Injection Method Failure Tested
Compute Fault Kill random pod / Terminate VM Self-healing controller & pod anti-affinity
Network Latency Add 500ms jitter with tc / eBPF Timeout settings & circuit breaker triggers
Packet Loss Drop 20% of TCP packets Retry storms & graceful degradation
Disk Exhaustion Fill partition with dd Log rotation & storage alerting
DNS Failure Block port 53 traffic DNS caching & fallback resolution

📖 Extended Guide

1. Kubernetes Chaos Mesh Manifest Example

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency-chaos
  namespace: production
spec:
  action: delay
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment-service
  delay:
    latency: '500ms'
    jitter: '100ms'
  duration: '5m'
⌂ All Guides