Incident Management, Command Hierarchy, and Blameless Postmortems

When production outages occur, structured incident management protocols minimize Mean Time to Resolve (MTTR), maintain transparent stakeholder communication, and establish a blameless learning culture through root-cause analysis (RCA).

⚡ Quick Dive

Incident Severity Classification Matrix

Severity Impact Scope Response SLA Command Protocol
SEV-1 (Critical) Core product down; widespread data loss or revenue impact ⚡ < 5 minutes (24/7 Page) Dedicated Incident Commander, war room, executive updates
SEV-2 (Major) Major feature degraded; workarounds available < 15 minutes Active on-call triages, dedicated Slack channel
SEV-3 (Minor) Minor bug, non-critical admin tool impacted Next Business Day Jira ticket, standard sprint backlog triage

The Incident Command System (ICS) Roles

                      ┌────────────────────────────┐
                      │    Incident Commander (IC) │ (Leads response, delegates, sets priorities)
                      └─────────────┬──────────────┘
                                    │
         ┌──────────────────────────┼──────────────────────────┐
         ▼                          ▼                          ▼
┌──────────────────┐       ┌──────────────────┐       ┌──────────────────┐
│ Operations Lead  │       │ Communications   │       │ Scribe / Log     │
│ (Executes fixes) │       │ (Statuspage/Exec)│       │ (Records timeline│
└──────────────────┘       └──────────────────┘       └──────────────────┘

📖 Extended Guide

1. Blameless Postmortem Culture

[!IMPORTANT] Core SRE Premise: Human error is never the root cause. If an engineer executed a catastrophic command, the root cause is that the system allowed a single human action to cause widespread damage without guardrails.

The 5-Whys Root Cause Analysis (RCA):

  1. Why did the website go down? The database ran out of disk space.
  2. Why did disk space run out? Binary query logs filled the /var/lib/mysql partition.
  3. Why did binary logs fill the partition? Log retention was set to 30 days instead of 3 days.
  4. Why was it set to 30 days? A recent Terraform upgrade reverted the custom configuration.
  5. Why did Terraform revert it? The parameter was not covered by automated regression tests or drift detection.
⌂ All Guides