Phase 5: Maintenance, Zero-Downtime Migrations, Runbooks, and Error Triage

Over 70% of total software lifecycle costs occur during the Maintenance Phase. Sustaining high reliability requires actionable Operational Runbooks, the Expand/Contract Pattern for database schema migrations, and proactive error tracking with Sentry.

⚡ Quick Dive

The Expand/Contract Migration Pattern

Phase 1 (Expand): Add new column/table in DB; write to BOTH old and new columns.
Phase 2 (Backfill): Run background worker to copy legacy data to new column.
Phase 3 (Switch Read): Point application queries to read exclusively from new column.
Phase 4 (Contract): Remove write to old column and safely drop legacy column.

[!IMPORTANT] Zero-Downtime Golden Rule: Never rename a database column in a single migration script! Always use the 4-phase Expand/Contract pattern.

📖 Extended Guide

1. Operational Runbook Template (docs/runbooks/high-error-rate.md)

# Runbook: High API 5xx Error Rate
* **Alert Trigger**: `HTTP_5xx_Rate > 1% for 5 minutes`
* **Severity**: SEV-1 / SEV-2
* **On-Call Role**: Primary Backend Engineer

## 1. Quick Diagnostics
```bash
# Check top error logs in Kubernetes
kubectl logs -l app=api-gateway --tail=100 | grep "500"

# Check database connection pool saturation
pg_stat_activity query to verify active connections

2. Mitigation Steps

  1. If errors started after recent deployment: Trigger Rollback via ArgoCD.
  2. If database CPU is 100%: Enable Read Replica Routing or disable non-critical background jobs.

3. Escalation

  • Escalate to Database SRE if RDS failover is required.
⌂ All Guides