Cloud Messaging: AWS SQS, SNS, and Google Cloud Pub/Sub

Cloud-managed messaging services eliminate the operational burden of hosting brokers while providing massive horizontal scale, multi-AZ redundancy, and serverless pay-per-request pricing. This guide covers AWS Simple Queue Service (SQS), Simple Notification Service (SNS), Fan-Out patterns, and Google Cloud Pub/Sub.

⚡ Quick Dive

Cloud Messaging Architectures Matrix

Service Architecture Ordering Replay Delivery Guarantee
AWS SQS Standard Distributed Queue Best-effort ❌ No At-least-once
AWS SQS FIFO Ordered Sharded Queue Strict FIFO ❌ No Exactly-once (5-min dedup)
AWS SNS Topic Pub/Sub Best-effort ❌ No Fan-out to SQS / Lambdas / HTTP
GCP Cloud Pub/Sub Topic & Subscription Optional FIFO ✅ 7-day replay At-least-once

The SNS + SQS Fan-Out Architecture

[ Checkout Service ] ── (Publish Event: 'OrderPlaced') ──► [ SNS Topic: orders ]
                                                                   │
                                ┌──────────────────────────────────┼──────────────────────────────────┐
                                ▼                                  ▼                                  ▼
                     [ SQS: Payment Queue ]             [ SQS: Inventory Queue ]           [ SQS: Shipping Queue ]
                                │                                  │                                  │
                                ▼                                  ▼                                  ▼
                       [ Payment Worker ]                 [ Inventory Worker ]               [ Shipping Worker ]

📖 Extended Guide

1. AWS SQS Visibility Timeout Mechanics

When a consumer worker pulls a message from an SQS queue, SQS does not delete it immediately. Instead, it hides the message from other workers for the Visibility Timeout (e.g. 30 seconds):

Worker pulls Msg 1 ──► Visibility Timeout Clock Starts (30s)
  ├── Worker succeeds: Sends DeleteMessage API call ──► SQS purges Msg 1 permanently.
  └── Worker crashes: Timeout expires (30s) ──► Msg 1 becomes visible again for another worker!

2. SQS FIFO: Message Group IDs and Deduplication

  • MessageGroupId: Defines strict FIFO ordering per entity (e.g. order-101). Different group IDs process concurrently across multiple workers without blocking each other.
  • MessageDeduplicationId: SQS drops duplicate messages received with the same deduplication hash within a 5-minute rolling window.
⌂ All Guides