Apache Kafka: Distributed Event Streaming, Partitions, and Consumer Groups

Apache Kafka is a distributed event store and stream-processing platform capable of handling trillions of events a day with millisecond latencies. This guide covers Kafka broker internals, topics and partitions, segment files, zero-copy OS transfers, KRaft consensus, consumer group rebalancing, and exactly-once processing (EOS).

⚡ Quick Dive

Kafka Core Terminology & Architecture

Concept Definition & Function
Topic Logical category/feed to which records are published
Partition Ordered, immutable commit log sequence; unit of parallelism and sharding
Offset Sequential 64-bit integer ID assigned to each record in a partition
Producer Application publishing records; computes partition via key hash
Consumer Group Set of consumers coordinating to consume a topic (1 consumer per partition)
Leader & Followers Each partition has 1 Leader handling read/writes and $N-1$ Followers syncing via ISR (In-Sync Replicas)
KRaft Event-driven Raft metadata quorum (eliminates ZooKeeper dependency)

Producer Delivery Guarantees Matrix

acks Setting Latency Durability Guarantee Failure Risk
acks=0 ⚡ Lowest None (Fire-and-forget) High (Silent message loss on network error)
acks=1 Medium Leader acknowledged Low (Data lost if Leader crashes before replication)
acks=all / -1 High All In-Sync Replicas (ISR) acknowledged 🔒 Zero Data Loss Guarantee

📖 Extended Guide

1. Partition Architecture & Consumer Scaling

Topic: 'orders' (3 Partitions)
┌─────────────────────────────────────────────────────────────┐
│ Partition 0: [0][1][2][3][4] ──► [ Consumer 1 in Group A ]  │
│ Partition 1: [0][1][2][3]    ──► [ Consumer 2 in Group A ]  │
│ Partition 2: [0][1][2][3][4] ──► [ Consumer 3 in Group A ]  │
└─────────────────────────────────────────────────────────────┘
  • Parallelism Rule: Maximum concurrent active consumers in a consumer group equals the total number of partitions in the topic. Excess consumers remain idle as hot standbys.

2. High-Throughput Storage Engine: Zero-Copy and Sequential I/O

Kafka achieves millions of writes/sec per node through low-level Linux kernel optimizations:

  1. Append-Only Sequential Disk I/O: Sequential writes to rotational disks or NVMe SSDs are as fast as random memory writes.
  2. Page Cache & Zero-Copy (sendfile syscall): 
    Traditional read: Disk ──► OS Page Cache ──► App Memory ──► Socket Buffer ──► NIC Buffer
Kafka sendfile:  Disk ──► OS Page Cache ──────────────────────────────────► NIC Buffer
    Data transfers directly from kernel page cache to the Network Interface Card (NIC) with zero user-space context switches.

3. Idempotent Producers & Exactly-Once Semantics (EOS)

To prevent duplicate messages caused by network retry loops:

  • The broker assigns each producer a unique Producer ID (PID) and tracks incremental Sequence Numbers per partition.
  • If the producer retries an already committed message, the broker drops the duplicate while returning success.
⌂ All Guides