Transport Layer Protocols: TCP, UDP, Congestion Control, and QUIC

Layer 4 (Transport) manages host-to-host communication, process multiplexing via ports, connection reliability, flow control, and network congestion mitigation. This guide covers the mechanics of the TCP 3-way handshake, 4-way teardown, state transitions, modern congestion control algorithms (BBR vs. Cubic), UDP, and next-generation QUIC.

⚡ Quick Dive

Protocol Comparison Matrix

Feature TCP (Transmission Control Protocol) UDP (User Datagram Protocol) QUIC (Quick UDP Internet Connections)
Connection Model Connection-Oriented (3-way handshake) Connectionless (Fire-and-forget) Connection-Oriented (0-RTT / 1-RTT Handshake)
Reliability Guaranteed (ACKs, Retransmissions) Unreliable (Best-effort delivery) Guaranteed (Packet-level encryption & ACKs)
Ordering Strict In-Order byte stream Out-of-order delivery possible Independent stream multiplexing (No HoL Blocking)
Flow & Congestion Full sliding window + BBR / Cubic None (Application managed) End-to-end BBR / Cubic congestion control
Header Overhead 20 to 60 bytes 8 bytes Variable encrypted payload over UDP
Common Protocols HTTP/1.1, HTTP/2, SSH, TLS, gRPC, BGP DNS, NTP, DHCP, VoIP, Video Streaming HTTP/3, Modern VPNs, Cloudflare Tunnel

📖 Extended Guide

1. TCP Lifecycle: 3-Way Handshake and 4-Way Teardown

TCP 3-Way Connection Handshake:
Client                                           Server
  │ ─── SYN (Seq=X) ───────────────────────────► │ (Server allocates TCB buffer: SYN_RCVD)
  │ ◄── SYN-ACK (Seq=Y, Ack=X+1) ─────────────── │
  │ ─── ACK (Seq=X+1, Ack=Y+1) ────────────────► │ (ESTABLISHED: Ready for data)

TCP 4-Way Connection Teardown:
Client                                           Server
  │ ─── FIN (Seq=U) ───────────────────────────► │ (State: CLOSE_WAIT)
  │ ◄── ACK (Ack=U+1) ────────────────────────── │ (State: FIN_WAIT_2)
  │ ◄── FIN (Seq=V) ─────────────────────────── │ (Server finished sending data)
  │ ─── ACK (Ack=V+1) ─────────────────────────► │ (Client enters TIME_WAIT for 2*MSL)

2. The TIME_WAIT State & Socket Exhaustion

When a client closes a TCP connection, the socket remains in TIME_WAIT for $2 \times \text{MSL}$ (Maximum Segment Lifetime $\approx 60\text{s}$):

  • Why it exists: Ensures the final ACK was received by the server, and prevents stale duplicate packets from an old connection interfering with a newly opened connection on the same port tuple.
  • High-Throughput Risk: High-traffic reverse proxies can exhaust all ephemeral client ports ($65,535$).
  • Remedy: Enable net.ipv4.tcp_tw_reuse = 1 in /etc/sysctl.conf to safely reuse sockets in TIME_WAIT for outgoing connections.

3. TCP Congestion Control: Cubic vs. Google BBR

Loss-Based Congestion Control (Cubic / Reno):
Increases throughput linearly ──► Packet Loss Occurs (Bufferbloat) ──► Slashes throughput by 50%
(Reactive: Oscillates wildly, causes latency spikes on lossy links)

Model-Based Congestion Control (Google BBR - Bottleneck Bandwidth & RTT):
Monitors real-time delivery rate & minimum round-trip time ──► Maximizes pipe without filling queues
(Proactive: Provides 2x-10x throughput on congested, transcontinental links)

Enabling BBR in Linux Kernel:

sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

4. QUIC: Eliminating Head-of-Line Blocking

In HTTP/2 over TCP, multiple multiplexed streams share a single TCP connection. If a single packet drops on the network, the TCP kernel buffer blocks all streams until the missing packet is retransmitted (Head-of-Line Blocking).

QUIC (HTTP/3 over UDP) runs in user-space on top of UDP:

  • Every stream has independent packet retransmission logic.
  • A dropped packet in Stream A does not stall Streams B or C.
  • Integrated TLS 1.3 encryption into the transport handshake (0-RTT connection resumption).
⌂ All Guides