Linux Containers Under the Hood: Namespaces, Cgroups, and OverlayFS

Containers are not virtual machines; they are standard Linux processes isolated by kernel Namespaces and resource-governed by Control Groups (cgroups) running on top of union filesystems (OverlayFS). This guide explores the low-level kernel building blocks that power container engines like Docker, Podman, and containerd.

⚡ Quick Dive

Container Primitives Cheat Sheet

Primitive Kernel Mechanism What It Isolates / Controls CLI Diagnostic
PID Namespace CLONE_NEWPID Process IDs (container has its own PID 1) lsns -t pid
NET Namespace CLONE_NEWNET Network interfaces, routes, firewall rules ip netns list
MNT Namespace CLONE_NEWNS Filesystem mount points and root trees lsns -t mnt
UTS Namespace CLONE_NEWUTS Hostname and NIS domain name lsns -t uts
IPC Namespace CLONE_NEWIPC System V IPC, POSIX message queues lsns -t ipc
USER Namespace CLONE_NEWUSER Maps container root (UID 0) to unprivileged host UID lsns -t user
Cgroups (v2) /sys/fs/cgroup/ CPU, Memory, I/O, PIDs hard resource quotas systemd-cgls
OverlayFS overlay driver Multi-layered Copy-on-Write union filesystem mount | grep overlay

Creating a Container by Hand with Standard Linux Utilities

# 1. Download Alpine Linux rootfs tarball and extract
mkdir -p /tmp/mycontainer/rootfs
curl -sL https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.0-x86_64.tar.gz | tar -xz -C /tmp/mycontainer/rootfs

# 2. Launch an isolated shell with new PID, UTS, Mount, and Network namespaces
sudo unshare --pid --uts --mount --net --fork /bin/sh -c "
    hostname container-box
    mount -t proc proc /tmp/mycontainer/rootfs/proc
    chroot /tmp/mycontainer/rootfs /bin/sh
"

📖 Extended Guide

1. The 7 Linux Kernel Namespaces

Namespaces partition kernel resources so that one set of processes sees one set of resources while another set of processes sees a completely different set.

+-------------------------------------------------------------------------+
|                              Linux Kernel                               |
+------------------------------------+------------------------------------+
|        Host Namespace              |        Container Namespace         |
|  - Hostname: host.production       |  - Hostname: web-app-7d8           |
|  - IP: 192.168.1.100               |  - IP: 172.17.0.2                  |
|  - Process List: 1...2500          |  - Process List: 1, 2, 3           |
|  - Root FS: /                      |  - Root FS: /var/lib/docker/...    |
+------------------------------------+------------------------------------+

Inspecting Namespaces on a Live System

# List all active namespaces across all processes
sudo lsns

# Inspect namespaces attached to a specific process PID
ls -l /proc/<PID>/ns/

Entering Existing Namespaces with nsenter

Troubleshoot container issues directly without a shell running inside the container:

# Enter the network namespace of process <PID>
sudo nsenter -t <PID> -n ip a

# Enter all namespaces of a target container process
sudo nsenter -t <PID> --all /bin/bash

2. Control Groups (cgroups v2): Resource Limitation

While namespaces control what a process can see, Control Groups (cgroups) control how many resources a process can use.

In modern Linux (cgroups v2 unified hierarchy), limits are managed via the pseudo-filesystem under /sys/fs/cgroup/:

/sys/fs/cgroup/
├── cgroup.controllers
├── memory.max
├── cpu.max
└── system.slice / docker.slice

Testing Cgroup Limits Manually:

# 1. Create a new cgroup named 'sandbox'
sudo mkdir /sys/fs/cgroup/sandbox

# 2. Limit memory to 256MB
echo "268435456" | sudo tee /sys/fs/cgroup/sandbox/memory.max

# 3. Limit CPU to 50% of one core (50000 microseconds out of 100000 period)
echo "50000 100000" | sudo tee /sys/fs/cgroup/sandbox/cpu.max

# 4. Attach a process (e.g. active shell) to the cgroup
echo $$ | sudo tee /sys/fs/cgroup/sandbox/cgroup.procs

3. OverlayFS: Layered Copy-on-Write Storage

Container images use OverlayFS to stack multiple read-only image layers under a single top writable container layer:

Merged View (Container Mount Point: /mnt/merged)
  ▲
  ├── Upper Layer (Writable Layer: /tmp/overlay/upper)
  └── Lower Layers (Read-Only Base Image: /tmp/overlay/lower1, /tmp/overlay/lower2)

Mounting an OverlayFS Pipeline Manually:

mkdir -p /tmp/overlay/{lower,upper,work,merged}
echo "Base Image Content" > /tmp/overlay/lower/base.txt

# Mount the union filesystem
sudo mount -t overlay overlay \
  -o lowerdir=/tmp/overlay/lower,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work \
  /tmp/overlay/merged

# Inspect merged directory
ls -la /tmp/overlay/merged
# Any modifications made inside /tmp/overlay/merged write only to /tmp/overlay/upper

4. Rootless Containers & User Namespaces

In traditional Docker, the daemon runs as root. If a container escapes its boundaries, it gains root privileges on the host.

User Namespaces (CLONE_NEWUSER) map UID 0 (root inside container) to an unprivileged UID on the host (e.g., UID 100000 via /etc/subuid and /etc/subgid):

  • Inside Container: whoami -> root (UID 0)
  • On Host Kernel: Process runs strictly as unprivileged user 100000.

This architecture enables tools like Podman and rootless Docker to run securely without root daemon privileges.

⌂ All Guides