Linux Filesystem Hierarchy: Server & Container Architecture

Understanding the Linux filesystem hierarchy (FHS 3.0), storage layers, and virtual filesystems is critical for designing reliable server infrastructure, container environments, and automated CI/CD pipelines. This guide provides an in-depth breakdown of standard directory topologies, mounting mechanics, filesystem drivers, and storage diagnostics.

⚡ Quick Dive

Filesystem Hierarchy Map

/ (Root Directory)
├── bin -> usr/bin          # Essential user binaries (symlinked on modern distros)
├── sbin -> usr/sbin        # Essential administrative binaries
├── boot/                   # Linux kernel images (vmlinuz), initramfs, and GRUB
├── dev/                    # Hardware device nodes managed by udev (e.g. /dev/sda)
├── etc/                    # System-wide configuration files
├── home/                   # Standard user home directories
├── root/                   # Root administrative user home directory
├── lib -> usr/lib          # Core shared libraries and kernel modules
├── media/ & mnt/           # Removable media & temporary manual mount points
├── opt/                    # Third-party add-on application packages
├── proc/                   # Virtual kernel & process information filesystem
├── sys/                    # Virtual sysfs hardware and kernel subsystem interface
├── run/                    # Ephemeral runtime volatile state (tmpfs, PID files)
├── srv/                    # Site-specific data served by this system
├── tmp/                    # Ephemeral temporary files (often tmpfs / auto-cleaned)
├── usr/                    # Read-only userland binaries, headers, and shared assets
└── var/                    # Variable persistent data (logs, databases, spools)

Server-Critical Directory Quick Reference

Essential Disk Diagnostics

# Check filesystem disk space and filesystem types
df -hT

# Check inode consumption (crucial for email/session servers)
df -ih

# Find the top 10 largest folders under /var
du -ah /var | sort -rh | head -n 10

# List block devices with UUIDs and mount points
lsblk -f

📖 Extended Guide

1. Filesystem Concepts: VFS, Inodes, and Superblocks

Linux abstracts diverse physical filesystems (ext4, XFS, Btrfs, NFS) behind the Virtual File System (VFS):

User Applications (read, write, open)
               │
               ▼
   Virtual File System (VFS)
  ┌────────────┬────────────┬────────────┐
  ▼            ▼            ▼            ▼
 ext4         XFS         Btrfs        tmpfs
  │            │            │            │
  └────────────┴─────┬──────┴────────────┘
                     ▼
            Block Device Layer
            (/dev/sda1, /dev/nvme0n1p1)

Core Components

• Superblock: Contains filesystem-wide metadata (total block count, block size, free blocks, mount count).
• Inode (Index Node): Stores metadata for an individual file (permissions, owner UID/GID, timestamps, file size, block pointers). It does NOT store the filename (the directory entry maps filename → inode).
• Data Blocks: Physical storage blocks holding the actual payload data.

2. Deep Dive into Server-Critical Directories

/etc — Host Configuration

• Plain-text configuration files controlling system daemon behaviors.
• Critical Files:
  • /etc/fstab: Filesystem mount tables.
  • /etc/hosts & /etc/resolv.conf: Name resolution and DNS servers.
  • /etc/sysctl.d/: Kernel runtime tuning parameters.
  • /etc/systemd/system/: Custom and overridden systemd unit definitions.

/var — Variable Runtime & Application State

• Contains data that dynamically grows during normal server operation:
  • /var/log/: Log destinations (journald, auth, application logs).
  • /var/lib/: State storage for databases (PostgreSQL /var/lib/postgresql, MySQL /var/lib/mysql) and container engines (/var/lib/docker, /var/lib/containerd).
  • /var/spool/: Queued jobs for cron (/var/spool/cron), mail, and print queues.

/run — Runtime Volatile Files (tmpfs)

• Mounted on a RAM-backed tmpfs. Replaces legacy /var/run.
• Automatically cleared on reboot.
• Hosts Unix domain sockets (e.g., /run/docker.sock, /run/php/php-fpm.sock) and PID lockfiles.

/proc and /sys — Virtual Kernel Filesystems

• /proc (procfs): Virtual window into the Linux kernel state and process table.
  • /proc/loadavg: System load averages (1, 5, 15 minutes).
  • /proc/meminfo: Detailed RAM/Swap memory allocation.
  • /proc/sys/net/ipv4/: Live network stack tunables.
  • /proc/<PID>/: In-depth state of process <PID> (cmdline, environ, fd, status).
• /sys (sysfs): Structured representation of hardware devices, block layers, and kernel drivers.

3. Storage Mounting & /etc/fstab Anatomy

The /etc/fstab file defines how storage partitions and remote shares mount at boot time.

Anatomy of /etc/fstab:

# <file system>                           <mount point>  <type>  <options>                  <dump> <pass>
UUID=3f9e8a71-1234-4b5c-89ab-0123456789ab /              ext4    noatime,errors=remount-ro  0      1
UUID=98765432-abcd-ef01-2345-6789abcdef01 /var/log       xfs     defaults,noexec,nosuid     0      2
tmpfs                                     /tmp           tmpfs   defaults,noexec,nosuid,nodev 0    0

Field Explanations:

1. File System Identifier: Always prefer UUID= or LABEL= over /dev/sdX because device letter assignments can shift across reboots.
2. Mount Point: Target directory where the filesystem attaches.
3. Type: Filesystem format (ext4, xfs, btrfs, nfs, tmpfs).
4. Security & Performance Options:
   • noatime: Disables access time updates on reads (significant I/O performance improvement for servers).
   • noexec: Disallows executing binaries on this filesystem (security hardening for /tmp or /var/log).
   • nosuid: Blocks SUID/SGID bits on this partition.
   • nodev: Disallows character/block device creation.
   • ro: Read-only mount.
5. Dump: Legacy backup flag (0 to disable).
6. Pass: fsck filesystem check order at boot (1 for root /, 2 for other partitions, 0 to skip).

4. Common Linux Filesystems

5. Disk Space & Inode Troubleshooting

Resolving "No space left on device" Errors

Often servers report disk full errors even when df -h shows available capacity. This is usually caused by:

1. Inode Exhaustion (df -i): Millions of tiny files (sessions, cache, queue files) consume all available inodes.

   # Identify directory consuming massive inode counts
   find /var -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n | tail -n 10
   

2. Deleted Files Held Open by Processes (lsof +L1): Deleting a large log file with rm does not free disk blocks if a running daemon still holds an open file descriptor.

   # Locate deleted files still held open in RAM
   lsof +L1
   
   # Truncate active log file safely instead of deleting
   truncate -s 0 /var/log/nginx/access.log
   

6. Distribution Layout Differences