Linux Service Orchestration: Systemd Units, Timers, and Journald

systemd is the central init system, process supervisor, and service manager across modern Linux distributions (PID 1). This guide covers authoring production .service unit files, scheduling automated workflows with .timer units (modern cron replacement), managing overrides via drop-ins, and querying structured logs with journalctl.

⚡ Quick Dive

Systemd Administration Cheat Sheet

Production Unit File Template (/etc/systemd/system/myapp.service)

[Unit]
Description=My Custom Backend API Service
Documentation=https://docs.example.com/api
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env.conf
ExecStartPre=/usr/bin/test -f /opt/myapp/config.yaml
ExecStart=/opt/myapp/bin/server --port=8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
KillMode=mixed
TimeoutStopSec=30s

# Security Hardening Directives
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

📖 Extended Guide

1. Systemd Architecture & Unit Anatomy

Systemd organizes managed objects into Units (.service, .timer, .socket, .target, .mount, .path).

Unit File Sections:

1. [Unit]: Metadata and dependency relationships.

   • Description=: Human-readable name.
   • After= / Before=: Determines ordering at boot (does NOT enforce dependency).
   • Requires=: Strict dependency (if the required unit fails, this unit fails).
   • Wants=: Soft dependency (attempts to start the target unit, but won't fail if target fails).

2. [Service]: Execution parameters and process lifecycle.

   • Type=:
     • simple (default): Executed binary is the main process.
     • exec: Process considered started only after binary is execve'd.
     • forking: Traditional daemons that call fork() into background (requires PIDFile=).
     • oneshot: Runs task to completion and exits (used with scripts/timers).
     • notify: Process signals systemd via sd_notify() when initialization is complete.
   • ExecStart=: Absolute path to binary and arguments.
   • Restart=: no, always, on-failure, on-abort.
   • RestartSec=: Time to wait before restarting (e.g., 5s).

3. [Install]: Installation configuration enabled via systemctl enable.

   • WantedBy=multi-user.target: Standard multi-user graphical/text boot runlevel.

2. Service Management Best Practices

Step-by-Step Service Creation Workflow

# 1. Write unit file
sudo nano /etc/systemd/system/worker.service

# 2. Tell systemd to re-index all unit files
sudo systemctl daemon-reload

# 3. Enable boot persistence and launch immediately
sudo systemctl enable --now worker.service

# 4. Check status and output
sudo systemctl status worker.service

Safe Customization via Drop-Ins (systemctl edit)

Never edit vendor-supplied unit files in /lib/systemd/system/ directly because package updates will overwrite your changes. Instead, use drop-in overrides:

# Opens safe override editor
sudo systemctl edit nginx

This creates /etc/systemd/system/nginx.service.d/override.conf:

[Service]
# Increase file descriptor limit for high-traffic web server
LimitNOFILE=65536

3. Systemd Timers (Modern cron Replacement)

Systemd timers provide superior capabilities over cron: dependency management, structured logging in journald, catch-up runs after reboot, and monotonic intervals.

A timer requires two paired files:

1. /etc/systemd/system/backup.service (defines what to execute)
2. /etc/systemd/system/backup.timer (defines when to execute)

1. The Service File (/etc/systemd/system/backup.service):

[Unit]
Description=Daily Database Backup Task

[Service]
Type=oneshot
User=postgres
ExecStart=/usr/local/bin/pg_backup.sh

2. The Timer File (/etc/systemd/system/backup.timer):

[Unit]
Description=Trigger Daily Database Backup

[Timer]
# Calendar event format: DayOfWeek Year-Month-Day Hour:Minute:Second
OnCalendar=*-*-* 03:00:00
# Randomize trigger within 10 minutes to avoid load spikes
RandomizedDelaySec=600
# Run immediately if server was offline during scheduled window
Persistent=true

[Install]
WantedBy=timers.target

# Enable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# View next execution time
systemctl list-timers backup.timer

4. Structured Logging with journalctl

journald captures stdout, stderr, syslog, and kernel events into indexed binary logs.

Practical Query Recipes

# 1. View logs from specific unit since last system boot
journalctl -u myapp.service -b

# 2. Stream logs live with high-precision timestamps
journalctl -u myapp.service -f -o short-full

# 3. Filter logs within an explicit time window
journalctl --since "2026-08-29 09:00:00" --until "2026-08-29 11:30:00"

# 4. View kernel logs only
journalctl -k -b

# 5. Filter by specific process PID
journalctl _PID=2450

Journal Log Retention Configuration (/etc/systemd/journald.conf)

Prevent journal logs from filling system disk partitions:

[Journal]
Storage=persistent
# Limit total journal log size to 2GB
SystemMaxUse=2G
# Rotate logs when individual file reaches 100MB
SystemMaxFileSize=100M
# Retain maximum 30 days of logs
MaxRetentionSec=30day

# Apply configuration
sudo systemctl restart systemd-journald

# Manually vacuum logs to free disk space immediately
sudo journalctl --vacuum-size=1G