Skip to main content
In this lesson we’ll cover working with system logs — a core skill for Linux system administrators. Logs record what happened on a system: service start/stop events, user logins, application errors, failed authentications, kernel messages, and hardware events. When troubleshooting, administrators inspect logs first because they show the actual sequence of events. After completing this lesson you will be able to:
  • Locate important log entries quickly.
  • Filter logs for specific services, severities, or time ranges.
  • Use logs to diagnose and correlate system issues across services or hosts.
A dark-themed slide titled "Working With System Logs" with a lined note area on the left. On the right are three rounded boxes reading "Locate important log entries", "Filter logs for specific services or events", and "Use logs to diagnose system issues."
Overview: first we explain the system log architecture (how messages are produced, collected, and stored), then we look at rsyslog and classic /var/log files, and finally we cover the systemd journal and journalctl for querying journald data. We’ll finish with best practices for timestamps and retention.
Accurate timestamps are essential. If system time is incorrect or drifting, correlating events across logs or multiple systems becomes unreliable. Ensure NTP or chrony is configured so logs carry consistent timestamps across servers.
Think of logs like a building with sensors: doors opening, alarms triggering, cameras recording. Each sensor or camera generates events that are sent to a central monitoring room. On Linux, applications, services, and the kernel are the sensors; the logging subsystem collects their messages; and administrators review logs to reconstruct the sequence of events.
A dark-themed infographic showing a building with a door and icons for cameras and sensors labeled "Applications" and "Services." Surrounding it are circular icons for "Log Messages" and "Admins Review Logs," with the caption "Record Events Through Logs."
Log sources and flow
  • Sources: kernel, system services (daemons), user applications, and security systems (PAM, auditd).
  • Collection: On systemd-based systems, systemd-journald collects kernel messages, syslog, and service stdout/stderr into a binary journal.
  • Processing & storage: The journal can be queried directly with journalctl and may forward messages to a syslog daemon (e.g., rsyslog) which writes plain-text logs under /var/log for compatibility and archival.
Common operational flow:
  1. Applications / kernel produce messages.
  2. systemd-journald aggregates messages into the journal.
  3. journald can forward messages to rsyslog.
  4. rsyslog applies filters/formatting and writes to files in /var/log or forwards to remote log collectors.
A dark-themed diagram titled "System Log Architecture" showing Applications/Services/Kernel generating logs that go to systemd-journald (collects logs), which forwards them to rsyslog (processes logs) and then stores them under /var/log. Example log files listed include messages, secure, cron, and boot.log.
Severity and facilities Not all log messages require equal attention. Syslog defines severity (priority) levels so you can focus on what’s important and route or alert based on severity.
A dark-themed diagram titled "Syslog Priority Levels" listing severity levels 0–7 with their names (Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug) and short descriptions. It maps highest severity ("System is unusable") down to lowest ("Detailed debugging information").
Syslog priority levels (0 = highest severity to 7 = lowest): You can filter logs by priority (severity) when querying or when creating rsyslog/journald rules to route messages. Practical journalctl examples
  • Tail live journal: journalctl -f
  • Show messages for a service (e.g., sshd): journalctl -u sshd.service
  • Show messages since boot with error or higher: journalctl -b -p err..emerg
  • Time-range query: journalctl --since "2026-10-01" --until "2026-10-02" -u nginx.service
  • Output with ISO timestamps: journalctl --output=short-iso -u nginx.service --since today
Example rsyslog selector rules (in /etc/rsyslog.conf or /etc/rsyslog.d/*.conf):
Facilities identify the subsystem that generated a message and can be used to route messages to different files or backends.
A presentation slide titled "Common Syslog Facilities" showing a two-column table of syslog facility names and their purposes. Entries include items like auth/authpriv, cron, daemon, kern, mail, syslog and user with brief descriptions (e.g., authentication events, kernel messages, mail server activity).
Common syslog facilities Using facility + priority you can create granular rules, for example sending authpriv.* to /var/log/secure and mail.* to /var/log/maillog. Retention, persistence, and best practices
  • The systemd journal is a binary store queried with journalctl. By default some distributions write journal data to volatile storage (/run/log/journal), so ensure Storage=persistent in /etc/systemd/journald.conf if you need on-disk persistence.
  • For long-term retention, archival, compliance, or centralized analysis, forward logs from journald/rsyslog to a remote log server or log aggregation pipeline (e.g., ELK/EFK, Splunk, or a cloud log service).
  • Rotate and compress /var/log files with logrotate to prevent disk exhaustion.
  • Correlate events between systems using consistent time: synchronize clocks with NTP or chrony.
By default the journal may be stored in volatile memory. Verify journald storage settings (/etc/systemd/journald.conf) and configure persistent storage or forwarding to avoid losing logs after reboot. Also limit journal size and configure rotation to prevent disk space issues.
References and further reading Use the commands and configuration patterns above to locate, filter, and retain logs efficiently. Properly structured logging, consistent timestamps, and a clear retention strategy make troubleshooting and incident response far more effective.

Watch Video