> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Working with the Journal > Guide to enabling persistent systemd journal storage and using journalctl filters and commands to inspect, follow, and manage system logs. Modern Linux distributions using systemd include a structured logging service called systemd-journald. The journal aggregates logs from multiple sources: the kernel, system services, applications, and the stdout/stderr streams of systemd-managed services. This guide shows how to enable persistent journal storage and how to use the `journalctl` command to query and follow logs effectively. What you'll learn * How to enable persistent on-disk journaling. * How to verify journal files and disk usage. * Common `journalctl` filters and usage patterns for troubleshooting. ## Understanding systemd-journald systemd-journald stores logs in a binary, indexed format that supports structured fields (e.g., `_PID=`, `_COMM=`, `_SYSTEMD_UNIT=`). You can query those fields with `journalctl` to produce focused, high-signal output for debugging and operational tasks. ## Enable persistent journal storage Many distributions default to in-memory journaling (volatile storage). To keep the journal across reboots, set `Storage=persistent` in `/etc/systemd/journald.conf`. Steps: 1. If an `/etc/systemd/journald.conf` doesn't already exist, copy the vendor-provided configuration: ```bash theme={null} root@control:~# cp /usr/lib/systemd/journald.conf /etc/systemd/journald.conf ``` 2. Edit the file and set `Storage=persistent`: ```bash theme={null} root@control:~# vim /etc/systemd/journald.conf # In the file, set: # Storage=persistent ``` 3. Reload systemd unit files and restart journald: ```bash theme={null} root@control:~# systemctl daemon-reload root@control:~# systemctl restart systemd-journald ``` Note: If your distribution already provides `/etc/systemd/journald.conf`, edit that file directly instead of copying from `/usr/lib/...`. ## Verify the journal directory and disk usage When `Storage=persistent` is enabled, journals are stored under `/var/log/journal`. Verify presence and usage: List `/var/log` to confirm a `journal` directory: ```bash theme={null} root@control:/var/log# ls anaconda chrony error.log maillog-20260415 rhsm spooler-20260422 audit cron firewalld maillog-20260422 samba sssd boot.log cron-20260415 gdm mail-no.log secure tuned boot.log-20260415 cron-20260422 info-nomail.log messages secure-20260422 user.log boot.log-20260420 cups insights-client messages-20260415 secure-20260422 warning.log boot.log-20260421 dnf.librepo.log journal messages-20260422 speech-dispatcher wtmp boot.log-20260422 dnf.log lastlog private spooler btmp dnf.rpm.log maillog qemu-ga spooler-20260415 ``` Check how much disk space the journal occupies: ```bash theme={null} root@control:/var/log# journalctl --disk-usage Archived and active journals take up 48M in the file system. ``` ## Common journalctl filters and examples `journalctl` supports a rich set of filters. Below is a quick reference of the most commonly used options, followed by examples. | Task | Command / Option | Description | | - | - | - | | Show entries from current boot | `journalctl -b` | Prints all entries for the current boot (use `-r` to reverse) | | Show entries for a unit | `journalctl -u sshd` | Filter by a systemd unit (service) | | Filter by priority | `journalctl -p err` | Show messages at `err` level and above (values: `emerg`, `alert`, `crit`, `err`, `warning`, `info`, `debug`) | | Show last N entries | `journalctl -n 10` | Display only the most recent 10 entries | | Follow logs in real time | `journalctl -f` | Like `tail -f`, updates as new entries arrive | Examples (kept in the original sequence for clarity): 1. Entries from the current boot: ```bash theme={null} root@control:/var/log# journalctl -b # (This prints all entries for the current boot; optionally add -r to reverse order) ``` 2. Entries for a specific unit (example: sshd): ```bash theme={null} root@control:/var/log# journalctl -u sshd # Shows all journal entries generated by the sshd.service unit. ``` 3. Filter by priority (show errors and worse): ```bash theme={null} root@control:/var/log# journalctl -p err # Equivalent to --priority=err; other values include emerg, alert, crit, warning, info, debug ``` 4. Show only the last N entries (example N=10): ```bash theme={null} root@control:/var/log# journalctl -n 10 Apr 23 14:33:59 control systemd-journald[4525]: Received SIGTERM from PID 1 (systemd). Apr 23 14:33:59 control systemd[1]: systemd-journald.service: Deactivated successfully. Apr 23 14:33:59 control systemd[1]: Stopped systemd-journald.service - Journal Service. Apr 23 14:33:59 control systemd[1]: Starting systemd-journald.service - Journal Service... Apr 23 14:33:59 control systemd-journald[4563]: Collecting audit messages is disabled. Apr 23 14:33:59 control systemd-journald[4563]: Journal started Apr 23 14:33:59 control systemd-journald[4563]: System Journal (/var/log/journal/500d9331afaf4b96957586d4650b73e2) is 48M, max 1.6G, 1.6G free. Apr 23 14:33:59 control systemd[1]: Started systemd-journald.service - Journal Service. Apr 23 14:34:01 control CROND[4569]: (root) CMD (echo "system job" >> /root/systemjob) Apr 23 14:34:01 control CROND[4568]: (root) CMDEND (echo "system job" >> /root/systemjob) ``` 5. Follow logs in real time: ```bash theme={null} root@control:/var/log# journalctl -f Apr 23 14:33:59 control systemd[1]: Stopped systemd-journald.service - Journal Service. Apr 23 14:33:59 control systemd[1]: Starting systemd-journald.service - Journal Service... Apr 23 14:33:59 control systemd-journald[4563]: Collecting audit messages is disabled. Apr 23 14:33:59 control systemd-journald[4563]: Journal started Apr 23 14:33:59 control systemd-journald[4563]: System Journal (/var/log/journal/500d9331afaf4b96957586d4650b73e2) is 48M, max 1.6G, 1.6G free. Apr 23 14:33:59 control systemd[1]: Started systemd-journald.service - Journal Service. Apr 23 14:34:01 control CROND[4569]: (root) CMD (echo "system job" >> /root/systemjob) Apr 23 14:36:01 control CROND[4649]: (root) CMD (echo "system job" >> /root/systemjob) # Press Ctrl-C to exit ``` ## Additional tips and configuration for long-term retention * Use structured-field matches: `journalctl _PID=1234` or `journalctl _COMM=sshd`. * Limit disk usage and retention by configuring `/etc/systemd/journald.conf`: * `SystemMaxUse=` — maximum disk space for all system journals. * `SystemKeepFree=` — amount of disk space to keep free for other uses. * `SystemMaxFileSize=` — maximum size for individual journal files. * For targeted troubleshooting of services, prefer: * `journalctl -u -n 100` (last 100 lines for a unit) * `journalctl -u -f` (follow unit logs in real time) For examinations or deeper troubleshooting, use [man journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html) to review all flags and structured-field matching. Familiarity with `journalctl` options is valuable for both ops and troubleshooting scenarios. ## Man page excerpt (reference) ```bash theme={null} root@control:/var/log# man journalctl # The manual describes usage and available options: # NAME # journalctl - Print log entries from the systemd journal # SYNOPSIS # journalctl [OPTIONS...] [MATCHES...] # DESCRIPTION # journalctl is used to print the log entries stored in the journal by systemd-journald.service(8) and # systemd-journal-remote.service(8). # # If called without parameters, it will show the contents of the journal accessible to the calling user, # starting with the oldest entry collected. # # Matches take the form "FIELD=VALUE", e.g. "_SYSTEMD_UNIT=httpd.service". See systemd.journal-fields(7) # for a list of well-known fields. ``` ## Links and references * [journalctl man page](https://www.freedesktop.org/software/systemd/man/journalctl.html) * [systemd.journal-fields(7) — structured journal fields](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) * systemd-journald configuration: `/etc/systemd/journald.conf` This concludes the lesson. You should now be able to enable persistent journaling and use `journalctl` effectively to inspect and follow system logs. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.