Red Hat Certified System Administrator(RHCSA)
Operate Running Systems
Locate and analyze system log files
In this guide, you will learn how to locate and analyze system log files on a Linux system. Logs are critical for understanding system events—such as user activity, errors, and system warnings—in server environments. Linux logs are stored as plain text files that record messages generated by the Linux kernel and various applications. The most common logging daemon, rsyslog (Rocket Fast System for Log Processing), organizes these messages in the /var/log directory.
Below is a simple command to list the files in the /var/log directory. Since these files are plain text, you can use utilities like grep or less to search through them. Note that many of these files require root privileges to access.
$ ls
For example, listing the contents of /var/log might display entries similar to this:
bash
$ ls /var/log/
anaconda dnf.rpm.log secure
audit firewalld secure-20211026
boot.log gdm secure-20211002
boot.log-20211026 glusterfs speech-dispatcher
boot.log-20211027 hawkey.log spooler
boot.log-20211028 hawkey.log-20211026 spooler-20211026
boot.log-20211101 hawkey.log-20211102 spooler-20211002
boot.log-20211102 kdump.log sssd
boot.log-20211104 lastlog swtpm
boot.log-20211108 libvirt tuned
If you need to work with restricted log files, you may have to log in as the root user. You can do this either by using the su command (and providing the root password) or with sudo --login:
bash
$ ls /var/log/
anaconda dnf.rpm.log secure-20211102
audit firewalld speech-dispatcher
boot.log gdm spooler
boot.log-20211026 glusterfs spooler-20211026
boot.log-20211027 hawkey.log spooler-20211102
boot.log-20211028 hawkey.log-20211102 spooler-20211002
boot.log-20211101 kdump.log sssd
boot.log-20211102 lastlog swtpm
boot.log-20211104 libvirt tuned
$ su
Note
Keep in mind that switching to the root user helps you access all log files, but also demands careful handling to avoid unintended modifications.
Analyzing Specific Log Files
Suppose you need to determine where SSH login events are recorded. SSH-related entries are typically found in the /var/log/secure file. You can search the entire /var/log directory for the keyword "ssh" as follows:
bash
$ grep -r 'ssh' /var/log/
/var/log/secure:Nov 15 14:47:28 LFCS-CentOS sshd[1021]: Server listening on :: port 22.
/var/log/secure:Nov 15 15:10:16 LFCS-CentOS sshd[1018]: Server listening on 0.0.0.0 port 22.
/var/log/secure:Nov 15 15:10:16 LFCS-CentOS sshd[1018]: Server listening on :: port 22.
/var/log/secure:Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on 0.0.0.0 port 22.
/var/log/secure:Nov 16 19:45:53 centos-vm sshd[1709]: Accepted password for aaron from 192.168.0.1 port 57626 ssh2
/var/log/boot.log-20211104:[ OK ] Reached target sshd-keygen.target.
/var/log/boot.log-20211108:[ OK ] Reached target sshd-keygen.target.
Since SSH logs are stored in /var/log/secure, you can view this file using a pager like less:
bash
$ less /var/log/secure
Inside the secure log, you will see details such as successful SSH logins, failed authentication attempts, use of sudo privileges, and password change events. A typical output might look like this:
bash
$ less /var/log/secure
Nov 16 17:48:31 LFCS-CentOS sshd[3380]: Accepted password for aaron from 192.168.0.3 port 63798 ssh2
Nov 16 17:49:16 LFCS-CentOS unix_chkpwd[3470]: password check failed for user (aaron)
Nov 16 17:36:09 LFCS-CentOS sudo[3113]: aaron : TTY=pts/0 ; PWD=/home/aaron ; USER=root ; COMMAND=/bin/bash
Nov 2 21:01:57 LFCS-CentOS sudo[6592]: aaron : TTY=pts/0 ; PWD=/home/aaron/Pictures ; USER=root ; COMMAND=/bin/killall less
Nov 16 17:56:44 LFCS-CentOS passwd[3581]: pam_unix(passwd:chauthtok): password changed for root
System events such as boot messages are generally stored in /var/log/messages. Additionally, you may come across rotated logs that include date information (for example, boot.log-20211026), which are archives of older log entries.
Below is an example showing how to view both secure and messages logs:
sh
$ less /var/log/secure
Nov 16 17:48:31 LFCS-CentOS sshd[3380]: Accepted password for aaron from 192.168.0.3 port 63798 ssh2
Nov 16 17:49:16 LFCS-CentOS unix_chkpwd[3470]: password check failed for user (aaron)
Nov 16 17:36:09 LFCS-CentOS sudo[3113]: aaron : TTY=pts/0 ; PWD=/home/aaron ; USER=root ; COMMAND=/bin/bash
Nov 2 21:01:57 LFCS-CentOS sudo[6592]: aaron : TTY=pts/0 ; PWD=/home/aaron/Pictures ; USER=root ; COMMAND=/bin/killall less
Nov 16 17:56:44 LFCS-CentOS passwd[3581]: pam_unix(passwd:chauthtok): password changed for root
$ less /var/log/messages
Nov 2 10:31:08 LFCS-CentOS systemd[1]: Starting dnf makecache...
Nov 2 10:31:09 LFCS-CentOS dnf[3572]: CentOS Stream 8 - AppStream 14 kB/s | 4.4 kB 00:00
Nov 2 10:31:10 LFCS-CentOS dnf[3572]: CentOS Stream 8 - BaseOS 3.3 kB/s | 3.9 kB 00:01
Nov 2 10:31:11 LFCS-CentOS dnf[3572]: CentOS Stream 8 - Extras 6.9 kB/s | 3.0 kB 00:00
Nov 2 10:31:11 LFCS-CentOS systemd[1]: dnf-makecache.service: Succeeded.
The rotated log file, such as boot.log-20211026, indicates that the logs in the current /var/log/boot.log were archived on October 26, 2021, preserving past log history.
Following Live Log Outputs
When troubleshooting an application or monitoring system activity, you might want to view log entries as they are recorded. The tail command with the -F flag enables follow mode, which displays new log entries in real time. To exit follow mode, simply press Control+C.
$ tail -F /var/log/secure
Nov 16 17:49:27 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168.0.3 port 63821 ssh2
Nov 16 17:49:28 LFCS-CentOS sshd[3468]: Connection reset by authenticating user aaron from 192.168.0.3 port 63821 [preauth]
Nov 16 17:49:28 LFCS-CentOS sshd[3468]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3 user=aaron
Nov 16 17:56:44 LFCS-CentOS passwd[3581]: pam_unix(passwd:chauthtok): password changed for root
Nov 16 17:56:44 LFCS-CentOS passwd[3581]: gkr-pam: couldn't update the login keyring password: no old password was entered
Nov 16 18:09:36 LFCS-CentOS gdm-password[3827]: gkr-pam: unlocked login keyring
Nov 16 18:21:11 LFCS-CentOS login[4116]: LOGIN ON tty1 BY aaron
Nov 16 18:21:16 LFCS-CentOS systemd[4249]: pam_unix(systemd-user:session): session opened for user gdm by (uid=0)
Using journalctl for Enhanced Log Analysis
Modern Linux systems use the systemd journal to manage structured logs. The journalctl command offers numerous powerful options for filtering and displaying logs.
Viewing Logs for a Specific Command
First, determine the full path of a command (e.g., sudo):
bash
$ which sudo
/bin/sudo
Then, view only the logs generated by that command:
bash
$ journalctl /bin/sudo
-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:31:22 CST --
Nov 16 17:36:09 LFCS-CentOS sudo[3113]: aaron : TTY=pts/0 ; PWD=/home/aaron ...
Nov 16 17:36:09 LFCS-CentOS sudo[3113]: pam_systemd(sudo-i:session): Cannot create...
Nov 16 17:36:09 LFCS-CentOS sudo[3113]: pam_unix(sudo-i:session): session opened
For viewing SSH daemon logs, run:
bash
$ journalctl -u sshd.service
-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:40:01 CST --
Nov 16 17:31:35 LFCS-CentOS systemd[1]: Starting OpenSSH server daemon...
Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on 0.0.0.0 port 22.
Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on :: port 22.
Nov 16 17:41:31 LFCS-CentOS sshd[1936]: Started OpenSSH server daemon.
Nov 16 17:48:31 LFCS-CentOS sshd[3388]: Accepted password for aaron from 192.168...
Nov 16 17:48:31 LFCS-CentOS sshd[3388]: pam_unix(sshd:session): session opened
Nov 16 17:49:28 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168...
Nov 16 17:49:33 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168...
Running journalctl without options displays all logs. During active troubleshooting, jump to the end of the log by using the -e flag:
bash
$ journalctl -e
Nov 16 18:39:05 LFCS-CentOS dbus-daemon[870]: [system] Successfully activated service...
Nov 16 18:39:05 LFCS-CentOS systemd[1]: Started Fingerprint Authentication Daemon.
Nov 16 18:39:08 LFCS-CentOS gdm-password[5133]: gkr-pam: unlocked login keyring...
Nov 16 18:39:08 LFCS-CentOS gnome-shell[2302]: Could not delete runtime/persistent state...
Nov 16 18:39:08 LFCS-CentOS NetworkManager[1015]: <info> [1637190548.889] agent...
Nov 16 18:40:01 LFCS-CentOS systemd[1]: fprintd.service: Succeeded.
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Job cron.weekly started
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Job cron.weekly terminated
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Normal exit (2 jobs run)
Similar to tail, journalctl supports follow mode for live log monitoring. Activate this mode with the -f flag and use Control+C to exit:
bash
$ journalctl -f
-- Logs begin at Tue 2021-11-16 17:31:32 CST. --
Nov 16 18:39:05 LFCS-CentOS systemd[1]: Starting Fingerprint Authentication Daemon...
Nov 16 18:39:05 LFCS-CentOS dbus-daemon[870]: [system] Successfully activated service 'net.reactivated.Fprint'
Nov 16 18:39:05 LFCS-CentOS systemd[1]: Started Fingerprint Authentication Daemon.
Nov 16 18:39:08 LFCS-CentOS gdm-password[5133]: gkr-pam: unlocked login keyring
Nov 16 18:39:08 LFCS-CentOS gnome-shell[2302]: Could not delete runtime/persistent state file: No such file or directory
Nov 16 18:39:08 LFCS-CentOS NetworkManager[1015]: <info>[1637109548.8989] agent-manager: agent registered
Nov 16 18:39:35 LFCS-CentOS systemd[1]: fprintd.service: Succeeded.
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Job 'cron.weekly' started
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Job 'cron.weekly' terminated
Nov 16 18:40:01 LFCS-CentOS anacron[3666]: Normal exit (2 jobs run)
Filtering Logs by Priority
Logs are tagged with priorities such as debug, info, notice, warning, error, crit, alert, and emerg. To view only the error-level messages, use the -p option like this:
$ journalctl -p err
-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:56:04 CST --
Nov 16 17:31:33 LFCS-CentOS kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to...
Nov 16 17:31:33 LFCS-CentOS kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to...
Nov 16 17:31:35 LFCS-CentOS alsactl[882]: alsa-lib main.c:1405:(snd_use_case_mgr)
Nov 16 17:31:42 LFCS-CentOS pulseaudio[1883]: module-rescue-stream is obsolete
To see all available priority codes, simply type:
$ journalctl -p
alert crit debug emerg err info notice warning
Additionally, you can filter logs similarly to grep using the -g flag. For example, to show info-level log messages beginning with the letter "B", use:
bash
$ journalctl -p info -g '^b'
-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 19:01:48 CST --
Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-provided physical RAM map:
Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fcff]
Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-e820: [mem 0x000000000009fd00-0x00000000000fffff]
Filtering by Time
You can restrict your log output to a specific time window using the -S (since) and -U (until) options. For example, to view logs recorded between 1 a.m. and 2 a.m.:
bash
$ journalctl -S 01:00 -U 02:00
Or, to display logs recorded after a particular date and time (e.g., November 16, 2021, at 12:04:55):
bash
$ journalctl -S '2021-11-16 12:04:55'
Viewing Logs from a Specific Boot
Often, you may want to see only the logs from the current boot session. Use the -b option with a boot offset, where 0 refers to the current boot:
bash
$ journalctl -b 0
-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 19:01:48 CST --
Nov 16 17:31:32 LFCS-CentOS kernel: Linux version 4.18.0-348.el8.x86_64 ...
Nov 16 17:31:32 LFCS-CentOS kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz
Nov 16 17:31:32 LFCS-CentOS kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x'
Nov 16 17:31:32 LFCS-CentOS kernel: x86/fpu: Supporting XSAVE feature 0x002: 'S'
To view logs from the previous boot, simply use -b -1. Note that on systems like CentOS the journal may be stored in memory by default, so persistent storage must be configured if you require logs from previous boots:
bash
$ journalctl -b -1
Specifying boot ID or boot offset has no effect, no persistent journal was found.
$ mkdir /var/log/journal/
Warning
Before configuring persistent journals, ensure you understand the disk space implications and security policies of your environment.
Viewing Login History
To review user login history, you can use the last command. This command displays recent session information, with the newest entries at the top, including system reboots. For example:
$ last
aaron tty2 tty2 Tue Nov 16 17:31 - still logged in
reboot system boot 4.18.0-348.el8.x Tue Nov 16 17:31 - still running
aaron tty2 tty2 Mon Nov 15 15:13 - down (00:01)
reboot system boot 4.18.0-348.el8.x Mon Nov 15 15:10 - 15:15 (00:05)
aaron tty2 tty2 Mon Nov 15 14:47 - down (00:22)
Alternatively, the lastlog command displays the most recent login for each user, including remote SSH login details:
$ lastlog
Username Port From Latest
setroubleshoot **Never logged in**
flatpak **Never logged in**
gdm tty1 Tue Nov 16 18:21:16 -0600 2021
clevis **Never logged in**
gnome-initial-setup **Never logged in**
tcpdump **Never logged in**
sshd **Never logged in**
aaron tty3 Tue Nov 16 18:21:11 -0600 2021
jane pts/1 192.168.0.3 Tue Nov 16 19:18:55 -0600 2021
Conclusion
This guide has explored multiple methods to locate and analyze system log files on Linux. Whether browsing the plain text files in /var/log using commands like grep and less or leveraging the powerful filtering and live monitoring features of journalctl, understanding your system logs is essential for effective troubleshooting and maintaining security.
Let's now proceed to some hands-on labs.
Watch Video
Watch video content
Practice Lab
Practice lab