Learn to locate and analyze system log files on Linux for understanding events like user activity, errors, and warnings in server environments.
In this guide, you will learn how to locate and analyze system log files on a Linux system. Logs are critical for understanding system events—such as user activity, errors, and system warnings—in server environments. Linux logs are stored as plain text files that record messages generated by the Linux kernel and various applications. The most common logging daemon, rsyslog (Rocket Fast System for Log Processing), organizes these messages in the /var/log directory.Below is a simple command to list the files in the /var/log directory. Since these files are plain text, you can use utilities like grep or less to search through them. Note that many of these files require root privileges to access.
Copy
Ask AI
$ ls
For example, listing the contents of /var/log might display entries similar to this:
If you need to work with restricted log files, you may have to log in as the root user. You can do this either by using the su command (and providing the root password) or with sudo --login:
Suppose you need to determine where SSH login events are recorded. SSH-related entries are typically found in the /var/log/secure file. You can search the entire /var/log directory for the keyword “ssh” as follows:
Copy
Ask AI
bash$ grep -r 'ssh' /var/log//var/log/secure:Nov 15 14:47:28 LFCS-CentOS sshd[1021]: Server listening on :: port 22./var/log/secure:Nov 15 15:10:16 LFCS-CentOS sshd[1018]: Server listening on 0.0.0.0 port 22./var/log/secure:Nov 15 15:10:16 LFCS-CentOS sshd[1018]: Server listening on :: port 22./var/log/secure:Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on 0.0.0.0 port 22./var/log/secure:Nov 16 19:45:53 centos-vm sshd[1709]: Accepted password for aaron from 192.168.0.1 port 57626 ssh2/var/log/boot.log-20211104:[ OK ] Reached target sshd-keygen.target./var/log/boot.log-20211108:[ OK ] Reached target sshd-keygen.target.
Since SSH logs are stored in /var/log/secure, you can view this file using a pager like less:
Copy
Ask AI
bash$ less /var/log/secure
Inside the secure log, you will see details such as successful SSH logins, failed authentication attempts, use of sudo privileges, and password change events. A typical output might look like this:
Copy
Ask AI
bash$ less /var/log/secureNov 16 17:48:31 LFCS-CentOS sshd[3380]: Accepted password for aaron from 192.168.0.3 port 63798 ssh2Nov 16 17:49:16 LFCS-CentOS unix_chkpwd[3470]: password check failed for user (aaron)Nov 16 17:36:09 LFCS-CentOS sudo[3113]: aaron : TTY=pts/0 ; PWD=/home/aaron ; USER=root ; COMMAND=/bin/bashNov 2 21:01:57 LFCS-CentOS sudo[6592]: aaron : TTY=pts/0 ; PWD=/home/aaron/Pictures ; USER=root ; COMMAND=/bin/killall lessNov 16 17:56:44 LFCS-CentOS passwd[3581]: pam_unix(passwd:chauthtok): password changed for root
System events such as boot messages are generally stored in /var/log/messages. Additionally, you may come across rotated logs that include date information (for example, boot.log-20211026), which are archives of older log entries.Below is an example showing how to view both secure and messages logs:
The rotated log file, such as boot.log-20211026, indicates that the logs in the current /var/log/boot.log were archived on October 26, 2021, preserving past log history.
When troubleshooting an application or monitoring system activity, you might want to view log entries as they are recorded. The tail command with the -F flag enables follow mode, which displays new log entries in real time. To exit follow mode, simply press Control+C.
Copy
Ask AI
$ tail -F /var/log/secureNov 16 17:49:27 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168.0.3 port 63821 ssh2Nov 16 17:49:28 LFCS-CentOS sshd[3468]: Connection reset by authenticating user aaron from 192.168.0.3 port 63821 [preauth]Nov 16 17:49:28 LFCS-CentOS sshd[3468]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3 user=aaronNov 16 17:56:44 LFCS-CentOS passwd[3581]: pam_unix(passwd:chauthtok): password changed for rootNov 16 17:56:44 LFCS-CentOS passwd[3581]: gkr-pam: couldn't update the login keyring password: no old password was enteredNov 16 18:09:36 LFCS-CentOS gdm-password[3827]: gkr-pam: unlocked login keyringNov 16 18:21:11 LFCS-CentOS login[4116]: LOGIN ON tty1 BY aaronNov 16 18:21:16 LFCS-CentOS systemd[4249]: pam_unix(systemd-user:session): session opened for user gdm by (uid=0)
Modern Linux systems use the systemd journal to manage structured logs. The journalctl command offers numerous powerful options for filtering and displaying logs.
First, determine the full path of a command (e.g., sudo):
Copy
Ask AI
bash$ which sudo/bin/sudo
Then, view only the logs generated by that command:
Copy
Ask AI
bash$ journalctl /bin/sudo-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:31:22 CST --Nov 16 17:36:09 LFCS-CentOS sudo[3113]: aaron : TTY=pts/0 ; PWD=/home/aaron ...Nov 16 17:36:09 LFCS-CentOS sudo[3113]: pam_systemd(sudo-i:session): Cannot create...Nov 16 17:36:09 LFCS-CentOS sudo[3113]: pam_unix(sudo-i:session): session opened
For viewing SSH daemon logs, run:
Copy
Ask AI
bash$ journalctl -u sshd.service-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:40:01 CST --Nov 16 17:31:35 LFCS-CentOS systemd[1]: Starting OpenSSH server daemon...Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on 0.0.0.0 port 22.Nov 16 17:31:35 LFCS-CentOS sshd[1026]: Server listening on :: port 22.Nov 16 17:41:31 LFCS-CentOS sshd[1936]: Started OpenSSH server daemon.Nov 16 17:48:31 LFCS-CentOS sshd[3388]: Accepted password for aaron from 192.168...Nov 16 17:48:31 LFCS-CentOS sshd[3388]: pam_unix(sshd:session): session openedNov 16 17:49:28 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168...Nov 16 17:49:33 LFCS-CentOS sshd[3468]: Failed password for aaron from 192.168...
Running journalctl without options displays all logs. During active troubleshooting, jump to the end of the log by using the -e flag:
Logs are tagged with priorities such as debug, info, notice, warning, error, crit, alert, and emerg. To view only the error-level messages, use the -p option like this:
Copy
Ask AI
$ journalctl -p err-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 18:56:04 CST --Nov 16 17:31:33 LFCS-CentOS kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to...Nov 16 17:31:33 LFCS-CentOS kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to...Nov 16 17:31:35 LFCS-CentOS alsactl[882]: alsa-lib main.c:1405:(snd_use_case_mgr)Nov 16 17:31:42 LFCS-CentOS pulseaudio[1883]: module-rescue-stream is obsolete
To see all available priority codes, simply type:
Copy
Ask AI
$ journalctl -palert crit debug emerg err info notice warning
Additionally, you can filter logs similarly to grep using the -g flag. For example, to show info-level log messages beginning with the letter “B”, use:
Copy
Ask AI
bash$ journalctl -p info -g '^b'-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 19:01:48 CST --Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-provided physical RAM map:Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fcff]Nov 16 17:31:32 LFCS-CentOS kernel: BIOS-e820: [mem 0x000000000009fd00-0x00000000000fffff]
You can restrict your log output to a specific time window using the -S (since) and -U (until) options. For example, to view logs recorded between 1 a.m. and 2 a.m.:
Copy
Ask AI
bash$ journalctl -S 01:00 -U 02:00
Or, to display logs recorded after a particular date and time (e.g., November 16, 2021, at 12:04:55):
Often, you may want to see only the logs from the current boot session. Use the -b option with a boot offset, where 0 refers to the current boot:
Copy
Ask AI
bash$ journalctl -b 0-- Logs begin at Tue 2021-11-16 17:31:32 CST, end at Tue 2021-11-16 19:01:48 CST --Nov 16 17:31:32 LFCS-CentOS kernel: Linux version 4.18.0-348.el8.x86_64 ...Nov 16 17:31:32 LFCS-CentOS kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuzNov 16 17:31:32 LFCS-CentOS kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x'Nov 16 17:31:32 LFCS-CentOS kernel: x86/fpu: Supporting XSAVE feature 0x002: 'S'
To view logs from the previous boot, simply use -b -1. Note that on systems like CentOS the journal may be stored in memory by default, so persistent storage must be configured if you require logs from previous boots:
Copy
Ask AI
bash$ journalctl -b -1Specifying boot ID or boot offset has no effect, no persistent journal was found.$ mkdir /var/log/journal/
Before configuring persistent journals, ensure you understand the disk space implications and security policies of your environment.
To review user login history, you can use the last command. This command displays recent session information, with the newest entries at the top, including system reboots. For example:
Copy
Ask AI
$ lastaaron tty2 tty2 Tue Nov 16 17:31 - still logged inreboot system boot 4.18.0-348.el8.x Tue Nov 16 17:31 - still runningaaron tty2 tty2 Mon Nov 15 15:13 - down (00:01)reboot system boot 4.18.0-348.el8.x Mon Nov 15 15:10 - 15:15 (00:05)aaron tty2 tty2 Mon Nov 15 14:47 - down (00:22)
Alternatively, the lastlog command displays the most recent login for each user, including remote SSH login details:
Copy
Ask AI
$ lastlogUsername Port From Latestsetroubleshoot **Never logged in**flatpak **Never logged in**gdm tty1 Tue Nov 16 18:21:16 -0600 2021clevis **Never logged in**gnome-initial-setup **Never logged in**tcpdump **Never logged in**sshd **Never logged in**aaron tty3 Tue Nov 16 18:21:11 -0600 2021jane pts/1 192.168.0.3 Tue Nov 16 19:18:55 -0600 2021
This guide has explored multiple methods to locate and analyze system log files on Linux. Whether browsing the plain text files in /var/log using commands like grep and less or leveraging the powerful filtering and live monitoring features of journalctl, understanding your system logs is essential for effective troubleshooting and maintaining security.Let’s now proceed to some hands-on labs.
