sodcalculator

10 Tech How-To

Linux System Logs: Check & Troubleshoot Like a Pro

Guide 10-55991 · Updated · Step by step

To troubleshoot Linux issues like a pro, you check system logs using commands like journalctl, dmesg, and tail on log files in /var/log. These logs record kernel messages, service events, authentication attempts, and application errors, giving you a chronological timeline of what happened before a problem occurred.

Why Linux Logs Matter

Linux generates logs for nearly everything: the kernel, systemd services, cron jobs, package managers, and individual applications. When something breaks—a service won’t start, the system freezes, or a user can’t log in—the logs are often the only record of the underlying cause. Unlike a desktop error dialog, logs persist across reboots and include timestamps, process IDs, and severity levels. This makes them indispensable for diagnosing recurring issues, security incidents, and performance problems.

Modern Linux distributions use systemd and its journal, which centralizes logs from the kernel and services. Older systems and many daemons still write plain-text files under /var/log. Knowing both systems is essential because some tools (like dmesg) read kernel ring buffers, while others (like journalctl) query the binary journal.

Key Log Locations and What They Contain

Most logs live in /var/log. Here are the ones you’ll check most often:

  • /var/log/syslog or /var/log/messages – general system activity and application messages (Debian/Ubuntu vs. RHEL/CentOS).
  • /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL) – authentication attempts, sudo usage, and SSH logins.
  • /var/log/kern.log – kernel messages, hardware errors, and driver issues.
  • /var/log/dmesg – kernel ring buffer messages, especially useful for hardware detection at boot.
  • /var/log/boot.log – boot-time messages from systemd.
  • /var/log/nginx/access.log and error.log – web server logs (if nginx is installed).
  • /var/log/mysql/ or /var/log/postgresql/ – database server logs.

Some applications (like Docker or Jenkins) write to their own directories. Always check the application’s documentation for its log path.

Using journalctl: The systemd Journal

If your distribution uses systemd (most do), journalctl is your primary tool. It reads the binary journal, which includes all service output and kernel messages, and it persists across reboots if Storage=persistent is set in /etc/systemd/journald.conf.

Basic usage:

  • journalctl – show all logs (from oldest to newest).
  • journalctl -b – show logs from the current boot only.
  • journalctl -b -1 – show logs from the previous boot (useful after a crash).
  • journalctl -u nginx.service – show logs for a specific service.
  • journalctl -p err – show only error-level and higher messages.
  • journalctl --since "1 hour ago" – filter by time.
  • journalctl -f – follow new log entries in real time (like tail -f).
  • journalctl -k – show kernel messages only.

Combine options: journalctl -u ssh -b -p err shows SSH errors from the current boot.

Reading Plain-Text Logs with tail, grep, and less

For traditional log files, use standard text tools. The most common workflow:

  1. See the last 50 lines: tail -n 50 /var/log/syslog
  2. Follow new lines as they appear: tail -f /var/log/syslog
  3. Search for a keyword: grep "error" /var/log/syslog
  4. Search case-insensitively: grep -i "fail" /var/log/auth.log
  5. View a large file page by page: less /var/log/kern.log (press q to quit, / to search).

Combine them: grep "error" /var/log/syslog | tail -n 20 shows the last 20 error lines. This is often the fastest way to find the most recent problem.

Kernel Messages with dmesg

dmesg prints the kernel ring buffer, which contains messages from the kernel itself—hardware detection, driver loads, disk errors, and OOM (out-of-memory) kills. It’s invaluable when diagnosing hardware issues or boot failures.

Common uses:

  • dmesg | tail -n 30 – see the most recent kernel messages.
  • dmesg | grep -i error – find kernel errors.
  • dmesg -T – show human-readable timestamps (instead of seconds since boot).
  • dmesg -w – watch new messages in real time (useful when plugging in a device).

Note: dmesg requires root privileges on many systems because kernel messages can reveal sensitive information.

Common Problems and How to Fix Them

Here are typical scenarios where log checking solves the issue:

  • Service won’t start: Run systemctl status myservice for a summary, then journalctl -u myservice -b for details. Look for lines like “Failed to start” or “Permission denied”.
  • Sudden reboot or freeze: Check journalctl -b -1 -p err for the previous boot. Look for kernel panics, hardware errors (e.g., mce), or OOM killer messages.
  • SSH login failures: Check /var/log/auth.log or /var/log/secure for “Failed password” or “Connection closed” lines. This can reveal brute-force attacks.
  • Disk full: Logs themselves can fill the disk. Use du -sh /var/log/* to find large files, then rotate or truncate them (with caution).
  • Application errors: For web apps, check both the system log and the app’s own log (e.g., /var/log/nginx/error.log).

A common mistake is ignoring timestamps. Always correlate log entries with the time of the incident. Also, remember that some logs are rotated (e.g., syslog.1, syslog.2.gz). Use zgrep to search compressed logs.

Log Rotation and Persistence

Logs can grow huge, so Linux uses logrotate to compress and rotate them. Configuration files live in /etc/logrotate.conf and /etc/logrotate.d/. By default, logs are rotated weekly or when they reach a certain size, with old versions compressed and eventually deleted.

For the systemd journal, persistence is controlled in /etc/systemd/journald.conf. If Storage=auto (the default), the journal is stored in memory (/run/log/journal) unless /var/log/journal exists. To make logs persistent across reboots, create that directory and restart systemd-journald:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

This ensures you can go back to previous boots with journalctl -b -1.

Frequently Asked Questions

How do I check logs from a previous boot?

Use journalctl -b -1 (or -2 for two boots ago). This works only if the journal is persistent. If it isn’t, you’ll need to have enabled persistence before the crash.

What’s the difference between /var/log/syslog and /var/log/messages?

They serve the same purpose—general system logs—but syslog is used on Debian/Ubuntu, while messages is used on RHEL/CentOS. The content is similar; just know which one your distribution uses.

Why can’t I read the journal without sudo?

By default, only root and users in the systemd-journal or adm group can read the journal. Add your user to the adm group (sudo usermod -aG adm $USER) and log out/in to gain read access.

How do I search for a specific error across all logs?

Use journalctl | grep "error" to search the entire journal, or grep -r "error" /var/log/ to search all text logs. For compressed logs, use zgrep instead of grep.

What does “OOM killer” mean in logs?

It means the kernel ran out of memory and killed a process to free RAM. Look for lines like Out of memory: Kill process in dmesg or the journal. This usually indicates a memory leak or insufficient RAM.

Can I clear logs to free up disk space?

Yes, but carefully. For the journal: sudo journalctl --vacuum-time=7d removes entries older than 7 days. For text logs, you can truncate them with sudo truncate -s 0 /var/log/syslog, but only after you’ve reviewed them. Never delete log files while services are running; use logrotate instead.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →