Skip to main content
In this lesson, we explore how to monitor the performance of storage devices—similar to how we track CPU and RAM usage. Just as a CPU consistently running at 100% can slow down your system, overusing storage devices can also degrade performance. While tools such as TOP and HTOP are common for monitoring CPU and RAM usage, various utilities exist specifically for storage devices. In this guide, we will review several simple tools to help you manage and diagnose storage performance issues. There is a package called sysstat that bundles many system monitoring utilities—including iostat and pidstat, which we will focus on. Install sysstat with:
The installation process may display output similar to this:
iostat, short for input/output statistics, monitors data being written to or read from a storage device. In contrast, pidstat shows statistics for process IDs (PID) interacting with the storage subsystem. Every process in Linux has a unique identifier, and pidstat helps you understand which processes create heavy I/O loads.
The image is an infographic about storage monitoring, showing how the sysstat package, including iostat and pidstat, is used to gather I/O and process ID statistics from read/write operations.
Below, we begin by running iostat without any arguments. This command produces a summary of storage device usage since boot:
The above output represents historical usage measured since the system booted. Note that the displayed values represent averages over the system’s uptime, which might not reflect the current instantaneous load. For example, if disk reads vary over consecutive seconds, the average value shown by iostat adjusts accordingly. Fields such as TPS (transfers per second), kB_read/s, and kB_wrtn/s are computed from these running averages, while kB_read and kB_wrtn show cumulative totals.
The image explains how the iostat command calculates the average kilobytes per second read over three seconds, resulting in 100 kilobytes per second.
In practical terms, a process that writes 1 kB repeatedly in rapid succession can cause high TPS values even if the individual write size is small. Conversely, operations involving large data transfers are more apparent when you review the cumulative values in kB_read and kB_wrtn. For instance, consider the following iostat output from a different kernel version:
Even a seemingly small write (such as 1 kB) may be reported as a larger block, particularly when larger transfers occur. A storage device can experience stress in two ways:
  1. When a process initiates frequent read/write operations (producing a high TPS).
  2. When large volumes of data are transferred during each operation (visible in the kB_read/kB_wrn fields).
For example, if a device capable of 2 GB/s is nearly saturated by one process, other processes may experience significant delays. Both iostat and pidstat are essential in identifying and diagnosing these issues.
For quick diagnostics, consider running iostat with a delay parameter (e.g., 1 second) to refresh the output and capture current activity.

​
Creating a Storage Load Scenario

To illustrate these concepts, let’s create a process that continuously writes to a storage device. We use the dd command to write zeros from /dev/zero into a file. The parameters for the command mean the following:
  • if=/dev/zero: Uses an infinite stream of zeros.
  • of=DELETEME: Specifies the output file.
  • bs=1: Sets the block size to 1 byte.
  • count=1000000: Writes 1 byte a million times.
  • oflag=dsync: Forces the write operation to bypass caching.
  • &: Runs the command in the background.
When executed, you might see output like:
This background process allows you to continue using the terminal. To monitor which process is writing to the device, run iostat with a delay parameter:
The output refreshes every second. A sample refreshed output may look like:
After a while, you may notice a sudden increase in TPS and kB_wrn/s for devices like dm-0 and sda. For example:
This indicates that certain devices are under heavy write activity. However, iostat alone does not reveal which process is responsible.

​
Identifying the Process with pidstat

To determine the process causing high I/O, exit iostat by pressing Ctrl+C and use pidstat:
The output will list I/O statistics for each process, similar to:
In the above example, the dd process (PID 1411) is responsible for heavy write activity. To continuously monitor, run pidstat with an interval:
Press Ctrl+C to exit. A sample repeated output might look like:
Both dm-0 and /dev/sda show significant write activity in the iostat output, and the dd process (PID 1411) is the main contributor based on pidstat.

​
Understanding Device Mapper (dm-0)

You may wonder what the dm-0 device represents. DM stands for Device Mapper, which is used by the Logical Volume Manager (LVM) to manage storage volumes. To find out what dm-0 maps to, use:
Example output:
Next, verify the relationship between dm-0 and the physical devices with lsblk:
Sample lsblk output:
From this output, dm-0 (shown as ubuntu—vg-ubuntu—lv) is a logical volume created on /dev/sda3, which explains why both dm-0 and sda are active.

​
Managing Processes with High I/O

To summarize our findings:
  • The dd process (PID 1411) is writing continuously to the storage device.
  • These writes are occurring on /dev/sda, accessed through the logical volume dm-0.
  • In a real-world scenario, a process such as a database (e.g., MariaDB) might be responsible, warranting further investigation into logs or even a restart using:
To view the full command of a given process, use:
If you decide to stop the process, execute:
Verify the process termination with:
Example output:
If the process does not terminate gracefully, you may force termination with the SIGKILL signal:
Using SIGKILL (kill -9) stops a process immediately without allowing it to perform any cleanup. Use this option only as a last resort.

​
Additional Command-Line Options

Here are a few useful options for both iostat and pidstat:

​
iostat Options:

  • Use -d to display only device utilization (omitting CPU statistics).
  • Use -h for human-readable output, where large numbers are automatically scaled (e.g., kilobytes, megabytes, gigabytes).
Examples:

​
pidstat Options:

  • Use --human to display I/O statistics in a human-readable format.
Example:
Sample output:

​
Monitoring Disk Partitions with iostat

By default, iostat displays statistics for whole devices (e.g., /dev/sda). To view statistics for every partition, use:
Or, to focus on partitions for a specific device like sda:
For more detailed options, always refer to the manual pages:

​
Conclusion

This lesson demonstrated the techniques to monitor storage performance, diagnose device stress, and identify processes causing heavy I/O activity using tools like iostat and pidstat. By combining these tools with commands such as dd, dmsetup, and lsblk, you can efficiently troubleshoot and resolve storage-related performance issues. Now that you have learned these valuable monitoring techniques, let’s move on to our next lesson!

Watch Video