The beginning portion of the output shows metrics such as CPU free and I/O waits as you have seen from the mpstat command.
The next part of the output shows very important metrics for each of the disk devices on the system.
The 'tps' column show the number of transfers per second, i.e. number of I/O operations per second. Note: this is just the number of I/O operations; each operation could be huge or small.
The 'Blk_read/s' column shows the number of blocks read from this device per second. Blocks are usually of 512 bytes in size. This is a better value of the disk’s utilization.
The 'Blk_wrtn/s' column shows the number of blocks written to this device per second.
The 'Blk_read' column shows the number of blocks read from this device so far. Be careful; this is not what is happening right now. These many blocks have already been read from the device. It’s possible that nothing is being read now. Watch this for some time to see if there is a change.
The 'Blk_wrtn' column shows the number of blocks written to the device.
To get the metrics for a specific device, passing that device as a parameter:
To suppress the CPU related stats shown in the beginning of the output, use the -d option.
You can place optional parameters at the end to let iostat display the device stats in regular intervals. To get the stats for this device every 5 seconds for 10 times, issue the following:
iostat -d sdaj 5 10
You can display the stats in kilobytes instead of just bytes using the -k option:
iostat -k -d sdaj
While the above output can be helpful, there is lot of information not readily displayed. For instance, one of the key causes of disk issues is the disk service time, i.e. how fast the disk gets the data to the process that is asking for it. To get that level of metrics, we have to get the “extended” stats on the disk, using the -x option:
iostat -x sdaj
The 'rrqm/s' column shows the number of read requests merged per second. The disk requests are queued. Whenever possible, the kernel tries to merge several requests to one. This metric measures the merge requests for read transfers.
The 'wrqm/s' column is similar to reads, this is the number of write requests merged.
The 'r/s' column shows the number of read requests per second issued to this device.
The 'w/s' column shows the number of write requests per second.
The 'rsec/s' column shows the number of sectors read from this device per second.
The 'wsec/s' column shows the number of sectors written to the device per second.
The 'rkB/s' column shows the data read per second from this device, in kilobytes per second
The 'wkB/s' column shows the data written to this device, in kb/s.
The 'avgrq-sz' column shows the average size of the read requests, in sectors
The 'avgqu-sz' column shows the average length of the request queue for this device
The 'await' column shows the average elapsed time (in milliseconds) for the device for I/O requests. This is a sum of service time + waiting time in the queue.
The 'svctm' column shows the average service time (in milliseconds) of the device.
The '%util' column shows the bandwidth utilization of the device. If this is close to 100 percent, the device is saturated.
Remember, disks could be slow in getting the request from the processes. The amount of time the disk takes to get the data from it to the queue is called service time.
If you want to find out the disks with the highest service times, you issue:
iostat -x | sort -nrk13
This shows that the disk sdat has the highest service time (64.05 ms). Why is it so high? There could be many possibilities but three are most likely:
- The disk gets a lot of requests so the average service time is high.
- The disk is being utilized to the maximum possible bandwidth
- The disk is inherently slow.
Looking at the output we see that reads/sec and writes/sec are 0.00 (almost nothing is happening), so we can rule out #1. The utilization is also 0.00% (the last column), so we can rule out #2. That leaves #3. However, before we draw a conclusion that the disk is inherently slow, we need to observe that disk a little more closely. We can examine that disk alone every 5 seconds for 10 times:
iostat -x sdat 5 10
If the output shows the same average service time, read rate and utilization, we can conclude that #3 is the most likely factor. If they change, then we can get further clues to understand why the service time is high for this device.
Similarly, you can sort on the read rate column to display the disk under constant read rates. The information helps you to locate a disk that is “hot”—that is, subject to a lot of reads or writes. If the disk is indeed hot, you should identify the reason for that; perhaps a filesystem defined on the disk is subject to a lot of reading. If that is the case, you should consider striping the filesystem across many disks to distribute the load, minimizing the possibility that one specific disk will be hot.
You can also use iostat command which report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions. It can be use to find out your system's average CPU utilization since the last reboot.
You may want to use following command, which gives you three outputs every 5 seconds (as previous command gives information since the last reboot):
iostat -xtc 5 3
How to benchmark disk performance? To compare disk performance (perhaps an lvm device against a raw device, or an ext2 device against ext3 devices), we can:
- copy a large (huge) file
- run query that update a large number of rows, if we are running a DBMS on the system (or we are going to put a DBMS on the system)