Displaying I/O Statistics

The initial report detail lines generated by iostat provide statistics encompassing the time since the system was last booted. Subsequent sets of detail lines cover the time since the previous report interval.

Each set of report lines starts with a header row with CPU statistics which represents the CPU usage across all processors.  Following the CPU information, a device header row is displayed with subsequent detail lines of statistics for each device in the system.

The following example shows the invocation of iostat specifying a three second interval or delay with a total of five samplings or counts:

$ iostat 3 5

Linux 2.6.5-1.358 (Dell-Linux)  10/18/2004

For the average CPU report, %user, %nice, %iowait, and %idle are defined the same as they were in the mpstat command output.  One remaining piece of information is defined as:

* %sys: The percentage of processor utilization occurring at the system kernel level.

For the device utilization report:

* device: The device name as listed in the /dev directory is displayed.  These device names are mapped to mount points in the file /etc/fstab and are also listed in the output of the df command.

* tps: The number of transfers (I/O requests) per second issued to the device.

* blk_read/s: The number of blocks per second read from the device.

* blk_wrtn/s: The number of blocks per second written to the device.

* blk_read: The total number of blocks read.

* blk_wrtn: The total number of blocks written.

This information can assist in the determination of which devices are more heavily used than others and perhaps help with the determination of how to better distribute data to balance the workload.

Displaying Virtual Memory Statistics

The vmstat command displays information about processes, memory, paging, block IO, and different levels of CPU activity. As with iostat, the first detail lines produce report averages since the last reboot. Subsequent detail lines report information using the interval specified on the command line.

As with the other commands in this section, the vmstat command is driven by delay and count options that determine the time interval between report lines and the totals number of intervals to be reported.

$ vmstat 3 5

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0  63492  94856  24996    0    0     8     3  484    29  1  0 99  0
 0  0      0  63492  94856  24996    0    0     0     0 1005    25  1  0 99  0
 0  0      0  63492  94860  24996    0    0     0    13 1005    24  1  0 99  0
 0  0      0  63492  94860  24996    0    0     0     0 1002    21  0  0 99  0
 0  0      0  63492  94864  24996    0    0     0     4 1003    22  1  0 99  0

The Linux man page for vmstat defines the fields displayed as follows:

* procs

* r: The number of processes waiting for run time

* b: The number of processes in uninterruptible sleep, which means they are waiting on a resource

* memory

* swpd: Virtual memory used

* free: Idle memory

* buff: Amount of memory used as buffers

* cache: Current memory used as cache

* swap

* si: Memory swapped in per second from disk

* so: Memory swapped out per second to disk

* io

* bi: Blocks per second received from a block device

* bo: Blocks per second sent to a block device

* system

* in:.Number of interrupts per second, including the clock

* cs: Number of context switches per second

* cpu: These statistics are percentages of total CPU time:

* us: User time spent running non-kernel code, includes nice time

* sy: System time spent running kernel code

* id: Idle time

* wa: Wait time spent waiting for I/O

The vmstat information can be invaluable when studying resource utilization trends.  Here are a few examples of how vmstat output can be interpreted:

If over time the run queue value, procs-r, remains consistently higher than the number of processors in the server and CPU idle time is low, the system is CPU bound and can benefit from the addition of more and/or faster processors. Alternatively a high number displayed in the procs-b column also indicates a bottleneck, but one where processes are waiting on other resources.

If the virtual memory used (memory-swpd) remains high and the free memory (memory-free) remains low, then the system is memory constrained and will benefit from additional RAM.

Consistently high I/O rates paired with consistently low CPU utilization (cpu-us) indicates an I/O bound system that could benefit from a highly buffered disk array or possibly solid-state disk.