We all know and love vmstat, but wouldn’t it be nice to get such information on a per process basis, to be able to better understand what is causing i/o problems? This is exactly what iopp, written by Mark Wong and released as open source does: “It’s a custom tool to go through the Linux process table to get i/o statistics per process. It is open source and can be downloaded from: http://git.postgresql.org/?p=~markwkm/iopp.git;a=summary”

Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):

1
2
3
4

git clone git://git.postgresql.org/git/~markwkm/iopp.git
cd iopp
cmake CMakeLists.txt
make

And install it:

1
2
3
4
5
6

make install DESTDIR=/usr
[100%] Built target iopp
Install the project...
-- Install configuration: ""
-- Installing /usr/bin/iopp
-- Installing /usr/share/man/man8/iopp.8

after this iopp will be installed into /usr/bin.

Let’s see what are the program parameters:

1
2
3
4
5
6
7
8
9

iopp -h
usage: iopp -h|--help
usage: iopp [-ci] [-k|-m] [delay [count]]
-c, --command display full command line
-h, --help display help
-i, --idle hides idle processes
-k, --kilobytes display data in kilobytes
-m, --megabytes display data in megabytes
-u, --human-readable display data in kilo-, mega-, or giga-bytes

And finally let’s test it:

1
2
3
4
5
6
7
8
9

iopp -i -u 5
pid rchar wchar    syscr    syscw reads writes cwrites command
2464    0B    0B        0        0    0B  8192B      0B kjournald
3364  515K    0B     1336        0    0B     0B      0B mysqld
4664    0B  134B        0        4    0B     0B      0B apache2
4685  803K  114K      454       80    0B     0B      0B collectd
20758   82K   82K      329      329    0B    84K      0B cronolog
26236   67K 3754B       59       27    0B     0B      0B apache2
...

The manual page explains each output field:

• pid: The process id.

• rchar: The number of bytes which this task has caused to be read from storage.

• wchar: The number of bytes which this task has caused, or shall cause to be written to disk.

• syscr: Count of the number of read I/O operations.

• syscw: Count of the number of write I/O operations.

• rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.

• wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.

• cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.

• command: Filename of the executable.

Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.