We all know and love vmstat, but wouldn’t it be nice to get such information on a per process basis, to be able to better understand what is causing i/o problems? This is exactly what iopp, written by Mark Wong and released as open source does:
“It’s a custom tool to go through the Linux process table to get i/o statistics per process. It is open source and can be downloaded from: http://git.postgresql.org/?p=~markwkm/iopp.git;a=summary”
Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):
git clone git://git.postgresql.org/git/~markwkm/iopp.git cd iopp cmake CMakeLists.txt make
And install it:
make install DESTDIR=/usr [100%] Built target iopp Install the project... -- Install configuration: "" -- Installing /usr/bin/iopp -- Installing /usr/share/man/man8/iopp.8
after this iopp will be installed into /usr/bin.
Let’s see what are the program parameters:
iopp -h usage: iopp -h|--help usage: iopp [-ci] [-k|-m] [delay [count]] -c, --command display full command line -h, --help display help -i, --idle hides idle processes -k, --kilobytes display data in kilobytes -m, --megabytes display data in megabytes -u, --human-readable display data in kilo-, mega-, or giga-bytes
And finally let’s test it:
iopp -i -u 5 pid rchar wchar syscr syscw reads writes cwrites command 2464 0B 0B 0 0 0B 8192B 0B kjournald 3364 515K 0B 1336 0 0B 0B 0B mysqld 4664 0B 134B 0 4 0B 0B 0B apache2 4685 803K 114K 454 80 0B 0B 0B collectd 20758 82K 82K 329 329 0B 84K 0B cronolog 26236 67K 3754B 59 27 0B 0B 0B apache2 ...
The manual page explains each output field:
pid: The process id.
rchar: The number of bytes which this task has caused to be read from storage.
wchar: The number of bytes which this task has caused, or shall cause to be written to disk.
syscr: Count of the number of read I/O operations.
syscw: Count of the number of write I/O operations.
rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.
wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.
cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.
command: Filename of the executable.
Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.