Linux Wait I/O what process is causing it ?

How to analyze Wait I/O problem ?

the first thing is to control what kernel version you have.

> 2.6.20 ?

You can use the IOTOP or LATENCYTOP utility that uses the IO_ACCOUNT capability of new kernel

< 2.6.20 ?

block_dump
block_dump enables block I/O debugging when set to a nonzero value.
When this flag is set, Linux reports all disk read and write operations that take place, and all block dirtyings done to files. The output of block_dump is written to the kernel output, and it can be retrieved using "dmesg".
When you use block_dump and your kernel logging level also includes kernel debugging messages, you probably want to turn off klogd, otherwise the output of block_dump will be logged, causing disk activity that is not normally there.

- First you must clear all log in "dmesg"

#dmesg -c 

- Stop any syslog you have for loggin only disk activity

 #/etc/init.d/syslog stop 

- Enable Block_Dump

# echo 1 > /proc/sys/vm/block_dump 

( also sysctl vm.block_dump for read the value and sysctl vm.block_dump=1 to enable the feature)

- Start the reporting: 

#while true; do sleep 1; dmesg -c; done | perl iodump (download iodump script )

- CTRL-C Stop the system from dumping these messages

- Disable Block_Dump after the analisys

#echo 0 > /proc/sys/vm/block_dump 

(also sysctl vm.block_dump for read the value and sysctl vm.block_dump=0 to disable the feature)

- Re-Start any syslog

# /etc/init.d/syslog start

If you prefer, you can save all the block_dump in a log file and then re-elaborate the data.

dmesg -c

/etc/init.d/syslog stop 

echo 1 > /proc/sys/vm/block_dump 

touch /tmp/disklog 

watch "dmesg -c >> /tmp/disklog" 

CTRL-C when you're done collecting data 

echo 0 > /proc/sys/vm/block_dump 

/etc/init.d/syslog start

cat /tmp/disklog | awk -F"[() \t]" '/(READ|WRITE|dirtied)/ {activity[$1]++} END {for (x in activity) print x, activity[x]}'| sort -nr -k2