The difference between %iowait from sar and %util from iostat
Recently I needed to explain the difference between %iowait from sar and %util from iostat. They both measure outstanding i/o requests but their method for measurement is slightly different. From the sar man page iowait is defined as:
iowait:
Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
Iostat has a slightly different definition of it’s disk metric which is %util (percent utilization).
%util:
Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.
The definitions are very similar but there is a slight difference. %iowait uses the per cpu timings from /proc/stat. /proc/stat breaks down cpu time into four parts. The fields are defined as:
- user: normal processes executing in user mode
- nice: niced processes executing in user mode
- system: processes executing in kernel mode
- idle: twiddling thumbs
- iowait: waiting for I/O to complete
- irq: servicing interrupts
- softirq: servicing softirqs
- steal: involuntary wait
The only one important for this discussion is iowait. Inside the kernel iowait time is bascially time processes spend waiting for reqeusts that have been sent into the i/o scheduler.
iostat’s %util is measured using /proc/diskstats which keeps track of the number of miliseconds devices spent with outstanding i/o requests. iostat compares uptime of the server to the number of miliseconds spent doing i/o to determine how busy the disk is.
The main difference I can see between these two approaches is that sar will count the time a i/o request spent in the i/o scheduler where iostat won’t.
A few test runs show wildly different results from the two different tools. While running bonnie++, iostat, and sar in parallel some times iostat returns 100 %util and sar returns 25% iowait. During the read phase of bonnie++ iostat can return 100% where sar returns 1%. It’s very strange…
I’m still a bit unclear about all the differences. I’ll update this post when I learn more.