Eric Bergen

Recently I needed to explain the difference between %iowait from sar and %util from iostat. They both measure outstanding i/o requests but their method for measurement is slightly different. From the sar man page iowait is defined as:

iowait:
Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

Iostat has a slightly different definition of it’s disk metric which is %util (percent utilization).

%util:
Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

The definitions are very similar but there is a slight difference. %iowait uses the per cpu timings from /proc/stat. /proc/stat breaks down cpu time into four parts. The fields are defined as:

• user: normal processes executing in user mode
• nice: niced processes executing in user mode
• system: processes executing in kernel mode
• idle: twiddling thumbs
• iowait: waiting for I/O to complete
• irq: servicing interrupts
• softirq: servicing softirqs
• steal: involuntary wait

The only one important for this discussion is iowait. Inside the kernel iowait time is bascially time processes spend waiting for reqeusts that have been sent into the i/o scheduler.

iostat’s %util is measured using /proc/diskstats which keeps track of the number of miliseconds devices spent with outstanding i/o requests. iostat compares uptime of the server to the number of miliseconds spent doing i/o to determine how busy the disk is.

The main difference I can see between these two approaches is that sar will count the time a i/o request spent in the i/o scheduler where iostat won’t.

A few test runs show wildly different results from the two different tools. While running bonnie++, iostat, and sar in parallel some times iostat returns 100 %util and sar returns 25% iowait. During the read phase of bonnie++ iostat can return 100% where sar returns 1%. It’s very strange…

I’m still a bit unclear about all the differences. I’ll update this post when I learn more.

For those of you that don’t live in the culturally diverse bay area, pho is a Vietnamese soup consisting of rice noodles in a clear broth and various meats. Around here it’s typically eaten with over sized plastic chopsticks. The problem is that wet noodles are very difficult to pick up with plastic chopsticks. The Pho Hoa in Mountain View recently switched from plastic to wooden chopsticks. Hooray! Chopstick challenged patrons like myself can now safely eat pho without risk of a large blob of slippery noodles sliding off the sticks into the bowl blasting them with scalding hot broth.

Next they need to get rid of their $10 minimum Visa charge.