Skip to contentSkip to conversation

What causes iowait other than waiting for i/o?

DISCUSSION Bill Davidsen 4 participants 6 messages May 28, 2007 – May 29, 2007

lore
Share & tools
lore
#0 lore raw

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#1 lore raw
Quoted 7 lines · Bill Davidsen

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

Quoted 6 lines

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

Nothing /should/ cause time to be attributed to i/o wait other than waiting for i/o. But the accounting isn't completely accurate. When the kernel records what the has happen (governed by when the timer interrupt triggers as far as I remember), many different processes may have run and done may different things, but only what the last one did is recorded. So if you are unlucky you may have the system doing all sorts of activities, but only recording i/o wait if the last process to run before the kernel looked was doing i/o wait... The numbers are not exactely accurate...

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#2 lore raw
In reply to Bill Davidsen

Hi Bill,

Quoted 7 lines · Bill Davidsen

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

Quoted 5 lines

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

What tool / kernel instrumentation / mechanism are you using to determine that some task(s) are indeed blocked waiting for i/o? Perhaps some userspace process accounting tools could be "broken" in the sense that they generalize all uninterruptible sleep as waiting for i/o ...

Satyam

#3 lore raw
Quoted 12 lines · Satyam Sharma

Hi Bill,

Quoted 9 lines · Bill Davidsen

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

At this point I'm really curious, does someone have a good clue?

Quoted 8 lines
Quoted 3 lines

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

What tool / kernel instrumentation / mechanism are you using to determine that some task(s) are indeed blocked waiting for i/o? Perhaps some userspace process accounting tools could be "broken" in the sense that they generalize all uninterruptible sleep as waiting for i/o ...

I wouldn't expect /proc/stat and similar to be broken in that way, but If no one has a better idea I guess I will assume there's a check needed of where time is added to iowait. I was hoping to avoid a full kernel search. Never thought of /proc data as a user space tool, but I guess.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#4 lore raw
In reply to Bill Davidsen
Quoted 5 lines · Bill Davidsen

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

All filesystem IO and direct disk IO can cause iowait.

This includes NFS activity.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#5 lore raw
Quoted 11 lines · Rik van Riel
Quoted 5 lines · Bill Davidsen

I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.

All filesystem IO and direct disk IO can cause iowait.

This includes NFS activity.

If I didn't note it before, I'm read the the data from /proc, cpustats, net/dev, and diskstats. I assume that all i/o would show up in one of those places. NFS isn't involved, although this machine is a fileserver as a side job the modules weren't even loaded during testing.

A puzzlement for future consideration. If I get a chance later this week I'll make a pretty graphic of all the stuff going on when the iowait spiked, ctx rate, inq rate, hell the last time I even grabbed the CPU temp to see if it told me anything (didn't, thermal throttling NOT).

Thanks for the feedback, I think that lets out the obvious stuff.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot