What causes iowait other than waiting for i/o?
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 7 lines · Bill Davidsen
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
Quoted 6 lines
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
Nothing /should/ cause time to be attributed to i/o wait other than waiting for i/o. But the accounting isn't completely accurate. When the kernel records what the has happen (governed by when the timer interrupt triggers as far as I remember), many different processes may have run and done may different things, but only what the last one did is recorded. So if you are unlucky you may have the system doing all sorts of activities, but only recording i/o wait if the last process to run before the kernel looked was doing i/o wait... The numbers are not exactely accurate...
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Hi Bill,
Quoted 7 lines · Bill Davidsen
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
Quoted 5 lines
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
What tool / kernel instrumentation / mechanism are you using to determine that some task(s) are indeed blocked waiting for i/o? Perhaps some userspace process accounting tools could be "broken" in the sense that they generalize all uninterruptible sleep as waiting for i/o ...
Satyam
Quoted 12 lines · Satyam Sharma
Hi Bill,
Quoted 9 lines · Bill Davidsen
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
At this point I'm really curious, does someone have a good clue?
Quoted 8 lines
Quoted 3 lines
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
What tool / kernel instrumentation / mechanism are you using to determine that some task(s) are indeed blocked waiting for i/o? Perhaps some userspace process accounting tools could be "broken" in the sense that they generalize all uninterruptible sleep as waiting for i/o ...
I wouldn't expect /proc/stat and similar to be broken in that way, but If no one has a better idea I guess I will assume there's a check needed of where time is added to iowait. I was hoping to avoid a full kernel search. Never thought of /proc data as a user space tool, but I guess.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Quoted 5 lines · Bill Davidsen
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
All filesystem IO and direct disk IO can cause iowait.
This includes NFS activity.
Signature · 3 lines
Politics is the struggle between those who want to make their country the best in the world, and those who believe it already is. Each group calls the other unpatriotic.
Quoted 11 lines · Rik van Riel
Quoted 5 lines · Bill Davidsen
I recently noted that my system was spending a lot of time in i/o wait when doing some tasks which I thought didn't involve i/o, as noted by the lack of disk light activity most of the time. I thought of network, certainly the NIC had no activity for this job. So I set up a little loop to capture all disk i/o and network activity (including loopback). That was no obvious help, and the program doesn't use pipes.
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the kernel is doing something and charging time to iowait instead of system I don't see anything to fix, but I would like to understand.
All filesystem IO and direct disk IO can cause iowait.
This includes NFS activity.
If I didn't note it before, I'm read the the data from /proc, cpustats, net/dev, and diskstats. I assume that all i/o would show up in one of those places. NFS isn't involved, although this machine is a fileserver as a side job the modules weren't even loaded during testing.
A puzzlement for future consideration. If I get a chance later this week I'll make a pretty graphic of all the stuff going on when the iowait spiked, ctx rate, inq rate, hell the last time I even grabbed the CPU temp to see if it told me anything (didn't, thermal throttling NOT).
Thanks for the feedback, I think that lets out the obvious stuff.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot