i/o wait eating all of CPU on 2.6.1

I have two servers, both of which have more than 300 gigabytes of hard drive space and those files are made available to the network with samba, nfs and http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I noticed that everything was VERY slow, from a machine that is connected to the other server with a 100M link, 57kB/s tops. i/o wait eats up all of the cpu. On the other hand, Apache (and everything else) works very fast when I only send /dev/zero to a client, since that doesn't need disk operations.

I don't notice anything suspicious in dmesg but since this happens on two machines and has only happened when upgraded to 2.6.1, it's most likely because of 2.6.1. I'm downgrading to 2.6.0 (with mremap-patch) today if I don't figure out what is wrong. Any ideas?

And since I'm not subscribed to Linux Kernel Mailing List, please forward any replies to me.

-Jaakko Helminen

Is DMA enabled with 2.6.1 on these two machines?

	Paolo Ornati
	Linux v2.6.2-rc1-mm3

Oh yes, I forgot to mention it, DMA is enabled on both machines.

-Jaakko Helminen

Hi,

A couple of things. IO wait does not 'eat cpu'. Its a wait state, recorded when the box has nothing to do and IO is pending. That being said I remember reading here about nfs bugs in 2.6.1 that slow it down. A search should find some patches for you to try.

Ed

In article <[email protected]>,
Jaakko Helminen <[email protected]> wrote:

| 
| I have two servers, both of which have more than 300 gigabytes of hard drive
| space and those files are made available to the network with samba, nfs and
| http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I noticed
| that everything was VERY slow, from a machine that is connected to the other
| server with a 100M link, 57kB/s tops. i/o wait eats up all of the cpu.
| On the other hand, Apache (and everything else) works very fast when I only
| send /dev/zero to a client, since that doesn't need disk operations.

Actually, io-wait is an indication that the CPU is idle with io outstanding, and doesn't eat the machine any more than idle time. It is an indication that the io is not keeping up, of course.

The fact that sending /dev/zero is fast sort of eliminates the reported problems with NFS and setting the read/write size to 8k on the client. Just for grins, hopefully you have followed the "NFS is slow" thread, but I saw people having the issue with 2.6.0, so that's not likely to be be involved.

I have some preliminary numbers which indicate 2.6.2-rc2 is faster at some disk operations than 2.6.1, but since that's one benchmark on one machine all I can say is that if you want to move forward, that kernel is working well for me. Of course if there's nothing in 2.6.1 you really want or need, falling back is a sure cure.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.