Skip to contentSkip to conversation

Getting lousy NFS + tar-pipe throughput on 2.4.20

DISCUSSION Timothy Miller 3 participants 6 messages Feb 13, 2004 – Feb 16, 2004

lore
Share & tools
lore
#0 lore raw

I'm running a fresh install of RH9 (kernel 2.4.20-something) on a workstation. The workstation is an Athlon 3200+ with 512 megs of RAM on an ABIT KV7 (KT600 chipset). The ethernet controller built into the KV7 is "VIA RhineII". The file system is ext3.

We are mounting an NFS filesystem from a Sun box using automount, and we're using a tar-pipe to move data from the server to the workstation.
Both tars of the tar-pipe are running on the workstation, so the
network traffic is all NFS.

(1) We have verified that the disk load on the server is very low. The disk is not being saturated.

(2) We have verified that the ethernet on the server is not being saturated.

(3) The workstation is connected to the server through a switch, so it's not competing for bandwidth with anything else.

In theory, we should get about 10 megabytes/sec throughput, but what we're measuring is about 1 to 2 megs/sec.

The workstation is using a single 120 gig WD IDE drive (WD1200JB), which as I was talking about in other emails should be able to do up to 30 megs/sec for writes.

While this tar-pipe is going on, the workstation is very unresponsive. "top" reports that kernel CPU usage is anywhere from 30% to 70%, but mostly around 40%. User space is using about 10%; that varies also. Despite the fact that there is some amount of idle time, the X cursor jumps about badly.

We're not compressing or anything. We're just doing the tar-pipe. Therefore, the workstation should be experiencing very little load while it transfers a mere 10 megs/sec to disk. Buffering in RAM should also allow the kernel to order writes efficiently.

Since the source tar process is talking to an NFS volume, the overhead of opening, reading, and closing small files could hurt throughput (would have been better to rsh the source tar so that the tar data is what was going over ethernet through a single socket). But that should _reduce_ the amount of I/O that is being accomplished, thereby reducing the work being done by the workstation. It would just WAIT more. It should not be unresponsive.

I would like to investigate this performance issue, but I don't know what tools I should run to investigate. If anyone could please give me some tips on it, I would be most appreciative.

Thanks!

#1 lore raw

IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]

I have discovered one major problem that I didn't think to check before. I have never before had trouble using DMA with WD drives, VIA chipsets, or RH9, so I didn't think to check before, but hdparm reports that DMA is disabled for the disk on the workstation I mentioned in my earlier post.

So, I tried this:
hdparm -d1 /dev/hda

And I got this result:
/dev/hda:
Setting using_dma to 1 (on)
HDIO_SET_DMA failed: Operation not permitted
using_dma = 0 (off)

How do I fix this? Do I have to unmount the filesystem before I can change the dma setting? Why would it be off to begin with? I know that the BIOS is set up right. Does 2.4.20 not work well with the KT600 chipset?

Thanks.

Quoted 1 line · Timothy Miller

[snip everything about slow NFS performance]

#2 lore raw

Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]

Quoted 15 lines · Timothy Miller

I have discovered one major problem that I didn't think to check before. I have never before had trouble using DMA with WD drives, VIA chipsets, or RH9, so I didn't think to check before, but hdparm reports that DMA is disabled for the disk on the workstation I mentioned in my earlier post.

So, I tried this:
hdparm -d1 /dev/hda

And I got this result:
/dev/hda:
Setting using_dma to 1 (on)
HDIO_SET_DMA failed: Operation not permitted
using_dma = 0 (off)

Quoted 4 lines

How do I fix this? Do I have to unmount the filesystem before I can change the dma setting? Why would it be off to begin with? I know that the BIOS is set up right. Does 2.4.20 not work well with the KT600 chipset?

It doesn't, upgrade to 2.4.24.

#3 lore raw

2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

Quoted 7 lines · Bartlomiej Zolnierkiewicz

On Saturday 14 of February 2004 01:16, Timothy Miller wrote:

Does 2.4.20 not work well with the KT600

Quoted 1 line

chipset?

It doesn't, upgrade to 2.4.24.

Ok, so we tried that. That caused all sorts of havoc, most of which we might be able to figure out. It seems that RedHat's utilities and stuff don't get along well with 2.4.24 if all you do is just boot the new kernel. Seems a bunch of other stuff needs to be upgraded.

Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems to be no way to enable it. Is this a RedHat only thing?

In any event, when we managed to get it to boot, it absolutely did NOT fix the DMA problem with the KT600. Trying to enable it with hdparm still says "Operation not permitted". We spend a LOT of time trying to make sure we got the kernel configured right with all of the right options, etc., but we're still without IDE DMA.

Any further suggestions?

Thanks.

#4 lore raw

Re: 2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

Quoted 9 lines · Timothy Miller
Quoted 7 lines · Bartlomiej Zolnierkiewicz

On Saturday 14 of February 2004 01:16, Timothy Miller wrote:

Does 2.4.20 not work well with the KT600

Quoted 1 line

chipset?

It doesn't, upgrade to 2.4.24.

Quoted 7 lines

Ok, so we tried that. That caused all sorts of havoc, most of which we might be able to figure out. It seems that RedHat's utilities and stuff don't get along well with 2.4.24 if all you do is just boot the new kernel. Seems a bunch of other stuff needs to be upgraded.

Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems to be no way to enable it. Is this a RedHat only thing?

This feature is not present in vanilla kernels.

Quoted 7 lines

In any event, when we managed to get it to boot, it absolutely did NOT fix the DMA problem with the KT600. Trying to enable it with hdparm still says "Operation not permitted". We spend a LOT of time trying to make sure we got the kernel configured right with all of the right options, etc., but we're still without IDE DMA.

Any further suggestions?

Do you have VIA IDE driver compiled in?
CONFIG_BLK_DEV_VIA82CXXX=y

If so, please send output of 'dmesg' command.

--bart

#5 lore raw

Re: 2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

Quoted 5 lines · Bartlomiej Zolnierkiewicz
Quoted 2 lines · Timothy Miller

Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems to be no way to enable it. Is this a RedHat only thing?

This feature is not present in vanilla kernels.

It's not a kernel feature.

Red Hat's initrd image makes "root=LABEL=/" work. The user probably needs to regen his initrd image...

Jeff