NFS access slow

Accessing disks locally on a server gives read speeds around 100MB/s, write speeds around 267MB/s.

Mounting the same disks on the same server via NFS (ie, not using the network at all) gives read speeds around 30MB/s, write speeds around 80MB/s.

That's about 30% of the local access speed.

Is that to be expected? I'd expect a 10-15% slowdown, but not this much.

This is using NFSv4; using NFSv3 improves the speeds slightly (36MB/s read, 95MB/s write).

Other parameters we've changed, none of which have a significant impact:

- UDP/TCP
- rsize and wsize
- noatime
- noacl
- nocto

If that's an unexpected slow down, where should we be looking?

Thanks.

What are your disks? How exactly are you getting those numbers? (Literally, step-by-step, what commands are you running?)

What kernel version?

Note loopback-mounts (client and server on same machine) aren't really fully supported.

--b.

They are Enterprise Nearline 6Gb/s SAS drives in an Infortrend disk array.

Using postmark:

pm> set location /mnt/tmp
pm> set size 10000 10000000
pm> run

The only difference is the 'set location' line, which points to either the NFS mountpoint or the local mountpoint.

A test using dd ("dd if=/dev/zero of=/mnt/tmp bs=1M count=8192") gave a difference of about five times faster for direct access versus access via NFS.

3.2

OK, I wasn't aware of that. We were only testing that way to try to eliminate switches, cables, etc. I've just run a test from another server, both connected via 10G links, and I'm getting a read speed of just under 20BM/s and a write speed of 52MB/s.

Keith Edmunds wrote:

OK, I wasn't aware of that. We were only testing that way to try to eliminate switches, cables, etc. I've just run a test from another server, both connected via 10G links, and I'm getting a read speed of just under 20BM/s and a write speed of 52MB/s.

Something's wrong. What numbers do you get from iperf, or even something like wget? Are you setting anything unusual with sysctl?

Note that NFS requires operations such as file creation and removal to be synchronous (for reboot/crash-recovery reasons). So e.g. if postmark is single threaded (I think it is), then the client has to wait for the server to respond to a file create before proceeding, and the server has to wait for the create to hit disk before responding.

Depending on exactly how postmark calculates those bandwidth numbers that could have a big effect.

If your array has a battery-backed cache that should help.

To make that an apples-to-apples comparison you should include the time to sync after the dd in both cases. (Though if your server doesn't have much memory that might not make a big difference.)

Have you tested the network speed? (E.g. with iperf.)

--b.