Skip to contentSkip to conversation

Strange NFS write performance Linux->Solaris-10/VXFS, maybe VW related

DISCUSSION Martin Knoblauch 2 participants 2 messages Dec 28, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

currently I am tracking down an "interesting" effect when writing to a Solars-10/Sparc based server. The server exports two filesystems. One UFS, one VXFS. The filesystems are mounted NFS3/TCP, no special options. Linux kernel in question is 2.6.24-rc6, but it happens with earlier kernels (2.6.19.2, 2.6.22.6) as well. The client is x86_64 with 8 GB of ram.

The problem: when writing to the VXFS based filesystem, performance drops dramatically when the the filesize reaches or exceeds "dirty_ratio". For a dirty_ratio of 10% (about 800MB) files below 750 MB are transfered with about 30 MB/sec. Anything above 770 MB drops down to below 10 MB/sec. If I perform the same tests on the UFS based FS, performance stays at about 30 MB/sec until 3GB and likely larger (I just stopped at 3 GB).

Any ideas what could cause this difference? Any suggestions on debugging it?

spsdm5:/lfs/test_ufs on /mnt/test_ufs type nfs (rw,proto=tcp,nfsvers=3,hard,intr,addr=160.50.118.37) spsdm5:/lfs/test_vxfs on /mnt/test_vxfs type nfs (rw,proto=tcp,nfsvers=3,hard,intr,addr=160.50.118.37)

Cheers
Martin
PS: Please CC me, as I am not subscribed. Don't worry about the spamtrap name :-)

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de
#1 lore raw
Quoted 8 lines · Martin Knoblauch

Hi,

currently I am tracking down an "interesting" effect when writing to a Solars-10/Sparc based server. The server exports two filesystems. One UFS, one VXFS. The filesystems are mounted NFS3/TCP, no special options. Linux kernel in question is 2.6.24-rc6, but it happens with earlier kernels (2.6.19.2, 2.6.22.6) as well. The client is x86_64 with 8 GB of ram.

Quoted 8 lines

The problem: when writing to the VXFS based filesystem, performance drops dramatically when the the filesize reaches or exceeds "dirty_ratio". For a dirty_ratio of 10% (about 800MB) files below 750 MB are transfered with about 30 MB/sec. Anything above 770 MB drops down to below 10 MB/sec. If I perform the same tests on the UFS based FS, performance stays at about 30 MB/sec until 3GB and likely larger (I just stopped at 3 GB).

Any ideas what could cause this difference? Any suggestions on debugging it?

1) Try normal NFS tuning, such as rsize/wsize tuning.

2) You're entering synchronous writeback mode, so you can delay the problem by raising dirty_ratio to 100, or reduce the size of the problem by lowering dirty_ratio to 1. Either one could help.

3) It sounds like the bottleneck is the vxfs filesystem. It only *appears* on the client side because writes up until dirty_ratio get buffered on the client. If you can confirm that the server is actually writing stuff to disk slower
when the client is in writeback mode, then it's possible the Linux NFS client is doing something inefficient in writeback mode.

-- Chris