Strange NFS write performance Linux->Solaris-10/VXFS, maybe VW related
Subsystems (2)
----- Original Message ----
Quoted 6 lines
From: Chris Snook <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: [email protected]; [email protected]
Sent: Friday, December 28, 2007 7:45:13 PM
Subject: Re: Strange NFS write performance Linux->Solaris-10/VXFS, maybe VW related
Quoted 6 lines
Quoted 3 lines · Martin Knoblauch
Hi,
currently I am tracking down an "interesting" effect when writing
to
a
Quoted 3 lines
Quoted 1 line
Solars-10/Sparc based server. The server exports two filesystems.
One
UFS,
Quoted 3 lines
Quoted 1 line
one VXFS. The filesystems are mounted NFS3/TCP, no special
options.
Linux
Quoted 6 lines
Quoted 4 lines
kernel in question is 2.6.24-rc6, but it happens with earlier kernels (2.6.19.2, 2.6.22.6) as well. The client is
x86_64with 8 GB of ram.The problem: when writing to the VXFS based filesystem,
performance
drops
Quoted 3 lines
Quoted 1 line
dramatically when the the filesize reaches or exceeds
"
dirty_ratio".
For a
Quoted 3 lines
Quoted 1 line
dirty_ratioof 10% (about 800MB) files below 750 MB aretransfered
with about
Quoted 3 lines
Quoted 1 line
30 MB/sec. Anything above 770 MB drops down to below 10 MB/sec. If
I
perform
Quoted 3 lines
Quoted 1 line
the same tests on the UFS based FS, performance stays at about
30
MB/sec
Quoted 5 lines
Quoted 3 lines
until 3GB and likely larger (I just stopped at 3 GB).
Any ideas what could cause this difference? Any suggestions
on
debugging it?
Quoted 3 lines
1) Try normal NFS tuning, such as rsize/wsize tuning.
rsize/wsize only have minimal effect. The negotiated size seems to be optimal.
Quoted 2 lines
2) You're entering synchronous writeback mode, so you can delay the
problem by raising dirty_ratio to 100, or reduce the size of the problem
Quoted 2 lines
by lowering
dirty_ratioto 1. Either one could help.
For experiments, sure. But I do not think that I want to have 8 GB of dirty pages [potentially] laying around. Are you sure that 1% is a useful value for dirty_ratio? Looking at the code, it seems a minimum of 5% is enforced in "page-writeback.c:get_dirty_limits":
dirty_ratio = vm_dirty_ratio;
if (dirty_ratio > unmapped_ratio / 2)
dirty_ratio = unmapped_ratio / 2;
if (dirty_ratio < 5)
dirty_ratio = 5;Quoted 2 lines
3) It sounds like the bottleneck is the vxfs filesystem. It only
*appears* on the client side because writes up until dirty_ratio get buffered on
Quoted 1 line
the client.
Sure, the fact that a UFS (or SAM-FS) based FS behaves well in the same situation points in that direction.
Quoted 2 lines
If you can confirm that the server is actually writing stuff to disk
slower when the client is in writeback mode, then it's possible the Linux
Quoted 2 lines
NFS client is doing something inefficient in writeback mode.
I will try to get an iostat trace from the Sun side. Thanks for the suggestion.
Cheers
Martin
PS: Happy Year 2008 to all Kernel Hackers and their families