Skip to contentSkip to conversation

Strange NFS write performance Linux->Solaris-10/VXFS, maybe VW related

DISCUSSION Martin Knoblauch 1 participant 1 message Dec 29, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

----- Original Message ----

Quoted 6 lines

From: Chris Snook <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: [email protected]; [email protected]
Sent: Friday, December 28, 2007 7:45:13 PM
Subject: Re: Strange NFS write performance Linux->Solaris-10/VXFS, maybe VW related

Quoted 6 lines
Quoted 3 lines · Martin Knoblauch

Hi,

currently I am tracking down an "interesting" effect when writing

to

a

Quoted 3 lines
Quoted 1 line

Solars-10/Sparc based server. The server exports two filesystems.

One

UFS,

Quoted 3 lines
Quoted 1 line

one VXFS. The filesystems are mounted NFS3/TCP, no special

options.

Linux

Quoted 6 lines
Quoted 4 lines

kernel in question is 2.6.24-rc6, but it happens with earlier kernels (2.6.19.2, 2.6.22.6) as well. The client is x86_64 with 8 GB of ram.

The problem: when writing to the VXFS based filesystem,

performance

drops

Quoted 3 lines
Quoted 1 line

dramatically when the the filesize reaches or exceeds

"dirty_ratio".

For a

Quoted 3 lines
Quoted 1 line

dirty_ratio of 10% (about 800MB) files below 750 MB are

transfered

with about

Quoted 3 lines
Quoted 1 line

30 MB/sec. Anything above 770 MB drops down to below 10 MB/sec. If

I

perform

Quoted 3 lines
Quoted 1 line

the same tests on the UFS based FS, performance stays at about

30

MB/sec

Quoted 5 lines
Quoted 3 lines

until 3GB and likely larger (I just stopped at 3 GB).

Any ideas what could cause this difference? Any suggestions

on

debugging it?

Quoted 3 lines

1) Try normal NFS tuning, such as rsize/wsize tuning.

rsize/wsize only have minimal effect. The negotiated size seems to be optimal.

Quoted 2 lines

2) You're entering synchronous writeback mode, so you can delay the

problem by raising dirty_ratio to 100, or reduce the size of the problem

Quoted 2 lines

by lowering dirty_ratio to 1. Either one could help.

For experiments, sure. But I do not think that I want to have 8 GB of dirty pages [potentially] laying around. Are you sure that 1% is a useful value for dirty_ratio? Looking at the code, it seems a minimum of 5% is enforced in "page-writeback.c:get_dirty_limits":

        dirty_ratio = vm_dirty_ratio;
        if (dirty_ratio > unmapped_ratio / 2)
                dirty_ratio = unmapped_ratio / 2;

        if (dirty_ratio < 5)
                dirty_ratio = 5;
Quoted 2 lines

3) It sounds like the bottleneck is the vxfs filesystem. It only

*appears* on the client side because writes up until dirty_ratio get buffered on

Quoted 1 line

the client.

Sure, the fact that a UFS (or SAM-FS) based FS behaves well in the same situation points in that direction.

Quoted 2 lines

If you can confirm that the server is actually writing stuff to disk

slower when the client is in writeback mode, then it's possible the Linux

Quoted 2 lines

NFS client is doing something inefficient in writeback mode.

I will try to get an iostat trace from the Sun side. Thanks for the suggestion.

Cheers
Martin
PS: Happy Year 2008 to all Kernel Hackers and their families