Skip to contentSkip to conversation

strange performance issues with OS X 10.6 client

DISCUSSION Stefan Krüger 2 participants 9 messages Apr 15, 2010 – Apr 22, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!

$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

#1 lore raw
Quoted 37 lines · Stefan Krüger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!

Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz 
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

Honestly, I don't see anything suspicious, but maybe you guys will catch something

Anyway, if you need more info or want to me to test something else, just drop me a line

TIA

#2 lore raw

Ok, how about this one..

FreeBSD nfsstat -s (statistics zeroed before running tar) after extracting Cocoa-3rd.tgz ->

Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove

    628      6419      5871         0       627      2488      1989 627

Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
0 0 0 227 0 0 0 5529
Mknod Fsstat Fsinfo PathConf Commit
0 2 1 1 2318
Server Ret-Failed

             5871
Server Faults
            0

Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved

     2488      2488         0

Fedora 12 nfsstat -s ->

Server rpc stats:

calls      badcalls   badauth    badclnt    xdrcall
33275      0          0          0          0       

Server nfs v3:
null getattr setattr lookup access readlink
0 0% 627 1% 6426 19% 8523 25% 9111 27% 0 0%
read write create mkdir symlink mknod
994 2% 2362 7% 1996 5% 221 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
660 1% 0 0% 0 0% 0 0% 0 0% 1 0%
fsstat fsinfo pathconf commit
1 0% 0 0% 0 0% 2353 7%

So... 33275 RPC calls for 1589 directories/files?

#3 lore raw
In reply to Stefan Krüger
Quoted 60 lines · Stefan Krüger
Quoted 37 lines · Stefan Krüger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!

Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running

Quoted 6 lines

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

The number of packets is around 1800 for the FreeBSD server and around 1940 for the Linux server. The RPC counts you posted in a later email show that Linux does more LOOKUP and ACCESS requests. But generally, it looks like your client is doing roughly the same amount of work in both cases.

But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETATTR or COMMIT. This looks normal, since disk writes can take a few milliseconds.

FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.

Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
#4 lore raw
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount op=

tions:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576
(nfsv3 and tcp are default, I have also tried udp, and with and wi=

thout

Quoted 1 line
Quoted 1 line
Quoted 1 line

rdirplus, with different read/write sizes (started with 32k, less =

for udp,

Quoted 1 line
Quoted 1 line
Quoted 1 line

and then cranked it up to 1m to make the beachball appear less oft=

en),

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unple=

asant

Quoted 1 line
Quoted 1 line
Quoted 1 line

experience on nfs, beachball every now and then together with a hu=

ge amount

Quoted 1 line
Quoted 1 line
Quoted 1 line

of network traffic (RX with 20MB/s+ peaks), not unusual to see sev=

eral

Quoted 1 line
Quoted 1 line
Quoted 1 line

gigabytes received after some minutes browsing, XCode shows a ''Th=

e

Quoted 1 line
Quoted 1 line
Quoted 1 line

document "SomeFile.m" could not be saved.''-error after some edits=

, Opera

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min=

!

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extract=

ing the

Quoted 1 line
Quoted 1 line

Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running in=

side

Quoted 1 line
Quoted 1 line

VMWare though), and still it was much faster (5:51.10 vs 0:09.35) t=

han

Quoted 8 lines
Quoted 8 lines

extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nf=

s.out

Quoted 1 line
Quoted 1 line

host nfssrv and port 2049) on both freebsd8 (interface for freebsd =

is a bit

Quoted 10 lines
Quoted 8 lines

different ofc) and fedora12 while running

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

=20
The number of packets is around 1800 for the FreeBSD server and aroun=

d=20

Quoted 1 line

1940 for the Linux server. The RPC counts you posted in a later emai=

l=20

Quoted 1 line

show that Linux does more LOOKUP and ACCESS requests. But generally,=

it=20

Quoted 1 line

looks like your client is doing roughly the same amount of work in bo=

th=20

Quoted 4 lines

cases.
=20
But what catches my eye in the F12 tcpdump is that there are pauses=20 where the server reply is delayed by a few milliseconds after a SETAT=

TR=20

Quoted 7 lines

or COMMIT. This looks normal, since disk writes can take a few=20 milliseconds.
=20
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is=20 doing something illegal. No NFS server can turn a SETATTR around in=20 just a few microseconds and claim that it is on permanent storage,=20 unless it has some kind of NVRAM.

=46irst of all, thanks for your answer Chuck :)

there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some e= xtra
reads etc. sneaked in the linux tcpdump/nfsstat -s)

Just for the record, Solaris 10u8 (UFS) extraction time is almost the s= ame as
=46reeBSDs:

$ time tar xfz Cocoa-3rd.tgz
0.111u 1.621s 0:09.03 19.1%	0+0k 8+4966io 0pf+0w

So I guess they're both cheating ;-)

Anyway, seems like I'm the only one with this problem on OS X (seeing 5= min
extraction times and huge rx peaks, beachball etc. during normal use), = so
thanks for your time

Cheers

#5 lore raw
Quoted 17 lines · Stefan Kr=FCger
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount op=

tions:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576
(nfsv3 and tcp are default, I have also tried udp, and with and wi=

thout

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rdirplus, with different read/write sizes (started with 32k, less =

for udp,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and then cranked it up to 1m to make the beachball appear less oft=

en),

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unple=

asant

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

experience on nfs, beachball every now and then together with a hu=

ge amount

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of network traffic (RX with 20MB/s+ peaks), not unusual to see sev=

eral

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

gigabytes received after some minutes browsing, XCode shows a ''Th=

e

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

document "SomeFile.m" could not be saved.''-error after some edits=

, Opera

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min=

!

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extract=

ing the

Quoted 1 line
Quoted 1 line
Quoted 1 line

Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running in=

side

Quoted 1 line
Quoted 1 line
Quoted 1 line

VMWare though), and still it was much faster (5:51.10 vs 0:09.35) t=

han

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nf=

s.out

Quoted 1 line
Quoted 1 line
Quoted 1 line

host nfssrv and port 2049) on both freebsd8 (interface for freebsd =

is a bit

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

different ofc) and fedora12 while running

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

The number of packets is around 1800 for the FreeBSD server and arou=

nd

Quoted 1 line
Quoted 1 line

1940 for the Linux server. The RPC counts you posted in a later ema=

il

Quoted 1 line
Quoted 1 line

show that Linux does more LOOKUP and ACCESS requests. But generally=

, it

Quoted 1 line
Quoted 1 line

looks like your client is doing roughly the same amount of work in b=

oth

Quoted 4 lines
Quoted 4 lines

cases.

But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETA=

TTR

Quoted 8 lines
Quoted 7 lines

or COMMIT. This looks normal, since disk writes can take a few milliseconds.

FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.

Quoted 4 lines

First of all, thanks for your answer Chuck :)

there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some=

extra

Quoted 3 lines

reads etc. sneaked in the linux tcpdump/nfsstat -s)

Just for the record, Solaris 10u8 (UFS) extraction time is almost the=

same as

Quoted 6 lines

FreeBSDs:

$ time tar xfz Cocoa-3rd.tgz
0.111u 1.621s 0:09.03 19.1%	0+0k 8+4966io 0pf+0w

So I guess they're both cheating ;-)

Is Solaris running in a guest too?

What physical file system are you using on the Linux NFS server?

--=20
chuck[dot]lever[at]oracle[dot]com

#6 lore raw
Quoted 18 lines · Chuck Lever
Quoted 17 lines · Stefan Kr=FCger
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount =

options:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576 (nfsv3 and tcp are default, I have also tried udp, and with and =

without

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rdirplus, with different read/write sizes (started with 32k, les=

s for udp,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and then cranked it up to 1m to make the beachball appear less o=

ften),

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unp=

leasant

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

experience on nfs, beachball every now and then together with a =

huge

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

amount of network traffic (RX with 20MB/s+ peaks), not unusual t=

o see

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

several gigabytes received after some minutes browsing, XCode sh=

ows a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

''The document "SomeFile.m" could not be saved.''-error after so=

me

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

edits, Opera hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3m=

in!

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extra=

cting

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (runn=

ing

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface fo=

r

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

freebsd is a bit different ofc) and fedora12 while running

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

The number of packets is around 1800 for the FreeBSD server and ar=

ound

Quoted 1 line
Quoted 1 line
Quoted 1 line

1940 for the Linux server. The RPC counts you posted in a later e=

mail

Quoted 1 line
Quoted 1 line
Quoted 1 line

show that Linux does more LOOKUP and ACCESS requests. But general=

ly, it

Quoted 1 line
Quoted 1 line
Quoted 1 line

looks like your client is doing roughly the same amount of work in=

both

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

cases.

But what catches my eye in the F12 tcpdump is that there are pause=

s

Quoted 1 line
Quoted 1 line
Quoted 1 line

where the server reply is delayed by a few milliseconds after a SE=

TATTR

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

or COMMIT. This looks normal, since disk writes can take a few milliseconds.

FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD =

is

Quoted 1 line
Quoted 1 line
Quoted 1 line

doing something illegal. No NFS server can turn a SETATTR around =

in

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.

First of all, thanks for your answer Chuck :)

there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so so=

me

Quoted 3 lines
Quoted 3 lines

extra reads etc. sneaked in the linux tcpdump/nfsstat -s)

Just for the record, Solaris 10u8 (UFS) extraction time is almost t=

he

Quoted 8 lines
Quoted 6 lines

same as FreeBSDs:

$ time tar xfz Cocoa-3rd.tgz
0.111u 1.621s 0:09.03 19.1%	0+0k 8+4966io 0pf+0w

So I guess they're both cheating ;-)

=20
Is Solaris running in a guest too?

Yes, this test was also done using VMWare
I also installed Fedora12/ext4 in VMWare and got ->

$ time tar xfz Cocoa-3rd.tgz
0.135u 2.151s 0:36.50 6.2%      0+0k 3+4770io 0pf+0w

Well, 36 seconds, still 3x longer than Solaris/FreeBSD but way better t= han
5min on bare metal

Anyway, to get some real data I splitted my raid1 mirror on the nfs ser= ver
yesterday and installed FreeBSD 8 on the (now) free disk
I don't know why but NFS performance is stunning, extracting the tgz on= ly
takes seconds, Opera doesn't hang for minutes when closing, no strange = rx
spikes and a lot less total bytes received on the same hardware with th= e same
files (no special tweaks done, only change on freebsd was to start 8 nf= sd
servers [default is 4], fs is UFS with softdeps)

I can't explain it...

Quoted 1 line

What physical file system are you using on the Linux NFS server?

See my first mail ("nfs shared /home is ext4 with default mount options= "),
so it's ext4 (I did some tests with xfs/ext3 using fedora12/VMWare, too= , same
bad results though)

#7 lore raw
Quoted 122 lines · Stefan Krüger
Quoted 96 lines · Chuck Lever
Quoted 95 lines · Stefan Krüger
Quoted 79 lines · Chuck Lever
Quoted 60 lines · Stefan Krüger
Quoted 37 lines · Stefan Krüger

Hello list,

I have some really strange nfs performance issues

NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options

/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)

nfs and nfslock are up and running

Nothing else touched on the server nfs-wise.

NFS client is Mac OS X, version 10.6.3

My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)

Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.

It's horrible :(

Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!

Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz
0.169u 3.198s 5:51.10 0.9%	0+0k 1+6972io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd/
0.014u 0.477s 0:45.59 1.0%	0+0k 1+1io 0pf+0w

So any help or hints really appreciated

So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:

$ time tar xfz Cocoa-3rd.tgz
0.104u 1.474s 0:09.35 16.7%	0+0k 0+4896io 0pf+0w
$ time rm -rf Solutions-Cocoa-3rd
0.006u 0.160s 0:01.24 12.9%	0+0k 0+0io 0pf+0w

I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running

Quoted 6 lines

tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted

(which extracts just a couple of files) , you can find them here:

Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/

The number of packets is around 1800 for the FreeBSD server and around 1940 for the Linux server. The RPC counts you posted in a later email show that Linux does more LOOKUP and ACCESS requests. But generally, it looks like your client is doing roughly the same amount of work in both cases.

But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETATTR or COMMIT. This looks normal, since disk writes can take a few milliseconds.

Quoted 4 lines

FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.

First of all, thanks for your answer Chuck :)

there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some extra reads etc. sneaked in the linux tcpdump/nfsstat -s)

Just for the record, Solaris 10u8 (UFS) extraction time is almost the same as FreeBSDs:

Quoted 6 lines
Quoted 4 lines
$ time tar xfz Cocoa-3rd.tgz
0.111u 1.621s 0:09.03 19.1%	0+0k 8+4966io 0pf+0w

So I guess they're both cheating ;-)

Is Solaris running in a guest too?

Yes, this test was also done using VMWare
I also installed Fedora12/ext4 in VMWare and got ->

$ time tar xfz Cocoa-3rd.tgz
0.135u 2.151s 0:36.50 6.2%      0+0k 3+4770io 0pf+0w

Well, 36 seconds, still 3x longer than Solaris/FreeBSD but way better than 5min on bare metal

Anyway, to get some real data I splitted my raid1 mirror on the nfs server yesterday and installed FreeBSD 8 on the (now) free disk
I don't know why but NFS performance is stunning, extracting the tgz only takes seconds, Opera doesn't hang for minutes when closing, no strange rx spikes and a lot less total bytes received on the same hardware with the same files (no special tweaks done, only change on freebsd was to start 8 nfsd servers [default is 4], fs is UFS with softdeps)

Quoted 7 lines

I can't explain it...

Quoted 1 line

What physical file system are you using on the Linux NFS server?

See my first mail ("nfs shared /home is ext4 with default mount options"), so it's ext4 (I did some tests with xfs/ext3 using fedora12/VMWare, too, same bad results though)

So, VMware is probably not waiting for disk writes to complete, since NFS servers in VMware guests run faster than native. Note to self: never put mission critical data on NFS servers running as VMware guests.

Still, the Linux NFS server has some kind of problem here when comparing apples to apples. Does increasing the number of nfsd threads on the server help?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#8 lore raw
Quoted 3 lines · Chuck Lever

So, VMware is probably not waiting for disk writes to complete, since NFS servers in VMware guests run faster than native. Note to self: never put mission critical data on NFS servers running as VMware guests.

yeah I only did this for testing...

Quoted 3 lines

Still, the Linux NFS server has some kind of problem here when comparing apples to apples. Does increasing the number of nfsd threads on the server help?

Increasing the number of nfsd threads did not help (I raised them from 8 to 32 but Fedora is still crawling when extracting the tgz file, nfsstat numbers below are with 8 nfsd threads on both Fedora and FreeBSD)

So speaking of nfsstat, I've got some new statistics, this time I extracted the whole tgz and not just a single directory (+ files) inside it (statistics zeroed before ofc, both tests run on bare metal and on same hardware, network, etc.)

Fedora 12:

Server rpc stats:

calls      badcalls   badauth    badclnt    xdrcall
45960      0          0          0          0       

Server nfs v3:
null getattr setattr lookup access readlink
1 0% 641 1% 6885 14% 10373 22% 18336 39% 0 0%
read write create mkdir symlink mknod
1917 4% 2552 5% 1999 4% 228 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
633 1% 0 0% 3 0% 0 0% 0 0% 3 0%
fsstat fsinfo pathconf commit
21 0% 1 0% 1 0% 2366 5%

FreeBSD 8:

Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove

      628      6421      5287         3       703      2490      1991 629

Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 229 2 0 2 6526
Mknod Fsstat Fsinfo PathConf Commit
0 5 1 1 2402
Server Ret-Failed

             5267
Server Faults
            0

Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved

     2490      2490         0

First thing that catches my eyes is Access, it's 3x higher on Fedora, next would be Lookup and Read, both 2x higher

MacOS 10.6 nfsstat client statistics after extracting on Fedora 12:

Client Info:
RPC Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove

      640      6885     10380         0      1919      2553      1999 634

Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 228 1 0 3 18353
Mknod Fsstat Fsinfo PathConf Commit
0 21 1 1 2367
RPC Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 45989
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses

    84529      5034      9021      9408      5235      1701       766 2553

BioRLHits Misses BioD Hits Misses DirE Hits Misses
0 0 1 3 4 0

... after extracting on FreeBSD 8:

Client Info:
RPC Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove

      628      6421      5290         3       704      2490      1991 630

Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 229 3 0 2 6538
Mknod Fsstat Fsinfo PathConf Commit
0 5 1 1 2402
RPC Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 27342
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses

    36508      1529      1817      4317       958       704       767 2490

BioRLHits Misses BioD Hits Misses DirE Hits Misses
1 3 1 7 3 0

Well this mostly matches what we've seen in the nfs server statistics, even though a lot more caching was done when using the Fedora nfs server

nfsstat manpage on OSX explains them as:
Attr Hits/Misses - Performance of the NFS file attribute cache. Lkup Hits/Misses - Performance of the directory name lookup cache. BioR Hits/Misses - Performance of block cache for reads. BioW Hits/Misses - Performance of block cache for writes. BioRL Hits/Misses - Performance of symbolic link cache. BioD Hits/Misses - Performance of directory cache.
DirE Hits/Misses - Performance of directory offset cache.

I've also captured the traffic from both sessions with tcpdump, it's quite huge though, 42MB for Fedora and 10MB for FreeBSD... let me know if you want to take a look

TIA

PS: Please let me say again that extraction time using FreeBSD nfs is 10 *seconds* while on Fedora it's over 5 *minutes*...