strange performance issues with OS X 10.6 client
Subsystems (2)
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)
nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)
Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0w
So any help or hints really appreciated
Quoted 37 lines · Stefan Krüger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!
Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0w
I captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/
Honestly, I don't see anything suspicious, but maybe you guys will catch something
Anyway, if you need more info or want to me to test something else, just drop me a line
TIA
Ok, how about this one..
FreeBSD nfsstat -s (statistics zeroed before running tar) after extracting Cocoa-3rd.tgz ->
Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove
628 6419 5871 0 627 2488 1989 627
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
0 0 0 227 0 0 0 5529
Mknod Fsstat Fsinfo PathConf Commit
0 2 1 1 2318
Server Ret-Failed
5871
Server Faults
0Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved
2488 2488 0
Fedora 12 nfsstat -s ->
Server rpc stats:
calls badcalls badauth badclnt xdrcall 33275 0 0 0 0
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 627 1% 6426 19% 8523 25% 9111 27% 0 0%
read write create mkdir symlink mknod
994 2% 2362 7% 1996 5% 221 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
660 1% 0 0% 0 0% 0 0% 0 0% 1 0%
fsstat fsinfo pathconf commit
1 0% 0 0% 0 0% 2353 7%
So... 33275 RPC calls for 1589 directories/files?
Quoted 60 lines · Stefan Krüger
Quoted 37 lines · Stefan Krüger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0wI captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running
Quoted 6 lines
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/
The number of packets is around 1800 for the FreeBSD server and around 1940 for the Linux server. The RPC counts you posted in a later email show that Linux does more LOOKUP and ACCESS requests. But generally, it looks like your client is doing roughly the same amount of work in both cases.
But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETATTR or COMMIT. This looks normal, since disk writes can take a few milliseconds.
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.
Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount op=
tions:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576
(nfsv3 and tcp are default, I have also tried udp, and with and wi=
thout
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdirplus, with different read/write sizes (started with 32k, less =
for udp,
Quoted 1 line
Quoted 1 line
Quoted 1 line
and then cranked it up to 1m to make the beachball appear less oft=
en),
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
but I still have issues no matter which options I chose)
Anyway, I'm stuck now, surfing the web with Safari is a very unple=
asant
Quoted 1 line
Quoted 1 line
Quoted 1 line
experience on nfs, beachball every now and then together with a hu=
ge amount
Quoted 1 line
Quoted 1 line
Quoted 1 line
of network traffic (RX with 20MB/s+ peaks), not unusual to see sev=
eral
Quoted 1 line
Quoted 1 line
Quoted 1 line
gigabytes received after some minutes browsing, XCode shows a ''Th=
e
Quoted 1 line
Quoted 1 line
Quoted 1 line
document "SomeFile.m" could not be saved.''-error after some edits=
, Opera
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min=
!
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extract=
ing the
Quoted 1 line
Quoted 1 line
Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running in=
side
Quoted 1 line
Quoted 1 line
VMWare though), and still it was much faster (5:51.10 vs 0:09.35) t=
han
Quoted 8 lines
Quoted 8 lines
extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0wI captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nf=
s.out
Quoted 1 line
Quoted 1 line
host nfssrv and port 2049) on both freebsd8 (interface for freebsd =
is a bit
Quoted 10 lines
Quoted 8 lines
different ofc) and fedora12 while running
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/=20
The number of packets is around 1800 for the FreeBSD server and aroun=
d=20
Quoted 1 line
1940 for the Linux server. The RPC counts you posted in a later emai=
l=20
Quoted 1 line
show that Linux does more LOOKUP and ACCESS requests. But generally,=
it=20
Quoted 1 line
looks like your client is doing roughly the same amount of work in bo=
th=20
Quoted 4 lines
cases.
=20
But what catches my eye in the F12 tcpdump is that there are pauses=20 where the server reply is delayed by a few milliseconds after a SETAT=
TR=20
Quoted 7 lines
or COMMIT. This looks normal, since disk writes can take a few=20 milliseconds.
=20
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is=20 doing something illegal. No NFS server can turn a SETATTR around in=20 just a few microseconds and claim that it is on permanent storage,=20 unless it has some kind of NVRAM.
=46irst of all, thanks for your answer Chuck :)
there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some e= xtra
reads etc. sneaked in the linux tcpdump/nfsstat -s)
Just for the record, Solaris 10u8 (UFS) extraction time is almost the s= ame as
=46reeBSDs:
$ time tar xfz Cocoa-3rd.tgz 0.111u 1.621s 0:09.03 19.1% 0+0k 8+4966io 0pf+0w
So I guess they're both cheating ;-)
Anyway, seems like I'm the only one with this problem on OS X (seeing 5= min
extraction times and huge rx peaks, beachball etc. during normal use), = so
thanks for your time
Cheers
Quoted 17 lines · Stefan Kr=FCger
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount op=
tions:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576
(nfsv3 and tcp are default, I have also tried udp, and with and wi=
thout
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdirplus, with different read/write sizes (started with 32k, less =
for udp,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and then cranked it up to 1m to make the beachball appear less oft=
en),
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
but I still have issues no matter which options I chose)
Anyway, I'm stuck now, surfing the web with Safari is a very unple=
asant
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
experience on nfs, beachball every now and then together with a hu=
ge amount
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
of network traffic (RX with 20MB/s+ peaks), not unusual to see sev=
eral
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
gigabytes received after some minutes browsing, XCode shows a ''Th=
e
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
document "SomeFile.m" could not be saved.''-error after some edits=
, Opera
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min=
!
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extract=
ing the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running in=
side
Quoted 1 line
Quoted 1 line
Quoted 1 line
VMWare though), and still it was much faster (5:51.10 vs 0:09.35) t=
han
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0wI captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nf=
s.out
Quoted 1 line
Quoted 1 line
Quoted 1 line
host nfssrv and port 2049) on both freebsd8 (interface for freebsd =
is a bit
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
different ofc) and fedora12 while running
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/The number of packets is around 1800 for the FreeBSD server and arou=
nd
Quoted 1 line
Quoted 1 line
1940 for the Linux server. The RPC counts you posted in a later ema=
il
Quoted 1 line
Quoted 1 line
show that Linux does more LOOKUP and ACCESS requests. But generally=
, it
Quoted 1 line
Quoted 1 line
looks like your client is doing roughly the same amount of work in b=
oth
Quoted 4 lines
Quoted 4 lines
cases.
But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETA=
TTR
Quoted 8 lines
Quoted 7 lines
or COMMIT. This looks normal, since disk writes can take a few milliseconds.
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.
Quoted 4 lines
First of all, thanks for your answer Chuck :)
there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some=
extra
Quoted 3 lines
reads etc. sneaked in the linux tcpdump/nfsstat -s)
Just for the record, Solaris 10u8 (UFS) extraction time is almost the=
same as
Quoted 6 lines
FreeBSDs:
$ time tar xfz Cocoa-3rd.tgz 0.111u 1.621s 0:09.03 19.1% 0+0k 8+4966io 0pf+0wSo I guess they're both cheating ;-)
Is Solaris running in a guest too?
What physical file system are you using on the Linux NFS server?
--=20
chuck[dot]lever[at]oracle[dot]com
Quoted 18 lines · Chuck Lever
Quoted 17 lines · Stefan Kr=FCger
Quoted 15 lines · Chuck Lever
Quoted 14 lines · Stefan Kr=FCger
Quoted 12 lines · Stefan Kr=FCger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount =
options:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* nosuid,nodev,resvport,rdirplus,rwsize=3D1048576 (nfsv3 and tcp are default, I have also tried udp, and with and =
without
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdirplus, with different read/write sizes (started with 32k, les=
s for udp,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and then cranked it up to 1m to make the beachball appear less o=
ften),
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
but I still have issues no matter which options I chose)
Anyway, I'm stuck now, surfing the web with Safari is a very unp=
leasant
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
experience on nfs, beachball every now and then together with a =
huge
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
amount of network traffic (RX with 20MB/s+ peaks), not unusual t=
o see
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
several gigabytes received after some minutes browsing, XCode sh=
ows a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
''The document "SomeFile.m" could not be saved.''-error after so=
me
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
edits, Opera hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3m=
in!
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extra=
cting
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (runn=
ing
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0wI captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface fo=
r
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
freebsd is a bit different ofc) and fedora12 while running
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/The number of packets is around 1800 for the FreeBSD server and ar=
ound
Quoted 1 line
Quoted 1 line
Quoted 1 line
1940 for the Linux server. The RPC counts you posted in a later e=
Quoted 1 line
Quoted 1 line
Quoted 1 line
show that Linux does more LOOKUP and ACCESS requests. But general=
ly, it
Quoted 1 line
Quoted 1 line
Quoted 1 line
looks like your client is doing roughly the same amount of work in=
both
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
cases.
But what catches my eye in the F12 tcpdump is that there are pause=
s
Quoted 1 line
Quoted 1 line
Quoted 1 line
where the server reply is delayed by a few milliseconds after a SE=
TATTR
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
or COMMIT. This looks normal, since disk writes can take a few milliseconds.
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD =
is
Quoted 1 line
Quoted 1 line
Quoted 1 line
doing something illegal. No NFS server can turn a SETATTR around =
in
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.
First of all, thanks for your answer Chuck :)
there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so so=
me
Quoted 3 lines
Quoted 3 lines
extra reads etc. sneaked in the linux tcpdump/nfsstat -s)
Just for the record, Solaris 10u8 (UFS) extraction time is almost t=
he
Quoted 8 lines
Quoted 6 lines
same as FreeBSDs:
$ time tar xfz Cocoa-3rd.tgz 0.111u 1.621s 0:09.03 19.1% 0+0k 8+4966io 0pf+0wSo I guess they're both cheating ;-)
=20
Is Solaris running in a guest too?
Yes, this test was also done using VMWare
I also installed Fedora12/ext4 in VMWare and got ->
$ time tar xfz Cocoa-3rd.tgz 0.135u 2.151s 0:36.50 6.2% 0+0k 3+4770io 0pf+0w
Well, 36 seconds, still 3x longer than Solaris/FreeBSD but way better t= han
5min on bare metal
Anyway, to get some real data I splitted my raid1 mirror on the nfs ser= ver
yesterday and installed FreeBSD 8 on the (now) free disk
I don't know why but NFS performance is stunning, extracting the tgz on= ly
takes seconds, Opera doesn't hang for minutes when closing, no strange = rx
spikes and a lot less total bytes received on the same hardware with th= e same
files (no special tweaks done, only change on freebsd was to start 8 nf= sd
servers [default is 4], fs is UFS with softdeps)
I can't explain it...
Quoted 1 line
What physical file system are you using on the Linux NFS server?
See my first mail ("nfs shared /home is ext4 with default mount options= "),
so it's ext4 (I did some tests with xfs/ext3 using fedora12/VMWare, too= , same
bad results though)
Quoted 122 lines · Stefan Krüger
Quoted 96 lines · Chuck Lever
Quoted 95 lines · Stefan Krüger
Quoted 79 lines · Chuck Lever
Quoted 60 lines · Stefan Krüger
Quoted 37 lines · Stefan Krüger
Hello list,
I have some really strange nfs performance issues
NFS server is Fedora 12, running
* kernel-2.6.32.11-99.fc12.x86_64 and
* nfs-utils-1.2.1-4.fc12.x86_64
* nfs shared /home is ext4 with default mount options
/etc/exports:
/home 192.168.1.0/255.255.255.0(rw,sync)nfs and nfslock are up and running
Nothing else touched on the server nfs-wise.
NFS client is Mac OS X, version 10.6.3
My /home dir is automounted on the Mac with the following mount options:
* nosuid,nodev,resvport,rdirplus,rwsize=1048576
(nfsv3 and tcp are default, I have also tried udp, and with and without rdirplus, with different read/write sizes (started with 32k, less for udp, and then cranked it up to 1m to make the beachball appear less often), but I still have issues no matter which options I chose)Anyway, I'm stuck now, surfing the web with Safari is a very unpleasant experience on nfs, beachball every now and then together with a huge amount of network traffic (RX with 20MB/s+ peaks), not unusual to see several gigabytes received after some minutes browsing, XCode shows a ''The document "SomeFile.m" could not be saved.''-error after some edits, Opera hangs for minutes when closing, etc etc.
It's horrible :(
Another example, extracting
http://www.bignerdranch.com/solutions/Cocoa-3rd.tgz took over 3min!Quoted 6 lines
$ time tar xzf Cocoa-3rd.tgz 0.169u 3.198s 5:51.10 0.9% 0+0k 1+6972io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd/ 0.014u 0.477s 0:45.59 1.0% 0+0k 1+1io 0pf+0wSo any help or hints really appreciated
So, no answers yet, but I did some more tests, i.e. I tried extracting the Cocoa-3rd.tgz (2.2MB, 12MB untar'ed) on FreeBSD 8.0-REL (running inside VMWare though), and still it was much faster (5:51.10 vs 0:09.35) than extracting on bare metal fedora12:
$ time tar xfz Cocoa-3rd.tgz 0.104u 1.474s 0:09.35 16.7% 0+0k 0+4896io 0pf+0w $ time rm -rf Solutions-Cocoa-3rd 0.006u 0.160s 0:01.24 12.9% 0+0k 0+0io 0pf+0wI captured the nfs traffic with tcpdump (tcpdump -i eth1 -s 0 -w nfs.out host nfssrv and port 2049) on both freebsd8 (interface for freebsd is a bit different ofc) and fedora12 while running
Quoted 6 lines
tar xfz Cocoa-3rd.tgz Solutions-Cocoa-3rd/02_GetStarted
(which extracts just a couple of files) , you can find them here:
Fedora 12 tcpdump -> http://www.dpaste.org/5cvp/
FreeBSD 8 tcpdump -> http://www.dpaste.org/uCGX/The number of packets is around 1800 for the FreeBSD server and around 1940 for the Linux server. The RPC counts you posted in a later email show that Linux does more LOOKUP and ACCESS requests. But generally, it looks like your client is doing roughly the same amount of work in both cases.
But what catches my eye in the F12 tcpdump is that there are pauses where the server reply is delayed by a few milliseconds after a SETATTR or COMMIT. This looks normal, since disk writes can take a few milliseconds.
Quoted 4 lines
FreeBSD doesn't appear to have these pauses, so I suspect FreeBSD is doing something illegal. No NFS server can turn a SETATTR around in just a few microseconds and claim that it is on permanent storage, unless it has some kind of NVRAM.
First of all, thanks for your answer Chuck :)
there are some additional packets because the .tgz was on the same (nfs-mounted) dir and on a local dir during the freebsd test (so some extra reads etc. sneaked in the linux tcpdump/nfsstat -s)
Just for the record, Solaris 10u8 (UFS) extraction time is almost the same as FreeBSDs:
Quoted 6 lines
Quoted 4 lines
$ time tar xfz Cocoa-3rd.tgz 0.111u 1.621s 0:09.03 19.1% 0+0k 8+4966io 0pf+0wSo I guess they're both cheating ;-)
Is Solaris running in a guest too?
Yes, this test was also done using VMWare
I also installed Fedora12/ext4 in VMWare and got ->$ time tar xfz Cocoa-3rd.tgz 0.135u 2.151s 0:36.50 6.2% 0+0k 3+4770io 0pf+0wWell, 36 seconds, still 3x longer than Solaris/FreeBSD but way better than 5min on bare metal
Anyway, to get some real data I splitted my raid1 mirror on the nfs server yesterday and installed FreeBSD 8 on the (now) free disk
I don't know why but NFS performance is stunning, extracting the tgz only takes seconds, Opera doesn't hang for minutes when closing, no strange rx spikes and a lot less total bytes received on the same hardware with the same files (no special tweaks done, only change on freebsd was to start 8 nfsd servers [default is 4], fs is UFS with softdeps)
Quoted 7 lines
I can't explain it...
Quoted 1 line
What physical file system are you using on the Linux NFS server?
See my first mail ("nfs shared /home is ext4 with default mount options"), so it's ext4 (I did some tests with xfs/ext3 using fedora12/VMWare, too, same bad results though)
So, VMware is probably not waiting for disk writes to complete, since NFS servers in VMware guests run faster than native. Note to self: never put mission critical data on NFS servers running as VMware guests.
Still, the Linux NFS server has some kind of problem here when comparing apples to apples. Does increasing the number of nfsd threads on the server help?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 3 lines · Chuck Lever
So, VMware is probably not waiting for disk writes to complete, since NFS servers in VMware guests run faster than native. Note to self: never put mission critical data on NFS servers running as VMware guests.
yeah I only did this for testing...
Quoted 3 lines
Still, the Linux NFS server has some kind of problem here when comparing apples to apples. Does increasing the number of nfsd threads on the server help?
Increasing the number of nfsd threads did not help (I raised them from 8 to 32 but Fedora is still crawling when extracting the tgz file, nfsstat numbers below are with 8 nfsd threads on both Fedora and FreeBSD)
So speaking of nfsstat, I've got some new statistics, this time I extracted the whole tgz and not just a single directory (+ files) inside it (statistics zeroed before ofc, both tests run on bare metal and on same hardware, network, etc.)
Fedora 12:
Server rpc stats:
calls badcalls badauth badclnt xdrcall 45960 0 0 0 0
Server nfs v3:
null getattr setattr lookup access readlink
1 0% 641 1% 6885 14% 10373 22% 18336 39% 0 0%
read write create mkdir symlink mknod
1917 4% 2552 5% 1999 4% 228 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
633 1% 0 0% 3 0% 0 0% 0 0% 3 0%
fsstat fsinfo pathconf commit
21 0% 1 0% 1 0% 2366 5%
FreeBSD 8:
Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove
628 6421 5287 3 703 2490 1991 629
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 229 2 0 2 6526
Mknod Fsstat Fsinfo PathConf Commit
0 5 1 1 2402
Server Ret-Failed
5267
Server Faults
0Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved
2490 2490 0
First thing that catches my eyes is Access, it's 3x higher on Fedora, next would be Lookup and Read, both 2x higher
MacOS 10.6 nfsstat client statistics after extracting on Fedora 12:
Client Info:
RPC Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove
640 6885 10380 0 1919 2553 1999 634
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 228 1 0 3 18353
Mknod Fsstat Fsinfo PathConf Commit
0 21 1 1 2367
RPC Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 45989
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
84529 5034 9021 9408 5235 1701 766 2553
BioRLHits Misses BioD Hits Misses DirE Hits Misses
0 0 1 3 4 0
... after extracting on FreeBSD 8:
Client Info:
RPC Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove
628 6421 5290 3 704 2490 1991 630
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
3 0 0 229 3 0 2 6538
Mknod Fsstat Fsinfo PathConf Commit
0 5 1 1 2402
RPC Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 27342
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
36508 1529 1817 4317 958 704 767 2490
BioRLHits Misses BioD Hits Misses DirE Hits Misses
1 3 1 7 3 0
Well this mostly matches what we've seen in the nfs server statistics, even though a lot more caching was done when using the Fedora nfs server
nfsstat manpage on OSX explains them as:
Attr Hits/Misses - Performance of the NFS file attribute cache. Lkup Hits/Misses - Performance of the directory name lookup cache. BioR Hits/Misses - Performance of block cache for reads. BioW Hits/Misses - Performance of block cache for writes. BioRL Hits/Misses - Performance of symbolic link cache. BioD Hits/Misses - Performance of directory cache.
DirE Hits/Misses - Performance of directory offset cache.
I've also captured the traffic from both sessions with tcpdump, it's quite huge though, 42MB for Fedora and 10MB for FreeBSD... let me know if you want to take a look
TIA
PS: Please let me say again that extraction time using FreeBSD nfs is 10 *seconds* while on Fedora it's over 5 *minutes*...