RH 7.3 kernels and NFS performance
Subsystems (2)
Hello,
I remember reading on the list about the terrible NFS
performance of the latest RH kernels. Would someone care to summarize this for me (UDP or TCP, etc.)? Also, does anyone have any rough estimates of the performance hit?
Thanks!
Jeff Layton
Lockheed-Martin Aeronautical Company - Marietta
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · Jeff Layton
I remember reading on the list about the terrible NFS
performance of the latest RH kernels. Would someone care to summarize this for me (UDP or TCP, etc.)? Also, does anyone have any rough estimates of the performance hit?
The Hit is big and bad. When using the default NFS export options (in=20 particular sync), I see NFS write performance of ~50k with the 2.4.18-5=20 kernels. When set async, performance comes back up to par. I can give y= ou=20
the bugzilla # I reported it under, if you like.
-- Rex
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Rex Dieter <[email protected]> writes:
> The Hit is big and bad. When using the default NFS export > options (in particular sync), I see NFS write performance of > ~50k with the 2.4.18-5 kernels. When set async, performance > comes back up to par. I can give you the bugzilla # I reported > it under, if you like.
That is *not* a bug. The NFS protocol does not actually allow for the sort of behaviour that 'async' provides. The main problem is that actions like 'fsync()' on the client can no longer guarantee that the data is physically written to disk on the server. IOW if the server crashes, your data is lost...
The new default of 'sync' for exports is quite correct and should only be changed by people who are aware of the consequences for data integrity.
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " =3D=3D Rex Dieter <[email protected]> writes:
> The Hit is big and bad. When using the default NFS export > options (in particular sync), I see NFS write performance of > ~50k with the 2.4.18-5 kernels. When set async, performance > comes back up to par. I can give you the bugzilla # I reported > it under, if you like.
That is *not* a bug.=20
It depends. I agree that defaulting to 'sync' is not a bug. I do *not*=20 agree that 50k writes is not a bug.
-- Rex
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Rex Dieter <[email protected]> writes:
> It depends. I agree that defaulting to 'sync' is not a bug. I > do *not* agree that 50k writes is not a bug.
NFSv2 or v3?
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " =3D=3D Rex Dieter <[email protected]> writes:
> It depends. I agree that defaulting to 'sync' is not a bug. I > do *not* agree that 50k writes is not a bug.
NFSv2 or v3?
I tried primarily v3, but I think I remember testing v2 as well, with=20 similar results.
-- Rex
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Rex Dieter <[email protected]> writes:
> On Wednesday 07 August 2002 11:17 am, Trond Myklebust wrote:
>> >>>>> " " == Rex Dieter <[email protected]> writes:
>> > It depends. I agree that defaulting to 'sync' is not a bug. >> > I do *not* agree that 50k writes is not a bug.
>>
>> NFSv2 or v3?
> I tried primarily v3, but I think I remember testing v2 as > well, with similar results.
If you are getting that performance with v3, then it does indeed sound like a bug somewhere. Try with a stock 2.4.19 kernel instead...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Rex Dieter <[email protected]> writes:
> On Wednesday 07 August 2002 11:17 am, Trond Myklebust wrote:
>> >>>>> " " == Rex Dieter <[email protected]> writes:
>> > It depends. I agree that defaulting to 'sync' is not a bug. >> > I do *not* agree that 50k writes is not a bug.
>>
Quoted 7 lines
>> NFSv2 or v3?
> I tried primarily v3, but I think I remember testing v2 as > well, with similar results.
If you are getting that performance with v3, then it does indeed sound like a bug somewhere. Try with a stock 2.4.19 kernel instead...
I just tested things with a test kernel (2.4.18-7) from redhat, and it sync write speeds were again way up there. Problem fixed, as far as I'm concerned... hopefully, they'll release an errata soon.
Signature · 14 lines
Rex A. Dieter [email protected] Computer System Administrator http://www.math.unl.edu/~rdieter/ Mathematics and Statistics University of Nebraska Lincoln ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Rex Dieter
Quoted 3 lines · Trond Myklebust
If you are getting that performance with v3, then it does indeed sound like a bug somewhere. Try with a stock 2.4.19 kernel instead...
I just tested things with a test kernel (2.4.18-7) from redhat, and it sync write speeds were again way up there. Problem fixed, as far as I'm concerned... hopefully, they'll release an errata soon.
Quoted 5 lines
--
Rex A. Dieter [email protected]
Computer System Administrator http://www.math.unl.edu/~rdieter/
Mathematics and Statistics
University of Nebraska Lincoln
and
Quoted 15 lines · Charles
hi jeff-
Quoted 5 lines
I remember reading on the list about the terrible NFS
performance of the latest RH kernels. Would someone care to summarize this for me (UDP or TCP, etc.)? Also, does anyone have any rough estimates of the performance hit?if you are referring to bad NFS client performance, this is due to a bug in the Linux IP fragmentation logic which causes it to send part of a fragmented packet, and drop the rest, if it runs out of socket buffer space during the fragmentation process.
Quoted 9 lines
thus it only affects NFS over UDP.
there is an easy workaround: enlarge the size of the RPC transport socket's buffers. see the NFS FAQ for instructions.
a bug was reported in the eepro100 driver too, and that may have some effect on client performance.
So if this is one and the same bug, then enlarging the transport socket size is actually discussed in the performance section of the howto documents (section 5.7) at
http://nfs.sourceforge.net/nfs-howto/performance.html#MEMLIMITS
Tom
--
------------------------------------------------------------
Tom McNeal [email protected] (650)906-0761 (cell)
------------------------------------------------------------
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · Rex Dieter
I just tested things with a test kernel (2.4.18-7) from redhat, and it sync write speeds were again way up there. Problem fixed, as far as I'm concerned... hopefully, they'll release an errata soon.
BTW, Arjan has included many of Trond's RPC patches in the latest Rawhide kernels:
$ rpm -qp --changelog kernel-2.4.18-7.93.src.rpm | head -4 * Fri Aug 02 2002 Arjan van de Ven <[email protected]>
- added most of the NFS patchkit
Based on the patch file names, these look like Trond's 2.4.19-rc3 patches:
linux-2.4.19-00-fix_clnt.patch
linux-2.4.19-01-fix_kmap1.patch
linux-2.4.19-02-fix_kmap2.patch
linux-2.4.19-03-fix_kmap3.patch
linux-2.4.19-04-rpc_rep.patch
linux-2.4.19-05-rpc_rtt1.patch
linux-2.4.19-06-rpc_rtt2.patch
linux-2.4.19-07-rpc_rtt3.patch
linux-2.4.19-08-xprt_write.patch
linux-2.4.19-09-rpc_cong1.patch
linux-2.4.19-10-rpc_cong2.patch
linux-2.4.19-11-rpc_cong3.patch
linux-2.4.19-12-rpc_wspace.patch
linux-2.4.19-13-rpc_cleanup.patch
linux-2.4.19-16-rpcbuf.patch
This should be nearly current, though Trond's 2.4.19 patches are broken down somewhat differently.
I hope that Trond is planning to submit most of this to Marcelo for 2.4.20, as Neil is submitting his NFS server patches, and it would be very nice to have a 2.4.20 that is fully up-to-date w.r.t current NFS patches for both client and server. [Those of us on this list for a long time will remember a similar exercise with Alan for 2.2.19 -- gosh, that seems like ancient history. :-P]
In any case, Trond's work should receive some more widespread "end-user" testing from the Limbo Beta and Rawhide crowd in the near future.
I've long pined [since RH 5.x] for a Red Hat kernel with working NFS that played nice with the NetApp and Solaris. It has since become a ritual to patch NFS into the Red Hat tree.
Many thanks to Trond and Neil for their tireless efforts!
It is difficult to underestimate how important working NFS is in deployments like ours ...
Regards
Bill Rugolsky
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines
Quoted 5 lines
> I tried primarily v3, but I think I remember testing v2 as > well, with similar results.
If you are getting that performance with v3, then it does indeed sound like a bug somewhere. Try with a stock 2.4.19 kernel instead...
I just tested things with a test kernel (2.4.18-7) from redhat, and it sync write speeds were again way up there. Problem fixed, as far as I'm concerned... hopefully, they'll release an errata soon.
look carefully - the only patches in 2.4.18-X (where X >5) are to misc network drivers. 2.4.18-5 with some network cards (specifically eepros and 3c905s) were terribly slow. 2.4.18-5e+ fix it.
-sv
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
----- Original Message -----
From: "Seth Vidal" <[email protected]>
Quoted 1 line
Quoted 1 line
I just tested things with a test kernel (2.4.18-7) from redhat, and i=
t
Quoted 1 line
Quoted 1 line
sync write speeds were again way up there. Problem fixed, as far as =
I'm
Quoted 4 lines
Quoted 1 line
concerned... hopefully, they'll release an errata soon.
look carefully - the only patches in 2.4.18-X (where X >5) are to misc network drivers. 2.4.18-5 with some network cards (specifically eepros =
and
Quoted 1 line
3c905s) were terribly slow. 2.4.18-5e+ fix it.
I didn't look carefully, and it's changelog didn't reveal anything, but I= =20
can tell you that I HAD tried kernel-2.4.18-5e, and the problem persisted= =2E =20
As of 2.4.18-7, the problem is gone. I may have to take a closer look=20 now... I'm curious.
-- Rex
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs