Performance expectations of NFS
Subsystems (2)
Hello there,
I run a small (10-node) diskless network in our company, and I've
been a user of Linux as server and client in it for many years now. Most of the time my users are very happy with the system, and it performs very reliably; however, I do get complaints that the network is slow, and I always want to advocate that diskless should be as fast enough as local disks (even though it really isn't!)
So a user today presented me with a real-world benchmark of an operation that he does regularly. It's basically issueing a copy of a code checkout:
% cp -r gazpacho gazpacho-new
This is a 33MB tree with 2674 files. Network is a switched 100Mbit/s, and you can netcat a file over HTTP at about 10MB/s from the server; I think that suggests the network is fine.
Unscientific average times for variations of this benchmark are as follows:
0:05 - Server: locally on RAID-5 0:06 - Client: copying from NFS mount to tmpfs 2:31 - Client: Copying from NFS mount to NFS mount
(i.e., the simple command above)
3:30 - Client: Copying from NFS mount to NFS mount over an existing
tree.
5:08 - Client: Copying from tmpfs to NFS mount
0:48 - Client: Deleting copy of tree on NFS mount
Now I know that file creation is synchronous, and that because of this, the NFS copy will /always/ run significantly slower. But I was amazed at just how much faster reading was than writing, and at how fast reading actually is -- it's much faster than I expected NFS could be.
I also found it interesting that copying from NFS mount to NFS mount was faster than copying from local filesystem to NFS mount; I imagine there is a protocol optimization for that common scenario.
But what I'd really like to know if there is a way of making the above operation run in significantly less time than it is today, or if our expectations are just unrealistic given the complexity of network filesystems. I'd be pretty happy with an operation that ran in, say, 40 seconds -- but 2:30 is a pretty long time. I've tried toying with rsize and wsize, async and sync, and tcp rmem and wmem proc settings. None of those changes makes a real difference. So my questions are:
Are there practical ways of speeding the operation up?
Would people with better hardware see significantly reduced times?
I appreciate any advice and discussion on the topic. Thanks!
Signature · 10 lines
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125 ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Hi Christian,
Quoted 8 lines
0:05 - Server: locally on RAID-5 0:06 - Client: copying from NFS mount to tmpfs 2:31 - Client: Copying from NFS mount to NFS mount(i.e., the simple command above)
3:30 - Client: Copying from NFS mount to NFS mount over an existing
tree.
5:08 - Client: Copying from tmpfs to NFS mount
0:48 - Client: Deleting copy of tree on NFS mount
I guess your exports has the "sync" option or your didn't set async or syn= c =
at all (sync is then the default)? Can you repeat the test again using the =
async option? But please, also read carefully the explanations of "man 5 =
exports" about this topic. =
Quoted 5 lines
Now I know that file creation is synchronous, and that because of this, the NFS copy will /always/ run significantly slower. But I was amazed at just how much faster reading was than writing, and at how fast reading actually is -- it's much faster than I expected NFS could be.
Just for fun, take ethereal and capture some data, once with the async and =
once with the sync option. Now have a look how much time each operation of =
the nfs client takes before it gets confirmed by the server. Especially loo= k =
at the commit calls, once with sync and once with async option.
Quoted 2 lines
But what I'd really like to know if there is a way of making the above operation run in significantly less time than it is today, or if
I guess if one could convince the nfs clients to reduce the number of commi= t =
calls, write operations would be much much faster. Don't know if that is =
possible at all, though.
Cheers,
Bernd
-- =
Bernd Schubert
PCI / Theoretische Chemie
Universit=E4t Heidelberg
INF 229
69120 Heidelberg
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE=
VDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Bernd Schubert
Hi Christian,
Quoted 8 lines
0:05 - Server: locally on RAID-5 0:06 - Client: copying from NFS mount to tmpfs 2:31 - Client: Copying from NFS mount to NFS mount(i.e., the simple command above)
3:30 - Client: Copying from NFS mount to NFS mount over an existing
tree.
5:08 - Client: Copying from tmpfs to NFS mount
0:48 - Client: Deleting copy of tree on NFS mount
Quoted 4 lines
I guess your exports has the "sync" option or your didn't set async or sync at all (sync is then the default)? Can you repeat the test again using the async option? But please, also read carefully the explanations of "man 5 exports" about this topic.
This /was/ using the async option. Using sync causes the times to worsen by about 70% (so the NFS to NFS copy takes about 5 minutes).
Quoted 4 lines
Just for fun, take ethereal and capture some data, once with the async and once with the sync option. Now have a look how much time each operation of the nfs client takes before it gets confirmed by the server. Especially look at the commit calls, once with sync and once with async option.
Yeah, that's a great idea; I should do that tomorrow.
Quoted 3 lines
I guess if one could convince the nfs clients to reduce the number of commit calls, write operations would be much much faster. Don't know if that is possible at all, though.
I get the feeling that the real bottleneck is synchronous file creation. What I would like to understand is whether this is done for reliability or to preserve filesystem semantics.
Signature · 10 lines
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125 ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs