NFS for millions of files
Subsystems (2)
Hi, I'm trying to setup a server that we can create millions of files on over NFS. When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Thanks in advance,
Jason
Have a look at these two kernel params - I'd recommend bumping them up to 128 (they're 16 by default).
sunrpc.tcp_slot_table_entries
sunrpc.udp_slot_table_entries
Keep in mind that this could also be a serialization issue. If you've got a 3ms latency, and you're performing all of your opens serially, you aren't going to get much faster. If you do the work in parallel you'll likely get substantially better numbers.
-Aaron
Quoted 23 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Thanks in advance,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Is this NFSv4 ? rsize and wsize > MTU size will cause fragmentation and performance issues...Try making it around 4k .....You used 1<<15 fir your example. if you don't do writes....then this shouldn't matter...and of course NFS is Nfs is not For Scalability.. You cannot get the same performance on NFS as you would get for localFS...May be you can try 10g....still there is TCP/UDP/IP stack overhead.....
Quoted 43 lines · Aaron Wiebe
Have a look at these two kernel params - I'd recommend bumping them up to 128 (they're 16 by default).
sunrpc.tcp_slot_table_entries
sunrpc.udp_slot_table_entriesKeep in mind that this could also be a serialization issue. If you've got a 3ms latency, and you're performing all of your opens serially, you aren't going to get much faster. If you do the work in parallel you'll likely get substantially better numbers.
-Aaron
Quoted 16 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Quoted 7 lines
Thanks in advance,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 8 lines
Best regards, Peter Chacko NetDiox computing systems, Network storage & OS training and research. Bangalore, India. www.netdiox.com 080 2664 0708
NFSv3, and I will try various other rsize/wsize.
Apologies for the accidental double post...
We are utilizing XFS with a separate logdev, dedicated raid 1 15k sas drives, and are already utilizing write caching. I guess I can try playing with a few other things. Thanks for all the pointers.
Jason
Quoted 64 lines · It is said that Peter Chacko, at Thu, 03 Sep 2009, wrote:
Is this NFSv4 ? rsize and wsize > MTU size will cause fragmentation and performance issues...Try making it around 4k .....You used 1<<15 fir your example. if you don't do writes....then this shouldn't matter...and of course NFS is Nfs is not For Scalability.. You cannot get the same performance on NFS as you would get for localFS...May be you can try 10g....still there is TCP/UDP/IP stack overhead.....
Quoted 43 lines · Aaron Wiebe
Have a look at these two kernel params - I'd recommend bumping them up to 128 (they're 16 by default).
sunrpc.tcp_slot_table_entries
sunrpc.udp_slot_table_entriesKeep in mind that this could also be a serialization issue. ?If you've got a 3ms latency, and you're performing all of your opens serially, you aren't going to get much faster. ?If you do the work in parallel you'll likely get substantially better numbers.
-Aaron
Quoted 16 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. ?When I run our creation benchmark locally ?I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. ?It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). ?Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Quoted 7 lines
Thanks in advance,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at ?http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at ?http://vger.kernel.org/majordomo-info.htmlSignature · 8 lines
Best regards, Peter Chacko NetDiox computing systems, Network storage & OS training and research. Bangalore, India. www.netdiox.com 080 2664 0708
Quoted 6 lines · Peter Chacko
Is this NFSv4 ? rsize and wsize > MTU size will cause fragmentation and performance issues...Try making it around 4k .....You used 1<<15 fir your example. if you don't do writes....then this shouldn't matter...and of course NFS is Nfs is not For Scalability.. You cannot get the same performance on NFS as you would get for localFS...May be you can try 10g....still there is TCP/UDP/IP stack overhead.....
Fragmentation won't hurt you that much, and it doesn't even apply to opens, since those operations are significantly smaller. The performance gains higher in the stack from a larger rsize/wsize generally outweigh the savings in frame size optimization. Of course, jumbo frames are always a good idea, but again, doesn't help here.
-Aaron
Of course fragmentation doesn't matter if there is no data IO...You true...I usually got significant performance with rszie/wsize around 4K, while MTU was at 1.5 k.....As you said right balance between the performance gains at the RPC/UDP stack and fragmentation overhead at L3/L2 stack....
May be you can try adding more threads on the server, and if it is MIPIO capable, you can find better performance....and lastly NFS is not meant for performance...Until you use NFSoRDMA pr pNFS....As Chuck mentioned, sequential LOG based file systems based NFS server offer better performance at a server storage level..
And if you have a real need, i have created an NFS load balancer that
brings superior performance(better than local FS for large meta data
intensive workloads or IO is spread to different files) for my previous employer...You can find more details here.. Its not open source...But i can have them contact you and give you a trial version...
http://www.calsoftlabs.com/whitepapers/net-nfs.html
thanks
Quoted 16 lines · Aaron Wiebe
Quoted 6 lines · Peter Chacko
Is this NFSv4 ? rsize and wsize > MTU size will cause fragmentation and performance issues...Try making it around 4k .....You used 1<<15 fir your example. if you don't do writes....then this shouldn't matter...and of course NFS is Nfs is not For Scalability.. You cannot get the same performance on NFS as you would get for localFS...May be you can try 10g....still there is TCP/UDP/IP stack overhead.....
Fragmentation won't hurt you that much, and it doesn't even apply to opens, since those operations are significantly smaller. The performance gains higher in the stack from a larger rsize/wsize generally outweigh the savings in frame size optimization. Of course, jumbo frames are always a good idea, but again, doesn't help here.
-Aaron
Signature · 8 lines
Best regards, Peter Chacko NetDiox computing systems, Network storage & OS training and research. Bangalore, India. www.netdiox.com 080 2664 0708
Quoted 5 lines · It is said that Aaron Wiebe, at Wed, 02 Sep 2009, wrote:
Have a look at these two kernel params - I'd recommend bumping them up to 128 (they're 16 by default).
sunrpc.tcp_slot_table_entries
sunrpc.udp_slot_table_entries
They were indeed 16. I've bumped them up to 128, and am trying again.
Quoted 4 lines
Keep in mind that this could also be a serialization issue. If you've got a 3ms latency, and you're performing all of your opens serially, you aren't going to get much faster. If you do the work in parallel you'll likely get substantially better numbers.
It appears it might just be serialization. When I run more than 1 thread, I can consistently get between 200 and 250 files/thread/second (up to about 10 threads, at which point I get diminishing returns per thread).
I seem to be doing a lot of GETATTR's, and it appears to be driven by perl. We are open()'ing files with +>filename and apparently perl will do an fstat for us after opening. Thanks for the tip, Aaron!
-j
Quoted 31 lines
-Aaron
Quoted 16 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. ?When I run our creation benchmark locally ?I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. ?It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). ?Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Quoted 7 lines
Thanks in advance,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at ?http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
Quoted 8 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
Hi.
Please keep in mind that the NFS stable storage requirements are probably causing a significant slowdown in activities such as this. All of the files being created over NFS are being flushed to stable storage, including modified directory contents and inode information, before the server responds to the client. All of those local file creates are simply manipulating in-memory buffers and are not being flushed to stable storage until some later time.
There really aren't any good solutions, except perhaps, for utilizing write caching on the server, and I would only recommend doing that if you have a good solution, from top to bottom in the file system and storage stack on the server. Most aren't.
ps
Quoted 2 lines · Peter Staubach
Please keep in mind that the NFS stable storage requirements are probably causing a significant slowdown in activities such as this.
My first thought too, but:
Quoted 4 lines
Quoted 3 lines · Jason Legate
When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
...
Quoted 2 lines
Quoted 2 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec).
The localhost NFS mount would be incurring the same sync latency, so all his latency must be due to network. (And with those numbers I guess he's either got lots of disk spindles, or an ssd, or (uh-oh) has the async option set?)
--b.
Quoted 19 lines · J. Bruce Fields
Quoted 3 lines · Peter Staubach
Please keep in mind that the NFS stable storage requirements are probably causing a significant slowdown in activities such as this.
My first thought too, but:
Quoted 6 lines
Quoted 4 lines · Jason Legate
When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
...
Quoted 3 lines
Quoted 3 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec).
Quoted 7 lines
The localhost NFS mount would be incurring the same sync latency, so all his latency must be due to network. (And with those numbers I guess he's either got lots of disk spindles, or an ssd, or (uh-oh) has the async option set?)
--b.
For small files without doing an fsync per file, getting 3000 files/sec is not that much. Ext3 can do it with a local s-ata disk. I suspect that Jason would run much slower if he ran with local fsync()'s enabled (similar to what NFS servers have to do).
Some quick testing on a local F12 (rawhide) box:
No fsync: 3729 files/sec
[root@ricdesktop rwheeler]# fs_mark -s 20480 -n 10000 -S 0 -d /test/test_dir
# fs_mark -s 20480 -n 10000 -S 0 -d /test/test_dir
# Version 3.3, 1 thread(s) starting at Thu Sep 3 14:26:26 2009 # Sync method: NO SYNC: Test does not issue sync() or fsync() calls. # Directories: no subdirectories used # File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name) # Files info: size 20480 bytes, written with an IO size of 16384 bytes per write # App overhead is time in microseconds spent in the test not doing file writing related system calls.
FSUse% Count Size Files/sec App Overhead
5 10000 20480 3729.0 315859
Fsync & working barriers: 24.8 files/sec
[root@ricdesktop rwheeler]# fs_mark -s 20480 -n 10000 -S 1 -d /test/test_dir
# fs_mark -s 20480 -n 10000 -S 1 -d /test/test_dir
# Version 3.3, 1 thread(s) starting at Thu Sep 3 14:27:00 2009 # Sync method: INBAND FSYNC: fsync() per file in write loop. # Directories: no subdirectories used # File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name) # Files info: size 20480 bytes, written with an IO size of 16384 bytes per write # App overhead is time in microseconds spent in the test not doing file writing related system calls.
FSUse% Count Size Files/sec App Overhead
5 10000 20480 24.8 350322
Fsync/no write barriers: 377.1 files/sec
[root@ricdesktop rwheeler]# umount /test/
[root@ricdesktop rwheeler]# mount -o barrier=0 /dev/sdb /test/ [root@ricdesktop rwheeler]# fs_mark -s 20480 -n 10000 -S 1 -d /test/test_dir
# fs_mark -s 20480 -n 10000 -S 1 -d /test/test_dir
# Version 3.3, 1 thread(s) starting at Thu Sep 3 14:36:27 2009 # Sync method: INBAND FSYNC: fsync() per file in write loop. # Directories: no subdirectories used # File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name) # Files info: size 20480 bytes, written with an IO size of 16384 bytes per write # App overhead is time in microseconds spent in the test not doing file writing related system calls.
FSUse% Count Size Files/sec App Overhead
5 10000 20480 377.1 328472
Ric
Quoted 25 lines · Ric Wheeler
Quoted 24 lines · J. Bruce Fields
Quoted 3 lines · Peter Staubach
Please keep in mind that the NFS stable storage requirements are probably causing a significant slowdown in activities such as this.
My first thought too, but:
Quoted 6 lines
Quoted 4 lines · Jason Legate
When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
...
Quoted 3 lines
Quoted 3 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec).
The localhost NFS mount would be incurring the same sync latency, so all his latency must be due to network. (And with those numbers I guess he's either got lots of disk spindles, or an ssd, or (uh-oh) has the async option set?)
Quoted 7 lines
Quoted 2 lines
--b.
For small files without doing an fsync per file, getting 3000 files/sec is not that much. Ext3 can do it with a local s-ata disk. I suspect that Jason would run much slower if he ran with local fsync()'s enabled (similar to what NFS servers have to do).
Actually, NFS servers have to do _two_ fsyncs. One when creating the file, and one when the client closes it. NFSv4 with write delegations could allow the client to delay the fsync on close, but does not allow it to eliminate the fsync-on-create.
In order help speed up those workloads we have therefore recently submitted a proposal to also delay the fsync-on-create. This proposal is for inclusion in the IETF's NFSv4 working group charter for NFSv4.2...
Cheers
Trond
Quoted 25 lines · Ric Wheeler
Quoted 24 lines · J. Bruce Fields
Quoted 3 lines · Peter Staubach
Please keep in mind that the NFS stable storage requirements are probably causing a significant slowdown in activities such as this.
My first thought too, but:
Quoted 6 lines
Quoted 4 lines · Jason Legate
When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
...
Quoted 3 lines
Quoted 3 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec).
The localhost NFS mount would be incurring the same sync latency, so all his latency must be due to network. (And with those numbers I guess he's either got lots of disk spindles, or an ssd, or (uh-oh) has the async option set?)
Quoted 7 lines
Quoted 2 lines
--b.
For small files without doing an fsync per file, getting 3000 files/sec is not that much. Ext3 can do it with a local s-ata disk. I suspect that Jason would run much slower if he ran with local fsync()'s enabled (similar to what NFS servers have to do).
Yeah, which is why I was suspecting they set "async" on the export. In which case I hope they know what they're doing....
--b.
Quoted 8 lines · Jason Legate
Hi, I'm trying to setup a server that we can create millions of files on over NFS. When I run our creation benchmark locally I can get around 3000 files/ second in the configuration we're using now, but only around 300/second over NFS. It's mounted as this:
rw,nosuid,nodev,noatime,nodiratime,hard,bg,nointr,rsize=32768,wsize=32768,tcp, nfsvers=3,timeo=600,actimeo=600,nocto
Quoted 7 lines
When I mount the same FS over localhost instead of across the lan, it performs about full speed (the 3000/sec). Anyone have any ideas what I might tweak or look at?
We're going to be testing various XFS/LVM configs to get the best performance, but right out the gate, NFS having a 10:1 penalty of performance doesn't bode well.
If you are using a slow LAN (like 100Mb/s) that might be a problem.
Metadata operations (like file creation) are always slower on NFS than on local file systems. There is significantly more serialization involved for NFS since access to the file system is shared across multiple systems.
You might consider a cluster file system instead of NFS if you are driving metadata-intensive workloads while sharing amongst only local clients. Or, if you can install a fast log device for your server file system, that might mitigate the disk waits during each file creation.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com