2.4 vs. 2.6 nfs client performance
Subsystems (2)
Hi,
I'm running some NFS performance tests to determine the best way to go to configure a few fileservers for some linux clusters I run. Right now I'm getting together a suite of test programs representative of the typical applications we see run on our clusters (scientific applications such as Ansys or Abaqus, custom MPI code, etc.) for use as a benchmark suite that I can disperse across 16 machines or so to create a decent load on the fileservers. I've just got started with a program called Gaussian03 (molecular modelling code that does a lot of I/O) and I'm already seeing some odd performance differences between the 2.4.22 client and the 2.6.0-test9-mm1 client (both against a 2.6.0-test9-mm2 server):
async mounts
------------
2.4.22: 110.87user 43.69system 4:04.58elapsed 63%CPU
2.6.0-test9-mm1: 111.88user 315.57system 9:51.87elapsed 72%CPU
sync mounts
-----------
2.4.22: 109.99user 45.49system 32:04.44elapsed 8%CPU
2.6.0-test9-mm1: 112.33user 197.76system 1:08:13elapsed 7%CPU
Note that the 1:08:13 in the sync 2.6.0-test9-mm1 is 1 *hour*, 8 minutes, not 1 *minute*. In both cases the 2.6 client came in at about twice the time. A local run not using NFS finishes in 2:13.49.
The nfsstats output for two async runs looks like:
-- 2.4.22 --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 880 0% 0 0% 160 0% 4745 0% 0 0%
read write create mkdir symlink mknod
780 0% 178588 32% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
3 0% 3 0% 0 0% 369185 66%
Client rpc stats:
calls retrans authrefrsh 554410 2099 0
-- 2.6.0-test9-mm1 --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 740 0% 0 0% 126 0% 249 0% 0 0%
read write create mkdir symlink mknod
855 0% 178574 14% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 1023777 85%
Client rpc stats:
calls retrans authrefrsh 1204390 5 0
The 2.6 client has over twice the number of RPC calls and almost 3 times the number of commits whereas the 2.4 client has alot more accesses. Mounts were done with rsize=32768,wsize=32768. The NFS filesystem is ext3. The two machines are connected via gigabit ethernet and the traffic between them never goes above 20-30 MB/s.
Can someone clue me in as to why this may be happening and if it's a "bug" or not?
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> Can someone clue me in as to why this may be happening and if > it's a "bug" or not?
Large numbers of 'commit' operations are a typical sign of the memory management going wild. Have you compared this to the standard 2.6.0-test9 kernel?
What sort of load did you otherwise have on that machine?
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Jason Holmes <[email protected]> writes:
> Can someone clue me in as to why this may be happening and if > it's a "bug" or not?
Large numbers of 'commit' operations are a typical sign of the memory management going wild. Have you compared this to the standard 2.6.0-test9 kernel?
What sort of load did you otherwise have on that machine?
The machines (both client and server) are idle other than the testing I'm doing and I'm doing only one test at a time at this point.
I did not do plain 2.6.0-test9 yet, but here is 2.6.0-test6 and 2.6.0-test9-bk17:
-- 2.6.0-test6 --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 749 0% 0 0% 130 0% 251 0% 0 0%
read write create mkdir symlink mknod
833 0% 178577 14% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 1 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 1024801 85%
Client rpc stats:
calls retrans authrefrsh 1205411 7 0
-- 2.6.0-test9-bk17 --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 744 0% 0 0% 129 0% 252 0% 0 0%
read write create mkdir symlink mknod
873 0% 178574 14% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 1023451 84%
Client rpc stats:
calls retrans authrefrsh 1204092 3 0
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> Client nfs v3: null getattr setattr lookup access readlink 0 0%
> 744 0% 0 0% 129 0% 252 0% 0 0% read write create mkdir symlink
> mknod 873 0% 178574 14% 24 0% 0 0% 0 0% 0 0% remove rmdir
> rename link readdir readdirplus 24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
> fsstat fsinfo pathconf commit 0 0% 3 0% 0 0% 1023451 84%
> Client rpc stats: calls retrans authrefrsh 1204092 3 0Something is seriously screwed up here: it should be impossible to get more commit requests than there are write requests.
Basically, commit is only sent if at least one unstable write has been sent, and is waiting on the commit list. The commit list itself is emptied before the RPC call gets sent. If the commit fails, it resends all pending writes.
IOW: it looks like you are seeing some form of list corruption. I suspect that the problem here might be something scribbling over the nfs_inode. Are you running with slab debugging, and stack overflow enabled?
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 19 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Jason Holmes <[email protected]> writes:
> Client nfs v3: null getattr setattr lookup access readlink 0 0% > 744 0% 0 0% 129 0% 252 0% 0 0% read write create mkdir symlink > mknod 873 0% 178574 14% 24 0% 0 0% 0 0% 0 0% remove rmdir > rename link readdir readdirplus 24 0% 0 0% 0 0% 0 0% 18 0% 0 0% > fsstat fsinfo pathconf commit 0 0% 3 0% 0 0% 1023451 84% > Client rpc stats: calls retrans authrefrsh 1204092 3 0Something is seriously screwed up here: it should be impossible to get more commit requests than there are write requests.
Basically, commit is only sent if at least one unstable write has been sent, and is waiting on the commit list. The commit list itself is emptied before the RPC call gets sent. If the commit fails, it resends all pending writes.
Quoted 4 lines
IOW: it looks like you are seeing some form of list corruption. I suspect that the problem here might be something scribbling over the
nfs_inode. Are you running with slab debugging, and stack overflow enabled?
I wasn't, but I'm recompiling test9-bk17 with all of the debugging options turned on right now. Is there anything in particular you want me to do?
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> I wasn't, but I'm recompiling test9-bk17 with all of the > debugging options turned on right now. Is there anything in > particular you want me to do?
Come to think of it, could you try running with and without the patch on
http://www.fys.uio.no/~trondmy/src/Linux-2.6.x/2.6.0-test9/linux-2.6.0-01-fix_deadlock.dif
That fixes a race that is likely to show up when memory is low. Normally it should cause a deadlock rather than corruption, but it is conceivable that there are other side-effects...
Please make sure that you run with slab debugging enabled...
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 10 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Jason Holmes <[email protected]> writes:
> I wasn't, but I'm recompiling test9-bk17 with all of the > debugging options turned on right now. Is there anything in > particular you want me to do?
Come to think of it, could you try running with and without the patch on
Quoted 7 lines
http://www.fys.uio.no/~trondmy/src/Linux-2.6.x/2.6.0-test9/linux-2.6.0-01-fix_deadlock.dif
That fixes a race that is likely to show up when memory is low. Normally it should cause a deadlock rather than corruption, but it is conceivable that there are other side-effects...
Please make sure that you run with slab debugging enabled...
With all debugging enabled, test9-bk17:
patched: 112.21user 259.86system 10:53.48elapsed 56%CPU unpatched: 112.17user 252.35system 10:37.43elapsed 57%CPU
-- patched --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 752 0% 0 0% 134 0% 260 0% 0 0%
read write create mkdir symlink mknod
873 0% 178571 14% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 1044537 85%
Client rpc stats:
calls retrans authrefrsh 1225196 8 0
-- unpatched --
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 753 0% 0 0% 134 0% 260 0% 0 0%
read write create mkdir symlink mknod
873 0% 178575 14% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 1050417 85%
Client rpc stats:
calls retrans authrefrsh 1231081 5 0
I didn't see anything to indicate that any debug code had triggered (I'm assuming that if it did the kernel would give an OOPS?)
FWIW, I ran the test with nfsvers=2 (with the export still being async) and it finished in 112.53user 45.02system 2:43.50elapsed 96%CPU.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> FWIW, I ran the test with nfsvers=2 (with the export still > being async) and it finished in 112.53user 45.02system > 2:43.50elapsed 96%CPU.
For comparison, on my own machine, I get stats of the form:
Client rpc stats:
calls retrans authrefrsh 248291 6225 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 77 0% 0 0% 60 0% 94 0% 0 0%
read write create mkdir symlink mknod
0 0% 240378 96% 8 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
33 0% 0 0% 0 0% 0 0% 0 0% 12 0%
fsstat fsinfo pathconf commit
0 0% 1 0% 0 0% 7628 3%
Any hints on how I can reproduce your results?
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 30 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Jason Holmes <[email protected]> writes:
> FWIW, I ran the test with nfsvers=2 (with the export still > being async) and it finished in 112.53user 45.02system > 2:43.50elapsed 96%CPU.
For comparison, on my own machine, I get stats of the form:
Client rpc stats:
calls retrans authrefrsh 248291 6225 0Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%Client nfs v3:
null getattr setattr lookup access readlink
0 0% 77 0% 0 0% 60 0% 94 0% 0 0%
read write create mkdir symlink mknod
0 0% 240378 96% 8 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
33 0% 0 0% 0 0% 0 0% 0 0% 12 0%
fsstat fsinfo pathconf commit
0 0% 1 0% 0 0% 7628 3%
Quoted 1 line
Any hints on how I can reproduce your results?
Unfortunately Gaussian's licensing would prohibit me from passing it along to reproduce this. I'll try to reproduce this with another program this morning and get back to you.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Jason Holmes
Quoted 2 lines · Trond Myklebust
Any hints on how I can reproduce your results?
Unfortunately Gaussian's licensing would prohibit me from passing it along to reproduce this. I'll try to reproduce this with another program this morning and get back to you.
OK, I haven't found any other program that behaves similar to Gaussian on this, but I did do a few more runs with 2.6.0-test9-bk17 vs 2.4.22, one set with nfs_debug on and one set with rpc_debug on. I haven't analyzed the files in depth (and doing so would really be beyond me at this point), but on a "dumb" level, the nfs files are of similar length in terms of lines, but the RPC file for 2.6.0-test9-bk17 is much larger:
[root@fs4 log]# wc -l kern.*
16236 kern.nfs-2.4.22
17188 kern.nfs-2.6.0-test9-bk17
20565 kern.rpc-2.4.22
32684 kern.rpc-2.6.0-test9-bk17
Note that there is probably some small amount of noise in these files, but 98% of it should be from the Gaussian runs. I put them up at:
http://magicbus.cac.psu.edu/nfs
if you want to look at them. One small observation is that 2.4.22 reaches much larger values of cwnd than test9-bk17, resulting in a lot fewer calls to functions such as xprt_cwnd_limited (1861 vs. 2715). Whether or not this means anything, I don't know.
Trond, if none of this helps and you really want to debug this, I can probably give you access to two machines to do so.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> if you want to look at them. One small observation is that > 2.4.22 reaches much larger values of cwnd than test9-bk17, > resulting in a lot fewer calls to functions such as > xprt_cwnd_limited (1861 vs. 2715). Whether or not this means > anything, I don't know.
It means that the UDP throughput is slightly more stable. However the thing that surprises me there is that in both cases 'xprt_cwnd_limited cong = 0' is the dominant case. That means basically that you only have one outstanding request on the wire at any point in time (there are only a few cases where cong=256 => 2 requests, and cong=512 => 3 requests).
IOW this all appears to be doing synchronous I/O only (both in the case of 2.4.22 and 2.6.0).
Note: most of these RPC calls do indeed appear to be something other than WRITE calls. Are they perhaps GETATTR calls? Have you for instance set 'noac', 'actimeo' or something like that?
Could you please try using
echo "9" >/proc/sys/sunrpc/nfs_debug
rather than echo "1". That should give us the debugging info from nfs/read.c and nfs/write.c in addition to what you've already got.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Trond Myklebust
IOW this all appears to be doing synchronous I/O only (both in the case of 2.4.22 and 2.6.0).
Note: most of these RPC calls do indeed appear to be something other than WRITE calls. Are they perhaps GETATTR calls? Have you for instance set 'noac', 'actimeo' or something like that?
This was my mistake - I had changed the export to sync for a test right before the debug runs and forgot to change it back.
Quoted 3 lines
Could you please try using
echo "9" >/proc/sys/sunrpc/nfs_debug
Files should be up there (http://magicbus.cac.psu.edu/nfs). I do have to admit that something odd is going on - 2.4.22 still is faster than 2.6.0-test9-bk17 and test9-bk17 is still showing the large amount of commits, but both of them are running faster than they did before on the sync mounts and I'm not sure what I might have changed to cause this.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jason Holmes <[email protected]> writes:
> Files should be up there (http://magicbus.cac.psu.edu/nfs). I > do have to admit that something odd is going on - 2.4.22 still > is faster than 2.6.0-test9-bk17 and test9-bk17 is still showing > the large amount of commits, but both of them are running > faster than they did before on the sync mounts and I'm not sure > what I might have changed to cause this.
Hmm... It looks like you are doing a succession of 4k commits. I'll bet the read() code is involved in this...
Could you try the following patch and see if it improves things?
Cheers,
Trond
fs/nfs/write.c renamed from fs/nfs/write.c.orig +1 -1
@@ -1103,7 +1103,7 @@ error =nfs_flush_file (inode, file, idx_start, npages, how);#if defined(CONFIG_NFS_V3 ) || defined(CONFIG_NFS_V4 )if (error ==0 ) - error =nfs_commit_file (inode, file, idx_start, npages, how); + error =nfs_commit_file (inode, file,0 ,0 , how);#endif }while (error >0 );return error;
-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 10 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Jason Holmes <[email protected]> writes:
> Files should be up there (http://magicbus.cac.psu.edu/nfs). I > do have to admit that something odd is going on - 2.4.22 still > is faster than 2.6.0-test9-bk17 and test9-bk17 is still showing > the large amount of commits, but both of them are running > faster than they did before on the sync mounts and I'm not sure > what I might have changed to cause this.
Quoted 4 lines
Hmm... It looks like you are doing a succession of 4k commits. I'll bet the
read()code is involved in this...Could you try the following patch and see if it improves things?
That made a huge difference:
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 714 0% 0 0% 110 0% 204 0% 0 0%
read write create mkdir symlink mknod
873 0% 178571 92% 24 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 3 0% 0 0% 13175 6%
Client rpc stats:
calls retrans authrefrsh 193716 13 0
It also cut the job time in about half. I put a level9 log of it up with the other ones. I'll have to rerun the 2.4 vs. this, but I think that this change makes the 2.6 client faster than the 2.4 for this case.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Trond Myklebust
Any hints on how I can reproduce your results?
Trond,
This is not the same test, nor the same results as Jason's, but here is some more data from bonnie and dd tests.This shows some write performance difference between 2.4 and 2.6
eric
SERVER: 2.4.21
CLIENT: 2.4.19 nfsv3
Bonnie
Writing with putc()... done: 9132 kB/s 33.4 %CPU Rewriting... done: 8713 kB/s 4.2 %CPU Writing intelligently...done: 9116 kB/s 4.6 %CPU
$ rm file;time dd if=/dev/zero of=file bs=1 count=10000000 10000000+0 records in 10000000+0 records out
real 0m30.404s user 0m3.930s sys 0m26.250s
CLIENT: 2.6.0-test6-mm4 Writing with putc()... done: 806 kB/s 5.4 %CPU Rewriting... done: 3321 kB/s 3.9 %CPU Writing intelligently...done: 2715 kB/s 2.4 %CPU
$ rm file;time dd if=/dev/zero of=file bs=1 count=10000000 10000000+0 records in 10000000+0 records out
real 1m37.848s user 0m17.355s sys 1m8.947s
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> This is not the same test, nor the same results as Jason's, but > here is some more data from bonnie and dd tests.This shows some > write performance difference between 2.4 and 2.6 eric
How about vs. 2.4.23?
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Trond Myklebust
> This is not the same test, nor the same results as Jason's, but > here is some more data from bonnie and dd tests.This shows some > write performance difference between 2.4 and 2.6 eric
How about vs. 2.4.23?
2.4.23 looks good -- see below -- these are acceptable numbers for a 100M network. I reran the 2.6.0-test6-mm4 benchmarks and it still is much slower than the the 2.4.23.
Question for Jason: -- are you running the 4/4 vm layout in your kernel? My 2.6.0-test6-mm4 has the 4/4 vm enabled -- which can have an impact on performance. I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.
eric
Linux host 2.4.23-rc1 #2 SMP Thu Nov 13 11:30:02 MST 2003 i686 athlon i386 GNU/Linux
/test> bonnie -s 100
Writing with putc()... done: 9228 kB/s 52.0 %CPU Rewriting... done: 9435 kB/s 6.0 %CPU Writing intelligently...done: 9235 kB/s 4.0 %CPU
/test> rm file;time dd if=/dev/zero of=file bs=1 count=10000000 10000000+0 records in
10000000+0 records out
real 0m20.086s user 0m3.120s sys 0m16.740s
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Eric Whiting
Quoted 6 lines · Trond Myklebust
> This is not the same test, nor the same results as Jason's, but > here is some more data from bonnie and dd tests.This shows some > write performance difference between 2.4 and 2.6 eric
How about vs. 2.4.23?
I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.
2.6.0-test9 (without mm1 4/4 vm) is even worse than 2.6.0-test6-mm4.
tcpdump shows very slow writes..
12:07:12.380001 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33858:1480@0+)
12:07:12.380012 172.16.17.176 > 172.16.32.181: udp (frag 33858:1480@1480+) 12:07:12.380014 172.16.17.176 > 172.16.32.181: udp (frag 33858:1284@2960) 12:07:17.979092 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33859:1480@0+)
12:07:17.979102 172.16.17.176 > 172.16.32.181: udp (frag 33859:1480@1480+) 12:07:17.979104 172.16.17.176 > 172.16.32.181: udp (frag 33859:1284@2960) 12:07:18.678981 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33860:1480@0+)
12:07:18.678992 172.16.17.176 > 172.16.32.181: udp (frag 33860:1480@1480+) 12:07:18.678994 172.16.17.176 > 172.16.32.181: udp (frag 33860:1284@2960) 12:07:20.078755 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33861:1480@0+)
12:07:20.078766 172.16.17.176 > 172.16.32.181: udp (frag 33861:1480@1480+) 12:07:20.078768 172.16.17.176 > 172.16.32.181: udp (frag 33861:1284@2960) 12:07:22.879310 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33862:1480@0+)
12:07:22.879321 172.16.17.176 > 172.16.32.181: udp (frag 33862:1480@1480+) 12:07:22.879323 172.16.17.176 > 172.16.32.181: udp (frag 33862:1284@2960) 12:07:28.479388 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33863:1480@0+)
12:07:28.479399 172.16.17.176 > 172.16.32.181: udp (frag 33863:1480@1480+) 12:07:28.479400 172.16.17.176 > 172.16.32.181: udp (frag 33863:1284@2960) 12:07:29.179275 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs] (frag 33864:1480@0+)
12:07:29.179286 172.16.17.176 > 172.16.32.181: udp (frag 33864:1480@1480+)
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Eric Whiting <[email protected]> writes:
> 2.6.0-test9 (without mm1 4/4 vm) is even worse than
> 2.6.0-test6-mm4.
> tcpdump shows very slow writes..Were you filtering away replies from the server, or were they really missing? If the latter, then that points towards a problem in the socket layer or possibly a problem with the device driver for your NIC.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · Eric Whiting
Quoted 6 lines · Trond Myklebust
> This is not the same test, nor the same results as Jason's, but > here is some more data from bonnie and dd tests.This shows some > write performance difference between 2.4 and 2.6 eric
How about vs. 2.4.23?
Quoted 7 lines
2.4.23 looks good -- see below -- these are acceptable numbers for a 100M network. I reran the 2.6.0-test6-mm4 benchmarks and it still is much slower than the the 2.4.23.
Question for Jason: -- are you running the 4/4 vm layout in your kernel? My 2.6.0-test6-mm4 has the 4/4 vm enabled -- which can have an impact on performance. I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.
Nope. I'm just doing normal 4GB highmem.
Thanks,
--
Jason Holmes
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Jason Holmes
<snip>
Quoted 4 lines
I'm
already seeing some odd performance differences between the 2.4.22 client and the 2.6.0-test9-mm1 client (both against a 2.6.0-test9-mm2 server):
Similar observations here. Our users have reported NFS slowdowns of 10x on 2.6.0-test6-mm4 nfs clients compared to the same box and setup running 2.4.22. Something is goofy in 2.6.0-test6-mm4 that has a bad effect on NFS client performance.
eric
-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs