NFS Still broken in 2.6.x?
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?
Basically, the problem I'm having is that downloads from an NFS server using kernel 2.6 are no more than half as fast as the same from a server using kernel 2.4. Write speed (uploading) seems to be about the same, but reading is slow.
I'm using tcp as my protocol, at the suggestion of many posts, but flipping over to udp doesn't seem to make any difference. I'm using version 3, although I did switch to 2 just to check (it's no better, usually slower). My read size is 32768 and my write size is 8192. Decreasing the read size only slows down the transfers. Increasing write size has no effect.
As for hardware, both machines are dual AMD Opterons, 100Mbps ethernet, and the NFS is serving space on a RAID array. The 2.4 (2.4.21 to be exact) kernel is running under SuSE 9.0, and the 2.6 (2.6.15 to be exact) kernel is running under SuSE 10.0. I saw the same speed drop when attempting to upgrade to SuSE 9.3. I stayed with 9.0 in hopes that the problem would be fixed in the future.
Anyone have any ideas?
-Bryan
Quoted 5 lines · Bryan Fink
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?
No it hasn't been fixed. ...and no, this is not a problem that only affects NFS: it just happens to give a more noticeable performance impact due to the larger latency of NFS over a 100Mbps link.
I will get round to this, but the general opacity of the current readahead code has been a bit of a put-off in the face of other NFS problems.
Cheers,
Trond
Quoted 11 lines · Trond Myklebust
Quoted 5 lines · Bryan Fink
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?No it hasn't been fixed. ...and no, this is not a problem that only affects NFS: it just happens to give a more noticeable performance impact due to the larger latency of NFS over a 100Mbps link.
iirc, last time we went round this loop Ram and I were unable to reproduce it.
Does anyone have a testcase?
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Trond Myklebust
Quoted 5 lines · Bryan Fink
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?
Quoted 7 lines
Quoted 3 lines
No it hasn't been fixed. ...and no, this is not a problem that only affects NFS: it just happens to give a more noticeable performance impact due to the larger latency of NFS over a 100Mbps link.
iirc, last time we went round this loop Ram and I were unable to reproduce it.
Does anyone have a testcase?
Yes. A dead simple one
run iozone in sequential read mode on a tcp link w/ rsize == 32k
Monitor the traffic using tcpdump. Pretty soon you will see the size of the NFS read requests drop from 32k to 4k, which indicates that there is no readahead at all.
Cheers,
Trond
Quoted 20 lines · Trond Myklebust
Quoted 17 lines · Andrew Morton
Quoted 7 lines · Trond Myklebust
Quoted 5 lines · Bryan Fink
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?Quoted 5 lines
No it hasn't been fixed. ...and no, this is not a problem that only affects NFS: it just happens to give a more noticeable performance impact due to the larger latency of NFS over a 100Mbps link.
iirc, last time we went round this loop Ram and I were unable to reproduce it.
Quoted 9 lines
Quoted 3 lines
Does anyone have a testcase?
Yes. A dead simple one
run iozone in sequential read mode on a tcp link w/ rsize == 32k
I'm sure Trond's testcase is much more useful, but for reference, I thought I'd add that I've been doing my testing with a simple "dd if=/nfsmount/file of=/dev/null bs=32k". /nfsmount/file is usually 2.5-3 GB, which makes the difference between NFS servers long enough that I feel safe throwing a "time" in front of the whole command. That is, the difference is nowhere near millisecond resolution (it's nearer a minute), so I like to start the test and then walk away to do other things.
Interesting that it's not an NFS-only bug. I assumed it was when I logged into each server so I could run "dd if=file of=/dev/null bs=32k" locally. When I did that, both servers gave roughly the same speed. Sorry I left this bit out of my first email. I assume this example only illustrates how opaque the code around this problem truly is.
Thanks very much for the help.
-Bryan
Quoted 16 lines · Andrew Morton
Quoted 7 lines · Trond Myklebust
Quoted 5 lines · Bryan Fink
Hi All. I'm running into a bit of trouble with NFS on 2.6. I see that at least Trond thought, mid-January, that "The readahead algorithm has been broken in 2.6.x for at least the past 6 months." (
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/0559.html) Anyone know if that has been fixed?Quoted 5 lines
No it hasn't been fixed. ...and no, this is not a problem that only affects NFS: it just happens to give a more noticeable performance impact due to the larger latency of NFS over a 100Mbps link.
Quoted 5 lines
iirc, last time we went round this loop Ram and I were unable to reproduce it.
Does anyone have a testcase?
Hi again. I just found some new, very interesting information. Until just a few minutes ago, I hadn't realized that one could change the I/O scheduler at runtime. Looking into it, my system was using "cfq", and I have three other options, "noop", "anticipatory", and "deadline". I've now run tests using all three of the other schedulers, and they all bring performance back up to the level I had with kernel 2.4. So, either NFS is incompatible with cfq, or cfq has some issues that show very vividly when used with NFS (or, I suppose, I just have my system tuned wrong for use with cfq).
Hope this helps the bug hunt. Special thanks to Asfand Yar Qazi for writing to the list this morning asking how to change schedulers at runtime (http://www.ussg.iu.edu/hypermail/linux/kernel/0602.3/0135.html). Off to find out exactly what the best scheduler is for my needs.
-Bryan
Quoted 9 lines · Bryan Fink
Hi again. I just found some new, very interesting information. Until just a few minutes ago, I hadn't realized that one could change the I/O scheduler at runtime. Looking into it, my system was using "cfq", and I have three other options, "noop", "anticipatory", and "deadline". I've now run tests using all three of the other schedulers, and they all bring performance back up to the level I had with kernel 2.4. So, either NFS is incompatible with cfq, or cfq has some issues that show very vividly when used with NFS (or, I suppose, I just have my system tuned wrong for use with cfq).
I run NFS for ages -- all linux boxen here mount a shared export from localnet controller box to get source + patches.
Only have 'deadline' installed on 2.6 kernels -- not seen any problems with NFS here (apart from back when I had data corruption due a faulty memory stick).
Grant.