Poor NFS client performance on 2.4.18?
Trond, et al.
I'm getting poor NFS performance (~250KBps read and write) on 2.4.18 and am wondering if I'm the only one. There is no performance drop under other OSs or other kernel versions, so I don't think it's the server.
Here's the the details:
2.4.18 patched with:
NFS client patches (linux-2.4.18-NFS_ALL.dif) xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these are dual Xeons)
If you need any more details, let me know.
Thanks,
Dan
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dan Yocum <[email protected]> writes:
> Trond, et al. I'm getting poor NFS performance (~250KBps read > and write) on 2.4.18 and am wondering if I'm the only one. > There is no performance drop under other OSs or other kernel > versions, so I don't think it's the server.
> Here's the the details:
> 2.4.18 patched with:
> NFS client patches (linux-2.4.18-NFS_ALL.dif)
> xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these
> are dual Xeons)
> If you need any more details, let me know.
The latest NFS_ALL patches include experimental code that changes the UDP congestion control. I'm basically trying to relax the algorithm to what is standard on *BSD (i.e. we follow the standard Van Jacobson).
This would mean that we don't wait for the reply from the server before we send off the next request. Unfortunately, there appears to be a lot of setups out there that start to drop packets when this occurs, and I haven't yet finished determining the root cause.
If I can manage to get my laptop to work again, I'll try to investigate a bit more this weekend...
Cheers,
Trond
Trond,
So, would I be correct in assuming that backing out the linux-2.4.18-ping.dif patch would solve the problem in the short term?
Thanks,
Dan
Quoted 31 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Dan Yocum <[email protected]> writes:
> Trond, et al. I'm getting poor NFS performance (~250KBps read > and write) on 2.4.18 and am wondering if I'm the only one. > There is no performance drop under other OSs or other kernel > versions, so I don't think it's the server.
> Here's the the details:
> 2.4.18 patched with:
> NFS client patches (linux-2.4.18-NFS_ALL.dif)
> xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these > are dual Xeons)> If you need any more details, let me know.
The latest
NFS_ALLpatches include experimental code that changes the UDP congestion control. I'm basically trying to relax the algorithm to what is standard on *BSD (i.e. we follow the standard Van Jacobson).This would mean that we don't wait for the reply from the server before we send off the next request. Unfortunately, there appears to be a lot of setups out there that start to drop packets when this occurs, and I haven't yet finished determining the root cause.
If I can manage to get my laptop to work again, I'll try to investigate a bit more this weekend...
Cheers,
Trond
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dan Yocum <[email protected]> writes:
> Trond, So, would I be correct in assuming that backing out the > linux-2.4.18-ping.dif patch would solve the problem in the > short term?
Not ping, but linux-2.4.18-rpc_tweaks.dif...
Cheers,
Trond
Trond,
OK, so backing out the rpc_tweaks dif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.
You can read and write to the NFS volume *before* trying to run something from there, but not after.
Ideas?
Thanks,
Dan
Quoted 31 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Dan Yocum <[email protected]> writes:
> Trond, et al. I'm getting poor NFS performance (~250KBps read > and write) on 2.4.18 and am wondering if I'm the only one. > There is no performance drop under other OSs or other kernel > versions, so I don't think it's the server.
> Here's the the details:
> 2.4.18 patched with:
> NFS client patches (linux-2.4.18-NFS_ALL.dif)
> xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these > are dual Xeons)> If you need any more details, let me know.
The latest
NFS_ALLpatches include experimental code that changes the UDP congestion control. I'm basically trying to relax the algorithm to what is standard on *BSD (i.e. we follow the standard Van Jacobson).This would mean that we don't wait for the reply from the server before we send off the next request. Unfortunately, there appears to be a lot of setups out there that start to drop packets when this occurs, and I haven't yet finished determining the root cause.
If I can manage to get my laptop to work again, I'll try to investigate a bit more this weekend...
Cheers,
Trond
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
Quoted 8 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.
Quoted 4 lines
You can read and write to the NFS volume *before* trying to run something from there, but not after.
Ideas?
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Cheers,
Trond
Quoted 10 lines · Trond Myklebust
Quoted 8 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.
Quoted 7 lines
Quoted 4 lines
You can read and write to the NFS volume *before* trying to run something from there, but not after.
Ideas?
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Ah, that may be the case - the problem also exists with a Linux server as well... let me check, and I'll let you know.
Thanks,
Dan
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
Quoted 15 lines · Dan Yocum
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.You can read and write to the NFS volume *before* trying to run something from there, but not after.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Ideas?
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Ah, that may be the case - the problem also exists with a Linux server as well... let me check, and I'll let you know.
I take that back - it's only hanging on the Linux server when the IRIX server is already hung.
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]
In reply to Dan YocumTrond, et al.
You're right, it's a driver (ns83820) issue. Strange that it only shows up when trying to execute an app that's mounted via NFS, but, whatever. Running apps from the the NFS volumes with the eepro100 adapter that's on the machine works fine with the updated NFS_all patch applied.
Thanks, again,
Dan
Quoted 27 lines · Dan Yocum
Quoted 15 lines · Dan Yocum
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.You can read and write to the NFS volume *before* trying to run something from there, but not after.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Ideas?
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Ah, that may be the case - the problem also exists with a Linux server as well... let me check, and I'll let you know.
I take that back - it's only hanging on the Linux server when the IRIX server is already hung.
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.
Re: ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]
In reply to Dan YocumUpgrade to 0.17 (which is in 2.4.19-pre5 or so and later) and you should find the issue resolved.
-ben
Quoted 51 lines · Dan Yocum
Trond, et al.
You're right, it's a driver (ns83820) issue. Strange that it only shows up when trying to execute an app that's mounted via NFS, but, whatever. Running apps from the the NFS volumes with the eepro100 adapter that's on the machine works fine with the updated
NFS_allpatch applied.Thanks, again,
DanQuoted 27 lines · Dan Yocum
Quoted 15 lines · Dan Yocum
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.You can read and write to the NFS volume *before* trying to run something from there, but not after.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Ideas?
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Ah, that may be the case - the problem also exists with a Linux server as well... let me check, and I'll let you know.
I take that back - it's only hanging on the Linux server when the IRIX server is already hung.
Signature · 9 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
"You will be reincarnated as a toad; and you will be much happier."
Re: ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]
In reply to Dan YocumBen, Trond,
Ugh. It's been a bad week in general. Anyway, yes, you're right, the 0.17 upgrade did the trick - I had actually tried that, but had accidentally re-applied the rpc_tweaks dif which knocked the performance down again. Backing out that patch, updating to v 0.17 of the ns83820 driver and all is well, again.
So, it's back in Trond's court, with the rpc_tweaks causing the slow read/write over NFS, problem.
Trond, if all I want is 32k block transfers from that dif, can I just apply the following, or is there more to it:
diff -u --recursive --new-file linux-2.4.18-svc_tcp/include/linux/nfsd/const.h linux-2.4.18-rpc_cong/include/linux/nfsd/const.h
--- linux-2.4.18-svc_tcp/include/linux/nfsd/const.h Sat Apr 1 18:04:27
2000+++ linux-2.4.18-rpc_cong/include/linux/nfsd/const.h Wed Feb 20
17:20:45 2002@@ -21,7 +21,7 @@
/*
* Maximum blocksize supported by daemon currently at 8K */
-#define NFSSVC_MAXBLKSIZE 8192
+#define NFSSVC_MAXBLKSIZE (32*1024)
#ifdef __KERNEL__
Thanks,
Dan
Quoted 61 lines · Benjamin LaHaise
Upgrade to 0.17 (which is in 2.4.19-pre5 or so and later) and you should find the issue resolved.
-ben
Quoted 41 lines · Dan Yocum
Trond, et al.
You're right, it's a driver (ns83820) issue. Strange that it only shows up when trying to execute an app that's mounted via NFS, but, whatever. Running apps from the the NFS volumes with the eepro100 adapter that's on the machine works fine with the updated
NFS_allpatch applied.Thanks, again,
DanQuoted 19 lines · Dan Yocum
Quoted 17 lines · Dan Yocum
Quoted 15 lines · Trond Myklebust
Quoted 12 lines · Dan Yocum
Trond,
OK, so backing out the
rpc_tweaksdif fixed the performance problem, however, seems to have introduced another problem that appears to be stemming from the seekdir.dif. Attempting to run an app from an IRIX client (that has the 32bitclients option set) freezes the NFS volume - one can't access it from the Linux side, anymore.You can read and write to the NFS volume *before* trying to run something from there, but not after.
Ideas?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
That smells like another network driver bug. Have you tcpdumped the traffic between client and server?
Ah, that may be the case - the problem also exists with a Linux server as well... let me check, and I'll let you know.
I take that back - it's only hanging on the Linux server when the IRIX server is already hung.
Quoted 10 lines
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org
SDSS. Mapping the Universe.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
"You will be reincarnated as a toad; and you will be much happier."
Signature · 4 lines
Dan Yocum Sloan Digital Sky Survey, Fermilab 630.840.6509 [email protected], http://www.sdss.org SDSS. Mapping the Universe.