Fix fallback from tcp to udp
Subsystems (2)
PATCH 1/1 Fix fallback from tcp to udp
Protocol negotiation in mount.nfs does not correctly negotiate with a server which only support NFSv3 and UDP.
When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED it does not fall back to NFSv3, as this is not recognised as a "does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.
Once we allow that error, NFSv3 is attempted and mount.nfs talks to rpcbind and discovers that UDP should be used for v3 and the mount succeeds.
utils/mount/stropts.c +3 -0
@@ -807,6 +807,9 @@ static int nfs_autonegotiate(struct nfsmount_info *mi)/* Linux servers prior to 2.6.25 may return * EPERM when NFS version 4 is not supported. */ goto fall_back; +case ECONNREFUSED : +/* UDP-Only servers won't support v4 */ + goto fall_back;default :return result; }
Quoted 10 lines · NeilBrown
Protocol negotiation in mount.nfs does not correctly negotiate with a server which only support NFSv3 and UDP.
When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED it does not fall back to NFSv3, as this is not recognised as a "does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.
I'm also pretty this is the error returned when the server is down or more pointy when server is rebooting... Do we really want to fallback at this point?
Secondly, its worrisome to me that we keep making this fallback list longer and longer... we really don't want to fall back to v3 but I do understand we want to be compatible with older servers...
steved.
Quoted 23 lines
Once we allow that error, NFSv3 is attempted and mount.nfs talks to rpcbind and discovers that UDP should be used for v3 and the mount succeeds.
Signed-off-by NeilBrown <[email protected]>Reported-by Carsten Ziepke <[email protected]>utils/mount/stropts.c +3
-0@@ -807,6 +807,9 @@ static int nfs_autonegotiate(struct nfsmount_info *mi)/* Linux servers prior to 2.6.25 may return * EPERM when NFS version 4 is not supported. */ goto fall_back; +case ECONNREFUSED : +/* UDP-Only servers won't support v4 */ + goto fall_back;default :return result; }
Quoted 7 lines · Steve Dickson
Quoted 4 lines · NeilBrown
Protocol negotiation in mount.nfs does not correctly negotiate with a server which only support NFSv3 and UDP.
Quoted 8 lines
Quoted 6 lines
When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED it does not fall back to NFSv3, as this is not recognised as a "does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.I'm also pretty this is the error returned when the server is down or more pointy when server is rebooting...
Probably worth checking that.
Quoted 3 lines
Do we really want to fallback at this point?
From a bz comment (#984901, not sure why it's private):
Any NFS server has to support either tcp or rpcbind. But it's OK for a server to support only of those two. So the only way to handle both cases while continuing to retry after ECONNREFUSED is to alternate between trying nfs4/tcp and rpcbind until you can connect to one or the other.
If it's the rpcbind call that succeeds first then I think we want to do one more try of nfs4/tcp just to make sure it didn't just come up, before falling back to v3.
The rpcbind call is done in userspace, if I understand right, so I think this is doable. Looking at utils/mount/ I don't understand the mount process well enough to understand exactly how to do it. Maybe everything but the final nfs_sys_mount needs to be moved out of nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?
Except the v3v2 mount logic seems to actually modify the mount_options, so probably that doesn't quite work.
?
--b.
Quoted 19 lines · bfields
Quoted 7 lines · Steve Dickson
Quoted 4 lines · NeilBrown
Protocol negotiation in mount.nfs does not correctly negotiate with a server which only support NFSv3 and UDP.
Quoted 8 lines
Quoted 6 lines
When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED it does not fall back to NFSv3, as this is not recognised as a "does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.I'm also pretty this is the error returned when the server is down or more pointy when server is rebooting...
Probably worth checking that.
Quoted 7 lines
Quoted 1 line
Do we really want to fallback at this point?
From a bz comment (#984901, not sure why it's private):
Any NFS server has to support either tcp or rpcbind. But it's OK for a server to support only of those two. So the only way to handle both cases while continuing to retry after ECONNREFUSED
(But I'm not actually convinced that's true. In particular I don't see ECONNREFUSED when rebooting a server. But I'm not clear when it's returned....)
--b.
Quoted 23 lines
is to alternate
between trying nfs4/tcp and rpcbind until you can connect to one or the other.If it's the rpcbind call that succeeds first then I think we want to do one more try of nfs4/tcp just to make sure it didn't just come up, before falling back to v3.
The rpcbind call is done in userspace, if I understand right, so I think this is doable. Looking at utils/mount/ I don't understand the mount process well enough to understand exactly how to do it. Maybe everything but the final
nfs_sys_mountneeds to be moved out ofnfs_do_mount_v3v2into a newnfs_do_probe_v3v2andnfs_autonegotiateshould alternate betweennfs_try_mount_v4andnfs_do_probe_v3v2as long as both return ECONNREFUSED, callingnfs_try_mount_v3v2only ifnfs_try_mount_v4has failed after a succesfulnfs_do_probe_v3v2?Except the v3v2 mount logic seems to actually modify the
mount_options, so probably that doesn't quite work.?
--b.
Quoted 14 lines · On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 13 lines · Steve Dickson
Quoted 10 lines · NeilBrown
Protocol negotiation in mount.nfs does not correctly negotiate with a server which only support NFSv3 and UDP.
When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED it does not fall back to NFSv3, as this is not recognised as a "does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.
Quoted 4 lines
Quoted 2 lines
I'm also pretty this is the error returned when the server is down or more pointy when server is rebooting...
Probably worth checking that.
It is certainly possible that there is a window during boot when a server will RST any SYN to port 2049. The window may be very small, but it will usually be there.
It is possible to configure a server to start listening before enabling the interface, and so close the window completely. But we certainly cannot assume any server does this.
Quoted 24 lines
Quoted 4 lines
Do we really want to fallback at this point?
From a bz comment (#984901, not sure why it's private):
Any NFS server has to support either tcp or rpcbind. But it's OK for a server to support only of those two. So the only way to handle both cases while continuing to retry after ECONNREFUSED is to alternate between trying nfs4/tcp and rpcbind until you can connect to one or the other.
If it's the rpcbind call that succeeds first then I think we want to do one more try of nfs4/tcp just to make sure it didn't just come up, before falling back to v3.
The rpcbind call is done in userspace, if I understand right, so I think this is doable. Looking at utils/mount/ I don't understand the mount process well enough to understand exactly how to do it. Maybe everything but the final
nfs_sys_mountneeds to be moved out ofnfs_do_mount_v3v2into a newnfs_do_probe_v3v2andnfs_autonegotiateshould alternate betweennfs_try_mount_v4andnfs_do_probe_v3v2as long as both return ECONNREFUSED, callingnfs_try_mount_v3v2only ifnfs_try_mount_v4has failed after a succesfulnfs_do_probe_v3v2?
Quoted 2 lines
Except the v3v2 mount logic seems to actually modify the
mount_options, so probably that doesn't quite work.
I had come to much the same conclusion after reading Steve's mail: when TCP fails we need rpcbind to be sure what to do.
I suspect it should be fairly straight forward to implement (I'm less pessimistic than you). I'll have a go on Monday.
Thanks,
NeilBrown
Quoted 21 lines · NeilBrown
Quoted 19 lines · On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <[email protected]>…
Any NFS server has to support either tcp or rpcbind. But it's OK for a server to support only of those two. So the only way to handle both cases while continuing to retry after ECONNREFUSED is to alternate between trying nfs4/tcp and rpcbind until you can connect to one or the other.
If it's the rpcbind call that succeeds first then I think we want to do one more try of nfs4/tcp just to make sure it didn't just come up, before falling back to v3.
The rpcbind call is done in userspace, if I understand right, so I think this is doable. Looking at utils/mount/ I don't understand the mount process well enough to understand exactly how to do it. Maybe everything but the final
nfs_sys_mountneeds to be moved out ofnfs_do_mount_v3v2into a newnfs_do_probe_v3v2andnfs_autonegotiateshould alternate betweennfs_try_mount_v4andnfs_do_probe_v3v2as long as both return ECONNREFUSED, callingnfs_try_mount_v3v2only ifnfs_try_mount_v4has failed after a succesfulnfs_do_probe_v3v2?
Quoted 7 lines
Quoted 2 lines
Except the v3v2 mount logic seems to actually modify the
mount_options, so probably that doesn't quite work.I had come to much the same conclusion after reading Steve's mail: when TCP fails we need rpcbind to be sure what to do.
I suspect it should be fairly straight forward to implement (I'm less pessimistic than you). I'll have a go on Monday.
OK, great!
Yeah, the mount code looked like a maze for me but I probably spent less than an hour trying to trace through it, I'm sure it's not that bad.
--b.
Quoted 30 lines · J. Bruce Fields
Quoted 21 lines · NeilBrown
Quoted 19 lines · On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <[email protected]>…
Any NFS server has to support either tcp or rpcbind. But it's OK for a server to support only of those two. So the only way to handle both cases while continuing to retry after ECONNREFUSED is to alternate between trying nfs4/tcp and rpcbind until you can connect to one or the other.
If it's the rpcbind call that succeeds first then I think we want to do one more try of nfs4/tcp just to make sure it didn't just come up, before falling back to v3.
The rpcbind call is done in userspace, if I understand right, so I think this is doable. Looking at utils/mount/ I don't understand the mount process well enough to understand exactly how to do it. Maybe everything but the final
nfs_sys_mountneeds to be moved out ofnfs_do_mount_v3v2into a newnfs_do_probe_v3v2andnfs_autonegotiateshould alternate betweennfs_try_mount_v4andnfs_do_probe_v3v2as long as both return ECONNREFUSED, callingnfs_try_mount_v3v2only ifnfs_try_mount_v4has failed after a succesfulnfs_do_probe_v3v2?Quoted 7 lines
Quoted 2 lines
Except the v3v2 mount logic seems to actually modify the
mount_options, so probably that doesn't quite work.I had come to much the same conclusion after reading Steve's mail: when TCP fails we need rpcbind to be sure what to do.
I suspect it should be fairly straight forward to implement (I'm less pessimistic than you). I'll have a go on Monday.
Quoted 4 lines
OK, great!
Yeah, the mount code looked like a maze for me but I probably spent less than an hour trying to trace through it, I'm sure it's not that bad.
Just a general comment. Mount negotiation is a maze because
a) we have so many legacy use cases that still MUST work, and
b) we have no regression test suite that can confirm that mount.nfs is still operating correctly after code changes
Thus it’s very difficult to clean up over time. It just accretes more and more logic. We add little bits here and there because it seems safe, but that adds up.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com