notes on VAULT 2017 NFS BOF
Subsystems (2)
Steve Dickson lead a quick NFS meeting Wednesday night at Boston during vault. I thought it might be worth posting my notes:
Flex file server: it's just there for testing. If someone wants to build on it, they can. It has no practical use, and should be configured out of distro kernels to avoid confusing users.
NFSv4-only server: some users want to minimize open ports, so we should support this configuration. But distros probably shouldn't be NFSv4-only by default. (And: a show of hands at Steve & Chuck's talk the next day confirmed that people still depend on NFSv3.)
What about turning off UDP? This looks more doable. Note client still needs to listen for lockd UDP. But we can keep that while turning off nfsd UDP. (Kernel lockd is currently hard-coded to listen on both UDP and TCP regardless of server configuration.)
Should the client by default try NFSv4.2 first? Consensus seems to be yes. When 4.2 fails, it tries 4.1, then 4.0, etc. It works transparently. Steved was worried that those retries might become a problem on clients with lots of NFS mounts. Trond suggested recording the result of the version negotiation across mounts, so a client doing a lot of mounts to the same server would only need the retries on the first mount.
The retries are driven by userspace which does a mount for a specific version and uses the return from the mount call to decide to negotiate down. So a new TCP connection happens for each mount attempt.
Miklos introduced a proposed new mount api at LSF earlier in the week. It would allow some communication with the file system driver to set up parameters before the system call that creates the mountpoint. If we moved the mount negotiation to that setup phase, that might make the negotiation phase more efficient while still leaving userspace in charge. (And we prefer leaving userspace in charge to give it maximum control over negotiation policy.)
Somebody asked about inotify implementation. Currently inotify only reports changes made on the same client. There is unimplemented protocol in RFC 5661 that would allow the client to get notifications of other changes from the server. Trond says it would be difficult and risks flooding the network with notifications, though the protocol does have some provision for batching them.
There were questions about NFS's uses of RDMA writes which I didn't follow, and my notes stopped there.
--b.
Quoted 2 lines · J. Bruce Fields
Steve Dickson lead a quick NFS meeting Wednesday night at Boston during vault. I thought it might be worth posting my notes:
Thank you for doing this...
Quoted 4 lines
Flex file server: it's just there for testing. If someone wants to build on it, they can. It has no practical use, and should be configured out of distro kernels to avoid confusing users.
+1... not really clear what the point was in even posting it.
Quoted 5 lines
NFSv4-only server: some users want to minimize open ports, so we should support this configuration. But distros probably shouldn't be NFSv4-only by default. (And: a show of hands at Steve & Chuck's talk the next day confirmed that people still depend on NFSv3.)
I did try hard... but there was definitely push back. :-)
Quoted 5 lines
What about turning off UDP? This looks more doable. Note client still needs to listen for lockd UDP. But we can keep that while turning off nfsd UDP. (Kernel lockd is currently hard-coded to listen on both UDP and TCP regardless of server configuration.)
Jeff's RFC patches are definitely on the my TODO list. My question is do we turn off UDP between services on the same host... I'm thinking not.
Quoted 8 lines
Should the client by default try NFSv4.2 first? Consensus seems to be yes. When 4.2 fails, it tries 4.1, then 4.0, etc. It works transparently. Steved was worried that those retries might become a problem on clients with lots of NFS mounts. Trond suggested recording the result of the version negotiation across mounts, so a client doing a lot of mounts to the same server would only need the retries on the first mount.
I just don't think this scales very well in large NFS mounted home directory server. Since the major enterprise servers do not support 4.2 and I don't see them supporting 4.2 anytime soon. Why try something when you know its going to fail? ;-)
Starting at v4.2 in non-enterprise environments works, at least it has for the last few years...
Quoted 4 lines
The retries are driven by userspace which does a mount for a specific version and uses the return from the mount call to decide to negotiate down. So a new TCP connection happens for each mount attempt.
Well it could be up to 3 connection (including the successful mount) when both the IPv4 and IPv6 address are tried.
Quoted 8 lines
Miklos introduced a proposed new mount api at LSF earlier in the week. It would allow some communication with the file system driver to set up parameters before the system call that creates the mountpoint. If we moved the mount negotiation to that setup phase, that might make the negotiation phase more efficient while still leaving userspace in charge. (And we prefer leaving userspace in charge to give it maximum control over negotiation policy.)
Any pointers to this?
steved.
Quoted 17 lines
Somebody asked about inotify implementation. Currently inotify only reports changes made on the same client. There is unimplemented protocol in RFC 5661 that would allow the client to get notifications of other changes from the server. Trond says it would be difficult and risks flooding the network with notifications, though the protocol does have some provision for batching them.
There were questions about NFS's uses of RDMA writes which I didn't follow, and my notes stopped there.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Steve Dickson
Quoted 7 lines · J. Bruce Fields
Should the client by default try NFSv4.2 first? Consensus seems to be yes. When 4.2 fails, it tries 4.1, then 4.0, etc. It works transparently. Steved was worried that those retries might become a problem on clients with lots of NFS mounts. Trond suggested recording the result of the version negotiation across mounts, so a client doing a lot of mounts to the same server would only need the retries on the first mount.
Quoted 4 lines
I just don't think this scales very well in large NFS mounted home directory server. Since the major enterprise servers do not support 4.2 and I don't see them supporting 4.2 anytime soon. Why try something when you know its going to fail? ;-)
I'm OK with sticking with 4.1 for now.
That said, the expense of negotiation shouldn't be an issue. We have ideas here for cutting that expense (if it really is an issue), and they'd help in the 4.1->4.0 case too.
Quoted 8 lines
Starting at v4.2 in non-enterprise environments works, at least it has for the last few years...
Quoted 3 lines
The retries are driven by userspace which does a mount for a specific version and uses the return from the mount call to decide to negotiate down. So a new TCP connection happens for each mount attempt.
Well it could be up to 3 connection (including the successful mount) when both the IPv4 and IPv6 address are tried.
Somebody correct me if I'm wrong, but I don't think that contributes to port exhaustion. Connections to the IPv4 and IPv6 addresses using the same ports are distinct, aren't they?
Quoted 8 lines
Quoted 7 lines
Miklos introduced a proposed new mount api at LSF earlier in the week. It would allow some communication with the file system driver to set up parameters before the system call that creates the mountpoint. If we moved the mount negotiation to that setup phase, that might make the negotiation phase more efficient while still leaving userspace in charge. (And we prefer leaving userspace in charge to give it maximum control over negotiation policy.)
Any pointers to this?
I googled around a little and can't find any. It's still just a proposal, so there's nothing for us to build on yet.
--b.
Quoted 26 lines · J. Bruce Fields
Quoted 8 lines · Steve Dickson
Quoted 7 lines · J. Bruce Fields
Should the client by default try NFSv4.2 first? Consensus seems to be yes. When 4.2 fails, it tries 4.1, then 4.0, etc. It works transparently. Steved was worried that those retries might become a problem on clients with lots of NFS mounts. Trond suggested recording the result of the version negotiation across mounts, so a client doing a lot of mounts to the same server would only need the retries on the first mount.
Quoted 4 lines
I just don't think this scales very well in large NFS mounted home directory server. Since the major enterprise servers do not support 4.2 and I don't see them supporting 4.2 anytime soon. Why try something when you know its going to fail? ;-)
I'm OK with sticking with 4.1 for now.
That said, the expense of negotiation shouldn't be an issue. We have ideas here for cutting that expense (if it really is an issue), and they'd help in the 4.1->4.0 case too.
Quoted 6 lines
Starting at v4.2 in non-enterprise environments works, at least it has for the last few years...
Quoted 3 lines
The retries are driven by userspace which does a mount for a specific version and uses the return from the mount call to decide to negotiate down. So a new TCP connection happens for each mount attempt.
Quoted 6 lines
Quoted 2 lines
Well it could be up to 3 connection (including the successful mount) when both the IPv4 and IPv6 address are tried.
Somebody correct me if I'm wrong, but I don't think that contributes to port exhaustion. Connections to the IPv4 and IPv6 addresses using the same ports are distinct, aren't they?
IDK... I was just looking at the for loop in nfs_try_mount_v4() count a connection for each loop...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Miklos introduced a proposed new mount api at LSF earlier in the week. It would allow some communication with the file system driver to set up parameters before the system call that creates the mountpoint. If we moved the mount negotiation to that setup phase, that might make the negotiation phase more efficient while still leaving userspace in charge. (And we prefer leaving userspace in charge to give it maximum control over negotiation policy.)
Quoted 4 lines
Quoted 1 line
Any pointers to this?
I googled around a little and can't find any. It's still just a proposal, so there's nothing for us to build on yet.
Fair enough... thanks for looking!
steved.
Quoted 7 lines
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html