NFSv4.0 allow nconnect for v4.0
Subsystems (2)
PATCH 1/1 NFSv4.0 allow nconnect for v4.0
From: Olga Kornievskaia <[email protected]>
--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4client.c +1 -1
@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT )
Signature · 1 line
1.8.3.1
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Have you done any testing with nconnect and the v4.0 replay caches? I did some digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
Thanks,
Anna
Quoted 19 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT )
Quoted 5 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Quoted 9 lines
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Quoted 22 lines
Thanks,
AnnaQuoted 19 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server, if (minorversion == 0)__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (
server->flags&NFS_MOUNT_NORESVPORT)
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 16 lines · Trond Myklebust
Quoted 5 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Quoted 9 lines
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
Quoted 5 lines
That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Thanks for the explanation! I'll add the patch.
Anna
Quoted 29 lines
Quoted 23 lines
Thanks,
AnnaQuoted 19 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server, if (minorversion == 0)__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (
server->flags&NFS_MOUNT_NORESVPORT)Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 17 lines · Anna
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
Quoted 7 lines
Quoted 5 lines
That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Thanks for the explanation! I'll add the patch.
What happened to this patch? As far as I can tell, the conclusion of this thread was that it should be applied.
--b.
Quoted 32 lines
Anna
Quoted 29 lines
Quoted 23 lines
Thanks,
AnnaQuoted 19 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server, if (minorversion == 0)__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (
server->flags&NFS_MOUNT_NORESVPORT)Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
________________________________________
From: [email protected] <[email protected]> on behalf of J. Bruce Fields <[email protected]>
Sent: Thursday, June 11, 2020 4:09 PM
To: Schumaker, Anna
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 1/1] NFSv4.0 allow nconnect for v4.0
CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to [email protected]
Quoted 17 lines · Anna
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
Quoted 7 lines
Quoted 5 lines
That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Thanks for the explanation! I'll add the patch.
What happened to this patch? As far as I can tell, the conclusion of this thread was that it should be applied.
--b.
Quoted 32 lines
Anna
Quoted 29 lines
Quoted 16 lines
Thanks,
AnnaQuoted 12 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +0
-0@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,Quoted 7 lines
Quoted 7 lines
fs/nfs/nfs4client.c +1
-1if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags & NFS_MOUNT_NORESVPORT)--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Quoted 18 lines · J. Bruce Fields
Quoted 17 lines · Anna
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Thanks for the explanation! I'll add the patch.
Maybe I am misinterpreting this discussion and, if so, please just ignore these comments.
Here are two snippets from RFC7530:
In Sec. 3.1.
Where an NFSv4 implementation supports operation over the IP network protocol, the supported transport layer between NFS and IP MUST be an IETF standardized transport protocol that is specified to avoid network congestion; such transports include TCP and the Stream Control Transmission Protocol (SCTP). To enhance the possibilities for interoperability, an NFSv4 implementation MUST support operation over the TCP transport protocol.
This clearly states that NFSv4.0 cannot operate over UDP. (See the MUST above.)
In Sec. 3.1.1.
When processing an NFSv4 request received over a reliable transport such as TCP, the NFSv4 server MUST NOT silently drop the request, except if the established transport connection has been broken. Given such a contract between NFSv4 clients and servers, clients MUST NOT retry a request unless one or both of the following are true:
o The transport connection has been broken
o The procedure being retried is the NULL procedure
If the TCP connection is broken, making a new TCP connection on the same port #s would be risky, unless a long delay is done. Normally a different port# would be used and the implication of the above is that any retry of an RPC (above the TCP layer retransmits of segments) will normally be from a different port#.
Long ago I played around with a DRC for TCP (which ended up in FreeBSD) and because retries happen after a long timeout and there can be many other RPCs done in between the first attempt for the RPC and a subsequent retry of the RPC, the design must be very different than a DRC for UDP. --> LRU caching doesn't work well unless the cache size is very large. --> For NFSv4.0 over TCP (not NFSv3), the DRC must assume (or at least
handle) retries from a different client port#, since they will be done on a different TCP connection and that will almost inevitably imply a different port#.
rick
What happened to this patch? As far as I can tell, the conclusion of this thread was that it should be applied.
--b.
Quoted 32 lines
Anna
Quoted 29 lines
Quoted 16 lines
Thanks,
AnnaQuoted 12 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +0
-0@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,Quoted 7 lines
Quoted 7 lines
fs/nfs/nfs4client.c +1
-1if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags & NFS_MOUNT_NORESVPORT)--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Quoted 25 lines · J. Bruce Fields
Quoted 23 lines · Anna
Quoted 21 lines · Trond Myklebust
Quoted 5 lines · Anna
Hi Olga,
Quoted 1 line · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Quoted 9 lines
Have you done any testing with nconnect and the v4.0 replay caches? I did some
digging on the mailing list and found this in one of the cover letters from
Trond: "The feature is only enabled for NFSv4.1 and NFSv4.2 for now; I don't
feel comfortable subjecting NFSv3/v4 replay caches to this treatment yet."That comment should be considered obsolete. The current code works hard to ensure that we replay using the same connection (or at least the same source/dest IP+ports) so that NFSv3/v4.0 DRCs work as expected. For that reason we've had NFSv3 support since the feature was merged. The NFSv4.0 support was just forgotten.
Quoted 4 lines
Quoted 1 line
Thanks for the explanation! I'll add the patch.
What happened to this patch? As far as I can tell, the conclusion of this thread was that it should be applied.
I decided not to submit this patch but anybody else is free to add that patch to add support for 4.0 nconnect as there is no reason it shouldn't be supported.
Quoted 35 lines
--b.
Quoted 32 lines
Anna
Quoted 29 lines
Quoted 16 lines
Thanks,
AnnaQuoted 12 lines
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +0
-0@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,Quoted 7 lines
Quoted 7 lines
fs/nfs/nfs4client.c +1
-1if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags & NFS_MOUNT_NORESVPORT)--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
PATCH 1/1 NFSv4.0 allow nconnect for v4.0
From: Olga Kornievskaia <[email protected]>
It looks like this "else" is just a typo. It turns off nconnect for NFSv4.0 even though it works for every other version.
--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 3 lines · Olga Kornievskaia
I decided not to submit this patch but anybody else is free to add that patch to add support for 4.0 nconnect as there is no reason it shouldn't be supported.
OK, resending, assuming Trond or Anna haven't already got this.--b.
fs/nfs/nfs4client.c +1 -1
@@ -880,7 +880,7 @@ static int nfs4_set_client(struct nfs_server *server,if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT )
Signature · 1 line
2.26.2
Hello,
Quoted 13 lines · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +0
-0
fs/nfs/nfs4client.c static int nfs4_set_client(struct nfs_server *server, · line 887
Quoted 8 lines
fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT )
With this patch v4.0 mounts act just like v4.1/v4.2 mounts But is that a good thing. :-)
Here is what I've found in my testing...
mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp
Will create 12 TCP connections and maintain those 12 connections until the umount happens. By maintain I mean if the connection times out, it is reconnected to maintain the 12 connections
# mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:667 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:746 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:672 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:832 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:895 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:673 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:732 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:795 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:918 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:674 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:953 172.31.1.54:2049 ESTABLISHED
# umount /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:667 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:746 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:672 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:832 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:895 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:673 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:732 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:795 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:918 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:674 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:953 172.31.1.54:2049 TIME_WAIT
Is this the expected behavior?
If so I have a few concerns...
* The connections walk all over the /etc/services namespace. Meaning using ports that are reserved for registered services, something we've tried to avoid in userland by not binding to privilege ports and use of backlist ports via /etc/bindresvport.blacklist
* When the unmount happens, all those connections go into TIME_WAIT on privilege ports and there are only so many of those. Not good during mount storms (when a server reboots and thousand of home dirs are remounted).
* No man page describing the new feature.
I realize there is not much we can do about some of these (aka umount==>TIME_WAIT) but I think we need to document what we are doing to people's connection namespace when they use this feature.
steved.
Quoted 114 lines · Steve Dickson
Hello,
Quoted 20 lines · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server, if (minorversion == 0)__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;Quoted 2 lines
if (
server->flags&NFS_MOUNT_NORESVPORT)Tested-by Steve Dickson <[email protected]>With this patch v4.0 mounts act just like v4.1/v4.2 mounts But is that a good thing. :-)
Here is what I've found in my testing...
mount -onconnect=12 172.31.1.54:/home/tmp
/mnt/tmpWill create 12 TCP connections and maintain those 12 connections until the umount happens. By maintain I mean if the connection times out, it is reconnected to maintain the 12 connections
# mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:667 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:746 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:672 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:832 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:895 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:673 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:732 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:795 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:918 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:674 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:953 172.31.1.54:2049 ESTABLISHED# umount /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:667 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:746 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:672 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:832 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:895 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:673 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:732 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:795 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:918 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:674 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:953 172.31.1.54:2049 TIME_WAITIs this the expected behavior?
If so I have a few concerns...
* The connections walk all over the
/etc/servicesnamespace. Meaning using ports that are reserved for registered services, something we've tried to avoid in userland by not binding to privilege ports and
use of backlist ports via/etc/bindresvport.blacklist* When the unmount happens, all those connections go into
TIME_WAITon
privilege ports and there are only so many of those. Not good during mount
storms (when a server reboots and thousand of home dirs are remounted).
Quoted 6 lines
* No man page describing the new feature.
I realize there is not much we can do about some of these (aka umount==>TIME_WAIT) but I think we need to document what we are doing to people's connection namespace when they use this feature.
I'm not sure that I understand the concern. The connections are limited to a specific window of ports by the min_resvport/max_resvport sunrpc module parameters just as they were before we added 'nconnect'. Nothing has changed in the way we choose ports...
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 122 lines · Trond Myklebust
Quoted 114 lines · Steve Dickson
Hello,
Quoted 20 lines · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-1@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server, if (minorversion == 0)__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;Quoted 2 lines
if (
server->flags&NFS_MOUNT_NORESVPORT)Tested-by Steve Dickson <[email protected]>With this patch v4.0 mounts act just like v4.1/v4.2 mounts But is that a good thing. :-)
Here is what I've found in my testing...
mount -onconnect=12 172.31.1.54:/home/tmp
/mnt/tmpWill create 12 TCP connections and maintain those 12 connections until the umount happens. By maintain I mean if the connection times out, it is reconnected to maintain the 12 connections
# mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:667 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:746 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:672 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:832 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:895 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:673 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:732 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:795 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:918 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:674 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:953 172.31.1.54:2049 ESTABLISHED# umount /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:667 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:746 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:672 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:832 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:895 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:673 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:732 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:795 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:918 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:674 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:953 172.31.1.54:2049 TIME_WAITIs this the expected behavior?
If so I have a few concerns...
* The connections walk all over the
/etc/servicesnamespace. Meaning using ports that are reserved for registered services, something we've tried to avoid in userland by not binding to privilege ports and
use of backlist ports via/etc/bindresvport.blacklist* When the unmount happens, all those connections go into
TIME_WAITon
privilege ports and there are only so many of those. Not good during mount
storms (when a server reboots and thousand of home dirs are remounted).Quoted 6 lines
* No man page describing the new feature.
I realize there is not much we can do about some of these (aka umount==>TIME_WAIT) but I think we need to document what we are doing to people's connection namespace when they use this feature.
Quoted 5 lines
I'm not sure that I understand the concern. The connections are limited to a specific window of ports by the min_resvport/max_resvport sunrpc module parameters just as they were before we added 'nconnect'. Nothing has changed in the way we choose ports...
Maybe this problem has existed for a while...
Here are the mins/max portsRPC_DEF_MIN_RESVPORT (665U)RPC_DEF_MAX_RESVPORT (1023U)
From /etc/services there are the services in that range acp(674), ha-cluster(694), kerberos-adm(749), kerberos-iv(750) webster(765), phonebook(767), rsync(873), rquotad(875), telnets(992), imaps(993), pop3s(995)
Granted a lot of these are unused/legacy services, but some of them, like imaps and rsync, are still used.
My point is since the nconnect connections are persistent, for the life of the mount, we could end up squatting on ports other services will needed.
Maybe there is not much we can do about this... But we should explain somewhere, like the man page, that nconnect will create up to 16 persistent connection on register privilege ports.
steved.
Quoted 152 lines · Steve Dickson
Quoted 128 lines · Trond Myklebust
Quoted 120 lines · Steve Dickson
Hello,
Quoted 19 lines · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +0
-0@@ -881,7 +881,7 @@ static int nfs4_set_client(struct
nfs_server
*server,fs/nfs/nfs4client.c +0
-0if (minorversion ==0 ) __set_bit(NFS_CS_REUSEPORT,&cl_init.init_flags);
fs/nfs/nfs4client.c static int nfs4_set_client(struct nfs_server *server, · line 887
Quoted 5 lines
fs/nfs/nfs4client.c +1
-1-else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT )Tested-by Steve Dickson <[email protected]>With this patch v4.0 mounts act just like v4.1/v4.2 mounts But is that a good thing. :-)
Here is what I've found in my testing...
mount -onconnect=12 172.31.1.54:/home/tmp
/mnt/tmpWill create 12 TCP connections and maintain those 12 connections until the umount happens. By maintain I mean if the connection times out, it is reconnected to maintain the 12 connections
# mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:667 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:746 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:672 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:832 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:895 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:673 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:732 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:795 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:918 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:674 172.31.1.54:2049 ESTABLISHED tcp 0 0 172.31.1.24:953 172.31.1.54:2049 ESTABLISHED# umount /mnt/tmp # netstat -an | grep 172.31.1.54 | wc -l 12 # netstat -an | grep 172.31.1.54 tcp 0 0 172.31.1.24:901 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:667 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:746 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:672 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:832 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:895 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:673 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:732 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:795 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:918 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:674 172.31.1.54:2049 TIME_WAIT tcp 0 0 172.31.1.24:953 172.31.1.54:2049 TIME_WAITIs this the expected behavior?
If so I have a few concerns...
* The connections walk all over the
/etc/servicesnamespace. Meaning
using ports that are reserved for registered services, something we've tried to avoid in userland by not binding to privilege ports
and
use of backlist ports via/etc/bindresvport.blacklist* When the unmount happens, all those connections go into
TIME_WAIT
on
privilege ports and there are only so many of those. Not good during
mount
storms (when a server reboots and thousand of home dirs are remounted).Quoted 6 lines
* No man page describing the new feature.
I realize there is not much we can do about some of these (aka umount==>TIME_WAIT) but I think we need to document what we are doing to people's connection namespace when they use this feature.
Quoted 8 lines
I'm not sure that I understand the concern. The connections are limited
to a specific window of ports by the min_resvport/max_resvport sunrpc
module parameters just as they were before we added 'nconnect'. Nothing
has changed in the way we choose ports...Maybe this problem has existed for a while...
Here are the mins/max ports
RPC_DEF_MIN_RESVPORT(665U)RPC_DEF_MAX_RESVPORT(1023U)From
/etc/servicesthere are the services in that range acp(674), ha-cluster(694), kerberos-adm(749), kerberos-iv(750) webster(765), phonebook(767), rsync(873), rquotad(875), telnets(992), imaps(993), pop3s(995)Granted a lot of these are unused/legacy services, but some of them, like imaps and rsync, are still used.
Quoted 7 lines
My point is since the nconnect connections are persistent, for the life of the mount, we could end up squatting on ports other services will needed.
Maybe there is not much we can do about this... But we should explain somewhere, like the man page, that nconnect will create up to 16 persistent connection on register privilege ports.
If users have a need to run servers on a port that might chosen by a kernel nfs, lockd or rpcbind client, then they can guarantee no collisions by redefining the 'privileged ports' available to sunrpc to any arbitrary range <portnr1> - <portnr2>:
Either
* Reserve the ports at module load time, by adding a line to a config file /etc/modprobe.d/foo.conf of the form
options sunrpc min_resvport=<portnr1> max_resvport=<portnr2>
or
* Change the port reservation after module load (but before mounting your first NFS filesystem) using
echo '<portnr1>' > /sys/module/sunrpc/parameters/min_resvport echo '<portnr2>' > /sys/module/sunrpc/parameters/max_resvport
This is something users ought to be doing already if they need guaranteed availability of specific ports in the range 665-1023 while using the NFS client. That is entirely independent of whether or not they are using nconnect.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Trond, Anna,
If it's not too late then I'd like to re-tract this patch. I understand if somebody else would really like to have this functionality in then please speak up. If it's too late to pull I also understand but I need to at least try.
Thank you.
Quoted 24 lines · Olga Kornievskaia
From: Olga Kornievskaia <[email protected]>
Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/nfs4client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4client.c +1
-2@@ -881,7 +881,7 @@ static int nfs4_set_client(struct nfs_server *server,if (minorversion ==0 )__set_bit (NFS_CS_REUSEPORT , &cl_init.init_flags); -else if (proto ==XPRT_TRANSPORT_TCP ) +if (proto ==XPRT_TRANSPORT_TCP ) cl_init.nconnect = nconnect;if (server->flags &NFS_MOUNT_NORESVPORT ) --1.8.3.1