xprt_bindresvport
Subsystems (3)
Quoted 6 lines
the current
xprt_bindresvportimplementation will search for=20 a privileged
port by counting down from 800 to 0. I think this is a bug, because it will potentially interfere with services trying to bind to=20 low ports as
well.
is this idle speculation, or do you actually have a test case that fails? :^)
Quoted 2 lines
The bindresvport implementation in glibc picks from the 600-1023 range.
we should review what other RPC implementations do (namely the reference implementation, Solaris).
but also notice this cuts the usable port range in half (from ~800 to ~420). we need some form of mitigation to ensure we aren't limiting the number of NFS mounts a client can have.
Quoted 9 lines
I also think it would be good to start at a "random" port. Otherwise, when you reboot, the server may still have a TCB for the old=20 connection
and send you an ACK probe when you try to connect (if all=20 goes well), and
the client's TCP stack will RST and fail the connect. If things go not-so-well you have a packet filter somewhere inbetween that eats the ACK probe because its connection tracking engine thinks the connection is in half-open and shouldn't see any SYN-less ACKs yet.
i don't agree. you're just making the bad behavior more rare by choosing ports at random; you are not actually addressing the root problem you describe. is it possible to fix the RPC client to retry the connection in this case? and/or fix the server to recognize and remove the context for the old connection?
introducing randomness here will make reproducing problems in this area very difficult.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
PATCH 1/1 xprt sharing (was Re: xprt_bindresvport)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 11 lines · Charles
Quoted 6 lines
the current
xprt_bindresvportimplementation will search for a privileged
port by counting down from 800 to 0. I think this is a bug, because it will potentially interfere with services trying to bind to low ports as
well.is this idle speculation, or do you actually have a test case that fails? :^)
Well, I haven't seen this 'interfere' with services yet, I can imagine that it is plausible for a service to want to grab some port at a later time only to have it in use by nfs.
Quoted 11 lines
Quoted 3 lines
The bindresvport implementation in glibc picks from the 600-1023 range.
we should review what other RPC implementations do (namely the reference implementation, Solaris).
but also notice this cuts the usable port range in half (from ~800 to ~420). we need some form of mitigation to ensure we aren't limiting the number of NFS mounts a client can have.
This has been bugging me for a while. The fact that we are limitting ourselves to a single nfs mount per port. From what I can tell, Solaris shares the transports between nfs mounts from the same server and saves themselves a lot of trouble with running out of port numbers in doing so.
The attached patch does the same for Linux against 2.6.9. We share xprts from existing connections, effectively removing any limit on the number of nfs mounts we have in the system.
The only thing to worry about now is any talking to the portmapper or mountd from userspace using tcp, which will put the reserved ports in TIME_WAIT state. This can limit the 'speed' at which we mount many mounts.
- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBt0VQdQs4kOxk3/MRAjo8AKCJjoqPxk1R3ev+o+UyquE0kiw0LQCfebi8 6mfhMSHLidFslVyt6jFeNis=
=0uCI
-----END PGP SIGNATURE-----
This patch allows for sharing of xprts. This is done by keeping a list of current xprts and passing them back to the caller of xprt_create_proto if they match the specifications required (IP X port X protocol X timeout).
We do this multiplexing at the xprt layer as it handles transport creation and destruction.
This patch has been tested in a test-only environment but has been able to handle a couple hundreds distinct nfs mounts from the same server over a single tcp stream.
This effectively gets rid of the 800 nfs mounts max problem, as long as you aren't mounting from many (800) nfs servers.
include/linux/sunrpc/xprt.h +5 -0
@@ -15,6 +15,8 @@#include <linux/sunrpc/sched.h> #include <linux/sunrpc/xdr.h> +#include <asm/atomic.h> +/* * The transport code maintains an estimate on the maximum number of out- * standing RPC requests, using a smoothed version of the congestion
@@ -194,6 +196,9 @@ struct rpc_xprt {void (*old_write_space)(struct sock *);wait_queue_head_t cong_wait; + +atomic_t count;/* shared xprt refcount */ +struct list_head shared;/* link to shared list */ };#ifdef __KERNEL__
net/sunrpc/xprt.c +90 -4
133 lines not on this page · raw message
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
Quoted 4 lines · Mike Waychison
This has been bugging me for a while. The fact that we are limitting ourselves to a single nfs mount per port. From what I can tell, Solaris shares the transports between nfs mounts from the same server and saves themselves a lot of trouble with running out of port numbers in doing so.
Shouldn't we allow NFS mounts to use non-privileged ports? Many environments don't really care about the "security" provided by privileged ports, but would be more than happy if they can run with a few hundred NFS mounts
Olaf
Signature · 13 lines
Olaf Kirch | Things that make Monday morning interesting, #2: [email protected] | "We have 8,000 NFS mount points, why do we keep ---------------+ running out of privileged ports?" ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
to den 09.12.2004 Klokka 12:31 (+0100) skreiv Olaf Kirch:
Quoted 4 lines
Shouldn't we allow NFS mounts to use non-privileged ports? Many environments don't really care about the "security" provided by privileged ports, but would be more than happy if they can run with a few hundred NFS mounts
Most AUTH_SYS based models still require it, however I agree that use of strong security makes the privileged port totally redundant.
Cheers,
Trond
Signature · 12 lines
Trond Myklebust <[email protected]> ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
Quoted 7 lines · Trond Myklebust
Quoted 4 lines
Shouldn't we allow NFS mounts to use non-privileged ports? Many environments don't really care about the "security" provided by privileged ports, but would be more than happy if they can run with a few hundred NFS mounts
Most
AUTH_SYSbased models still require it, however I agree that use of strong security makes the privileged port totally redundant.
The Linux auth_sys works fine with unprivileged ports if you allow it to; so why shouldn't we make that configurable on the client too? It'd sure help some installations who for some reason or other habe an excessive number of exported file systems (see .sig below :).
Olaf
Signature · 13 lines
Olaf Kirch | Things that make Monday morning interesting, #2: [email protected] | "We have 8,000 NFS mount points, why do we keep ---------------+ running out of privileged ports?" ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
to den 09.12.2004 Klokka 14:44 (+0100) skreiv Olaf Kirch:
Quoted 4 lines
The Linux
auth_sysworks fine with unprivileged ports if you allow it to; so why shouldn't we make that configurable on the client too? It'd sure help some installations who for some reason or other habe an excessive number of exported file systems (see .sig below :).
_Another_ mount option? Urgh... 8-)
Seriously: if you have 8000 NFS mount points, you do not want 8000 different superblocks, 8000 different RPC client structs, and 8000 different sockets.
Apart from being a ridiculous waste of resources, that does little to pop the contention bubble. It just ends up pushing it down to the (shared) device layer.
So while I am open to the idea of making use of unprivileged ports, I do not accept it as a substitute for socket sharing.
Cheers,
Trond
Signature · 12 lines
Trond Myklebust <[email protected]> ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
In reply to Olaf KirchQuoted 10 lines · Olaf Kirch
Quoted 4 lines · Mike Waychison
This has been bugging me for a while. The fact that we are limitting ourselves to a single nfs mount per port. From what I can tell, Solaris shares the transports between nfs mounts from the same server and saves themselves a lot of trouble with running out of port numbers in doing so.
Shouldn't we allow NFS mounts to use non-privileged ports? Many environments don't really care about the "security" provided by privileged ports, but would be more than happy if they can run with a few hundred NFS mounts
IMO, this is a good time to apply the principle: "Be liberal in what you accept, and conservative in what you send".
Last I heard, windows didn't even have a concept of a reserved port.
When I wrote a BSD-compatible printsystem in python, I made it accept connections from any port, but generate connections only from reserved ports.
It'd probably be worthwhile to have options to make NFS (and my printsystem) generate any port (not just reserved ones), and accept only reserved ports - but the default probably should be to accept any port, and send only reserved ports - not because reserved ports are effective at all, but because it'll avoid never ending questions about why NFS isn't working.
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
to den 09.12.2004 Klokka 11:34 (-0800) skreiv Dan Stromberg:
Quoted 6 lines
It'd probably be worthwhile to have options to make NFS (and my printsystem) generate any port (not just reserved ones), and accept only reserved ports - but the default probably should be to accept any port, and send only reserved ports - not because reserved ports are effective at all, but because it'll avoid never ending questions about why NFS isn't working.
Sure. The questions will no longer read "why isn't NFS working". They'll read "why can any Tom, Dick and Harry suddenly read my private email directly from the NFS server?". 8-)
The standard AUTH_SYS/AUTH_UNIX authentication scheme only checks the source IP address, and trusts the client 100% when it comes to supplying the correct uid/gid/.... (see RFC1831). By placing the additional requirement that the source must be a privileged port, one is at least able to prevent ordinary users on an authorized client from being able to spoof NFS requests.
Cheers,
Trond
Signature · 12 lines
Trond Myklebust <[email protected]> ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)
Quoted 20 lines · Trond Myklebust
to den 09.12.2004 Klokka 11:34 (-0800) skreiv Dan Stromberg:
Quoted 6 lines
It'd probably be worthwhile to have options to make NFS (and my printsystem) generate any port (not just reserved ones), and accept only reserved ports - but the default probably should be to accept any port, and send only reserved ports - not because reserved ports are effective at all, but because it'll avoid never ending questions about why NFS isn't working.
Sure. The questions will no longer read "why isn't NFS working". They'll read "why can any Tom, Dick and Harry suddenly read my private email directly from the NFS server?". 8-)
The standard AUTH_SYS/AUTH_UNIX authentication scheme only checks the source IP address, and trusts the client 100% when it comes to supplying the correct uid/gid/.... (see RFC1831). By placing the additional requirement that the source must be a privileged port, one is at least able to prevent ordinary users on an authorized client from being able to spoof NFS requests.
Quoted 2 lines
Cheers,
Trond
I bow before your superior geekiness. :)
Quoted 7 lines · Charles
Quoted 4 lines
port by counting down from 800 to 0. I think this is a bug, because it will potentially interfere with services trying to bind to low ports as
well.is this idle speculation, or do you actually have a test case that fails? :^)
I've seen many cases of glibc's bindresvport (which allocates from 600-1023) snatching ports used by other services, e.g. cups. The way we solve this in user space is by having a file in /etc with a blacklist of ports bindresvport isn't allowed to touch.
I admit that I haven't seen NFS step on someone else toes yet. But it all depends on the number of mounts you have, and on coincidence.
At any rate, it really looks like a bug to me. Why should our scan include ports in the low range (which is a no-no) but refuse to touch ports in the 800-1023 range? It's just an inverted test, IMO.
Quoted 5 lines
i don't agree. you're just making the bad behavior more rare by choosing ports at random; you are not actually addressing the root problem you describe. is it possible to fix the RPC client to retry the connection in this case? and/or fix the server to recognize and remove the context for the old connection?
You cannot fix the RPC client. It will not see any packets from the server, so it simply sits there until it times out and retries the connection. This can cause system boot to take quite a long time.
You can fix the firewall, this is currently being tossed around on the netfilter list. Unfortunately not all routers out there run the latest and greatest netfilter code; so we don't really have much control over that.
Quoted 2 lines
introducing randomness here will make reproducing problems in this area very difficult.
I agree. And I would only enable randomization if we change the port range to 600-up or something like this.
Olaf
Signature · 13 lines
Olaf Kirch | Things that make Monday morning interesting, #2: [email protected] | "We have 8,000 NFS mount points, why do we keep ---------------+ running out of privileged ports?" ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs