NFSv3 client RDMA multipath enhancements
Subsystems (3)
cover NFSv3 client RDMA multipath enhancements
Hi,
The purpose of the following changes is to allow specifying multiple target IP addresses in a single mount. Combining this with nconnect and servers that support exposing multiple ports, we can achieve load balancing and much greater throughput, especially on RDMA setups, even with the older NFSv3 protocol.
The changes allow specifing a new remoteports=<IP-addresses-ranges> mount option providing a group of IP addresses, from which nconnect at sunrpc scope picks target transport address in round-robin. There's also an accompanying localports parameter that allows local address bind so that the source port is better controlled in a way to ensure that transports are not hogging a single local interface.
This patchset targets the linux-next tree.
Dan Aloni (5):
sunrpc: Allow specifying a vector of IP addresses for nconnect
xprtrdma: Bind to a local address if requested
nfs: Extend nconnect with remoteports and localports mount params
sunrpc: Add srcaddr to xprt sysfs debug
nfs: Increase NFS_MAX_CONNECTIONS
fs/nfs/client.c | 24 +++ fs/nfs/fs_context.c | 173 ++++++++++++++++++++- fs/nfs/internal.h | 4 + include/linux/nfs_fs_sb.h | 2 + include/linux/sunrpc/clnt.h | 9 ++ include/linux/sunrpc/xprt.h | 1 + net/sunrpc/clnt.c | 47 ++++++ net/sunrpc/debugfs.c | 8 +- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/transport.c | 17 +- net/sunrpc/xprtrdma/verbs.c | 15 +- net/sunrpc/xprtrdma/xprt_rdma.h | 5 +- net/sunrpc/xprtsock.c | 49 +++--- 13 files changed, 329 insertions(+), 27 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/5 sunrpc: Allow specifying a vector of IP addresses for nconnect
This adds an rpc_portgroup structure to describe a group of IP addresses comprising one logical server with multiple ports. The remote endpoint can be in a single server exposing multiple network interfaces, or multiple remote machines implementing a distributed server architecture.
Combined with nconnect, the multiple transports try to make use of the multiple addresses given so that the connections are spread across the ports.
--- include/linux/sunrpc/clnt.h | 9 +++++++ net/sunrpc/clnt.c | 47 +++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/sunrpc/clnt.h +9 -0
@@ -115,12 +115,21 @@ struct rpc_procinfo {const char * p_name;/* name of procedure */ }; +#define RPC_MAX_PORTS 64 + +struct rpc_portgroup { +int nr; +struct sockaddr_storage addrs[RPC_MAX_PORTS ]; +}; +struct rpc_create_args {struct net *net;int protocol;struct sockaddr *address;size_t addrsize;struct sockaddr *saddress; +struct rpc_portgroup *localports;/* additional local addresses */ +struct rpc_portgroup *remoteports;/* additional remote addresses */ const struct rpc_timeout *timeout;const char *servername;const char *nodename;
net/sunrpc/clnt.c +47 -0
71 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 2/5 xprtrdma: Bind to a local address if requested
In reply to Dan AloniUntil now, rpcrdma did not make use of the local address when binding a QP. For subnets where the local machine has multiple IP addresses that are all connected to the same subnet, it may be desired to tell from which interface a QP is going out.
--- net/sunrpc/xprtrdma/transport.c | 7 +++++++ net/sunrpc/xprtrdma/verbs.c | 15 ++++++++++++++- net/sunrpc/xprtrdma/xprt_rdma.h | 2 ++ 3 files changed, 23 insertions(+), 1 deletion(-)
net/sunrpc/xprtrdma/transport.c +7 -0
@@ -355,6 +355,13 @@ xprt_setup_rdma(struct xprt_create *args)xprt_rdma_format_addresses (xprt, sap); new_xprt =rpcx_to_rdmax (xprt); + +/* Copy source address if specified */ +if (args->srcaddr) { + new_xprt->rx_has_srcaddr = true; +memcpy (&new_xprt->rx_saddr, args->srcaddr, args->addrlen); + } + rc =rpcrdma_buffer_create (new_xprt);if (rc) {xprt_rdma_free_addresses (xprt);
net/sunrpc/xprtrdma/verbs.c +14 -1
@@ -314,6 +314,7 @@ static struct rdma_cm_id *rpcrdma_create_id(struct rpcrdma_xprt *r_xprt,unsigned long wtimeout =msecs_to_jiffies (RDMA_RESOLVE_TIMEOUT ) +1 ;struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct rdma_cm_id *id; +struct sockaddr *saddr =NULL ;int rc;init_completion (&ep->re_done);
@@ -324,7 +325,19 @@ static struct rdma_cm_id *rpcrdma_create_id(struct rpcrdma_xprt *r_xprt,return id; ep->re_async_rc = -ETIMEDOUT ; - rc =rdma_resolve_addr (id,NULL , (struct sockaddr *)&xprt->addr, +if (r_xprt->rx_has_srcaddr) { +char buf[0x50 ] = {0 , }; + saddr = (struct sockaddr *)&r_xprt->rx_saddr; + +rpc_ntop (saddr, buf,sizeof (buf)); + +dprintk ("xprt=%p, source address port %s\n" , xprt, buf); + } + +dprintk ("xprt=%p, %s:%s resolving addr\n" , xprt, +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt)); + + rc =rdma_resolve_addr (id, saddr, (struct sockaddr *)&xprt->addr,RDMA_RESOLVE_TIMEOUT );if (rc) goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +2 -0
@@ -421,6 +421,8 @@ struct rpcrdma_stats {*/ struct rpcrdma_xprt {struct rpc_xprt rx_xprt; +struct sockaddr_storage rx_saddr; +bool rx_has_srcaddr;struct rpcrdma_ep *rx_ep;struct rpcrdma_buffer rx_buf;struct delayed_work rx_connect_worker;
Signature · 1 line
2.26.2
PATCH 3/5 nfs: Extend nconnect with remoteports and localports mount params
In reply to Dan AloniThe new added mount parameters allow passing a vector of IP addresses to be used with the extra transports that nconnect creates. The remoteports parameter provides the destination addresses, and localports specifies local address binds.
--- fs/nfs/client.c | 24 ++++++ fs/nfs/fs_context.c | 171 ++++++++++++++++++++++++++++++++++++++ fs/nfs/internal.h | 4 + include/linux/nfs_fs_sb.h | 2 + 4 files changed, 201 insertions(+)
fs/nfs/client.c +24 -0
@@ -166,6 +166,18 @@ struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *cl_init)memcpy (&clp->cl_addr, cl_init->addr, cl_init->addrlen); clp->cl_addrlen = cl_init->addrlen; +if (cl_init->localports) { + clp->cl_localports =vmalloc (sizeof (*cl_init->localports)); +if (!clp->cl_localports) + goto error_cleanup; + *clp->cl_localports = *cl_init->localports; + } +if (cl_init->remoteports) { + clp->cl_remoteports =vmalloc (sizeof (*cl_init->remoteports)); +if (!clp->cl_remoteports) + goto error_cleanup; + *clp->cl_remoteports = *cl_init->remoteports; + }if (cl_init->hostname) { err = -ENOMEM ;
@@ -187,6 +199,10 @@ struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *cl_init)return clp; error_cleanup: +if (clp->cl_remoteports) +vfree (clp->cl_remoteports); +if (clp->cl_localports) +vfree (clp->cl_localports);put_nfs_version (clp->cl_nfs_mod); error_dealloc:kfree (clp);
@@ -245,6 +261,10 @@ void nfs_free_client(struct nfs_client *clp)put_net (clp->cl_net);put_nfs_version (clp->cl_nfs_mod); +if (clp->cl_localports) +vfree (clp->cl_localports); +if (clp->cl_remoteports) +vfree (clp->cl_remoteports);kfree (clp->cl_hostname);kfree (clp->cl_acceptor);kfree (clp);
@@ -508,6 +528,8 @@ int nfs_create_rpc_client(struct nfs_client *clp, .nconnect = clp->cl_nconnect, .address = (struct sockaddr *)&clp->cl_addr, .addrsize = clp->cl_addrlen, + .localports = clp->cl_localports, + .remoteports = clp->cl_remoteports, .timeout = cl_init->timeparms, .servername = clp->cl_hostname, .nodename = cl_init->nodename,
@@ -678,6 +700,8 @@ static int nfs_init_server(struct nfs_server *server, .timeparms = &timeparms, .cred = server->cred, .nconnect = ctx->nfs_server.nconnect, + .localports = ctx->localports, + .remoteports = ctx->remoteports, .init_flags = (1UL << NFS_CS_REUSEPORT), };struct nfs_client *clp;
fs/nfs/fs_context.c +171 -0
225 lines not on this page · raw message
fs/nfs/internal.h +4 -0
@@ -63,6 +63,8 @@ struct nfs_client_initdata {const char *nodename;/* Hostname of the client */ const char *ip_addr;/* IP address of the client */ size_t addrlen; +struct rpc_portgroup *localports;/* Local addresses to bind */ +struct rpc_portgroup *remoteports;/* Remote server addresses */ struct nfs_subversion *nfs_mod;int proto;u32 minorversion;
@@ -96,6 +98,8 @@ struct nfs_fs_context {char *fscache_uniq;unsigned short protofamily;unsigned short mountfamily; +struct rpc_portgroup *localports; +struct rpc_portgroup *remoteports; struct { union {
include/linux/nfs_fs_sb.h +2 -0
@@ -50,6 +50,8 @@ struct nfs_client {#define NFS_CS_REUSEPORT 8/* - reuse src port on reconnect */ struct sockaddr_storage cl_addr;/* server identifier */ size_t cl_addrlen; +struct rpc_portgroup * cl_localports;/* Local addresses to bind */ +struct rpc_portgroup * cl_remoteports;/* Remote server addresses */ char * cl_hostname;/* hostname of server */ char * cl_acceptor;/* GSSAPI acceptor name */ struct list_head cl_share_link;/* link in global client list */
Signature · 1 line
2.26.2
PATCH 4/5 sunrpc: Add srcaddr to xprt sysfs debug
In reply to Dan AloniFor easier inspection of transport state with regard to nconnect, with remoteports and localports of NFS, this helps us to know what is the source address used for each established transport.
--- include/linux/sunrpc/xprt.h | 1 + net/sunrpc/debugfs.c | 8 ++-- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/transport.c | 10 ++++- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +- net/sunrpc/xprtsock.c | 49 ++++++++++++++-------- 6 files changed, 48 insertions(+), 25 deletions(-)
include/linux/sunrpc/xprt.h +1 -0
@@ -43,6 +43,7 @@ struct rpc_timeout {enum rpc_display_format_t {RPC_DISPLAY_ADDR =0 , +RPC_DISPLAY_SRC_ADDR ,RPC_DISPLAY_PORT ,RPC_DISPLAY_PROTO ,RPC_DISPLAY_HEX_ADDR ,
net/sunrpc/debugfs.c +5 -3
@@ -175,9 +175,11 @@ xprt_info_show(struct seq_file *f, void *v) {struct rpc_xprt *xprt = f->private; -seq_printf (f,"netid: %s\n" , xprt->address_strings[RPC_DISPLAY_NETID ]); -seq_printf (f,"addr: %s\n" , xprt->address_strings[RPC_DISPLAY_ADDR ]); -seq_printf (f,"port: %s\n" , xprt->address_strings[RPC_DISPLAY_PORT ]); +seq_printf (f,"netid: %s\n" , xprt->address_strings[RPC_DISPLAY_NETID ]); +seq_printf (f,"addr: %s\n" , xprt->address_strings[RPC_DISPLAY_ADDR ]); +seq_printf (f,"port: %s\n" , xprt->address_strings[RPC_DISPLAY_PORT ]); +seq_printf (f,"srcaddr: %s\n" , xprt->address_strings[RPC_DISPLAY_SRC_ADDR ] + ? xprt->address_strings[RPC_DISPLAY_SRC_ADDR ] :"" );seq_printf (f,"state: 0x%lx\n" , xprt->state);return 0 ; }
net/sunrpc/xprtrdma/svc_rdma_backchannel.c +1 -1
@@ -261,7 +261,7 @@ xprt_setup_rdma_bc(struct xprt_create *args)memcpy (&xprt->addr, args->dstaddr, args->addrlen); xprt->addrlen = args->addrlen; -xprt_rdma_format_addresses (xprt, (struct sockaddr *)&xprt->addr); +xprt_rdma_format_addresses (xprt, (struct sockaddr *)&xprt->addr,NULL ); xprt->resvport =0 ; xprt->max_payload = xprt_rdma_max_inline_read;
net/sunrpc/xprtrdma/transport.c +8 -2
@@ -181,7 +181,8 @@ xprt_rdma_format_addresses6(struct rpc_xprt *xprt, struct sockaddr *sap) }void -xprt_rdma_format_addresses (struct rpc_xprt *xprt,struct sockaddr *sap) +xprt_rdma_format_addresses (struct rpc_xprt *xprt,struct sockaddr *sap, +struct sockaddr *src_sap) {char buf[128 ];
@@ -200,6 +201,11 @@ xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap) (void )rpc_ntop (sap, buf,sizeof (buf)); xprt->address_strings[RPC_DISPLAY_ADDR ] =kstrdup (buf,GFP_KERNEL ); +if (src_sap) { + (void )rpc_ntop (src_sap, buf,sizeof (buf)); + xprt->address_strings[RPC_DISPLAY_SRC_ADDR ] =kstrdup (buf,GFP_KERNEL ); + } +snprintf (buf,sizeof (buf),"%u" ,rpc_get_port (sap)); xprt->address_strings[RPC_DISPLAY_PORT ] =kstrdup (buf,GFP_KERNEL );
@@ -352,7 +358,7 @@ xprt_setup_rdma(struct xprt_create *args)if (rpc_get_port (sap))xprt_set_bound (xprt); -xprt_rdma_format_addresses (xprt, sap); +xprt_rdma_format_addresses (xprt, sap, args->srcaddr); new_xprt =rpcx_to_rdmax (xprt);
net/sunrpc/xprtrdma/xprt_rdma.h +2 -1
@@ -573,7 +573,8 @@ static inline void rpcrdma_set_xdrlen(struct xdr_buf *xdr, size_t len)*/ extern unsigned int xprt_rdma_max_inline_read;extern unsigned int xprt_rdma_max_inline_write; -void xprt_rdma_format_addresses (struct rpc_xprt *xprt,struct sockaddr *sap); +void xprt_rdma_format_addresses (struct rpc_xprt *xprt,struct sockaddr *sap, +struct sockaddr *src_sap);void xprt_rdma_free_addresses (struct rpc_xprt *xprt);void xprt_rdma_close (struct rpc_xprt *xprt);void xprt_rdma_print_stats (struct rpc_xprt *xprt,struct seq_file *seq);
net/sunrpc/xprtsock.c +31 -18
129 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 5/5 nfs: Increase NFS_MAX_CONNECTIONS
In reply to Dan AloniThis matches it with the new RPC_MAX_PORTS value.
--- fs/nfs/fs_context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/fs_context.c +1 -1
@@ -28,7 +28,7 @@#define NFS_DEFAULT_VERSION 2#endif -#define NFS_MAX_CONNECTIONS 16 +#define NFS_MAX_CONNECTIONS RPC_MAX_PORTSenum nfs_param { Opt_ac,
Signature · 1 line
2.26.2
Hey Dan-
First, thanks for posting patches!
Quoted 7 lines · Dan Aloni
Hi,
The purpose of the following changes is to allow specifying multiple target IP addresses in a single mount. Combining this with nconnect and servers that support exposing multiple ports,
"port" is probably a bad term to use here, as that term already has a particular meaning when it comes to IP addresses. In standards documents, we've stuck with the term "endpoint".
I worked with the IETF's nfsv4 WG a couple years ago to produce a document that describes how we want NFS servers to advertise their network configuration to clients.
https://datatracker.ietf.org/doc/rfc8587/
That gives a flavor for what we've done for NFSv4. IMO anything done for NFSv3 ought to leverage similar principles and tactics.
Quoted 3 lines
we can achieve load
balancing and much greater throughput, especially on RDMA setups, even with the older NFSv3 protocol.
I support the basic goal of increasing transport parallelism.
As you probably became aware as you worked on these patches, the Linux client shares one or a small set of connections across all mount points of the same server. So a mount option that adds this kind of control is going to be awkward.
Anna has proposed a /sys API that would enable this information to be programmed into the kernel for all mount points sharing the same set of connections. That would be a little nicer for building separate administrator tools against, or even for providing an automation mechanism (like an orchestrator) that would enable clients to automatically fail over to a different server interface.
IMO I'd prefer to see a user space policy / tool that manages endpoint lists and passes them to the kernel client dynamically via Anna's API instead of adding one or more mount options, which would be fixed for the life of the mount and shared with other mount points that use the same transports to communicate with the NFS server.
As far as the NUMA affinity issues go, in the past I've attempted to provide some degree of CPU affinity between RPC Call and Reply handling only to find that it reduced performance unacceptably. Perhaps something that is node-aware or LLC-aware would be better than CPU affinity, and I'm happy to discuss that and any other ways we think can improve NFS behavior on NUMA systems. It's quite true that RDMA transports are more sensitive to NUMA than traditional socket-based ones.
Quoted 16 lines
The changes allow specifing a new
remoteports=<IP-addresses-ranges>mount option providing a group of IP addresses, from whichnconnectat sunrpc scope picks target transport address in round-robin. There's also an accompanyinglocalportsparameter that allows local address bind so that the source port is better controlled in a way to ensure that transports are not hogging a single local interface.This patchset targets the linux-next tree.
Dan Aloni (5):
sunrpc: Allow specifying a vector of IP addresses for nconnect
xprtrdma: Bind to a local address if requested
nfs: Extend nconnect with remoteports and localports mount params
sunrpc: Add srcaddr to xprt sysfs debug
nfs: IncreaseNFS_MAX_CONNECTIONS
Quoted 18 lines
fs/nfs/client.c | 24 +++ fs/nfs/fs_context.c | 173 ++++++++++++++++++++- fs/nfs/internal.h | 4 + include/linux/nfs_fs_sb.h | 2 + include/linux/sunrpc/clnt.h | 9 ++ include/linux/sunrpc/xprt.h | 1 + net/sunrpc/clnt.c | 47 ++++++ net/sunrpc/debugfs.c | 8 +- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/transport.c | 17 +- net/sunrpc/xprtrdma/verbs.c | 15 +- net/sunrpc/xprtrdma/xprt_rdma.h | 5 +- net/sunrpc/xprtsock.c | 49 +++--- 13 files changed, 329 insertions(+), 27 deletions(-)Signature · 1 line
2.26.2
--
Chuck Lever
Quoted 8 lines · Chuck Lever
I worked with the IETF's nfsv4 WG a couple years ago to produce a document that describes how we want NFS servers to advertise their network configuration to clients.
https://datatracker.ietf.org/doc/rfc8587/
That gives a flavor for what we've done for NFSv4. IMO anything done for NFSv3 ought to leverage similar principles and tactics.
Thanks for the pointer - I'll read and take it into consideration.
Quoted 10 lines
Quoted 3 lines
we can achieve load
balancing and much greater throughput, especially on RDMA setups, even with the older NFSv3 protocol.I support the basic goal of increasing transport parallelism.
As you probably became aware as you worked on these patches, the Linux client shares one or a small set of connections across all mount points of the same server. So a mount option that adds this kind of control is going to be awkward.
I tend to agree, from a developer perspective, but just to give an idea that from an admin POV it is often is not immediately apparent that this is what happens behind the scenes (i.e. the nfs_match_client function), so in our case the users have not reported back that our addition to the mount parameters looked weird, considering it as naturally extending nconnect, which I think falls under similar considerations - giving deeper details regarding how transports should behave during the mount command and not afterwards, regarding what actual NFS sessions are established.
Surely there may be better ways to do this, following from what's discussed next.
Quoted 7 lines
Anna has proposed a /sys API that would enable this information to be programmed into the kernel for all mount points sharing the same set of connections. That would be a little nicer for building separate administrator tools against, or even for providing an automation mechanism (like an orchestrator) that would enable clients to automatically fail over to a different server interface.
Quoted 6 lines
IMO I'd prefer to see a user space policy / tool that manages endpoint lists and passes them to the kernel client dynamically via Anna's API instead of adding one or more mount options, which would be fixed for the life of the mount and shared with other mount points that use the same transports to communicate with the NFS server.
I see now that these are fairly recent patches that I've unfortunately missed while working on other things. If this is the intended API to help manage active NFS sessions, I would very much like to help on testing and extending this code.
So a good way to go with this would be to look into supporting an 'add transport' op by extending on the new interface, and for optionally specifying local address bind similarly to the work I've done for the mount options.
I'll also be glad to contribute to nfs-utils so that we'd have the anticipated userspace tool, maybe 'nfs' (like /sbin/ip from iproute), that can executed for this purpose, e.g. 'nfs transport add <IP> mnt <PATH>'.
Also, from a lower level API perspective, we would need a way to figure out client ID from a mount point, so that ID can be used at the relevant sysfs directory. Perhaps this can be done via a new ioctl on the mount point itself?
Quoted 8 lines
As far as the NUMA affinity issues go, in the past I've attempted to provide some degree of CPU affinity between RPC Call and Reply handling only to find that it reduced performance unacceptably. Perhaps something that is node-aware or LLC-aware would be better than CPU affinity, and I'm happy to discuss that and any other ways we think can improve NFS behavior on NUMA systems. It's quite true that RDMA transports are more sensitive to NUMA than traditional socket-based ones.
Also to consider that RDMA is special for this as CPU memory caching can be skipped, and even main memory - for example a special case where the NFS read/write payload memory is not the main system memory but mapped from PCI, and the kernel's own PCI_P2PDMA distance matrix can be used for better xprt selection.
Signature · 1 line
Dan Aloni