SUNRPC: Avoid address overwrite with eBPF NAT
Subsystems (3)
PATCH 1/1 SUNRPC: Avoid address overwrite with eBPF NAT
kernel_connect() will modify the rpc_xprt socket address in contexts where eBPF programs perform NAT instead of iptables. In these contexts, it is common for an NFS mount to be mounted to be a static virtual IP while the server has an ephemeral IP leading to a problem where the virtual IP gets overwritten and forgotten. When the endpoint IP changes, reconnect attempts fail and the mount never recovers.
This patch protects addr from being modified in these scenarios, allowing NFS reconnects to work as intended.
--- include/linux/sunrpc/xprt.h | 1 + net/sunrpc/xprtsock.c | 17 +++++++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
include/linux/sunrpc/xprt.h +1 -0
@@ -211,6 +211,7 @@ struct rpc_xprt {const struct rpc_timeout *timeout;/* timeout parms */ struct sockaddr_storage addr;/* server address */ +struct sockaddr_storage m_addr;/* mutable server address */ size_t addrlen;/* size of server address */ int prot;/* IP protocol */
net/sunrpc/xprtsock.c +15 -2
@@ -236,6 +236,18 @@ static inline struct sockaddr *xs_addr(struct rpc_xprt *xprt)return (struct sockaddr *) &xprt->addr; } +static inline struct sockaddr *xs_m_addr (struct rpc_xprt *xprt) +{ +/* kernel_connect() may modify the address in contexts where NAT is +* performed by eBPF programs instead of iptables. Make a copy to ensure +* that our original address, xprt->addr, is not modified. Without this, +* NFS reconnects may fail if the endpoint address changes. +*/ +memcpy (&xprt->m_addr, &xprt->addr, xprt->addrlen); + +return (struct sockaddr *) &xprt->m_addr; +} +static inline struct sockaddr_un *xs_addr_un (struct rpc_xprt *xprt) {return (struct sockaddr_un *) &xprt->addr;
@@ -1954,7 +1966,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,xs_stream_start_connect (transport); -return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,0 ); +return kernel_connect (sock,xs_m_addr (xprt), xprt->addrlen,0 ); }/**
@@ -2334,7 +2346,8 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)/* Tell the socket layer to start connecting... */ set_bit (XPRT_SOCK_CONNECTING , &transport->sock_state); -return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,O_NONBLOCK ); + +return kernel_connect (sock,xs_m_addr (xprt), xprt->addrlen,O_NONBLOCK ); }/**
Signature · 1 line
2.42.0.rc1.204.g551eb34607-goog
Quoted 11 lines · Jordan Rife
kernel_connect()will modify therpc_xprtsocket address in contexts where eBPF programs perform NAT instead of iptables. In these contexts, it is common for an NFS mount to be mounted to be a static virtual IP while the server has an ephemeral IP leading to a problem where the virtual IP gets overwritten and forgotten. When the endpoint IP changes, reconnect attempts fail and the mount never recovers.
Quoted 5 lines
This patch protects addr from being modified in these scenarios, allowing NFS reconnects to work as intended.
Hello Jordan, since kernel_connect() is used exclusively by the RPC client, I suggest directing your patch to Trond and Anna.
<[email protected] <mailto:[email protected]>> <[email protected] <mailto:[email protected]>>
Does the RPC/RDMA client also have this issue? It does not use kernel_connect().
Quoted 62 lines
---
include/linux/sunrpc/xprt.h| 1 +net/sunrpc/xprtsock.c| 17 +++++++++++++++--2 files changed, 16 insertions(+), 2 deletions(-)include/linux/sunrpc/xprt.h +0
-0@@ -211,6 +211,7 @@ struct rpc_xprt {const struct
rpc_timeout*timeout; /* timeout parms */ structsockaddr_storageaddr; /* server address */
+ structsockaddr_storagem_addr; /* mutable server address */size_taddrlen; /* size of server address */ int prot; /* IP protocol */net/sunrpc/xprtsock.c +15
-2@@ -236,6 +236,18 @@ static inline struct sockaddr *xs_addr(struct rpc_xprt *xprt)return (struct sockaddr *) &xprt->addr; } +static inline struct sockaddr *xs_m_addr (struct rpc_xprt *xprt) +{ +/* kernel_connect() may modify the address in contexts where NAT is +* performed by eBPF programs instead of iptables. Make a copy to ensure +* that our original address, xprt->addr, is not modified. Without this, +* NFS reconnects may fail if the endpoint address changes. +*/ +memcpy (&xprt->m_addr, &xprt->addr, xprt->addrlen); + +return (struct sockaddr *) &xprt->m_addr; +} +static inline struct sockaddr_un *xs_addr_un (struct rpc_xprt *xprt) {return (struct sockaddr_un *) &xprt->addr;@@ -1954,7 +1966,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,xs_stream_start_connect (transport); -return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,0 ); +return kernel_connect (sock,xs_m_addr (xprt), xprt->addrlen,0 ); }/** @@ -2334,7 +2346,8 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)/* Tell the socket layer to start connecting... */ set_bit (XPRT_SOCK_CONNECTING , &transport->sock_state); -return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,O_NONBLOCK ); + +return kernel_connect (sock,xs_m_addr (xprt), xprt->addrlen,O_NONBLOCK ); }/** Signature · 1 line
2.42.0.rc1.204.g551eb34607-goog
--
Chuck Lever
Quoted 12 lines · Jordan Rife
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
kernel_connect()will modify therpc_xprtsocket address in contexts where eBPF programs perform NAT instead of iptables. In these contexts,
it is common for an NFS mount to be mounted to be a static virtual IP while the server has an ephemeral IP leading to a problem where the virtual IP gets overwritten and forgotten. When the endpoint IP changes,
reconnect attempts fail and the mount never recovers.
Quoted 3 lines
This patch protects addr from being modified in these scenarios, allowing
NFS reconnects to work as intended.
What? No! A connect() call should not be allowed to modify its own call parameters.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 15 lines · Trond Myklebust
Quoted 14 lines · Jordan Rife
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
kernel_connect()will modify therpc_xprtsocket address in contexts
where eBPF programs perform NAT instead of iptables. In these contexts,
it is common for an NFS mount to be mounted to be a static virtual IP
while the server has an ephemeral IP leading to a problem where the virtual IP gets overwritten and forgotten. When the endpoint IP changes,
reconnect attempts fail and the mount never recovers.
Quoted 8 lines
Quoted 3 lines
This patch protects addr from being modified in these scenarios, allowing
NFS reconnects to work as intended.What? No! A
connect()call should not be allowed to modify its own call
parameters.
To put it more succinctly, the struct rpc_xprt is one of many private kernel structures. Parts of it can be exposed through public APIs, such as the sysfs API that we're building, but when you use eBPF to hack your way around those public APIs, then you're on your own. We're not going to commit to support your hacks.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]