xprt_wait_for_buffer_space changes causes a hang.
Subsystems (3)
Hi,
We have a customer who reports occasional but reproducible hangs on our 3.0 based kernel.
I managed to deduce that
commit a9a6b52ee1baa865283a91eb8d443ee91adfca56
Author: Trond Myklebust <[email protected]> Date: Fri Feb 22 14:57:57 2013 -0500
SUNRPC: Don't start the retransmission timer when out of socket space
was to blame (it got into our kernel through -stable ... not sure why it deserved to be in -stable). Reverting that patch fixes the problem. However I don't fully understand why.
Analysing the logs suggests that the NFS client had tried to write a request to the socket, failed as the transmit queue was temporarily full, and never got the call-back when the queue emptied. All other requests piled up behind this one and all traffic stopped.
With the above commit reverted, the client will timeout and retry the transmit even if the call-back hasn't arrived. With the patch, it waits indefinitely.
This suggests that there is a race in the SOCK_ASYNC_NOSPACE handling.
The most obvious place for a race would be in xs_nospace() between
if (test_bit(SOCK_ASYNC_NOSPACE, &transport->sock->flags)) {and before
xprt_wait_for_buffer_space(task, xs_nospace_callback);
However xprt->transport_lock is held here, and xprt_write_space() will wait for the lock before calling rpc_wake_up_queued_task(), so the wakeup cannot happen before the task is put on the pending list.
My only remaining theory is that the space is freed before SOCK_NOSPACE is set in xs_nospace(). I think this would mean that the callback never happens.
In net/ipv4/tcp.c, in tcp_poll() there is code:
if (sk_stream_is_writeable(sk)) {
mask |= POLLOUT | POLLWRNORM;
} else { /* send SIGIO later */
set_bit(SOCK_ASYNC_NOSPACE,
&sk->sk_socket->flags);
set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
/* Race breaker. If space is freed after
* wspace test but before the flags are set,
* IO signal will be lost.
*/
if (sk_stream_is_writeable(sk))
mask |= POLLOUT | POLLWRNORM;
}which suggests there is room for a race, and seems to suggest that it is safest to test sk_stream_is_writeable() after setting SOCK_NOSPACE. This is what the sunrpc server side does for udp (svc_udp_has_wspace). For tcp it doesn't something which I don't really understand, so maybe I'm missing something.
Any thoughts? I could try asking the customer to test with an extra check for sk_stream_is_writeable before calling xprt_wait_for_buffer_space(), but I wouldn't be surprised if they'd rather not given they have a working solution. Does someone understand this code enough to be confident that such a test would be correct?
Thanks,
NeilBrown
Quoted 6 lines · NeilBrown
Hi,
We have a customer who reports occasional but reproducible hangs on our 3.0 based kernel.
I managed to deduce thatcommit
a9a6b52ee1baa865283a91eb8d443ee91adfca56
Quoted 9 lines
Author: Trond Myklebust <[email protected]> Date: Fri Feb 22 14:57:57 2013 -0500SUNRPC: Don't start the retransmission timer when out of socket space
was to blame (it got into our kernel through -stable ... not sure why it deserved to be in -stable). Reverting that patch fixes the problem. However I don't fully understand why.
The reason why that patch was put into stable was that the connection breakage triggered by the timeouts was causing nasty behaviour when servers (or the network) are heavily loaded. Instead of clearing the logjam, breaking the connection and then reconnecting would aggravate it, causing hangs.
Anyhow, does the following patch help to break the race? 8<------------------------------------------------------------------
Quoted 1 line
From
e4c0373be4b8deae2667a7478d34415b99924abcMon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Tue, 11 Feb 2014 09:15:54 -0500
Subject: [PATCH] SUNRPC: Fix races in xs_nospace()
When a send failure occurs due to the socket being out of buffer space, we call xs_nospace() in order to have the RPC task wait until the socket has drained enough to make it worth while trying again. The current patch fixes a race in which the socket is drained before we get round to setting up the machinery in xs_nospace(), and which is reported to cause hangs.
--- net/sunrpc/xprtsock.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
net/sunrpc/xprtsock.c +5 -1
@@ -510,6 +510,7 @@ static int xs_nospace(struct rpc_task *task)struct rpc_rqst *req = task->tk_rqstp;struct rpc_xprt *xprt = req->rq_xprt; struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); +struct sock *sk = transport->inet;int ret = -EAGAIN ; dprintk("RPC: %5u xmit incomplete (%u left of %u)\n",
@@ -527,7 +528,7 @@ static int xs_nospace(struct rpc_task *task)* window size */ set_bit (SOCK_NOSPACE , &transport->sock->flags); - transport->inet->sk_write_pending++; + sk->sk_write_pending++;/* ...and wait for more buffer space */ xprt_wait_for_buffer_space (task, xs_nospace_callback); }
@@ -537,6 +538,9 @@ static int xs_nospace(struct rpc_task *task) }spin_unlock_bh (&xprt->transport_lock); + +/* Race breaker in case memory is freed before above code is called */ + sk->sk_write_space (sk);return ret; }
Signature · 7 lines
1.8.5.3 -- Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Quoted 17 lines · On Tue, 11 Feb 2014 09:33:31 -0500 Trond Myklebust…
Quoted 6 lines · NeilBrown
Hi,
We have a customer who reports occasional but reproducible hangs on our 3.0 based kernel.
I managed to deduce thatcommit
a9a6b52ee1baa865283a91eb8d443ee91adfca56Quoted 9 lines
Author: Trond Myklebust <[email protected]> Date: Fri Feb 22 14:57:57 2013 -0500SUNRPC: Don't start the retransmission timer when out of socket space
was to blame (it got into our kernel through -stable ... not sure why it deserved to be in -stable). Reverting that patch fixes the problem. However I don't fully understand why.
Quoted 5 lines
The reason why that patch was put into stable was that the connection breakage triggered by the timeouts was causing nasty behaviour when servers (or the network) are heavily loaded. Instead of clearing the logjam, breaking the connection and then reconnecting would aggravate it, causing hangs.
Ahh, that make sense. Thanks.
Quoted 46 lines
Anyhow, does the following patch help to break the race? 8<------------------------------------------------------------------
Quoted 1 line
From
e4c0373be4b8deae2667a7478d34415b99924abcMon Sep 17 00:00:00 2001From: Trond Myklebust <[email protected]>
Date: Tue, 11 Feb 2014 09:15:54 -0500
Subject: [PATCH] SUNRPC: Fix races inxs_nospace()When a send failure occurs due to the socket being out of buffer space, we call
xs_nospace()in order to have the RPC task wait until the socket has drained enough to make it worth while trying again. The current patch fixes a race in which the socket is drained before we get round to setting up the machinery inxs_nospace(), and which is reported to cause hangs.Fixes a9a6b52ee1ba (SUNRPC: Don't start the retransmission timer...)Reported-by Neil Brown <[email protected]>Signed-off-by Trond Myklebust <[email protected]>--- net/sunrpc/xprtsock.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/sunrpc/xprtsock.c +2
-1@@ -510,6 +510,7 @@ static int xs_nospace(struct rpc_task *task)struct rpc_rqst *req = task->tk_rqstp;struct rpc_xprt *xprt = req->rq_xprt; struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); +struct sock *sk = transport->inet;int ret = -EAGAIN ; dprintk("RPC: %5u xmit incomplete (%u left of %u)\n",@@ -527,7 +528,7 @@ static int xs_nospace(struct rpc_task *task)* window size */ set_bit (SOCK_NOSPACE , &transport->sock->flags); - transport->inet->sk_write_pending++; + sk->sk_write_pending++;/* ...and wait for more buffer space */ xprt_wait_for_buffer_space (task, xs_nospace_callback); }
net/sunrpc/xprtsock.c static int xs_nospace(struct rpc_task *task) · line 546
Quoted 9 lines
net/sunrpc/xprtsock.c +3
-0@@ -537,6 +538,9 @@ static int xs_nospace(struct rpc_task *task) }spin_unlock_bh (&xprt->transport_lock); + +/* Race breaker in case memory is freed before above code is called */ + sk->sk_write_space (sk);return ret; }
I looks good. I've asked if the customer is willing to test it and provided the patch.
By the way, this bug is the first time that I've found the tasklisk - printed when you enable rpc debuging - useful. And it was *really* useful!
There were lots of
nfsv3 WRITE a:call_reserveresult q:xprt_backlog
and a few
nfsv3 WRITE a:call_status q:xprt_sending
and one
nfsv3 WRITE a:call_transmit_status q:xprt_pending
That last one was put to sleep by xprt_wait_for_buffer_space() and is blocking all the rest.
Now I've got another bug with vaguely similar symptoms and only
9bfb000 (null) 0 ffffffffa03aa0a0 nfsv3 READ a:call_reserveresult q:xprt_sending 9bfb000 (null) 0 ffffffffa03aa0a0 nfsv3 READ a:call_reserveresult q:xprt_sending a933000 (null) 0 ffffffffa03084e0 nfsv3 ACCESS a:call_reserveresult q:xprt_sending
I suspect it is a different bug. Maybe some missing backports or something, though we have 961a828df64979d2a which is related code.
Thanks,
NeilBrown
NeilBrown
Reviewed-byQuoted 77 lines · On Wed, 12 Feb 2014 15:05:55 +1100 NeilBrown <[email protected]> wrote:
Quoted 17 lines · On Tue, 11 Feb 2014 09:33:31 -0500 Trond Myklebust…
Quoted 6 lines · NeilBrown
Hi,
We have a customer who reports occasional but reproducible hangs on our 3.0 based kernel.
I managed to deduce thatcommit
a9a6b52ee1baa865283a91eb8d443ee91adfca56Quoted 9 lines
Author: Trond Myklebust <[email protected]> Date: Fri Feb 22 14:57:57 2013 -0500SUNRPC: Don't start the retransmission timer when out of socket space
was to blame (it got into our kernel through -stable ... not sure why it deserved to be in -stable). Reverting that patch fixes the problem. However I don't fully understand why.
Quoted 5 lines
The reason why that patch was put into stable was that the connection breakage triggered by the timeouts was causing nasty behaviour when servers (or the network) are heavily loaded. Instead of clearing the logjam, breaking the connection and then reconnecting would aggravate it, causing hangs.
Ahh, that make sense. Thanks.
Quoted 48 lines
Anyhow, does the following patch help to break the race? 8<------------------------------------------------------------------
Quoted 1 line
From
e4c0373be4b8deae2667a7478d34415b99924abcMon Sep 17 00:00:00 2001From: Trond Myklebust <[email protected]>
Date: Tue, 11 Feb 2014 09:15:54 -0500
Subject: [PATCH] SUNRPC: Fix races inxs_nospace()When a send failure occurs due to the socket being out of buffer space, we call
xs_nospace()in order to have the RPC task wait until the socket has drained enough to make it worth while trying again. The current patch fixes a race in which the socket is drained before we get round to setting up the machinery inxs_nospace(), and which is reported to cause hangs.Fixes a9a6b52ee1ba (SUNRPC: Don't start the retransmission timer...)Reported-by Neil Brown <[email protected]>Signed-off-by Trond Myklebust <[email protected]>--- net/sunrpc/xprtsock.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/sunrpc/xprtsock.c +2
-1@@ -510,6 +510,7 @@ static int xs_nospace(struct rpc_task *task)struct rpc_rqst *req = task->tk_rqstp;struct rpc_xprt *xprt = req->rq_xprt; struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); +struct sock *sk = transport->inet;int ret = -EAGAIN ; dprintk("RPC: %5u xmit incomplete (%u left of %u)\n",@@ -527,7 +528,7 @@ static int xs_nospace(struct rpc_task *task)* window size */ set_bit (SOCK_NOSPACE , &transport->sock->flags); - transport->inet->sk_write_pending++; + sk->sk_write_pending++;/* ...and wait for more buffer space */ xprt_wait_for_buffer_space (task, xs_nospace_callback); }@@ -537,6 +538,9 @@ static int xs_nospace(struct rpc_task *task) }
Quoted 10 lines
net/sunrpc/xprtsock.c static int xs_nospace(struct rpc_task *task) · line 546
Quoted 7 lines
net/sunrpc/xprtsock.c +3
-0spin_unlock_bh (&xprt->transport_lock); + +/* Race breaker in case memory is freed before above code is called */ + sk->sk_write_space (sk);return ret; }I looks good. I've asked if the customer is willing to test it and provided the patch.
Unfortunately the customer was not able to get internal approval to test this patch, as they already have a working solution.
So we might have to go with this patch without confirmation that it fixes this particular problem. I've examined it again and I cannot see any room for any new races, and it should fix a credible problem.
So
Thanks,
NeilBrown