SVC: Guard call to xpo_release_rqst in svc_send
Subsystems (3)
PATCH 1/1 SVC: Guard call to xpo_release_rqst in svc_send
The svc_send path is calling xpo_release_rqst without checking the XPT_DEAD bit. It is illegal to call transport methods on a dead transport. In practice, if the transport gets an error and shuts down while there are still RPC in svc_process the resulting svc_send could crash calling into a transport that is being shut down.
---
net/sunrpc/svc_xprt.c +4 -4
@@ -729,9 +729,6 @@ int svc_send(struct svc_rqst *rqstp)if (!xprt)return -EFAULT ; -/* release the receive skb before sending the reply */ - rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); -/* calculate over-all length */ xb = &rqstp->rq_res; xb->len = xb->head[0].iov_len +
@@ -742,8 +739,11 @@ int svc_send(struct svc_rqst *rqstp)mutex_lock (&xprt->xpt_mutex);if (test_bit (XPT_DEAD , &xprt->xpt_flags)) len = -ENOTCONN ; -else +else { +/* release the receive skb before sending the reply */ + rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); len = xprt->xpt_ops->xpo_sendto (rqstp); + }mutex_unlock (&xprt->xpt_mutex);svc_xprt_release (rqstp);
A message that is not in the archive (sent off-list or not yet received): 1204142339.24762.94.camel-SMNkleLxa3ZimH42XvhXlA@public.gmane.org
Quoted 6 lines · Tom Tucker
The
svc_sendpath is callingxpo_release_rqstwithout checking theXPT_DEADbit. It is illegal to call transport methods on a dead transport. In practice, if the transport gets an error and shuts down while there are still RPC insvc_processthe resultingsvc_sendcould crash calling into a transport that is being shut down.
As long as we have a pointer to an xprt (say in rqstp->rq_xprt), we should have a reference on the corresponding xprt class, shouldn't we? Anything else seems like a problem.
Quoted 18 lines
Signed-off-by Tom Tucker <[email protected]>---
net/sunrpc/svc_xprt.c +0
-3@@ -729,9 +729,6 @@ int svc_send(struct svc_rqst *rqstp)if (!xprt)return -EFAULT ; -/* release the receive skb before sending the reply */ - rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); -/* calculate over-all length */ xb = &rqstp->rq_res; xb->len = xb->head[0].iov_len +
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 746
Quoted 9 lines
net/sunrpc/svc_xprt.c +4
-1@@ -742,8 +739,11 @@ int svc_send(struct svc_rqst *rqstp)mutex_lock (&xprt->xpt_mutex);if (test_bit (XPT_DEAD , &xprt->xpt_flags)) len = -ENOTCONN ; -else +else { +/* release the receive skb before sending the reply */ + rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); len = xprt->xpt_ops->xpo_sendto (rqstp); + }
So in the case where XPT_DEAD is set, doesn't the receive skb also need to be freed, or does somebody else do that?
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 748
Quoted 2 lines
net/sunrpc/svc_xprt.c +0
-0mutex_unlock (&xprt->xpt_mutex);svc_xprt_release (rqstp);
Also something else is odd here: svc_xprt_release(rqstp) already calls rqstp->rq_xprt->xpt_ops->xpo_release_rqst(rqstp) itself, so we seem to have two calls to that method.
--b.
Quoted 6 lines
net/sunrpc/svc_xprt.c +0
-1-To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Bruce:
Summary response: Mia culpa....please revert this patch. Although it made the test failure go away, this patch just moves the problem, it doesn't fix anything.
More below inline....
Quoted 8 lines · J. Bruce Fields
Quoted 6 lines · Tom Tucker
The
svc_sendpath is callingxpo_release_rqstwithout checking theXPT_DEADbit. It is illegal to call transport methods on a dead transport. In practice, if the transport gets an error and shuts down while there are still RPC insvc_processthe resultingsvc_sendcould crash calling into a transport that is being shut down.
Quoted 4 lines
As long as we have a pointer to an xprt (say in
rqstp->rq_xprt), we should have a reference on the corresponding xprt class, shouldn't we? Anything else seems like a problem.
Generally yes. In particular, at entry to svc_recv, we should have a minimum of two references. svc_xprt_enqueue gets you one, and there is a one ref bias that comes from svc_xprt_init. The bias gets removed on close.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Tom Tucker <[email protected]>---
net/sunrpc/svc_xprt.c +0
-3@@ -729,9 +729,6 @@ int svc_send(struct svc_rqst *rqstp)if (!xprt)return -EFAULT ; -/* release the receive skb before sending the reply */ - rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); -/* calculate over-all length */ xb = &rqstp->rq_res;
xb->len= xb->head[0].iov_len +net/sunrpc/svc_xprt.c +0
-1@@ -742,8 +739,11 @@ int svc_send(struct svc_rqst *rqstp)mutex_lock (&xprt->xpt_mutex);if (test_bit (XPT_DEAD , &xprt->xpt_flags)) len = -ENOTCONN ; -else
Quoted 9 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 746
Quoted 5 lines
net/sunrpc/svc_xprt.c +4
-0+else { +/* release the receive skb before sending the reply */ + rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); len = xprt->xpt_ops->xpo_sendto (rqstp); + }So in the case where
XPT_DEADis set, doesn't the receive skb also need to be freed, or does somebody else do that?
Only reboot does that :-\. I need to think this through more carefully. On RDMA transports this function is used to return send credits to the user, and that involves talking to the transport. Sorry for this hasty patch...it's not right.
Quoted 7 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 748
Quoted 2 lines
net/sunrpc/svc_xprt.c +0
-0mutex_unlock (&xprt->xpt_mutex);svc_xprt_release (rqstp);Also something else is odd here:
svc_xprt_release(rqstp)already callsrqstp->rq_xprt->xpt_ops->xpo_release_rqst(rqstp)itself, so we seem to have two calls to that method.
Hmm. Busted. This patch is broken in three dimensions. Is that a record?
Tom
Quoted 12 lines
--b.
Quoted 6 lines
net/sunrpc/svc_xprt.c +0
-1-To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Tom Tucker
Bruce:
Summary response: Mia culpa....please revert this patch.
OK! I hadn't gotten around to applying it, so no need to revert.
Quoted 2 lines
Although it made the test failure go away, this patch just moves the problem, it doesn't fix anything.
Remind me what the test failure was? (You saw that the problem Torsten Kaiser reported turned out to be a bug in the memory allocator, right?:
http://bugzilla.kernel.org/show_bug.cgi?id=9973)
--b.
Quoted 90 lines
More below inline....
Quoted 8 lines · J. Bruce Fields
Quoted 6 lines · Tom Tucker
The
svc_sendpath is callingxpo_release_rqstwithout checking theXPT_DEADbit. It is illegal to call transport methods on a dead transport. In practice, if the transport gets an error and shuts down while there are still RPC insvc_processthe resultingsvc_sendcould crash calling into a transport that is being shut down.Quoted 4 lines
As long as we have a pointer to an xprt (say in
rqstp->rq_xprt), we should have a reference on the corresponding xprt class, shouldn't we? Anything else seems like a problem.Generally yes. In particular, at entry to
svc_recv, we should have a minimum of two references.svc_xprt_enqueuegets you one, and there is a one ref bias that comes fromsvc_xprt_init. The bias gets removed on close.Quoted 22 lines
Quoted 22 lines
Signed-off-by Tom Tucker <[email protected]>---
net/sunrpc/svc_xprt.c +0
-3@@ -729,9 +729,6 @@ int svc_send(struct svc_rqst *rqstp)if (!xprt)return -EFAULT ; -/* release the receive skb before sending the reply */ - rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); -/* calculate over-all length */ xb = &rqstp->rq_res;
xb->len= xb->head[0].iov_len +net/sunrpc/svc_xprt.c +0
-1@@ -742,8 +739,11 @@ int svc_send(struct svc_rqst *rqstp)mutex_lock (&xprt->xpt_mutex);if (test_bit (XPT_DEAD , &xprt->xpt_flags)) len = -ENOTCONN ; -else Quoted 9 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 746
Quoted 5 lines
net/sunrpc/svc_xprt.c +4
-0+else { +/* release the receive skb before sending the reply */ + rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); len = xprt->xpt_ops->xpo_sendto (rqstp); + }So in the case where
XPT_DEADis set, doesn't the receive skb also need to be freed, or does somebody else do that?Only reboot does that :-\. I need to think this through more carefully. On RDMA transports this function is used to return send credits to the user, and that involves talking to the transport. Sorry for this hasty patch...it's not right.
Quoted 7 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 748
Quoted 2 lines
net/sunrpc/svc_xprt.c +0
-0mutex_unlock (&xprt->xpt_mutex);svc_xprt_release (rqstp);Also something else is odd here:
svc_xprt_release(rqstp)already callsrqstp->rq_xprt->xpt_ops->xpo_release_rqst(rqstp)itself, so we seem to have two calls to that method.Hmm. Busted. This patch is broken in three dimensions. Is that a record?
Tom
Quoted 4 lines
--b.
Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · J. Bruce Fields
Quoted 3 lines · Tom Tucker
Bruce:
Summary response: Mia culpa....please revert this patch.
OK! I hadn't gotten around to applying it, so no need to revert.
Awesome.
Quoted 7 lines
Quoted 2 lines
Although it made the test failure go away, this patch just moves the problem, it doesn't fix anything.
Remind me what the test failure was? (You saw that the problem Torsten Kaiser reported turned out to be a bug in the memory allocator, right?:
http://bugzilla.kernel.org/show_bug.cgi?id=9973)
The real problem was found by Sandia labs. But it is easily reproducible here. Something seems to have changed in the Infiniband verbs core that exposed a latent bug in the RDMA transport driver. The initial "fix" fixed the original crash, but in addition to being incorrect, additional testing revealed that it still crashed in other ways -- that's why I sent this note.
So the problem is that there is a race between the I/O event callbacks from the RDMA core and the close path in the transport driver. The incorrect assumption made in the code is that after the call to rdma_destroy_id returns, you will receive no more events from lower level. Well .... that's not true. You won't receive any more CM events, but you may still receive CQ and QP events.
I have a fix and am thoroughly testing it now. I'll send it to you after I've had 24 hrs on it ;-).
Tom
Quoted 92 lines
--b.
Quoted 90 lines
More below inline....
Quoted 8 lines · J. Bruce Fields
Quoted 6 lines · Tom Tucker
The
svc_sendpath is callingxpo_release_rqstwithout checking theXPT_DEADbit. It is illegal to call transport methods on a dead transport. In practice, if the transport gets an error and shuts down while there are still RPC insvc_processthe resultingsvc_sendcould crash calling into a transport that is being shut down.Quoted 4 lines
As long as we have a pointer to an xprt (say in
rqstp->rq_xprt), we should have a reference on the corresponding xprt class, shouldn't we? Anything else seems like a problem.Generally yes. In particular, at entry to
svc_recv, we should have a minimum of two references.svc_xprt_enqueuegets you one, and there is a one ref bias that comes fromsvc_xprt_init. The bias gets removed on close.Quoted 22 lines
Quoted 22 lines
Signed-off-by Tom Tucker <[email protected]>---
net/sunrpc/svc_xprt.c +0
-3@@ -729,9 +729,6 @@ int svc_send(struct svc_rqst *rqstp)if (!xprt)return -EFAULT ; -/* release the receive skb before sending the reply */ - rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); -/* calculate over-all length */ xb = &rqstp->rq_res;
xb->len= xb->head[0].iov_len +net/sunrpc/svc_xprt.c +0
-1@@ -742,8 +739,11 @@ int svc_send(struct svc_rqst *rqstp)mutex_lock (&xprt->xpt_mutex);if (test_bit (XPT_DEAD , &xprt->xpt_flags)) len = -ENOTCONN ; -else Quoted 9 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 746
Quoted 5 lines
net/sunrpc/svc_xprt.c +4
-0+else { +/* release the receive skb before sending the reply */ + rqstp->rq_xprt->xpt_ops->xpo_release_rqst (rqstp); len = xprt->xpt_ops->xpo_sendto (rqstp); + }So in the case where
XPT_DEADis set, doesn't the receive skb also need to be freed, or does somebody else do that?Only reboot does that :-\. I need to think this through more carefully. On RDMA transports this function is used to return send credits to the user, and that involves talking to the transport. Sorry for this hasty patch...it's not right.
Quoted 7 lines
net/sunrpc/svc_xprt.c int svc_send(struct svc_rqst *rqstp) · line 748
Quoted 2 lines
net/sunrpc/svc_xprt.c +0
-0mutex_unlock (&xprt->xpt_mutex);svc_xprt_release (rqstp);Also something else is odd here:
svc_xprt_release(rqstp)already callsrqstp->rq_xprt->xpt_ops->xpo_release_rqst(rqstp)itself, so we seem to have two calls to that method.Hmm. Busted. This patch is broken in three dimensions. Is that a record?
Tom
Quoted 4 lines
--b.
Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html