Race in RPC code
Subsystems (3)
Hello all,
I think there is a race in the RPC code in 2.4.20, related to timeout and congestion window handling.
The problem code is in net/sunrpc/xprt.c:
static voidxprt_timer(struct rpc_task *task)
{
struct rpc_rqst *req = task->tk_rqstp;
struct rpc_xprt *xprt = req->rq_xprt;
spin_lock(&xprt->sock_lock);
if (req->rq_received)
goto out;
if (!xprt->nocong) {
if (xprt_expbackoff(task, req)) {
rpc_add_timer(task, xprt_timer);
goto out_unlock;
}
rpc_inc_timeo(&task->tk_client->cl_rtt);
xprt_adjust_cwnd(req->rq_xprt, -ETIMEDOUT);
}
req->rq_nresend++;The call to xprt_adjust_cwnd is the problem - I experienced a panic (null pointer dereference) in
static voidxprt_adjust_cwnd(struct rpc_xprt *xprt, int result)
{
unsigned long cwnd;
cwnd = xprt->cwnd;
if (result >= 0 && cwnd <= xprt->cong) {Here it is the "cwnd = xprt->cwnd" that causes the panic. xprt was 0.
This means, in the xprt_timer code, that req->rq_xprt must have been 0.
I guess this can happen because of the sequence:
xprt = req->rq_xprt;
spin_lock(&xprt->sock_lock);
...
xprt_adjust_cwnd(req->rq_xprt);
We don't know whether req has been modified between the assignment and the spin_lock.
Attached is a patch to solve the problem - please comment.
It does not solve the other potential (?) problem with:
xprt = req->rq_xprt;
spin_lock(&xprt->sock_lock);
...
if (xprt_expbackoff(task, req)) {...
Any suggestions to that one?
I cannot test whether my patch solve the problem, because this panic has happened once on a *heavily* loaded box which has run 2.4.20 ever since it came out. The race is extremely rare. It is an SMP box by the way.
Thanks a lot to "baldrick" on kernel-newbies for the help!
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
net/sunrpc/xprt.c +1 -1
@@ -1021,7 +1021,7 @@ goto out_unlock; }rpc_inc_timeo (&task->tk_client->cl_rtt); -xprt_adjust_cwnd (req->rq_xprt, -ETIMEDOUT ); +xprt_adjust_cwnd (xprt, -ETIMEDOUT ); } req->rq_nresend++;
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jakob Oestergaard <[email protected]> writes:
> We don't know whether req has been modified between the > assignment and the spin_lock.
It had better not be. If it is, then I want to know where so that we can fix it.
req->rq_xprt is set up when the request is initialized. It is not meant to change until the request gets released. This again should not happen while the request is still on the wait queue.
IOW the fix you propose would just be papering over another problem.
Cheers,
Trond
Quoted 6 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jakob Oestergaard <[email protected]> writes:
> We don't know whether req has been modified between the > assignment and the
spin_lock.
Quoted 8 lines
It had better not be. If it is, then I want to know where so that we can fix it.
req->rq_xprtis set up when the request is initialized. It is not meant to change until the request gets released. This again should not happen while the request is still on the wait queue.IOW the fix you propose would just be papering over another problem.
Any suggestions as to how it could happen?
The box is running huge compile jobs (>100MB memory used by each compiler - runs 2-3 compilers concurrently) all day long - we never had a GCC sig11 error. It has 512 MB of ECC memory (and ECC is enabled) - I seriously doubt that we have a memory corruption problem.
The panic has happened once, just today.
I will be happy to try other solutions, but I can't verify whether they work - I mean, if the box runs another few months without crashing that doesn't really prove anything...
Thanks for commenting!
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Hi Jakob, Trond,
Quoted 1 line · Jakob Oestergaard
The panic has happened once, just today.
I've seen this multiple times, and even reported it to the nfs mailing list.
I'm just glad someone else is seeing the same kind of oops with a vanilla kernel, because I was seeing it with a heavily patched redhat+nfs2.4.20+nfsall2.4.20 kernel and wasn't sure if that had anything to do with it.
I have at least 5-6 such oopsen recorded, if anyone cares.
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Quoted 8 lines · Ion Badulescu
Hi Jakob, Trond,
Quoted 1 line · Jakob Oestergaard
The panic has happened once, just today.
I've seen this multiple times, and even reported it to the nfs mailing list.
Quoted 6 lines
I'm just glad someone else is seeing the same kind of oops with a vanilla kernel, because I was seeing it with a heavily patched redhat+nfs2.4.20+nfsall2.4.20 kernel and wasn't sure if that had anything to do with it.
I have at least 5-6 such oopsen recorded, if anyone cares.
Please send them.
Duncan.