NFS weirdness in 2.6.0-test1
Subsystems (3)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Paul Mundt <[email protected]> writes:
> Any other suggestions?
I think I've found the problem. It is due to us overwriting req->rq_rcv_buf in call_encode() while the RPC request is still on the list waiting for a reply from the server.
Actually, this *is* likely to be an issue on 2.4.x too, but as it is also going to be very timing related, you are probably being lucky in one case, and not the other.
I'll draw up a patch some time in the next 24 hours...
Cheers,
Trond
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Paul Mundt <[email protected]> writes:
> I'm still left with corrupted data every time I get:
> NFS: server cheating in read reply: count 1526 > recvd 1000 > NFS: server cheating in read reply: count 4096 > recvd 1000 > NFS: server cheating in read reply: count 1583 > recvd 1000
> Any other suggestions?
Does the following patch fix it?
Note: I'm not entirely sure about whether or not we need those 2 smp_rmb(), but since the value of req->rq_received may be changed from inside a softirq while we're working, I'm assuming that it is.
Cheers,
Trond
include/linux/sunrpc/xprt.h +4 -0
@@ -98,6 +98,10 @@struct list_head rq_list; +struct xdr_buf rq_private_buf;/* The receive buffer +* used in the softirq. +*/ +/* * For authentication (e.g. auth_des) */
net/sunrpc/clnt.c +17 -8
@@ -659,7 +659,7 @@if (task->tk_status <0 )return ; task->tk_status =xprt_prepare_transmit (task); -if (task->tk_status <0 ) +if (task->tk_status !=0 )return ;/* Encode here so that rpcsec_gss can use correct sequence number. */ if (!task->tk_rqstp->rq_bytes_sent)
@@ -685,7 +685,8 @@struct rpc_rqst *req = task->tk_rqstp;int status; -if (req->rq_received !=0 ) +smp_rmb (); +if (req->rq_received >0 && !req->rq_bytes_sent) task->tk_status = req->rq_received; dprintk("RPC: %4d call_status (status %d)\n",
@@ -789,17 +790,21 @@if (!clnt->cl_softrtry) { task->tk_action = call_transmit; clnt->cl_stats->rpcretrans++; - }else { -printk (KERN_WARNING "%s: too small RPC reply size (%d bytes)\n" , - clnt->cl_protname, task->tk_status); -rpc_exit (task, -EIO ); + goto out_retry; } +printk (KERN_WARNING "%s: too small RPC reply size (%d bytes)\n" , + clnt->cl_protname, task->tk_status); +rpc_exit (task, -EIO );return ; } +/* Check that the softirq receive buffer is valid */ +WARN_ON (memcmp (&req->rq_rcv_buf, &req->rq_private_buf, +sizeof (req->rq_rcv_buf)) !=0 ); +/* Verify the RPC header */ if (!(p =call_verify (task))) -return ; + goto out_retry;/* * The following is an NFS-specific hack to cater for setuid
@@ -812,7 +817,7 @@ task->tk_flags ^=RPC_CALL_REALUID ; task->tk_action = call_bind; task->tk_suid_retry--; -return ; + goto out_retry; } }
@@ -822,6 +827,10 @@ task->tk_status =decode (req, p, task->tk_msg.rpc_resp);dprintk ("RPC: %4d call_decode result %d\n" , task->tk_pid, task->tk_status); +return ; +out_retry: + req->rq_received =0 ; + task->tk_status =0 ; }/*
net/sunrpc/xprt.c +43 -23
168 lines not on this page · raw message
Quoted 6 lines · Trond Myklebust
Quoted 3 lines
NFS: server cheating in read reply: count 1526 > recvd 1000
NFS: server cheating in read reply: count 4096 > recvd 1000
NFS: server cheating in read reply: count 1583 > recvd 1000Does the following patch fix it?
Yes, this seems to work fine. I've been running this for awhile now under relative load, and haven't seen any data corruption.
Thanks!