Skip to contentSkip to conversation

NFS weirdness in 2.6.0-test1

DISCUSSION Trond Myklebust 2 participants 3 messages Jul 28, 2003 – Jul 30, 2003 3 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#3 lore raw
In reply to a message not in the archive
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Paul Mundt <[email protected]> writes:

> Any other suggestions?

I think I've found the problem. It is due to us overwriting req->rq_rcv_buf in call_encode() while the RPC request is still on the list waiting for a reply from the server.

Actually, this *is* likely to be an issue on 2.4.x too, but as it is also going to be very timing related, you are probably being lucky in one case, and not the other.

I'll draw up a patch some time in the next 24 hours...

Cheers,
Trond

#4 lore raw
In reply to a message not in the archive
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Paul Mundt <[email protected]> writes:

> I'm still left with corrupted data every time I get:

> NFS: server cheating in read reply: count 1526 > recvd 1000 > NFS: server cheating in read reply: count 4096 > recvd 1000 > NFS: server cheating in read reply: count 1583 > recvd 1000

> Any other suggestions?

Does the following patch fix it?

Note: I'm not entirely sure about whether or not we need those 2 smp_rmb(), but since the value of req->rq_received may be changed from inside a softirq while we're working, I'm assuming that it is.

Cheers,
Trond

include/linux/sunrpc/xprt.h +4 -0
diff -u --recursive --new-file linux-2.6.0-test2/include/linux/sunrpc/xprt.h linux-2.6.0-01-fix_rcv/include/linux/sunrpc/xprt.h
@@ -98,6 +98,10 @@
 
 	struct list_head	rq_list;
 
+	struct xdr_buf		rq_private_buf;		/* The receive buffer
+							 * used in the softirq.
+							 */
+
 	/*
 	 * For authentication (e.g. auth_des)
 	 */
net/sunrpc/clnt.c +17 -8
diff -u --recursive --new-file linux-2.6.0-test2/net/sunrpc/clnt.c linux-2.6.0-01-fix_rcv/net/sunrpc/clnt.c
@@ -659,7 +659,7 @@
 	if (task->tk_status < 0)
 		return;
 	task->tk_status = xprt_prepare_transmit(task);
-	if (task->tk_status < 0)
+	if (task->tk_status != 0)
 		return;
 	/* Encode here so that rpcsec_gss can use correct sequence number. */
 	if (!task->tk_rqstp->rq_bytes_sent)
@@ -685,7 +685,8 @@
 	struct rpc_rqst	*req = task->tk_rqstp;
 	int		status;
 
-	if (req->rq_received != 0)
+	smp_rmb();
+	if (req->rq_received > 0 && !req->rq_bytes_sent)
 		task->tk_status = req->rq_received;
 
 	dprintk("RPC: %4d call_status (status %d)\n", 
@@ -789,17 +790,21 @@
 		if (!clnt->cl_softrtry) {
 			task->tk_action = call_transmit;
 			clnt->cl_stats->rpcretrans++;
-		} else {
-			printk(KERN_WARNING "%s: too small RPC reply size (%d bytes)\n",
-				clnt->cl_protname, task->tk_status);
-			rpc_exit(task, -EIO);
+			goto out_retry;
 		}
+		printk(KERN_WARNING "%s: too small RPC reply size (%d bytes)\n",
+			clnt->cl_protname, task->tk_status);
+		rpc_exit(task, -EIO);
 		return;
 	}
 
+	/* Check that the softirq receive buffer is valid */
+	WARN_ON(memcmp(&req->rq_rcv_buf, &req->rq_private_buf,
+				sizeof(req->rq_rcv_buf)) != 0);
+
 	/* Verify the RPC header */
 	if (!(p = call_verify(task)))
-		return;
+		goto out_retry;
 
 	/*
 	 * The following is an NFS-specific hack to cater for setuid
@@ -812,7 +817,7 @@
 			task->tk_flags ^= RPC_CALL_REALUID;
 			task->tk_action = call_bind;
 			task->tk_suid_retry--;
-			return;
+			goto out_retry;
 		}
 	}
 
@@ -822,6 +827,10 @@
 		task->tk_status = decode(req, p, task->tk_msg.rpc_resp);
 	dprintk("RPC: %4d call_decode result %d\n", task->tk_pid,
 					task->tk_status);
+	return;
+out_retry:
+	req->rq_received = 0;
+	task->tk_status = 0;
 }
 
 /*
net/sunrpc/xprt.c +43 -23

168 lines not on this page · raw message

#5 lore raw
Quoted 6 lines · Trond Myklebust
Quoted 3 lines

NFS: server cheating in read reply: count 1526 > recvd 1000
NFS: server cheating in read reply: count 4096 > recvd 1000
NFS: server cheating in read reply: count 1583 > recvd 1000

Does the following patch fix it?

Yes, this seems to work fine. I've been running this for awhile now under relative load, and haven't seen any data corruption.

Thanks!