Skip to content

NFS: Fix -EREMOTEIO error on interrupted slots

[email protected] linux-nfs 2 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest1 patch 0 repliesNew
  2. v11 patch 4 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/1NFS: Fix interrupted slots by sending a solo SEQUENCE operationfs/nfs/nfs4proc.c+18 -20 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/1] NFS: Fix -EREMOTEIO error on interrupted slots

From: Anna Schumaker <[email protected]>

The scenario is as follows:
- The client attempts to remove a file on the server, but the remove is interrupted AFTER the server receives it.
- At the same time, another thread removes the same file on the server before NFSD has a chance to remove it
- The client then attempts another NFS operation with the same slot.

Because another thread removed the file the vfs returns -ENOENT to NFSD, which causes NFSD to reply to the next operation on the same slot with the result of the REMOVE (even if we asked for an OPEN). The client detects the mismatched operations during decoding, and returns -EREMOTEIO to the application.

The timing is tricky to get right on this, so I added a 3-second sleep to nfsd4_remove() before calling nfsd_unlink():

fs/nfsd/nfs4proc.c +2 -0
index a09c35f0f6f0..bd93be50eaa8 100644
@@ -851,6 +851,8 @@ nfsd4_remove(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 
 	if (opens_in_grace(SVC_NET(rqstp)))
 		return nfserr_grace;
+
+	ssleep(3);
 	status = nfsd_unlink(rqstp, &cstate->current_fh, 0,
 			     remove->rm_name, remove->rm_namelen);
 	if (!status) {

I'm able to hit this every time using the following script combined with the artifical delay on the server:

#!/bin/bash
SERVER=192.168.111.200
SERVER_DIR=/srv/test
CLIENT_DIR=/mnt/test

ssh $SERVER "echo test > $SERVER_DIR/test1"
rm -v $CLIENT_DIR/test1 &
sleep 1
killall -9 rm
ssh $SERVER "rm $SERVER_DIR/test1"
echo "test2" > $CLIENT_DIR/test2

I was able to solve the issue by sending a SEQUENCE using the same slot. The server replies to this with NFS4ERR_SEQ_FALSE_RETRY instead of an operation from the reply cache, and we are able to recover from here.

Thoughts?
Anna

Anna Schumaker (1):
NFS: Fix interrupted slots by sending a solo SEQUENCE operation

 fs/nfs/nfs4proc.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0