nfs41_sequence_done

pnfs In review Benny Halevy 2 participants 6 messages Jul 12, 2010 4 messages not in the archive

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Benny Halevy

author

2010-07-12 18:58 UTC #2 lore raw

[[email protected] -> [email protected]]

Quoted 3 lines · Jim Rees

Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message

Heh, need to update hard-coded instructions to point to the new list...

Quoted 6 lines

I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.

I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress

It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.

Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?

Benny

Quoted 8 lines

_______________________________________________
NOTE: THIS LIST IS DEPRECATED. Please use [email protected] instead. (To subscribe to [email protected]: send "subscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TaiAVqoAR/[email protected])
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

Trond Myklebust

2010-07-12 19:15 UTC #3 lore raw

Quoted 20 lines · Benny Halevy

[[email protected] -> [email protected]]
Quoted 3 lines · Jim Rees
Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message
Heh, need to update hard-coded instructions to point to the new list...
Quoted 6 lines
I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.
I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress

Quoted 7 lines

It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.
Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?

So what is the root cause here? Is it the known issue that we don't deal correctly with an NFS4ERR_DELAY on the SEQUENCE operation?

Trond

A message that is not in the archive (sent off-list or not yet received): 1278962046.12559.3.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

Benny Halevy

author

2010-07-12 19:16 UTC #5 lore raw

Quoted 25 lines · Trond Myklebust

Quoted 24 lines · Benny Halevy
[[email protected] -> [email protected]]
Quoted 3 lines · Jim Rees
Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message
Heh, need to update hard-coded instructions to point to the new list...
Quoted 6 lines
I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.
I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress
It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.

Quoted 6 lines

Quoted 3 lines
Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?
So what is the root cause here? Is it the known issue that we don't deal correctly with an NFS4ERR_DELAY on the SEQUENCE operation?

Yes.

Quoted 6 lines

Trond
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Trond Myklebust

2010-07-12 19:26 UTC #6 lore raw

Quoted 26 lines · Benny Halevy

Quoted 25 lines · Trond Myklebust
Quoted 24 lines · Benny Halevy
[[email protected] -> [email protected]]
Quoted 3 lines · Jim Rees
Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message
Heh, need to update hard-coded instructions to point to the new list...
Quoted 6 lines
I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.
I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress
It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?
So what is the root cause here? Is it the known issue that we don't deal correctly with an NFS4ERR_DELAY on the SEQUENCE operation?
Yes.

I'm happy to take patches to fix that.

Trond

A message that is not in the archive (sent off-list or not yet received): 1278962763.12559.10.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

Benny Halevy

author

2010-07-12 19:49 UTC #8 lore raw

PATCH 1/1 nfs41: Do not free slot if retried while operation was in progress

Quoted 34 lines · Trond Myklebust

Quoted 33 lines · Benny Halevy
Quoted 31 lines · Trond Myklebust
Quoted 20 lines · Benny Halevy
[[email protected] -> [email protected]]
Quoted 3 lines · Jim Rees
Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message
Heh, need to update hard-coded instructions to point to the new list...
Quoted 6 lines
I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.
I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress
Quoted 7 lines
It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.
Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?
So what is the root cause here? Is it the known issue that we don't deal correctly with an NFS4ERR_DELAY on the SEQUENCE operation?

Quoted 7 lines

Quoted 1 line
Yes.
I'm happy to take patches to fix that.
Trond
From 7dc3c468463a337dabff7f714a3475e3f51380f6 Mon Sep 17 00:00:00 2001

From: Benny Halevy <[email protected]>
Date: Mon, 12 Jul 2010 22:42:15 +0300
Subject: [PATCH] nfs41: Do not free slot if retried while operation was in progress

Getting NFS4ERR_DELAY on OP_SEQUENCE means that the compound was retried while it's still in progress on the server. Therefore its respective slot must not be freed and reused for other compounds until it either succeeds or fails with another error status.

Signed-off-by Benny Halevy <[email protected]>

---

That fixed, do we ensure that the client either closes or loses the connection before retrying?

 fs/nfs/nfs4proc.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

fs/nfs/nfs4proc.c +6 -0

index 70015dd..baf86b9 100644
@@ -425,6 +425,12 @@ static void nfs41_sequence_done(struct nfs_client *clp,
 		/* Check sequence flags */
 		if (atomic_read(&clp->cl_count) > 1)
 			nfs41_handle_sequence_flag_errors(clp, res->sr_status_flags);
+	} else if (unlikely(res->sr_status == -NFS4ERR_DELAY)) {
+		/* Do not free slot if retried while operation was in progress */
+		tbl = &res->sr_session->fc_slot_table;
+		dprintk("%s: slot=%d seq=%d: Operation in progress\n", __func__,
+			res->sr_slotid, tbl->slots[res->sr_slotid].seq_nr);
+		return;
 	}
 out:
 	/* The session may be reset by one of the error handlers. */

Trond Myklebust

2010-07-12 20:00 UTC #9 lore raw

Re: [PATCH] nfs41: Do not free slot if retried while operation was in progress

Quoted 67 lines · Benny Halevy

Quoted 34 lines · Trond Myklebust
Quoted 33 lines · Benny Halevy
Quoted 31 lines · Trond Myklebust
Quoted 20 lines · Benny Halevy
[[email protected] -> [email protected]]
Quoted 3 lines · Jim Rees
Does anyone still care about this?
WARNING: nfs41_sequence_done: Operation in progress slot=1 seq=7 highest_used_slotid=1: please report to [email protected] if you saw this message
Heh, need to update hard-coded instructions to point to the new list...
Quoted 6 lines
I'm getting this on the client side of a pnfs block layout mount against the spnfs server. Kernel is benny's pnfs-all-2.6.35-rc3-2010-07-01 plus EMC complex block layout patches. It's possible the complex layout code is to blame, but I doubt it because this isn't a complex layout mount. I can provide more details.
I agree. This is a generic issue.
The patch that adds this check is
d6ce9ad DEVONLY: nfs41: Do not free slot if retried while operation was in progress
Quoted 7 lines
It was originally rejected (http://www.spinics.net/lists/linux-nfs/msg09562.html)
due to noise regarding where nfs41_sequence_free_slot is called
but that masked the real issue.
Can you readily reproduce this?
Can you debug also the server side to see if indeed the client retries the RPC while it is in progress on the server?
So what is the root cause here? Is it the known issue that we don't deal correctly with an NFS4ERR_DELAY on the SEQUENCE operation?
Quoted 7 lines
Quoted 1 line
Yes.
I'm happy to take patches to fix that.
Trond
From 7dc3c468463a337dabff7f714a3475e3f51380f6 Mon Sep 17 00:00:00 2001
From: Benny Halevy <[email protected]>
Date: Mon, 12 Jul 2010 22:42:15 +0300
Subject: [PATCH] nfs41: Do not free slot if retried while operation was in progress
Getting NFS4ERR_DELAY on OP_SEQUENCE means that the compound was retried while it's still in progress on the server. Therefore its respective slot must not be freed and reused for other compounds until it either succeeds or fails with another error status.
Signed-off-by Benny Halevy <[email protected]>
---
That fixed, do we ensure that the client either closes or loses the connection before retrying?
 fs/nfs/nfs4proc.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
fs/nfs/nfs4proc.c +0 -0
index 70015dd..baf86b9 100644
@@ -425,6 +425,12 @@ static void nfs41_sequence_done(struct nfs_client *clp,
 		/* Check sequence flags */
 		if (atomic_read(&clp->cl_count) > 1)

fs/nfs/nfs4proc.c static void nfs41_sequence_done(struct nfs_client *clp, · line 436

Quoted 10 lines

fs/nfs/nfs4proc.c +6 -0

 			nfs41_handle_sequence_flag_errors(clp, res->sr_status_flags);
+	} else if (unlikely(res->sr_status == -NFS4ERR_DELAY)) {
+		/* Do not free slot if retried while operation was in progress */
+		tbl = &res->sr_session->fc_slot_table;
+		dprintk("%s: slot=%d seq=%d: Operation in progress\n", __func__,
+			res->sr_slotid, tbl->slots[res->sr_slotid].seq_nr);
+		return;
 	}
 out:
 	/* The session may be reset by one of the error handlers. */

No. That is very clearly insufficient...

Never mind. I'll do it...