Skip to contentSkip to conversation

nfsd: clean up CB_LAYOUTRECALL handling

RFC PATCH In review Jeff Layton 3 participants 9 messages Dec 8, 2015 – Jan 8, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover nfsd: clean up CB_LAYOUTRECALL handling

This is a proposal that should follow the broad outlines of what HCH was suggesting for CB_LAYOUTRECALL. It changes the code to treat 0 and NFS4ERR_DELAY returns equivalently, and also implements a scheme to give up on the client after two lease periods.

Note that this is _not_ tested, since I don't have a pnfs test rig on which to test this.

Jeff Layton (2):
nfsd: don't hold ls_mutex across a layout recall
nfsd: give up on CB_LAYOUTRECALLs after two lease periods

 fs/nfsd/nfs4layouts.c | 37 ++++++++++++++++++++++++++-----------
 1 file changed, 26 insertions(+), 11 deletions(-)
Signature · 1 line
2.5.0
#1 lore raw

RFC PATCH 1/2 nfsd: don't hold ls_mutex across a layout recall

We do need to serialize layout stateid morphing operations, but we currently hold the ls_mutex across a layout recall which is pretty ugly. It's also unnecessary -- once we've bumped the seqid and copied it, we don't need to serialize the rest of the CB_LAYOUTRECALL vs. anything else. Just drop the mutex once the copy is done.

Fixes cc8a55320b5f "nfsd: serialize layout stateid morphing operations"
Reported-and-Tested-by Kinglong Mee <[email protected]>
---
 fs/nfsd/nfs4layouts.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfs4layouts.c +1 -1
index 9ffef06b30d5..c9d6c715c0fb 100644
@@ -616,6 +616,7 @@ nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 
 	mutex_lock(&ls->ls_mutex);
 	nfs4_inc_and_copy_stateid(&ls->ls_recall_sid, &ls->ls_stid);
+	mutex_unlock(&ls->ls_mutex);
 }
 
 static int
@@ -659,7 +660,6 @@ nfsd4_cb_layout_release(struct nfsd4_callback *cb)
 
 	trace_layout_recall_release(&ls->ls_stid.sc_stateid);
 
-	mutex_unlock(&ls->ls_mutex);
 	nfsd4_return_all_layouts(ls, &reaplist);
 	nfsd4_free_layouts(&reaplist);
 	nfs4_put_stid(&ls->ls_stid);
Signature · 1 line
2.5.0
#3 lore raw

RFC PATCH 2/2 nfsd: give up on CB_LAYOUTRECALLs after two lease periods

In reply to Jeff Layton

Have the CB_LAYOUTRECALL code treat NFS4_OK and NFS4ERR_DELAY returns equivalently. Change the code to periodically resend CB_LAYOUTRECALLS until the ls_layouts list is empty or the client returns a different error code.

If we go for two lease periods without the list being emptied or the client sending a hard error, then we give up and clean out the list anyway.

---
 fs/nfsd/nfs4layouts.c | 35 +++++++++++++++++++++++++----------
 1 file changed, 25 insertions(+), 10 deletions(-)
fs/nfsd/nfs4layouts.c +25 -10
index c9d6c715c0fb..f72abc3ba013 100644
@@ -624,24 +624,39 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 {
 	struct nfs4_layout_stateid *ls =
 		container_of(cb, struct nfs4_layout_stateid, ls_recall);
+	struct nfsd_net *nn;
+	ktime_t now, cutoff;
 	LIST_HEAD(reaplist);
 
+
 	switch (task->tk_status) {
 	case 0:
-		return 1;
+	case -NFS4ERR_DELAY:
+		/*
+		 * Anything left? If not, then call it done. Note that we don't
+		 * take the spinlock since this is an optimization and nothing
+		 * should get added until the cb counter goes to zero.
+		 */
+		if (list_empty(&ls->ls_layouts))
+			return 1;
+
+		/* Poll the client until it's done with the layout */
+		now = ktime_get();
+		nn = net_generic(ls->ls_stid.sc_client->net, nfsd_net_id);
+
+		/* Client gets 2 lease periods to return it */
+		cutoff = ktime_add_ns(task->tk_start,
+					 nn->nfsd4_lease * NSEC_PER_SEC * 2);
+
+		if (ktime_before(now, cutoff)) {
+			rpc_delay(task, HZ/100); /* 10 mili-seconds */
+			return 0;
+		}
+		/* Fallthrough */
 	case -NFS4ERR_NOMATCHING_LAYOUT:
 		trace_layout_recall_done(&ls->ls_stid.sc_stateid);
 		task->tk_status = 0;
 		return 1;
-	case -NFS4ERR_DELAY:
-		/* Poll the client until it's done with the layout */
-		/* FIXME: cap number of retries.
-		 * The pnfs standard states that we need to only expire
-		 * the client after at-least "lease time" .eg lease-time * 2
-		 * when failing to communicate a recall
-		 */
-		rpc_delay(task, HZ/100); /* 10 mili-seconds */
-		return 0;
 	default:
 		/*
 		 * Unknown error or non-responding client, we'll need to fence.
Signature · 1 line
2.5.0
#4 lore raw

Re: [RFC PATCH 2/2] nfsd: give up on CB_LAYOUTRECALLs after two lease periods

Quoted 8 lines · Jeff Layton

Have the CB_LAYOUTRECALL code treat NFS4_OK and NFS4ERR_DELAY returns equivalently. Change the code to periodically resend CB_LAYOUTRECALLS until the ls_layouts list is empty or the client returns a different error code.

If we go for two lease periods without the list being emptied or the client sending a hard error, then we give up and clean out the list anyway.

This looks reasonable to me, but I'll need to actually test it before giving an ACK.

Btw, it seems like the delegation and layoutrecall code would benefit from some more code sharing for timeouts. For example delegation returns currently don't support NFS4ERR_DELAY at all.

#5 lore raw

Re: [RFC PATCH 2/2] nfsd: give up on CB_LAYOUTRECALLs after two lease periods

Quoted 6 lines · On Tue, 8 Dec 2015 16:21:23 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 5 lines · Jeff Layton

Have the CB_LAYOUTRECALL code treat NFS4_OK and NFS4ERR_DELAY returns equivalently. Change the code to periodically resend CB_LAYOUTRECALLS until the ls_layouts list is empty or the client returns a different error code.

Quoted 7 lines
Quoted 3 lines

If we go for two lease periods without the list being emptied or the client sending a hard error, then we give up and clean out the list anyway.

This looks reasonable to me, but I'll need to actually test it before giving an ACK.

Please do, I don't have a good way to test this at the moment...

Quoted 3 lines

Btw, it seems like the delegation and layoutrecall code would benefit from some more code sharing for timeouts. For example delegation returns currently don't support NFS4ERR_DELAY at all.

Yes...

I also wonder -- are we handling revoked layouts correctly? Shouldn't we be handling revoked layouts like we would a revoked delegation? Stop allowing the stateid to be used and morph it appropriately so that a TEST_STATEID against it gives you an error?

Right now, it doesn't look like TEST_STATEID is set up to handle layout stateids at all so they all get back NFS4ERR_BAD_STATEID.

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw

Re: [RFC PATCH 2/2] nfsd: give up on CB_LAYOUTRECALLs after two lease periods

Quoted 5 lines · Jeff Layton
Quoted 3 lines

This looks reasonable to me, but I'll need to actually test it before giving an ACK.

Please do, I don't have a good way to test this at the moment...

Looks like the baseline got broken once again, so I'll need some time to track that down first.

Quoted 10 lines
Quoted 3 lines

Btw, it seems like the delegation and layoutrecall code would benefit from some more code sharing for timeouts. For example delegation returns currently don't support NFS4ERR_DELAY at all.

Yes...

I also wonder -- are we handling revoked layouts correctly? Shouldn't we be handling revoked layouts like we would a revoked delegation? Stop allowing the stateid to be used and morph it appropriately so that a TEST_STATEID against it gives you an error?

Probably, but I'd need to take a deeper look at this.

#8 lore raw

Re: [RFC PATCH 2/2] nfsd: give up on CB_LAYOUTRECALLs after two lease periods

Quoted 3 lines · Christoph Hellwig

Ok, this tests fine for me:

Thanks, applying for 4.5.

(Missed your reviewed-by on the other patch when I submitted that for stable, apologies.)

I did a little work on some pynfs tests to exercise some of the failure cases here, but didn't get very far; I'll post something if I get a chance to test them. I think those tests are what we'd need before doing something more sophisticated.

--b.