rados_cluster: add a "design" manpage

Quoted 45 lines · Jeff Layton

From: Jeff Layton <[email protected]>

Bruce asked for better design documentation, so this is my attempt at it. Let me know what you think. I'll probably end up squashing this into one of the code patches but for now I'm sending this separately to see if it helps clarify things.

Suggestions and feedback are welcome.

Change-Id I53cc77f66b2407c2083638e5760666639ba1fd57

Signed-off-by Jeff Layton <[email protected]>

---
 src/doc/man/ganesha-rados-cluster.rst | 227 ++++++++++++++++++++++++++
 1 file changed, 227 insertions(+)
 create mode 100644 src/doc/man/ganesha-rados-cluster.rst

src/doc/man/ganesha-rados-cluster.rst new file +23 -0

new file mode 100644
index 000000000000..1ba2d3c29093
@@ -0,0 +1,227 @@
+==============================================================================
+ganesha-rados-cluster-design -- Clustered RADOS Recovery Backend Design
+==============================================================================
+
+.. program:: ganesha-rados-cluster-design
+
+This document aims to explain the theory and design behind the
+rados_cluster recovery backend, which coordinates grace period
+enforcement among multiple, independent NFS servers.
+
+In order to understand the clustered recovery backend, it's first necessary
+to understand how recovery works with a single server:
+
+Singleton Server Recovery
+-------------------------
+NFSv4 is a lease-based protocol. Clients set up a relationship to the
+server and must periodically renew their lease in order to maintain
+their ephemeral state (open files, locks, delegations or layouts).
+
+When a singleton NFS server is restarted, any ephemeral state is lost. When
+the server comes comes back online, NFS clients detect that the server has
+been restarted and will reclaim the ephemeral state that they held at the
+time of their last contact with the server.

src/doc/man/ganesha-rados-cluster.rst line 33

Quoted 10 lines

src/doc/man/ganesha-rados-cluster.rst new file +10 -0

+
+Singleton Grace Period
+----------------------
+
+In order to ensure that we don't end up with conflicts, clients are
+barred from acquiring any new state while in the Recovery phase. Only
+reclaim operations are allowed.
+
+This period of time is called the **grace period**. Most NFS servers
+have a grace period that lasts around two lease periods, however

knfsd's is one lease period, who does two?

(Still catching up on the rest. Looks good.)

--b.

Thanks for having a look. Hmm...you're right.

        nn->nfsd4_lease = 90;   /* default lease time */                        
        nn->nfsd4_grace = 90;                                                   

nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?

In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.

ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.

Even worse:

        $ cat /proc/sys/fs/lease-break-time 
        45

Maybe we should be basing the v4 lease time on the lease-break-time value? It seems like we ought to revoke delegations after two lease periods rather than after half of one.

Quoted 8 lines · Jeff Layton

Thanks for having a look. Hmm...you're right.

        nn->nfsd4_lease = 90;   /* default lease time */                        
        nn->nfsd4_grace = 90;                                                   

nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?

Quoted 8 lines

In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.

ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.

OK, I'd be interested in any of that.

If we decide we need a larger grace/lease ratio, we may want to adjust default lease down at same time to avoid increasing grace period, as that's always an annoyance.

Quoted 8 lines

Even worse:

        $ cat /proc/sys/fs/lease-break-time 
        45

Maybe we should be basing the v4 lease time on the lease-break-time value? It seems like we ought to revoke delegations after two lease periods rather than after half of one.

No, we ignore lease-break-time on delegations. See the comment in fs/nfsd/nfs4state.c:nfsd_break_deleg_cb():

	/*
	 * We don't want the locks code to timeout the lease for us;
	 * we'll remove it ourself if a delegation isn't returned
	 * in time:
	 */
	fl->fl_break_time = 0;

Quoted 18 lines · J. Bruce Fields

Quoted 8 lines · Jeff Layton

Thanks for having a look. Hmm...you're right.

        nn->nfsd4_lease = 90;   /* default lease time */                        
        nn->nfsd4_grace = 90;                                                   

nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?

Quoted 8 lines

In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.

ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.

Quoted 6 lines

OK, I'd be interested in any of that.

If we decide we need a larger grace/lease ratio, we may want to adjust default lease down at same time to avoid increasing grace period, as that's always an annoyance.

I think we just figured that we ought to give the client a full lease period to reclaim everything, in case it holds a lot of state. You also need to give it a full lease period to figure out that the server has rebooted, and then I think we added a few seconds to account for "other factors" (transport delays and whatnot).

I think it would also be easy to extend it on demand.

So, for example: end the grace period when:

(one lease period has passed *and* we've received no reclaim request in the last second) *or*
two lease periods have passed

Maybe think about the exact choice of numbers a little.

Quoted 8 lines · J. Bruce Fields

I think it would also be easy to extend it on demand.

So, for example: end the grace period when:

(one lease period has passed *and* we've received no reclaim request in the last second) *or*
two lease periods have passed

Quoted 1 line

Maybe think about the exact choice of numbers a little.

Something like this. (Totally untested.)

I also wonder if 90 seconds is overkill as our default lease time. What does anyone else do? Maybe I'll just half it to 45s at the same time.

--b.

commit 90c471ab0150

Author: J. Bruce Fields <[email protected]>
Date:   Fri Jun 8 12:28:47 2018 -0400

nfsd4: extend reclaim period for reclaiming clients

If the client is only renewing state a little sooner than once a lease period, then it might not discover the server has restarted till close to the end of the grace period, and might run out of time to do the actual reclaim.

Extend the grace period by a second each time we notice there are clients still trying to reclaim, up to a limit of another whole lease period.

Signed-off-by J. Bruce Fields <[email protected]>

fs/nfsd/netns.h +1 -0

index 36358d435cb0..426f55005697 100644
@@ -102,6 +102,7 @@ struct nfsd_net {
 
 	time_t nfsd4_lease;
 	time_t nfsd4_grace;
+	bool somebody_reclaimed;
 
 
 	bool nfsd_net_up;

fs/nfsd/netns.h +0 -0

 	bool lockd_up;

fs/nfsd/nfs4proc.c +0 -0

index 5d99e8810b85..1929f85b8269 100644

fs/nfsd/nfs4proc.c +3 -0

 	dprintk("NFSD: nfsd4_open filename %.*s op_openowner %p\n",
 		(int)open->op_fname.len, open->op_fname.data,

@@ -424,6 +425,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 			if (status)
 				goto out;
 			open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED;
+			reclaim = true;
 		case NFS4_OPEN_CLAIM_FH:
 		case NFS4_OPEN_CLAIM_DELEG_CUR_FH:
 			status = do_open_fhandle(rqstp, cstate, open);

@@ -452,6 +454,8 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	WARN(status && open->op_created,
 	     "nfsd4_process_open2 failed to open newly-created file! status=%u\n",
 	     be32_to_cpu(status));
+	if (reclaim && !status)
+		nn->somebody_reclaimed = true;
 out:
 	if (resfh && resfh != &cstate->current_fh) {
 		fh_dup2(&cstate->current_fh, resfh);

fs/nfsd/nfs4state.c +30 -1

index 59ae65d3eec3..ffe816fe6e89 100644
@@ -4689,6 +4689,28 @@ nfsd4_end_grace(struct nfsd_net *nn)
 	 */
 }
 
+/*
+ * If we've waited a lease period but there are still clients trying to
+ * reclaim, wait a little longer to give them a chance to finish.
+ */
+static bool clients_still_reclaiming(struct nfsd_net *nn)
+{
+	unsigned long now = get_seconds();
+	unsigned long double_grace_period_end = nn->boot_time +
+						2 * nn->nfsd4_lease;
+
+	if (!nn->somebody_reclaimed)
+		return false;
+	nn->somebody_reclaimed = false;
+	/*
+	 * If we've given them *two* lease times to reclaim, and they're
+	 * still not done, give up:
+	 */
+	if (time_after(now, double_grace_period_end))
+		return false;
+	return true;
+}
+
 static time_t
 nfs4_laundromat(struct nfsd_net *nn)
 {

@@ -4702,6 +4724,11 @@ nfs4_laundromat(struct nfsd_net *nn)
 	time_t t, new_timeo = nn->nfsd4_lease;
 
 	dprintk("NFSD: laundromat service - starting\n");
+
+	if (clients_still_reclaiming(nn)) {
+		new_timeo = 0;
+		goto out;
+	}
 	nfsd4_end_grace(nn);
 	INIT_LIST_HEAD(&reaplist);
 	spin_lock(&nn->client_lock);

@@ -4799,7 +4826,7 @@ nfs4_laundromat(struct nfsd_net *nn)
 		posix_unblock_lock(&nbl->nbl_lock);
 		free_blocked_lock(nbl);
 	}
-
+out:
 	new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT);
 	return new_timeo;
 }

@@ -6049,6 +6076,8 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	case 0: /* success! */
 		nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid);
 		status = 0;
+		if (lock->lk_reclaim)
+			nn->somebody_reclaimed = true;
 		break;
 	case FILE_LOCK_DEFERRED:
 		nbl = NULL;

fs/nfsd/nfsctl.c +1 -0

index d107b4426f7e..5f22476cf371 100644
@@ -1239,6 +1239,7 @@ static __net_init int nfsd_init_net(struct net *net)
 		goto out_idmap_error;
 	nn->nfsd4_lease = 90;	/* default lease time */
 	nn->nfsd4_grace = 90;
+	nn->somebody_reclaimed = false;
 	nn->clverifier_counter = prandom_u32();
 	nn->clientid_counter = prandom_u32();
 

I know this is a "best effort" sort of thing, but should this be done with atomic loads and stores (READ_ONCE/WRITE_ONCE)?

Quoted 96 lines

fs/nfsd/nfs4proc.c +3 -0

 	dprintk("NFSD: nfsd4_open filename %.*s op_openowner %p\n",
 		(int)open->op_fname.len, open->op_fname.data,

@@ -424,6 +425,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 			if (status)
 				goto out;
 			open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED;
+			reclaim = true;
 		case NFS4_OPEN_CLAIM_FH:
 		case NFS4_OPEN_CLAIM_DELEG_CUR_FH:
 			status = do_open_fhandle(rqstp, cstate, open);

@@ -452,6 +454,8 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	WARN(status && open->op_created,
 	     "nfsd4_process_open2 failed to open newly-created file! status=%u\n",
 	     be32_to_cpu(status));
+	if (reclaim && !status)
+		nn->somebody_reclaimed = true;
 out:
 	if (resfh && resfh != &cstate->current_fh) {
 		fh_dup2(&cstate->current_fh, resfh);

fs/nfsd/nfs4state.c +30 -1

index 59ae65d3eec3..ffe816fe6e89 100644
@@ -4689,6 +4689,28 @@ nfsd4_end_grace(struct nfsd_net *nn)
 	 */
 }
 
+/*
+ * If we've waited a lease period but there are still clients trying to
+ * reclaim, wait a little longer to give them a chance to finish.
+ */
+static bool clients_still_reclaiming(struct nfsd_net *nn)
+{
+	unsigned long now = get_seconds();
+	unsigned long double_grace_period_end = nn->boot_time +
+						2 * nn->nfsd4_lease;
+
+	if (!nn->somebody_reclaimed)
+		return false;
+	nn->somebody_reclaimed = false;
+	/*
+	 * If we've given them *two* lease times to reclaim, and they're
+	 * still not done, give up:
+	 */
+	if (time_after(now, double_grace_period_end))
+		return false;
+	return true;
+}
+
 static time_t
 nfs4_laundromat(struct nfsd_net *nn)
 {

@@ -4702,6 +4724,11 @@ nfs4_laundromat(struct nfsd_net *nn)
 	time_t t, new_timeo = nn->nfsd4_lease;
 
 	dprintk("NFSD: laundromat service - starting\n");
+
+	if (clients_still_reclaiming(nn)) {
+		new_timeo = 0;
+		goto out;
+	}
 	nfsd4_end_grace(nn);
 	INIT_LIST_HEAD(&reaplist);
 	spin_lock(&nn->client_lock);

@@ -4799,7 +4826,7 @@ nfs4_laundromat(struct nfsd_net *nn)
 		posix_unblock_lock(&nbl->nbl_lock);
 		free_blocked_lock(nbl);
 	}
-
+out:
 	new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT);
 	return new_timeo;
 }

@@ -6049,6 +6076,8 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	case 0: /* success! */
 		nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid);
 		status = 0;
+		if (lock->lk_reclaim)
+			nn->somebody_reclaimed = true;
 		break;
 	case FILE_LOCK_DEFERRED:
 		nbl = NULL;

fs/nfsd/nfsctl.c +1 -0

index d107b4426f7e..5f22476cf371 100644
@@ -1239,6 +1239,7 @@ static __net_init int nfsd_init_net(struct net *net)
 		goto out_idmap_error;
 	nn->nfsd4_lease = 90;	/* default lease time */
 	nn->nfsd4_grace = 90;
+	nn->somebody_reclaimed = false;
 	nn->clverifier_counter = prandom_u32();
 	nn->clientid_counter = prandom_u32();
 

Signature · 1 line

Jeff Layton <[email protected]>