Skip to contentSkip to conversation

NFSv4.1 Server DRC rewrite Version 4

PATCH In review [email protected] 5 participants 26 messages Jun 24, 2009 – Jul 31, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover NFSv4.1 Server DRC rewrite Version 4

NFSv4.1 Server DRC rewrite Version 4

This patch set applies to linux-nfs.org/~bfields/linux, for-2.6.31 branch updated on 6-22-2009.

In response to bfield's comments I moved the DRC limit bug fixes to the front of the series, consolidated patches replacing the single slot clientid DRC into one patch and the replacement of the mulitple slot sequeqnce operation DRC into another single patch.

The NFSv4.1 DRC is changed from a page based cache to
a buffer based cache. The logic for the single slot clientid cache has been separated from the session slot logic to handle the CREATE_SESSION call preceeded by a SEQUENCE and all the replay combinations therein.

The session DRC now caches encoded operations with the exception of the SEQUENCE operation which for a replay is encoded with the current slot and session values. A review of message sizes indicates that a 512 byte buffer for the operations is adequate.

Not addressed is replacing the nfsd4_check_drc_limit() post-operation checking with a pre-operation processing estimate of the encoded per operation result

Testing:
4.1 mount: Connectathon and 4.1pynfs including the new create session replay tests.
4.0 mount; Connectathon.

0001-nfsd41-use-globals-for-DRC-limits.patch 0002-nfsd41-change-from-page-to-memory-based-drc-limits.patch 0003-nfsd41-reclaim-DRC-memory-on-session-free.patch 0004-nfsd41-set-the-session-maximum-response-size-cached.patch 0005-nfsd41-remove-redundant-forechannel-max-requests-ch.patch 0006-nfsd41-change-check_slot_seqid-parameters.patch 0007-nfsd41-replace-create_session-DRC-with-xdr-structur.patch 0008-nfsd41-replace-page-based-DRC-with-buffer-based-DRC.patch 0009-nfsd41-rename-nfsd4_enc_uncached_replay.patch 0010-nfsd41-encode-replay-sequence-from-the-slot-values.patch 0011-nfsd41-fix-nfsd4_replay_cache_entry-comments.patch 0012-nfsd41-support-16-slots-per-session.patch 0013-nfsd41-add-test-for-failed-sequence-operation.patch 0014-nfsd41-remove-redundant-failed-sequence-check.patch

Comments welcome.

-->Andy

#1 lore raw

PATCH 1/5 nfsd41: use globals for DRC limits

From: Andy Adamson <[email protected]>

The version 4.1 DRC memory limit and tracking variables are server wide and session specific. Replace struct svc_serv fields with globals. Stop using the svc_serv sv_lock.

Add a spinlock to serialize access to the DRC limit management variables which change on session creation and deletion (usage counter) or (future) administrative action to adjust the total DRC memory limit.

---
 fs/nfsd/nfs4state.c        |   10 +++++-----
 fs/nfsd/nfssvc.c           |   19 +++++++++++++++----
 include/linux/nfsd/nfsd.h  |    3 +++
 include/linux/sunrpc/svc.h |    2 --
 4 files changed, 23 insertions(+), 11 deletions(-)
fs/nfsd/nfs4state.c +5 -5
index 980a216..2e6a44e 100644
@@ -430,11 +430,11 @@ static int set_forechannel_maxreqs(struct nfsd4_channel_attrs *fchan)
 	else if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
 		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;
 
-	spin_lock(&nfsd_serv->sv_lock);
-	if (np + nfsd_serv->sv_drc_pages_used > nfsd_serv->sv_drc_max_pages)
-		np = nfsd_serv->sv_drc_max_pages - nfsd_serv->sv_drc_pages_used;
-	nfsd_serv->sv_drc_pages_used += np;
-	spin_unlock(&nfsd_serv->sv_lock);
+	spin_lock(&nfsd_drc_lock);
+	if (np + nfsd_drc_pages_used > nfsd_drc_max_pages)
+		np = nfsd_drc_max_pages - nfsd_drc_pages_used;
+	nfsd_drc_pages_used += np;
+	spin_unlock(&nfsd_drc_lock);
 
 	if (np <= 0) {
 		status = nfserr_resource;
fs/nfsd/nfssvc.c +15 -4
index d4c9884..78d8fcd 100644
@@ -67,6 +67,16 @@ struct timeval			nfssvc_boot;
 DEFINE_MUTEX(nfsd_mutex);
 struct svc_serv 		*nfsd_serv;
 
+/*
+ * nfsd_drc_lock protects nfsd_drc_max_pages and nfsd_drc_pages_used.
+ * nfsd_drc_max_pages limits the total amount of memory available for
+ * version 4.1 DRC caches.
+ * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage.
+ */
+spinlock_t	nfsd_drc_lock;
+unsigned int	nfsd_drc_max_pages;
+unsigned int	nfsd_drc_pages_used;
+
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
 static struct svc_stat	nfsd_acl_svcstats;
 static struct svc_version *	nfsd_acl_version[] = {
@@ -238,11 +248,12 @@ static void set_max_drc(void)
 {
 	/* The percent of nr_free_buffer_pages used by the V4.1 server DRC */
 	#define NFSD_DRC_SIZE_SHIFT	7
-	nfsd_serv->sv_drc_max_pages = nr_free_buffer_pages()
+	nfsd_drc_max_pages = nr_free_buffer_pages()
 						>> NFSD_DRC_SIZE_SHIFT;
-	nfsd_serv->sv_drc_pages_used = 0;
-	dprintk("%s svc_drc_max_pages %u\n", __func__,
-		nfsd_serv->sv_drc_max_pages);
+	nfsd_drc_pages_used = 0;
+	spin_lock_init(&nfsd_drc_lock);
+	dprintk("%s nfsd_drc_max_pages %u\n", __func__,
+		nfsd_drc_max_pages);
 }
 
 int nfsd_create_serv(void)
include/linux/nfsd/nfsd.h +3 -0
index 2b49d67..2571f85 100644
@@ -56,6 +56,9 @@ extern struct svc_version	nfsd_version2, nfsd_version3,
 extern u32			nfsd_supported_minorversion;
 extern struct mutex		nfsd_mutex;
 extern struct svc_serv		*nfsd_serv;
+extern spinlock_t		nfsd_drc_lock;
+extern unsigned int		nfsd_drc_max_pages;
+extern unsigned int		nfsd_drc_pages_used;
 
 extern struct seq_operations nfs_exports_op;
 
include/linux/sunrpc/svc.h +0 -2
index 2a30775..d0d8bf4 100644
@@ -94,8 +94,6 @@ struct svc_serv {
 	struct module *		sv_module;	/* optional module to count when
 						 * adding threads */
 	svc_thread_fn		sv_function;	/* main function for threads */
-	unsigned int		sv_drc_max_pages; /* Total pages for DRC */
-	unsigned int		sv_drc_pages_used;/* DRC pages used */
 };
 
 /*
Signature · 1 line
1.5.4.3
#2 lore raw

PATCH 2/5 nfsd41: change from page to memory based drc limits

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE is the size of all encoded operation responses (excluding the sequence operation) that we want to cache.

Adjust NFSD_DRC_SIZE_SHIFT to reflect using 512 bytes instead of PAGE_SIZE.

---
 fs/nfsd/nfs4state.c        |   30 ++++++++++++++----------------
 fs/nfsd/nfssvc.c           |   17 +++++++++--------
 include/linux/nfsd/nfsd.h  |    4 ++--
 include/linux/nfsd/state.h |    1 +
 4 files changed, 26 insertions(+), 26 deletions(-)
fs/nfsd/nfs4state.c +14 -16
index 2e6a44e..b4a536d 100644
@@ -414,35 +414,33 @@ gen_sessionid(struct nfsd4_session *ses)
 
 /*
  * Give the client the number of slots it requests bound by
- * NFSD_MAX_SLOTS_PER_SESSION and by sv_drc_max_pages.
+ * NFSD_MAX_SLOTS_PER_SESSION and by nfsd_drc_max_mem.
  *
- * If we run out of pages (sv_drc_pages_used == sv_drc_max_pages) we
- * should (up to a point) re-negotiate active sessions and reduce their
- * slot usage to make rooom for new connections. For now we just fail the
- * create session.
+ * If we run out of reserved DRC memory we should (up to a point) re-negotiate
+ * active sessions and reduce their slot usage to make rooom for new
+ * connections. For now we just fail the create session.
  */
 static int set_forechannel_maxreqs(struct nfsd4_channel_attrs *fchan)
 {
-	int status = 0, np = fchan->maxreqs * NFSD_PAGES_PER_SLOT;
+	int mem;
 
 	if (fchan->maxreqs < 1)
 		return nfserr_inval;
 	else if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
 		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;
 
+	mem = fchan->maxreqs * NFSD_SLOT_CACHE_SIZE;
+
 	spin_lock(&nfsd_drc_lock);
-	if (np + nfsd_drc_pages_used > nfsd_drc_max_pages)
-		np = nfsd_drc_max_pages - nfsd_drc_pages_used;
-	nfsd_drc_pages_used += np;
+	if (mem + nfsd_drc_mem_used > nfsd_drc_max_mem)
+		mem = nfsd_drc_max_mem - nfsd_drc_mem_used;
+	nfsd_drc_mem_used += mem;
 	spin_unlock(&nfsd_drc_lock);
 
-	if (np <= 0) {
-		status = nfserr_resource;
-		fchan->maxreqs = 0;
-	} else
-		fchan->maxreqs = np / NFSD_PAGES_PER_SLOT;
-
-	return status;
+	fchan->maxreqs = mem / NFSD_SLOT_CACHE_SIZE;
+	if (fchan->maxreqs == 0)
+		return nfserr_resource;
+	return 0;
 }
 
 /*
fs/nfsd/nfssvc.c +9 -8
index 78d8fcd..1793dba 100644
@@ -74,8 +74,8 @@ struct svc_serv 		*nfsd_serv;
  * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage.
  */
 spinlock_t	nfsd_drc_lock;
-unsigned int	nfsd_drc_max_pages;
-unsigned int	nfsd_drc_pages_used;
+unsigned int	nfsd_drc_max_mem;
+unsigned int	nfsd_drc_mem_used;
 
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
 static struct svc_stat	nfsd_acl_svcstats;
@@ -247,13 +247,14 @@ void nfsd_reset_versions(void)
 static void set_max_drc(void)
 {
 	/* The percent of nr_free_buffer_pages used by the V4.1 server DRC */
-	#define NFSD_DRC_SIZE_SHIFT	7
-	nfsd_drc_max_pages = nr_free_buffer_pages()
-						>> NFSD_DRC_SIZE_SHIFT;
-	nfsd_drc_pages_used = 0;
+	#define NFSD_DRC_SIZE_SHIFT	10
+	nfsd_drc_max_mem = (nr_free_buffer_pages()
+					>> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE;
+	nfsd_drc_mem_used = 0;
 	spin_lock_init(&nfsd_drc_lock);
-	dprintk("%s nfsd_drc_max_pages %u\n", __func__,
-		nfsd_drc_max_pages);
+	dprintk("%s nfsd_drc_max_mem %u [in pages %lu]\n", __func__,
+		nfsd_drc_max_mem,
+		nfsd_drc_max_mem / PAGE_SIZE);
 }
 
 int nfsd_create_serv(void)
include/linux/nfsd/nfsd.h +2 -2
index 2571f85..2812ed5 100644
@@ -57,8 +57,8 @@ extern u32			nfsd_supported_minorversion;
 extern struct mutex		nfsd_mutex;
 extern struct svc_serv		*nfsd_serv;
 extern spinlock_t		nfsd_drc_lock;
-extern unsigned int		nfsd_drc_max_pages;
-extern unsigned int		nfsd_drc_pages_used;
+extern unsigned int		nfsd_drc_max_mem;
+extern unsigned int		nfsd_drc_mem_used;
 
 extern struct seq_operations nfs_exports_op;
 
include/linux/nfsd/state.h +1 -0
index f5a95fd..10b6b30 100644
@@ -98,6 +98,7 @@ struct nfs4_cb_conn {
 /* Maximum number of pages per slot cache entry */
 #define NFSD_PAGES_PER_SLOT	1
 /* Maximum number of operations per session compound */
+#define NFSD_SLOT_CACHE_SIZE		512
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
 struct nfsd4_cache_entry {
Signature · 1 line
1.5.4.3
#3 lore raw

PATCH 3/5 nfsd41: reclaim DRC memory on session free

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfsd/nfs4state.c +3 -0
index b4a536d..991c3cc 100644
@@ -583,6 +583,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_mem_used -= ses->se_fchannel.maxreqs * NFSD_SLOT_CACHE_SIZE;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);
 }
 
Signature · 1 line
1.5.4.3
#4 lore raw

PATCH 4/5 nfsd41: set the session maximum response size cached

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE holds the encoded operations past the SEQUENCE operation. ca_maxresponsesize_cached (draft-ietf-nfsv4-minorversion1-29) is the xdr encoded size of the request including the RPC header. Since the RPC header size varies with security flavor credential and verifier. we cannot set an accurate ca_maxresponsesize_cached. Use NFSD_SLOT_CACHE_SIZE as an approximate ca_maxresponsesize_cached - we will have at least that much space.

---
 fs/nfsd/nfs4state.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +8 -3
index 991c3cc..a0bd6da 100644
@@ -464,9 +464,14 @@ static int init_forechannel_attrs(struct svc_rqst *rqstp,
 		fchan->maxresp_sz = maxcount;
 	session_fchan->maxresp_sz = fchan->maxresp_sz;
 
-	/* Set the max response cached size our default which is
-	 * a multiple of PAGE_SIZE and small */
-	session_fchan->maxresp_cached = NFSD_PAGES_PER_SLOT * PAGE_SIZE;
+	/*
+	 * The ca_maxresponssize_cached definition includes the xdr
+	 * encoded size of the rpc header with the variable length security
+	 * flavor credential plus verifier as well as the encoded SEQUENCE
+	 * operation response size which are not included in
+	 * NFSD_SLOT_CACHE_SIZE. We err on the side of being a bit small.
+	 */
+	session_fchan->maxresp_cached = NFSD_SLOT_CACHE_SIZE;
 	fchan->maxresp_cached = session_fchan->maxresp_cached;
 
 	/* Use the client's maxops if possible */
Signature · 1 line
1.5.4.3
#5 lore raw

PATCH 5/5 nfsd41: remove redundant forechannel max requests check

From: Andy Adamson <[email protected]>

This check is done in set_forechannel_maxreqs.

---
 fs/nfsd/nfs4state.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)
fs/nfsd/nfs4state.c +0 -4
index a0bd6da..dc91544 100644
@@ -479,10 +479,6 @@ static int init_forechannel_attrs(struct svc_rqst *rqstp,
 		fchan->maxops = NFSD_MAX_OPS_PER_COMPOUND;
 	session_fchan->maxops = fchan->maxops;
 
-	/* try to use the client requested number of slots */
-	if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
-		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;
-
 	/* FIXME: Error means no more DRC pages so the server should
 	 * recover pages from existing sessions. For now fail session
 	 * creation.
Signature · 1 line
1.5.4.3
#6 lore raw

PATCH 6/14 nfsd41: change check_slot_seqid parameters

From: Andy Adamson <[email protected]>

For separation of session slot and clientid slot processing.

---
 fs/nfsd/nfs4state.c |   24 +++++++++++++-----------
 1 files changed, 13 insertions(+), 11 deletions(-)
fs/nfsd/nfs4state.c +13 -11
index dc91544..0eab5e2 100644
@@ -1315,26 +1315,26 @@ error:
 }
 
 static int
-check_slot_seqid(u32 seqid, struct nfsd4_slot *slot)
+check_slot_seqid(u32 seqid, u32 slot_seqid, int slot_inuse)
 {
-	dprintk("%s enter. seqid %d slot->sl_seqid %d\n", __func__, seqid,
-		slot->sl_seqid);
+	dprintk("%s enter. seqid %d slot_seqid %d\n", __func__, seqid,
+		slot_seqid);
 
 	/* The slot is in use, and no response has been sent. */
-	if (slot->sl_inuse) {
-		if (seqid == slot->sl_seqid)
+	if (slot_inuse) {
+		if (seqid == slot_seqid)
 			return nfserr_jukebox;
 		else
 			return nfserr_seq_misordered;
 	}
 	/* Normal */
-	if (likely(seqid == slot->sl_seqid + 1))
+	if (likely(seqid == slot_seqid + 1))
 		return nfs_ok;
 	/* Replay */
-	if (seqid == slot->sl_seqid)
+	if (seqid == slot_seqid)
 		return nfserr_replay_cache;
 	/* Wraparound */
-	if (seqid == 1 && (slot->sl_seqid + 1) == 0)
+	if (seqid == 1 && (slot_seqid + 1) == 0)
 		return nfs_ok;
 	/* Misordered replay or misordered new request */
 	return nfserr_seq_misordered;
@@ -1357,7 +1357,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 
 	if (conf) {
 		slot = &conf->cl_slot;
-		status = check_slot_seqid(cr_ses->seqid, slot);
+		status = check_slot_seqid(cr_ses->seqid, slot->sl_seqid,
+					  slot->sl_inuse);
 		if (status == nfserr_replay_cache) {
 			dprintk("Got a create_session replay! seqid= %d\n",
 				slot->sl_seqid);
@@ -1382,7 +1383,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 		}
 
 		slot = &unconf->cl_slot;
-		status = check_slot_seqid(cr_ses->seqid, slot);
+		status = check_slot_seqid(cr_ses->seqid, slot->sl_seqid,
+					  slot->sl_inuse);
 		if (status) {
 			/* an unconfirmed replay returns misordered */
 			status = nfserr_seq_misordered;
@@ -1483,7 +1485,7 @@ nfsd4_sequence(struct svc_rqst *rqstp,
 	slot = &session->se_slots[seq->slotid];
 	dprintk("%s: slotid %d\n", __func__, seq->slotid);
 
-	status = check_slot_seqid(seq->seqid, slot);
+	status = check_slot_seqid(seq->seqid, slot->sl_seqid, slot->sl_inuse);
 	if (status == nfserr_replay_cache) {
 		cstate->slot = slot;
 		cstate->session = session;
Signature · 1 line
1.5.4.3
#7 lore raw

PATCH 7/14 nfsd41: replace create_session DRC with xdr structure

From: Andy Adamson <[email protected]>

The nfs41 single slot clientid DRC holds the results of create session processing. CREATE_SESSION can be preceeded by a SEQUENCE operation (an embedded CREATE_SESSION) and the create session single slot cache must be maintained. nfsd4_replay_cache_entry() and nfsd4_store_cache_entry() do not implement the replay of an embedded CREATE_SESSION.

The cliendid DRC slot does not need the inuse, cachethis or other fields that the multiple slot session cache uses. Replace the clientid DRC cache struct nfs4_slot cache with a new nfsd4_clid_slot cache. Save the xdr struct nfsd4_create_session into the cache at the end of processing, and on a replay, replace the struct for the replay request with the cached version all while under the state lock.

nfsd4_proc_compound will handle both the solo and embedded CREATE_SESSION case via the normal use of encode_operation.

Errors that do not change the create session cache:
A create session NFS4ERR_STALE_CLIENTID error means that a client record (and associated create session slot) could not be found and therefore can't be changed. NFSERR_SEQ_MISORDERED errors do not change the slot cache.

All other errors get cached.

Remove the clientid DRC specific check in nfs4svc_encode_compoundres to put the session only if cstate.session is set which will now always be true.

---
 fs/nfsd/nfs4proc.c         |    2 +-
 fs/nfsd/nfs4state.c        |   64 ++++++++++++++++++++++++++------------------
 fs/nfsd/nfs4xdr.c          |    3 +-
 include/linux/nfsd/state.h |   21 +++++++++++++-
 include/linux/nfsd/xdr4.h  |   12 --------
 5 files changed, 60 insertions(+), 42 deletions(-)
fs/nfsd/nfs4proc.c +1 -1
index 7c88017..b70a77d 100644
@@ -1083,7 +1083,7 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 			BUG_ON(op->status == nfs_ok);
 
 encode_op:
-		/* Only from SEQUENCE or CREATE_SESSION */
+		/* Only from SEQUENCE */
 		if (resp->cstate.status == nfserr_replay_cache) {
 			dprintk("%s NFS4.1 replay from cache\n", __func__);
 			if (nfsd4_not_cached(resp))
fs/nfsd/nfs4state.c +38 -26

126 lines not on this page · raw message

fs/nfsd/nfs4xdr.c +1 -2
index 2dcc7fe..fdf632b 100644
@@ -3313,8 +3313,7 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 			dprintk("%s: SET SLOT STATE TO AVAILABLE\n", __func__);
 			resp->cstate.slot->sl_inuse = 0;
 		}
-		if (resp->cstate.session)
-			nfsd4_put_session(resp->cstate.session);
+		nfsd4_put_session(resp->cstate.session);
 	}
 	return 1;
 }
include/linux/nfsd/state.h +20 -1
index 10b6b30..c45a7bd 100644
@@ -128,6 +128,25 @@ struct nfsd4_channel_attrs {
 	u32		rdma_attrs;
 };
 
+struct nfsd4_create_session {
+	clientid_t			clientid;
+	struct nfs4_sessionid		sessionid;
+	u32				seqid;
+	u32				flags;
+	struct nfsd4_channel_attrs	fore_channel;
+	struct nfsd4_channel_attrs	back_channel;
+	u32				callback_prog;
+	u32				uid;
+	u32				gid;
+};
+
+/* The single slot clientid cache structure */
+struct nfsd4_clid_slot {
+	u32				sl_seqid;
+	__be32				sl_status;
+	struct nfsd4_create_session	sl_cr_ses;
+};
+
 struct nfsd4_session {
 	struct kref		se_ref;
 	struct list_head	se_hash;	/* hash by sessionid */
@@ -194,7 +213,7 @@ struct nfs4_client {
 
 	/* for nfs41 */
 	struct list_head	cl_sessions;
-	struct nfsd4_slot	cl_slot;	/* create_session slot */
+	struct nfsd4_clid_slot	cl_cs_slot;	/* create_session slot */
 	u32			cl_exchange_flags;
 	struct nfs4_sessionid	cl_sessionid;
 };
include/linux/nfsd/xdr4.h +0 -12
index 2bacf75..5e4beb0 100644
@@ -366,18 +366,6 @@ struct nfsd4_exchange_id {
 	int		spa_how;
 };
 
-struct nfsd4_create_session {
-	clientid_t		clientid;
-	struct nfs4_sessionid	sessionid;
-	u32			seqid;
-	u32			flags;
-	struct nfsd4_channel_attrs fore_channel;
-	struct nfsd4_channel_attrs back_channel;
-	u32			callback_prog;
-	u32			uid;
-	u32			gid;
-};
-
 struct nfsd4_sequence {
 	struct nfs4_sessionid	sessionid;		/* request/response */
 	u32			seqid;			/* request/response */
Signature · 1 line
1.5.4.3
#8 lore raw

PATCH 8/14 nfsd41: replace page based DRC with buffer based DRC

From: Andy Adamson <[email protected]>

Use NFSD_SLOT_CACHE_SIZE size buffers for sessions DRC instead of holding nfsd pages in cache.

Allocate memory for the session DRC in the CREATE_SESSION operation because of the guarantee to the client that the memory resource is available for caching responses.

Never cache the SEQUENCE operation, encode it from the session slot values. Only cache the encoded results of operations past the SEQUENCE operation.

Remove struct nfsd4_cache_entry and helper functions for the old page-based DRC.

The iov_len calculation in nfs4svc_encode_compoundres is now always correct, clean up the nfs4svc_encode_compoundres session logic.

The nfsd4_compound_state statp pointer is also not used. Remove nfsd4_set_statp().

Move useful nfsd4_cache_entry fields into nfsd4_slot.

---
 fs/nfsd/nfs4proc.c         |    6 +-
 fs/nfsd/nfs4state.c        |  150 ++++++++------------------------------------
 fs/nfsd/nfs4xdr.c          |   13 ++--
 fs/nfsd/nfssvc.c           |    4 -
 include/linux/nfsd/state.h |   23 ++-----
 include/linux/nfsd/xdr4.h  |    4 +-
 6 files changed, 45 insertions(+), 155 deletions(-)
fs/nfsd/nfs4proc.c +3 -3
index b70a77d..6285106 100644
@@ -957,8 +957,8 @@ nfsd4_enc_uncached_replay(struct nfsd4_compoundargs *args,
 {
 	struct nfsd4_op *op;
 
-	dprintk("--> %s resp->opcnt %d ce_cachethis %u \n", __func__,
-		resp->opcnt, resp->cstate.slot->sl_cache_entry.ce_cachethis);
+	dprintk("--> %s resp->opcnt %d sl_cachethis %u \n", __func__,
+		resp->opcnt, resp->cstate.slot->sl_cachethis);
 
 	/* Encode the replayed sequence operation */
 	BUG_ON(resp->opcnt != 1);
@@ -966,7 +966,7 @@ nfsd4_enc_uncached_replay(struct nfsd4_compoundargs *args,
 	nfsd4_encode_operation(resp, op);
 
 	/*return nfserr_retry_uncached_rep in next operation. */
-	if (resp->cstate.slot->sl_cache_entry.ce_cachethis == 0) {
+	if (resp->cstate.slot->sl_cachethis == 0) {
 		op = &args->ops[resp->opcnt++];
 		op->status = nfserr_retry_uncached_rep;
 		nfsd4_encode_operation(resp, op);
fs/nfsd/nfs4state.c +27 -123

217 lines not on this page · raw message

fs/nfsd/nfs4xdr.c +6 -7
index fdf632b..49824ea 100644
@@ -3064,6 +3064,7 @@ nfsd4_encode_sequence(struct nfsd4_compoundres *resp, int nfserr,
 	WRITE32(0);
 
 	ADJUST_ARGS();
+	resp->cstate.datap = p; /* DRC cache data pointer */
 	return 0;
 }
 
@@ -3166,7 +3167,7 @@ static int nfsd4_check_drc_limit(struct nfsd4_compoundres *resp)
 		return status;
 
 	session = resp->cstate.session;
-	if (session == NULL || slot->sl_cache_entry.ce_cachethis == 0)
+	if (session == NULL || slot->sl_cachethis == 0)
 		return status;
 
 	if (resp->opcnt >= args->opcnt)
@@ -3291,6 +3292,7 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 	/*
 	 * All that remains is to write the tag and operation count...
 	 */
+	struct nfsd4_compound_state *cs = &resp->cstate;
 	struct kvec *iov;
 	p = resp->tagp;
 	*p++ = htonl(resp->taglen);
@@ -3304,14 +3306,11 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 		iov = &rqstp->rq_res.head[0];
 	iov->iov_len = ((char*)resp->p) - (char*)iov->iov_base;
 	BUG_ON(iov->iov_len > PAGE_SIZE);
-	if (nfsd4_has_session(&resp->cstate)) {
-		if (resp->cstate.status == nfserr_replay_cache &&
-				!nfsd4_not_cached(resp)) {
-			iov->iov_len = resp->cstate.iovlen;
-		} else {
+	if (nfsd4_has_session(cs)) {
+		if (cs->status != nfserr_replay_cache) {
 			nfsd4_store_cache_entry(resp);
 			dprintk("%s: SET SLOT STATE TO AVAILABLE\n", __func__);
-			resp->cstate.slot->sl_inuse = 0;
+			resp->cstate.slot->sl_inuse = false;
 		}
 		nfsd4_put_session(resp->cstate.session);
 	}
fs/nfsd/nfssvc.c +0 -4
index 1793dba..f0ef241 100644
@@ -580,10 +580,6 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 		+ rqstp->rq_res.head[0].iov_len;
 	rqstp->rq_res.head[0].iov_len += sizeof(__be32);
 
-	/* NFSv4.1 DRC requires statp */
-	if (rqstp->rq_vers == 4)
-		nfsd4_set_statp(rqstp, statp);
-
 	/* Now call the procedure handler, and encode NFS status. */
 	nfserr = proc->pc_func(rqstp, rqstp->rq_argp, rqstp->rq_resp);
 	nfserr = map_new_errors(rqstp->rq_vers, nfserr);
include/linux/nfsd/state.h +7 -16
index c45a7bd..09592db 100644
@@ -95,26 +95,17 @@ struct nfs4_cb_conn {
 
 /* Maximum number of slots per session. 128 is useful for long haul TCP */
 #define NFSD_MAX_SLOTS_PER_SESSION	128
-/* Maximum number of pages per slot cache entry */
-#define NFSD_PAGES_PER_SLOT	1
-/* Maximum number of operations per session compound */
 #define NFSD_SLOT_CACHE_SIZE		512
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
-struct nfsd4_cache_entry {
-	__be32		ce_status;
-	struct kvec	ce_datav; /* encoded NFSv4.1 data in rq_res.head[0] */
-	struct page	*ce_respages[NFSD_PAGES_PER_SLOT + 1];
-	int		ce_cachethis;
-	short		ce_resused;
-	int		ce_opcnt;
-	int		ce_rpchdrlen;
-};
-
 struct nfsd4_slot {
-	bool				sl_inuse;
-	u32				sl_seqid;
-	struct nfsd4_cache_entry	sl_cache_entry;
+	bool	sl_inuse;
+	u32	sl_seqid;
+	int	sl_cachethis;
+	int	sl_opcnt;
+	__be32	sl_status;
+	u32	sl_datalen;
+	char	sl_data[NFSD_SLOT_CACHE_SIZE];
 };
 
 struct nfsd4_channel_attrs {
include/linux/nfsd/xdr4.h +2 -2
index 5e4beb0..ae325ad 100644
@@ -51,7 +51,7 @@ struct nfsd4_compound_state {
 	/* For sessions DRC */
 	struct nfsd4_session	*session;
 	struct nfsd4_slot	*slot;
-	__be32			*statp;
+	__be32			*datap;
 	size_t			iovlen;
 	u32			minorversion;
 	u32			status;
@@ -472,7 +472,7 @@ static inline bool nfsd4_is_solo_sequence(struct nfsd4_compoundres *resp)
 
 static inline bool nfsd4_not_cached(struct nfsd4_compoundres *resp)
 {
-	return !resp->cstate.slot->sl_cache_entry.ce_cachethis ||
+	return !resp->cstate.slot->sl_cachethis ||
 			nfsd4_is_solo_sequence(resp);
 }
 
Signature · 1 line
1.5.4.3
#9 lore raw

PATCH 9/14 nfsd41: rename nfsd4_enc_uncached_replay

From: Andy Adamson <[email protected]>

This function is only used for SEQUENCE replay.

---
 fs/nfsd/nfs4proc.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/nfsd/nfs4proc.c +2 -2
index 6285106..386deb5 100644
@@ -952,7 +952,7 @@ static const char *nfsd4_op_name(unsigned opnum);
  * encode the uncache rep error on the next operation.
  */
 static __be32
-nfsd4_enc_uncached_replay(struct nfsd4_compoundargs *args,
+nfsd4_enc_sequence_replay(struct nfsd4_compoundargs *args,
 			 struct nfsd4_compoundres *resp)
 {
 	struct nfsd4_op *op;
@@ -1087,7 +1087,7 @@ encode_op:
 		if (resp->cstate.status == nfserr_replay_cache) {
 			dprintk("%s NFS4.1 replay from cache\n", __func__);
 			if (nfsd4_not_cached(resp))
-				status = nfsd4_enc_uncached_replay(args, resp);
+				status = nfsd4_enc_sequence_replay(args, resp);
 			else
 				status = op->status;
 			goto out;
Signature · 1 line
1.5.4.3
#10 lore raw

PATCH 10/14 nfsd41: encode replay sequence from the slot values

From: Andy Adamson <[email protected]>

The sequence operation is not cached; always encode the sequence operation on a replay from the slot table and session values. This simplifies the sessions replay logic in nfsd4_proc_compound.

If this is a replay of a compound that was specified not to be cached, return NFS4ERR_RETRY_UNCACHED_REP.

---
 fs/nfsd/nfs4proc.c  |   33 +--------------------------------
 fs/nfsd/nfs4state.c |   40 ++++++++++++++++++++++++++++++++++++----
 2 files changed, 37 insertions(+), 36 deletions(-)
fs/nfsd/nfs4proc.c +1 -32
index 386deb5..87be518 100644
@@ -947,34 +947,6 @@ static struct nfsd4_operation nfsd4_ops[];
 static const char *nfsd4_op_name(unsigned opnum);
 
 /*
- * This is a replay of a compound for which no cache entry pages
- * were used. Encode the sequence operation, and if cachethis is FALSE
- * encode the uncache rep error on the next operation.
- */
-static __be32
-nfsd4_enc_sequence_replay(struct nfsd4_compoundargs *args,
-			 struct nfsd4_compoundres *resp)
-{
-	struct nfsd4_op *op;
-
-	dprintk("--> %s resp->opcnt %d sl_cachethis %u \n", __func__,
-		resp->opcnt, resp->cstate.slot->sl_cachethis);
-
-	/* Encode the replayed sequence operation */
-	BUG_ON(resp->opcnt != 1);
-	op = &args->ops[resp->opcnt - 1];
-	nfsd4_encode_operation(resp, op);
-
-	/*return nfserr_retry_uncached_rep in next operation. */
-	if (resp->cstate.slot->sl_cachethis == 0) {
-		op = &args->ops[resp->opcnt++];
-		op->status = nfserr_retry_uncached_rep;
-		nfsd4_encode_operation(resp, op);
-	}
-	return op->status;
-}
-
-/*
  * Enforce NFSv4.1 COMPOUND ordering rules.
  *
  * TODO:
@@ -1086,10 +1058,7 @@ encode_op:
 		/* Only from SEQUENCE */
 		if (resp->cstate.status == nfserr_replay_cache) {
 			dprintk("%s NFS4.1 replay from cache\n", __func__);
-			if (nfsd4_not_cached(resp))
-				status = nfsd4_enc_sequence_replay(args, resp);
-			else
-				status = op->status;
+			status = op->status;
 			goto out;
 		}
 		if (op->status == nfserr_replay_me) {
fs/nfsd/nfs4state.c +36 -4
index 2232b7b..51a51d2 100644
@@ -1036,6 +1036,36 @@ out_warn:
 }
 
 /*
+ * Encode the replay sequence operation from the slot values.
+ * If cachethis is FALSE encode the uncached rep error on the next
+ * operation which sets resp->p and increments resp->opcnt for
+ * nfs4svc_encode_compoundres.
+ *
+ */
+static __be32
+nfsd4_enc_sequence_replay(struct nfsd4_compoundargs *args,
+			  struct nfsd4_compoundres *resp)
+{
+	struct nfsd4_op *op;
+	struct nfsd4_slot *slot = resp->cstate.slot;
+
+	dprintk("--> %s resp->opcnt %d cachethis %u \n", __func__,
+		resp->opcnt, resp->cstate.slot->sl_cachethis);
+
+	/* Encode the replayed sequence operation */
+	op = &args->ops[resp->opcnt - 1];
+	nfsd4_encode_operation(resp, op);
+
+	/* Return nfserr_retry_uncached_rep in next operation. */
+	if (args->opcnt > 1 && slot->sl_cachethis == 0) {
+		op = &args->ops[resp->opcnt++];
+		op->status = nfserr_retry_uncached_rep;
+		nfsd4_encode_operation(resp, op);
+	}
+	return op->status;
+}
+
+/*
  * Keep the first page of the replay. Copy the NFSv4.1 data from the first
  * cached page.  Replace any futher replay pages from the cache.
  */
@@ -1058,10 +1088,12 @@ nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp,
 	 * session inactivity timer fires and a solo sequence operation
 	 * is sent (lease renewal).
 	 */
-	if (seq && nfsd4_not_cached(resp)) {
-		seq->maxslots = resp->cstate.session->se_fchannel.maxreqs;
-		return nfs_ok;
-	}
+	seq->maxslots = resp->cstate.session->se_fchannel.maxreqs;
+
+	/* Either returns 0 or nfserr_retry_uncached */
+	status = nfsd4_enc_sequence_replay(resp->rqstp->rq_argp, resp);
+	if (status == nfserr_retry_uncached_rep)
+		return status;
 
 	/* The sequence operation has been encoded, cstate->datap set. */
 	memcpy(resp->cstate.datap, slot->sl_data, slot->sl_datalen);
Signature · 1 line
1.5.4.3
#11 lore raw

PATCH 11/14 nfsd41: fix nfsd4_replay_cache_entry comments

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |   15 +++------------
 1 files changed, 3 insertions(+), 12 deletions(-)
fs/nfsd/nfs4state.c +3 -12
index 51a51d2..7662ca8 100644
@@ -1066,8 +1066,8 @@ nfsd4_enc_sequence_replay(struct nfsd4_compoundargs *args,
 }
 
 /*
- * Keep the first page of the replay. Copy the NFSv4.1 data from the first
- * cached page.  Replace any futher replay pages from the cache.
+ * The sequence operation is not cached because we can use the slot and
+ * session values.
  */
 __be32
 nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp,
@@ -1078,16 +1078,7 @@ nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp,
 
 	dprintk("--> %s slot %p\n", __func__, slot);
 
-	/*
-	 * If this is just the sequence operation, we did not keep
-	 * a page in the cache entry because we can just use the
-	 * slot info stored in struct nfsd4_sequence that was checked
-	 * against the slot in nfsd4_sequence().
-	 *
-	 * This occurs when seq->cachethis is FALSE, or when the client
-	 * session inactivity timer fires and a solo sequence operation
-	 * is sent (lease renewal).
-	 */
+	/* Target max slot and flags will be set once they are implemented */
 	seq->maxslots = resp->cstate.session->se_fchannel.maxreqs;
 
 	/* Either returns 0 or nfserr_retry_uncached */
Signature · 1 line
1.5.4.3
#12 lore raw

PATCH 12/14 nfsd41: support 16 slots per session

From: Andy Adamson <[email protected]>

128 slots is a bit much without proof that they are needed.

---
 include/linux/nfsd/state.h |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)
include/linux/nfsd/state.h +1 -2
index 09592db..7cf3810 100644
@@ -93,8 +93,7 @@ struct nfs4_cb_conn {
 	struct rpc_cred	*	cb_cred;
 };
 
-/* Maximum number of slots per session. 128 is useful for long haul TCP */
-#define NFSD_MAX_SLOTS_PER_SESSION	128
+#define NFSD_MAX_SLOTS_PER_SESSION	16
 #define NFSD_SLOT_CACHE_SIZE		512
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
Signature · 1 line
1.5.4.3
#13 lore raw

PATCH 13/14 nfsd41: add test for failed sequence operation

From: Andy Adamson <[email protected]>

Failed sequence operations are not cached.

---
 include/linux/nfsd/xdr4.h |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)
include/linux/nfsd/xdr4.h +9 -1
index ae325ad..aae3a01 100644
@@ -464,6 +464,13 @@ struct nfsd4_compoundres {
 	struct nfsd4_compound_state	cstate;
 };
 
+static inline bool nfsd4_is_failed_sequence(struct nfsd4_compoundres *resp)
+{
+	struct nfsd4_compoundargs *args = resp->rqstp->rq_argp;
+	return resp->opcnt == 1 && args->ops[0].opnum == OP_SEQUENCE &&
+		resp->cstate.status;
+}
+
 static inline bool nfsd4_is_solo_sequence(struct nfsd4_compoundres *resp)
 {
 	struct nfsd4_compoundargs *args = resp->rqstp->rq_argp;
@@ -473,7 +480,8 @@ static inline bool nfsd4_is_solo_sequence(struct nfsd4_compoundres *resp)
 static inline bool nfsd4_not_cached(struct nfsd4_compoundres *resp)
 {
 	return !resp->cstate.slot->sl_cachethis ||
-			nfsd4_is_solo_sequence(resp);
+			nfsd4_is_solo_sequence(resp) ||
+			nfsd4_is_failed_sequence(resp);
 }
 
 #define NFS4_SVC_XDRSIZE		sizeof(struct nfsd4_compoundargs)
Signature · 1 line
1.5.4.3
#14 lore raw

PATCH 14/14 nfsd41: remove redundant failed sequence check

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |    7 -------
 1 files changed, 0 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +0 -7
index 7662ca8..bb85d0d 100644
@@ -1003,17 +1003,10 @@ void
 nfsd4_store_cache_entry(struct nfsd4_compoundres *resp)
 {
 	struct nfsd4_slot *slot = resp->cstate.slot;
-	struct svc_rqst *rqstp = resp->rqstp;
-	struct nfsd4_compoundargs *args = rqstp->rq_argp;
-	struct nfsd4_op *op = &args->ops[resp->opcnt];
 	unsigned int base;
 
 	dprintk("--> %s slot %p\n", __func__, slot);
 
-	/* Don't cache a failed OP_SEQUENCE. */
-	if (resp->opcnt == 1 && op->opnum == OP_SEQUENCE && resp->cstate.status)
-		return;
-
 	slot->sl_opcnt = resp->opcnt;
 	slot->sl_status = resp->cstate.status;
 
Signature · 1 line
1.5.4.3
#15 lore raw

Re: [pnfs] [PATCH 4/5] nfsd41: set the session maximum response sizecached

In reply to [email protected]
Quoted 38 lines · Andy

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE holds the encoded operations past the SEQUENCE operation. ca_maxresponsesize_cached (draft-ietf-nfsv4-minorversion1-29) is the xdr encoded size of the request including the RPC header. Since the RPC header size varies with security flavor credential and verifier. we cannot set an accurate ca_maxresponsesize_cached. Use NFSD_SLOT_CACHE_SIZE as an approximate ca_maxresponsesize_cached - we will have at least that much space.

---
 fs/nfsd/nfs4state.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +3 -3
index 991c3cc..a0bd6da 100644
@@ -464,9 +464,14 @@ static int init_forechannel_attrs(struct  
 svc_rqst *rqstp,
                fchan->maxresp_sz = maxcount;
        session_fchan->maxresp_sz = fchan->maxresp_sz;
 
-       /* Set the max response cached size our default which is
-        * a multiple of PAGE_SIZE and small */
-       session_fchan->maxresp_cached = NFSD_PAGES_PER_SLOT *  
 PAGE_SIZE;
+       /*
+        * The ca_maxresponssize_cached definition includes the xdr
+        * encoded size of the rpc header with the variable length  

security
+ * flavor credential plus verifier as well as the encoded
SEQUENCE

fs/nfsd/nfs4state.c +2 -0
+        * operation response size which are not included in
+        * NFSD_SLOT_CACHE_SIZE. We err on the side of being a bit  

small.

fs/nfsd/nfs4state.c static int init_forechannel_attrs(struct svc_rqst *rqstp, · line 475

Quoted 3 lines
fs/nfsd/nfs4state.c +2 -0
+        */
+       session_fchan->maxresp_cached = NFSD_SLOT_CACHE_SIZE;
        fchan->maxresp_cached = session_fchan->maxresp_cached;

Here we set the maxresp_cached equal to 512 which is normally much less than the maxresp_sz, and we return this value to client in CREATE_SESSION. This leads to a bunch of NFSERR_TOO_BIG_TO_CACHE errors for OPENs, READs, etc.
In theory the client should be able to recover (or better prevent this by not setting cachethis=1).
Is there a reason that maxresp_cached is so small ?

-alexandros

Quoted 9 lines
        /* Use the client's maxops if possible */

--
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

#16 lore raw

Re: [pnfs] [PATCH 4/5] nfsd41: set the session maximum response sizecached

Quoted 37 lines · Alexandros Batsakis
Quoted 34 lines · Andy

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE holds the encoded operations past the SEQUENCE operation. ca_maxresponsesize_cached (draft-ietf-nfsv4-minorversion1-29) is the xdr encoded size of the request including the RPC header. Since the RPC header size varies with security flavor credential and verifier. we cannot set an accurate ca_maxresponsesize_cached. Use NFSD_SLOT_CACHE_SIZE as an approximate ca_maxresponsesize_cached - we will have at least that much space.

---
fs/nfsd/nfs4state.c | 11 ++++++++---

1 files changed, 8 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +3 -3
index 991c3cc..a0bd6da 100644
@@ -464,9 +464,14 @@ static int init_forechannel_attrs(struct  
 svc_rqst *rqstp,
               fchan->maxresp_sz = maxcount;
       session_fchan->maxresp_sz = fchan->maxresp_sz;
 
-       /* Set the max response cached size our default which is
-        * a multiple of PAGE_SIZE and small */
-       session_fchan->maxresp_cached = NFSD_PAGES_PER_SLOT *  
 PAGE_SIZE;
+       /*
+        * The ca_maxresponssize_cached definition includes the xdr
+        * encoded size of the rpc header with the variable length  

security
+ * flavor credential plus verifier as well as the encoded
SEQUENCE
+ * operation response size which are not included in

Quoted 8 lines

fs/nfsd/nfs4state.c static int init_forechannel_attrs(struct svc_rqst *rqstp, · line 475

Quoted 7 lines

+ * NFSD_SLOT_CACHE_SIZE. We err on the side of being a bit
small.

fs/nfsd/nfs4state.c +2 -0
+        */
+       session_fchan->maxresp_cached = NFSD_SLOT_CACHE_SIZE;
       fchan->maxresp_cached = session_fchan->maxresp_cached;

Here we set the maxresp_cached equal to 512 which is normally much less than the maxresp_sz, and we return this value to client in CREATE_SESSION.

The max response size cached is supposed to be a lot less than the max response size. Since it is a nfsd memory resource, we would like to keep it as low as possible.

NFSv4.1 clients should not expect READ and READDIR results to be cached, and the linux client does not ask for these operations to be cached. GETATTR with an acl request should also not be cached - there is no way to tell how large the acl is.

Quoted 2 lines

This leads to a bunch of NFSERR_TOO_BIG_TO_CACHE errors for OPENs, READs, etc.

Argh. I chose the 512 byte DRC buffer size because it does service the responses of the Linux (and Solaris) clients. The 512 bytes were designed to hold operation results past the sequence operation. This was tested at the last bakeathon.

The maxresponse_size_cached is supposed to hold the RPC response, including verifier and the encoded operations. In the previous DRC patch set, which was tested at the bakeathon, the max response size cached was set to 512+size of encoded SEQUENCE + RPC max size.

Bruce suggested that we just use the DRC buffer size for the max response size cached which is what the current code uses.

I just ran connectathon tests against the nfsd41-all 2.6.31-rc1 server and saw the NFS4ERR_TOO_BIG_TO_CACHE on OPEN, CREATE, RENAME etc - all of which had a total response of 548 bytes or so - just past the max response size cached limit. This is with AUTH_SYS, and a NULL RPC verifier.

The sequence operation always returns 44 bytes. The minimal RPC reply is 32 bytes (NULL RPC verifier).

Here is what I suggest.

1) max response size should be the DRC buffer slot size (currently 512) + 44 + 32. The RPC GSS verifier will still not be accounted for, but the rest of the known sizes are.
2) Perhaps bump the 512 to 512+64. The current code shows we are cutting it a bit close, we want to ensure that nfsd can service normal operation responses with several GETATTR's with a lot of attributes.

-->Andy

Quoted 15 lines

In theory the client should be able to recover (or better prevent this by not setting cachethis=1). Is there a reason that maxresp_cached is so small ?

-alexandros

Quoted 9 lines
       /* Use the client's maxops if possible */

--
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

#17 lore raw

Re: [pnfs] [PATCH 4/5] nfsd41: set the session maximum response sizecached

Quoted 5 lines · Andy Adamson

Here is what I suggest.

1) max response size should be the DRC buffer slot size (currently 512) + 44 + 32. The RPC GSS verifier will still not be accounted for, but the rest of the known sizes are.

Sounds good.

Quoted 3 lines

2) Perhaps bump the 512 to 512+64. The current code shows we are cutting it a bit close, we want to ensure that nfsd can service normal operation responses with several GETATTR's with a lot of attributes.

576 sounds a little close too. 768? 1k?

--b.

#18 lore raw

Re: [pnfs] [PATCH 3/5] nfsd41: reclaim DRC memory on session free

In reply to [email protected]
Quoted 12 lines · On Wed, Jun 24, 2009 at 03:37:47PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfsd/nfs4state.c +0 -0
index b4a536d..991c3cc 100644

fs/nfsd/nfs4state.c free_session(struct kref *kref) · line 589

Quoted 7 lines
fs/nfsd/nfs4state.c +3 -0
@@ -583,6 +583,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_mem_used -= ses->se_fchannel.maxreqs * NFSD_SLOT_CACHE_SIZE;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);

This patch has to be combined with the previous one.

--b.

Quoted 9 lines
fs/nfsd/nfs4state.c +0 -0
 }
 
Signature · 6 lines
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs
#19 lore raw

Re: [pnfs] [PATCH 3/5] nfsd41: reclaim DRC memory on session free

Quoted 8 lines · J. Bruce Fields
Quoted 7 lines · On Wed, Jun 24, 2009 at 03:37:47PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
Quoted 14 lines

fs/nfsd/nfs4state.c free_session(struct kref *kref) · line 589

Quoted 12 lines
fs/nfsd/nfs4state.c +3 -0
index b4a536d..991c3cc 100644
@@ -583,6 +583,9 @@ free_session(struct kref *kref)
              struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
              nfsd4_release_respages(e->ce_respages, e->ce_resused);
      }
+     spin_lock(&nfsd_drc_lock);
+     nfsd_drc_mem_used -= ses->se_fchannel.maxreqs * NFSD_SLOT_CACHE_SIZE;
+     spin_unlock(&nfsd_drc_lock);
      kfree(ses);

This patch has to be combined with the previous one.

OK, makes sense.

-->Andy

Quoted 17 lines

--b.

Quoted 9 lines
fs/nfsd/nfs4state.c +0 -1
 }
 
--

1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

#20 lore raw

Re: [pnfs] [PATCH 3/5] nfsd41: reclaim DRC memory on session free

In reply to J. Bruce Fields
Quoted 8 lines · J. Bruce Fields
Quoted 7 lines · On Wed, Jun 24, 2009 at 03:37:47PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

---
 fs/nfsd/nfs4state.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
Quoted 14 lines

fs/nfsd/nfs4state.c free_session(struct kref *kref) · line 589

Quoted 12 lines
fs/nfsd/nfs4state.c +3 -0
index b4a536d..991c3cc 100644
@@ -583,6 +583,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_mem_used -= ses->se_fchannel.maxreqs * NFSD_SLOT_CACHE_SIZE;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);

This patch has to be combined with the previous one.

I thought this was fixing a bug from the previous patch! But instead I see this is a fix for a preexisting bugfix. So better to move this to the head of the patch series; applied the following to my for-2.6.32.

Apologies--I've just done this myself and applied the following. Look OK?

--b.

commit 0a515ffa6f29e9a0f0f60373576bd4d3d5608ae8

Author: Andy Adamson <[email protected]>
Date:   Mon Jul 27 18:49:05 2009 -0400

nfsd41: reclaim DRC memory on session free

This fixes a leak which would eventually lock out new clients.

fs/nfsd/nfs4state.c +3 -0
index 2e6a44e..69bd37e 100644
@@ -585,6 +585,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_pages_used -= ses->se_fchannel.maxreqs * NFSD_PAGES_PER_SLOT;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);
 }
 
#21 lore raw

Re: [pnfs] [PATCH 3/5] nfsd41: reclaim DRC memory on session free

Quoted 25 lines · J. Bruce Fields
Quoted 8 lines · J. Bruce Fields
Quoted 7 lines · On Wed, Jun 24, 2009 at 03:37:47PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

---
fs/nfsd/nfs4state.c | 3 +++

1 files changed, 3 insertions(+), 0 deletions(-)
Quoted 15 lines

fs/nfsd/nfs4state.c free_session(struct kref *kref) · line 590

Quoted 13 lines
fs/nfsd/nfs4state.c +3 -0
index b4a536d..991c3cc 100644
@@ -583,6 +583,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_mem_used -= ses->se_fchannel.maxreqs *  
 NFSD_SLOT_CACHE_SIZE;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);

This patch has to be combined with the previous one.

Quoted 6 lines

I thought this was fixing a bug from the previous patch! But instead I see this is a fix for a preexisting bugfix. So better to move this to the head of the patch series; applied the following to my for-2.6.32.

Apologies--I've just done this myself and applied the following. Look OK?

Looks fine. Thanks

-->Andy

Quoted 30 lines

--b.

commit 0a515ffa6f29e9a0f0f60373576bd4d3d5608ae8

Author: Andy Adamson <[email protected]>
Date:   Mon Jul 27 18:49:05 2009 -0400

nfsd41: reclaim DRC memory on session free

This fixes a leak which would eventually lock out new clients.

fs/nfsd/nfs4state.c +3 -0
index 2e6a44e..69bd37e 100644
@@ -585,6 +585,9 @@ free_session(struct kref *kref)
 		struct nfsd4_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
 		nfsd4_release_respages(e->ce_respages, e->ce_resused);
 	}
+	spin_lock(&nfsd_drc_lock);
+	nfsd_drc_pages_used -= ses->se_fchannel.maxreqs *  
 NFSD_PAGES_PER_SLOT;
+	spin_unlock(&nfsd_drc_lock);
 	kfree(ses);
 }
#22 lore raw

Re: [pnfs] [PATCH 2/5] nfsd41: change from page to memory based drc limits

In reply to [email protected]
Quoted 43 lines · On Wed, Jun 24, 2009 at 03:37:46PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE is the size of all encoded operation responses (excluding the sequence operation) that we want to cache.

Adjust NFSD_DRC_SIZE_SHIFT to reflect using 512 bytes instead of PAGE_SIZE.

---
 fs/nfsd/nfs4state.c        |   30 ++++++++++++++----------------
 fs/nfsd/nfssvc.c           |   17 +++++++++--------
 include/linux/nfsd/nfsd.h  |    4 ++--
 include/linux/nfsd/state.h |    1 +
 4 files changed, 26 insertions(+), 26 deletions(-)
fs/nfsd/nfs4state.c +5 -6
index 2e6a44e..b4a536d 100644
@@ -414,35 +414,33 @@ gen_sessionid(struct nfsd4_session *ses)
 
 /*
  * Give the client the number of slots it requests bound by
- * NFSD_MAX_SLOTS_PER_SESSION and by sv_drc_max_pages.
+ * NFSD_MAX_SLOTS_PER_SESSION and by nfsd_drc_max_mem.
  *
- * If we run out of pages (sv_drc_pages_used == sv_drc_max_pages) we
- * should (up to a point) re-negotiate active sessions and reduce their
- * slot usage to make rooom for new connections. For now we just fail the
- * create session.
+ * If we run out of reserved DRC memory we should (up to a point) re-negotiate
+ * active sessions and reduce their slot usage to make rooom for new
+ * connections. For now we just fail the create session.
  */
 static int set_forechannel_maxreqs(struct nfsd4_channel_attrs *fchan)
 {
-	int status = 0, np = fchan->maxreqs * NFSD_PAGES_PER_SLOT;
+	int mem;
 
 	if (fchan->maxreqs < 1)
 		return nfserr_inval;
 	else if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
 		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;

fs/nfsd/nfs4state.c gen_sessionid(struct nfsd4_session *ses) · line 437

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 
+	mem = fchan->maxreqs * NFSD_SLOT_CACHE_SIZE;
+
 	spin_lock(&nfsd_drc_lock);
-	if (np + nfsd_drc_pages_used > nfsd_drc_max_pages)
-		np = nfsd_drc_max_pages - nfsd_drc_pages_used;
-	nfsd_drc_pages_used += np;
+	if (mem + nfsd_drc_mem_used > nfsd_drc_max_mem)
+		mem = nfsd_drc_max_mem - nfsd_drc_mem_used;
+	nfsd_drc_mem_used += mem;

There's a small bug here: in the case where

0 < mem < NFSD_PAGES_PER_SLOT,

we add mem into nfsd_drc_mem_used, but never subtract it back off.

Quoted 39 lines
fs/nfsd/nfs4state.c +4 -7
 	spin_unlock(&nfsd_drc_lock);
 
-	if (np <= 0) {
-		status = nfserr_resource;
-		fchan->maxreqs = 0;
-	} else
-		fchan->maxreqs = np / NFSD_PAGES_PER_SLOT;
-
-	return status;
+	fchan->maxreqs = mem / NFSD_SLOT_CACHE_SIZE;
+	if (fchan->maxreqs == 0)
+		return nfserr_resource;
+	return 0;
 }
 
 /*
fs/nfsd/nfssvc.c +2 -6
index 78d8fcd..1793dba 100644
@@ -74,8 +74,8 @@ struct svc_serv 		*nfsd_serv;
  * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage.
  */
 spinlock_t	nfsd_drc_lock;
-unsigned int	nfsd_drc_max_pages;
-unsigned int	nfsd_drc_pages_used;
+unsigned int	nfsd_drc_max_mem;
+unsigned int	nfsd_drc_mem_used;
 
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
 static struct svc_stat	nfsd_acl_svcstats;
@@ -247,13 +247,14 @@ void nfsd_reset_versions(void)
 static void set_max_drc(void)
 {
 	/* The percent of nr_free_buffer_pages used by the V4.1 server DRC */
-	#define NFSD_DRC_SIZE_SHIFT	7
-	nfsd_drc_max_pages = nr_free_buffer_pages()
-						>> NFSD_DRC_SIZE_SHIFT;
-	nfsd_drc_pages_used = 0;

fs/nfsd/nfssvc.c void nfsd_reset_versions(void) · line 257

Quoted 10 lines
fs/nfsd/nfssvc.c +7 -2
+	#define NFSD_DRC_SIZE_SHIFT	10
+	nfsd_drc_max_mem = (nr_free_buffer_pages()
+					>> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE;
+	nfsd_drc_mem_used = 0;
 	spin_lock_init(&nfsd_drc_lock);
-	dprintk("%s nfsd_drc_max_pages %u\n", __func__,
-		nfsd_drc_max_pages);
+	dprintk("%s nfsd_drc_max_mem %u [in pages %lu]\n", __func__,
+		nfsd_drc_max_mem,
+		nfsd_drc_max_mem / PAGE_SIZE);

The reader can do the "in pages" for him/herself if necessary.

Quoted 23 lines
fs/nfsd/nfssvc.c +0 -0
 }
 
 int nfsd_create_serv(void)
include/linux/nfsd/nfsd.h +2 -2
index 2571f85..2812ed5 100644
@@ -57,8 +57,8 @@ extern u32			nfsd_supported_minorversion;
 extern struct mutex		nfsd_mutex;
 extern struct svc_serv		*nfsd_serv;
 extern spinlock_t		nfsd_drc_lock;
-extern unsigned int		nfsd_drc_max_pages;
-extern unsigned int		nfsd_drc_pages_used;
+extern unsigned int		nfsd_drc_max_mem;
+extern unsigned int		nfsd_drc_mem_used;
 
 extern struct seq_operations nfs_exports_op;
 
include/linux/nfsd/state.h +0 -0
index f5a95fd..10b6b30 100644

include/linux/nfsd/state.h struct nfs4_cb_conn { · line 101

Quoted 4 lines
include/linux/nfsd/state.h +1 -0
@@ -98,6 +98,7 @@ struct nfs4_cb_conn {
 /* Maximum number of pages per slot cache entry */
 #define NFSD_PAGES_PER_SLOT	1
 /* Maximum number of operations per session compound */
+#define NFSD_SLOT_CACHE_SIZE		512

Whoops--the above comment's misplaced now.

Also, this patch leaves the code in an inconsistent state where some places still uses page size, others don't. (E.g. we promise to cache page_size, and still will, I think, and allocate that much, but we calculate the maximum requests by the new smaller 512 byte value.)

Either fix all that at once, or just keep NFSD_SLOT_CACHE_SIZE 4k for now.

--b.

Quoted 10 lines
include/linux/nfsd/state.h +0 -0
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
 struct nfsd4_cache_entry {
Signature · 6 lines
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs
#23 lore raw

Re: [pnfs] [PATCH 2/5] nfsd41: change from page to memory based drc limits

Quoted 57 lines · J. Bruce Fields
Quoted 43 lines · On Wed, Jun 24, 2009 at 03:37:46PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE is the size of all encoded operation responses (excluding the sequence operation) that we want to cache.

Adjust NFSD_DRC_SIZE_SHIFT to reflect using 512 bytes instead of PAGE_SIZE.

---
 fs/nfsd/nfs4state.c        |   30 ++++++++++++++----------------
 fs/nfsd/nfssvc.c           |   17 +++++++++--------
 include/linux/nfsd/nfsd.h  |    4 ++--
 include/linux/nfsd/state.h |    1 +
 4 files changed, 26 insertions(+), 26 deletions(-)
fs/nfsd/nfs4state.c +5 -6
index 2e6a44e..b4a536d 100644
@@ -414,35 +414,33 @@ gen_sessionid(struct nfsd4_session *ses)
 
 /*
  * Give the client the number of slots it requests bound by
- * NFSD_MAX_SLOTS_PER_SESSION and by sv_drc_max_pages.
+ * NFSD_MAX_SLOTS_PER_SESSION and by nfsd_drc_max_mem.
  *
- * If we run out of pages (sv_drc_pages_used == sv_drc_max_pages) we
- * should (up to a point) re-negotiate active sessions and reduce their
- * slot usage to make rooom for new connections. For now we just fail the
- * create session.
+ * If we run out of reserved DRC memory we should (up to a point) re-negotiate
+ * active sessions and reduce their slot usage to make rooom for new
+ * connections. For now we just fail the create session.
  */
 static int set_forechannel_maxreqs(struct nfsd4_channel_attrs *fchan)
 {
-	int status = 0, np = fchan->maxreqs * NFSD_PAGES_PER_SLOT;
+	int mem;
 
 	if (fchan->maxreqs < 1)
 		return nfserr_inval;
 	else if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
 		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;

fs/nfsd/nfs4state.c gen_sessionid(struct nfsd4_session *ses) · line 437

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 
+	mem = fchan->maxreqs * NFSD_SLOT_CACHE_SIZE;
+
 	spin_lock(&nfsd_drc_lock);
-	if (np + nfsd_drc_pages_used > nfsd_drc_max_pages)
-		np = nfsd_drc_max_pages - nfsd_drc_pages_used;
-	nfsd_drc_pages_used += np;
+	if (mem + nfsd_drc_mem_used > nfsd_drc_max_mem)
+		mem = nfsd_drc_max_mem - nfsd_drc_mem_used;
+	nfsd_drc_mem_used += mem;

There's a small bug here: in the case where

Quoted 3 lines

0 < mem < NFSD_PAGES_PER_SLOT,

we add mem into nfsd_drc_mem_used, but never subtract it back off.

(Actually this is true whenever there's roundoff error, since we're adding in the un-rounded-off version but subtracting off (in the next patch) the rounded-off version.)

--b.

Quoted 107 lines
Quoted 40 lines
fs/nfsd/nfs4state.c +4 -7
 	spin_unlock(&nfsd_drc_lock);
 
-	if (np <= 0) {
-		status = nfserr_resource;
-		fchan->maxreqs = 0;
-	} else
-		fchan->maxreqs = np / NFSD_PAGES_PER_SLOT;
-
-	return status;
+	fchan->maxreqs = mem / NFSD_SLOT_CACHE_SIZE;
+	if (fchan->maxreqs == 0)
+		return nfserr_resource;
+	return 0;
 }
 
 /*
fs/nfsd/nfssvc.c +2 -6
index 78d8fcd..1793dba 100644
@@ -74,8 +74,8 @@ struct svc_serv 		*nfsd_serv;
  * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage.
  */
 spinlock_t	nfsd_drc_lock;
-unsigned int	nfsd_drc_max_pages;
-unsigned int	nfsd_drc_pages_used;
+unsigned int	nfsd_drc_max_mem;
+unsigned int	nfsd_drc_mem_used;
 
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
 static struct svc_stat	nfsd_acl_svcstats;
@@ -247,13 +247,14 @@ void nfsd_reset_versions(void)
 static void set_max_drc(void)
 {
 	/* The percent of nr_free_buffer_pages used by the V4.1 server DRC */
-	#define NFSD_DRC_SIZE_SHIFT	7
-	nfsd_drc_max_pages = nr_free_buffer_pages()
-						>> NFSD_DRC_SIZE_SHIFT;
-	nfsd_drc_pages_used = 0;

fs/nfsd/nfssvc.c void nfsd_reset_versions(void) · line 257

Quoted 10 lines
fs/nfsd/nfssvc.c +7 -2
+	#define NFSD_DRC_SIZE_SHIFT	10
+	nfsd_drc_max_mem = (nr_free_buffer_pages()
+					>> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE;
+	nfsd_drc_mem_used = 0;
 	spin_lock_init(&nfsd_drc_lock);
-	dprintk("%s nfsd_drc_max_pages %u\n", __func__,
-		nfsd_drc_max_pages);
+	dprintk("%s nfsd_drc_max_mem %u [in pages %lu]\n", __func__,
+		nfsd_drc_max_mem,
+		nfsd_drc_max_mem / PAGE_SIZE);

The reader can do the "in pages" for him/herself if necessary.

Quoted 23 lines
fs/nfsd/nfssvc.c +0 -0
 }
 
 int nfsd_create_serv(void)
include/linux/nfsd/nfsd.h +2 -2
index 2571f85..2812ed5 100644
@@ -57,8 +57,8 @@ extern u32			nfsd_supported_minorversion;
 extern struct mutex		nfsd_mutex;
 extern struct svc_serv		*nfsd_serv;
 extern spinlock_t		nfsd_drc_lock;
-extern unsigned int		nfsd_drc_max_pages;
-extern unsigned int		nfsd_drc_pages_used;
+extern unsigned int		nfsd_drc_max_mem;
+extern unsigned int		nfsd_drc_mem_used;
 
 extern struct seq_operations nfs_exports_op;
 
include/linux/nfsd/state.h +0 -0
index f5a95fd..10b6b30 100644

include/linux/nfsd/state.h struct nfs4_cb_conn { · line 101

Quoted 4 lines
include/linux/nfsd/state.h +1 -0
@@ -98,6 +98,7 @@ struct nfs4_cb_conn {
 /* Maximum number of pages per slot cache entry */
 #define NFSD_PAGES_PER_SLOT	1
 /* Maximum number of operations per session compound */
+#define NFSD_SLOT_CACHE_SIZE		512

Whoops--the above comment's misplaced now.

Also, this patch leaves the code in an inconsistent state where some places still uses page size, others don't. (E.g. we promise to cache page_size, and still will, I think, and allocate that much, but we calculate the maximum requests by the new smaller 512 byte value.)

Either fix all that at once, or just keep NFSD_SLOT_CACHE_SIZE 4k for now.

--b.

Quoted 10 lines
include/linux/nfsd/state.h +0 -0
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
 struct nfsd4_cache_entry {
Signature · 6 lines
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw

Re: [pnfs] [PATCH 2/5] nfsd41: change from page to memory based drc limits

In reply to [email protected]
Quoted 85 lines · On Wed, Jun 24, 2009 at 03:37:46PM -0400, [email protected] wrote:

From: Andy Adamson <[email protected]>

NFSD_SLOT_CACHE_SIZE is the size of all encoded operation responses (excluding the sequence operation) that we want to cache.

Adjust NFSD_DRC_SIZE_SHIFT to reflect using 512 bytes instead of PAGE_SIZE.

---
 fs/nfsd/nfs4state.c        |   30 ++++++++++++++----------------
 fs/nfsd/nfssvc.c           |   17 +++++++++--------
 include/linux/nfsd/nfsd.h  |    4 ++--
 include/linux/nfsd/state.h |    1 +
 4 files changed, 26 insertions(+), 26 deletions(-)
fs/nfsd/nfs4state.c +14 -16
index 2e6a44e..b4a536d 100644
@@ -414,35 +414,33 @@ gen_sessionid(struct nfsd4_session *ses)
 
 /*
  * Give the client the number of slots it requests bound by
- * NFSD_MAX_SLOTS_PER_SESSION and by sv_drc_max_pages.
+ * NFSD_MAX_SLOTS_PER_SESSION and by nfsd_drc_max_mem.
  *
- * If we run out of pages (sv_drc_pages_used == sv_drc_max_pages) we
- * should (up to a point) re-negotiate active sessions and reduce their
- * slot usage to make rooom for new connections. For now we just fail the
- * create session.
+ * If we run out of reserved DRC memory we should (up to a point) re-negotiate
+ * active sessions and reduce their slot usage to make rooom for new
+ * connections. For now we just fail the create session.
  */
 static int set_forechannel_maxreqs(struct nfsd4_channel_attrs *fchan)
 {
-	int status = 0, np = fchan->maxreqs * NFSD_PAGES_PER_SLOT;
+	int mem;
 
 	if (fchan->maxreqs < 1)
 		return nfserr_inval;
 	else if (fchan->maxreqs > NFSD_MAX_SLOTS_PER_SESSION)
 		fchan->maxreqs = NFSD_MAX_SLOTS_PER_SESSION;
 
+	mem = fchan->maxreqs * NFSD_SLOT_CACHE_SIZE;
+
 	spin_lock(&nfsd_drc_lock);
-	if (np + nfsd_drc_pages_used > nfsd_drc_max_pages)
-		np = nfsd_drc_max_pages - nfsd_drc_pages_used;
-	nfsd_drc_pages_used += np;
+	if (mem + nfsd_drc_mem_used > nfsd_drc_max_mem)
+		mem = nfsd_drc_max_mem - nfsd_drc_mem_used;
+	nfsd_drc_mem_used += mem;
 	spin_unlock(&nfsd_drc_lock);
 
-	if (np <= 0) {
-		status = nfserr_resource;
-		fchan->maxreqs = 0;
-	} else
-		fchan->maxreqs = np / NFSD_PAGES_PER_SLOT;
-
-	return status;
+	fchan->maxreqs = mem / NFSD_SLOT_CACHE_SIZE;
+	if (fchan->maxreqs == 0)
+		return nfserr_resource;
+	return 0;
 }
 
 /*
fs/nfsd/nfssvc.c +2 -2
index 78d8fcd..1793dba 100644
@@ -74,8 +74,8 @@ struct svc_serv 		*nfsd_serv;
  * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage.
  */
 spinlock_t	nfsd_drc_lock;
-unsigned int	nfsd_drc_max_pages;
-unsigned int	nfsd_drc_pages_used;
+unsigned int	nfsd_drc_max_mem;
+unsigned int	nfsd_drc_mem_used;
 
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
 static struct svc_stat	nfsd_acl_svcstats;

fs/nfsd/nfssvc.c void nfsd_reset_versions(void) · line 250

Quoted 4 lines
fs/nfsd/nfssvc.c +0 -1
@@ -247,13 +247,14 @@ void nfsd_reset_versions(void)
 static void set_max_drc(void)
 {
 	/* The percent of nr_free_buffer_pages used by the V4.1 server DRC */
-	#define NFSD_DRC_SIZE_SHIFT	7

Not really a "percent". Let's just let the calculation speak for itself.

--b.

Quoted 50 lines
fs/nfsd/nfssvc.c +7 -5
-	nfsd_drc_max_pages = nr_free_buffer_pages()
-						>> NFSD_DRC_SIZE_SHIFT;
-	nfsd_drc_pages_used = 0;
+	#define NFSD_DRC_SIZE_SHIFT	10
+	nfsd_drc_max_mem = (nr_free_buffer_pages()
+					>> NFSD_DRC_SIZE_SHIFT) * PAGE_SIZE;
+	nfsd_drc_mem_used = 0;
 	spin_lock_init(&nfsd_drc_lock);
-	dprintk("%s nfsd_drc_max_pages %u\n", __func__,
-		nfsd_drc_max_pages);
+	dprintk("%s nfsd_drc_max_mem %u [in pages %lu]\n", __func__,
+		nfsd_drc_max_mem,
+		nfsd_drc_max_mem / PAGE_SIZE);
 }
 
 int nfsd_create_serv(void)
include/linux/nfsd/nfsd.h +2 -2
index 2571f85..2812ed5 100644
@@ -57,8 +57,8 @@ extern u32			nfsd_supported_minorversion;
 extern struct mutex		nfsd_mutex;
 extern struct svc_serv		*nfsd_serv;
 extern spinlock_t		nfsd_drc_lock;
-extern unsigned int		nfsd_drc_max_pages;
-extern unsigned int		nfsd_drc_pages_used;
+extern unsigned int		nfsd_drc_max_mem;
+extern unsigned int		nfsd_drc_mem_used;
 
 extern struct seq_operations nfs_exports_op;
 
include/linux/nfsd/state.h +1 -0
index f5a95fd..10b6b30 100644
@@ -98,6 +98,7 @@ struct nfs4_cb_conn {
 /* Maximum number of pages per slot cache entry */
 #define NFSD_PAGES_PER_SLOT	1
 /* Maximum number of operations per session compound */
+#define NFSD_SLOT_CACHE_SIZE		512
 #define NFSD_MAX_OPS_PER_COMPOUND	16
 
 struct nfsd4_cache_entry {
Signature · 6 lines
1.5.4.3

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs
#25 lore raw
In reply to [email protected]

Andy,

I've merged this patchset onto the nfsd41 branch which is now based on 2.6.31-rc1, replacing the respective patches you sent previously.

Benny

Quoted 51 lines · On Jun. 24, 2009, 22:37 +0300, [email protected] wrote:

NFSv4.1 Server DRC rewrite Version 4

This patch set applies to linux-nfs.org/~bfields/linux, for-2.6.31 branch updated on 6-22-2009.

In response to bfield's comments I moved the DRC limit bug fixes to the front of the series, consolidated patches replacing the single slot clientid DRC into one patch and the replacement of the mulitple slot sequeqnce operation DRC into another single patch.

The NFSv4.1 DRC is changed from a page based cache to
a buffer based cache. The logic for the single slot clientid cache has been separated from the session slot logic to handle the CREATE_SESSION call preceeded by a SEQUENCE and all the replay combinations therein.

The session DRC now caches encoded operations with the exception of the SEQUENCE operation which for a replay is encoded with the current slot and session values. A review of message sizes indicates that a 512 byte buffer for the operations is adequate.

Not addressed is replacing the nfsd4_check_drc_limit() post-operation checking with a pre-operation processing estimate of the encoded per operation result

Testing:
4.1 mount: Connectathon and 4.1pynfs including the new create session replay tests.
4.0 mount; Connectathon.

0001-nfsd41-use-globals-for-DRC-limits.patch 0002-nfsd41-change-from-page-to-memory-based-drc-limits.patch 0003-nfsd41-reclaim-DRC-memory-on-session-free.patch 0004-nfsd41-set-the-session-maximum-response-size-cached.patch 0005-nfsd41-remove-redundant-forechannel-max-requests-ch.patch 0006-nfsd41-change-check_slot_seqid-parameters.patch 0007-nfsd41-replace-create_session-DRC-with-xdr-structur.patch 0008-nfsd41-replace-page-based-DRC-with-buffer-based-DRC.patch 0009-nfsd41-rename-nfsd4_enc_uncached_replay.patch 0010-nfsd41-encode-replay-sequence-from-the-slot-values.patch 0011-nfsd41-fix-nfsd4_replay_cache_entry-comments.patch 0012-nfsd41-support-16-slots-per-session.patch 0013-nfsd41-add-test-for-failed-sequence-operation.patch 0014-nfsd41-remove-redundant-failed-sequence-check.patch

Comments welcome.

-->Andy

_______________________________________________
pNFS mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs