Skip to contentSkip to conversation

NLM server can not process retransmited request correctly

RFC New Mi Jinlong 2 participants 7 messages Nov 2, 2009 – Nov 6, 2009

Subsystems (6)
lore
Share & tools
lore

Mi Jinlong

author
#0 lore raw

Hi Trond et al:

When i test the nfslock of NFSv3 at RHEL5.3GA (kernel: 2.6.18-128.el5), NLM server can not process client's retransmited request correctly.

Steps of my test likes followed:
client server
| |
step1 | open file |
open |------------------------------->|
| ok |
|<-------------------------------|
| | step2
| -> | <- service nfslock stop
| |
step3 | WL1: write lock request{0, 0} |
fcntl |------------------------------->|
| |
| WL1_re: WL1 retransmit |
|------------------------------->|
| |
| WL1.reply ENOLCK |
|<-------------------------------|
| | step4
| -> | <- service nfslock start
| |
step5 | WL2: write lock request{0, 0} |
fcntl |------------------------------->|
| |
| WL1_re.reply OK |
|<-------------------------------|
| WL2.reply EBLOCKD |
|<-------------------------------|
V V

Client can not acquire for write lock any more after step4.

Reason:
Server reply ENOLCK for WL1 to client because nfslock service stoped, but it can not distinguish retransmited request with normal request, so it reply OK for WL1_re to client after nfslock service start. But fcntl client called will return when it receive WL1.reply, WL1_re.reply will be droped at Client. So that, the lock between client and server is different. When client send a some lock request to server again, it will get a EBLOCKD error from Server.

Suggestion:
1. I suggest add an DRC(Duplicate request cache) for NLM. Because, as mentioned above, when a client send a lock request to server through NLM. If server isn't reply in time, a retransmit request will be send from client.

Maybe the retransmit request will get different reply from server as the lock request, the lock between client and server will different. So that, the DRC for NLM is necessary.

What's your opinion ?

2. If everybody thinks it's necessary to add it, here is some ideas in my opinion and some patchs implement at 2.6.32-rc5:

a. Because the implement of NLM's DRC is same as NFS's DRC almost,
many codes will be repeated.
I think, could we implement a common DRC for sunrpc? NLM and NFS will use this DRC.

Do you have some good ideas ?

b. A brief description of the patch at the reply:

[0001-Add-a-common-DRC-for-sunrpc.patch]
Add a common DRC for sunrpc, NLM and NFS implement their DRC using it.

[0002-Add-DRC-for-NLM-using-sunrpc-s-common-DRC.patch] [0003-Modify-nfs-s-DRC-to-use-sunrpc-s-common-DRC.patch]

Particular description will be implement at corresponding patch.

Waiting for your reply ......

thanks,

Mi Jinlong

author
#1 lore raw

PATCH 1/3 Add a common DRC for sunrpc

The implemention of the sunrpc's DRC is same as NFS's DRC almost, so i used file "include/linux/nfsd/cache.h" to create a file named "include/linux/sunrpc/drc.h".
The different between drc.h and cache.h are:
1. add a struct(named: drc_cache) to record the request. (the rchits, rcmissed, rcnocache is used to record the process status of the requests.)

struct drc_cache {
struct hlist_head *cache_hash;
struct list_head lru_head;
int cache_disabled;

/* Record the cache */
unsigned int rchits; /* repcache hits */
unsigned int rcmisses; /* repcache misses */
unsigned int rcnocache; /* uncached reqs */

/*
* locking for the reply cache:
* A cache entry is "single use" if c_state == RC_INPROG
* Otherwise, it when accessing _prev or _next, the lock must be held. */
spinlock_t cache_lock;
};

It can be used to create a global variable, and will be initialized when corresponding module be load and used to record those request.

2. drop the nfsd's declared function, modify other declared function name and accept a struct drc_cache argument.

Using file "fs/nfsd/nfscache.c" to create file "net/sunrpc/drc.c". Modify the function to process the struct drc_cache, and modify the nfsd's rchits, rcmissed, rcnocache to record at drc_cache.

---
 include/linux/sunrpc/drc.h |   97 +++++++++++++
 net/sunrpc/Makefile        |    2 +-
 net/sunrpc/drc.c           |  326 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 424 insertions(+), 1 deletions(-)
 create mode 100644 include/linux/sunrpc/drc.h
 create mode 100644 net/sunrpc/drc.c
include/linux/sunrpc/drc.h new file +97 -0
new file mode 100644
index 0000000..b581a4d
@@ -0,0 +1,97 @@
+/*
+ * include/linux/sunrpc/drc.h
+ *
+ * Request reply cache. This was heavily inspired by the
+ * implementation in 4.3BSD/4.4BSD.
+ *
+ * Copyright (C) 1995, 1996 Olaf Kirch <[email protected]>
+ */
+
+#ifndef DRC_H
+#define DRC_H
+
+#include <linux/in.h>
+#include <linux/uio.h>
+#include <linux/spinlock.h>
+
+/*
+ * Representation of a reply cache entry.
+ */
+struct svc_cacherep {
+	struct hlist_node	c_hash;
+	struct list_head	c_lru;
+
+	unsigned char		c_state,	/* unused, inprog, done */
+				c_type,		/* status, buffer */
+				c_secure : 1;	/* req came from port < 1024 */
+	struct sockaddr_in	c_addr;
+	__be32			c_xid;
+	u32			c_prot;
+	u32			c_proc;
+	u32			c_vers;
+	unsigned long		c_timestamp;
+	union {
+		struct kvec	u_vec;
+		__be32		u_status;
+	}			c_u;
+};
+
+#define c_replvec		c_u.u_vec
+#define c_replstat		c_u.u_status
+
+struct drc_cache {
+	struct hlist_head 	*cache_hash;
+	struct list_head	lru_head;
+	int 			cache_disabled;
+
+	/* Record the cache */
+	unsigned int		rchits;		/* repcache hits */
+	unsigned int		rcmisses;	/* repcache misses */
+	unsigned int		rcnocache;	/* uncached reqs */
+
+	/*
+	 * locking for the reply cache:
+	 * A cache entry is "single use" if c_state == RC_INPROG
+	 * Otherwise, it when accessing _prev or _next, the lock must be held.
+	 */
+	spinlock_t		cache_lock;
+};
+
+/* cache entry states */
+enum {
+	RC_UNUSED,
+	RC_INPROG,
+	RC_DONE
+};
+
+/* return values */
+enum {
+	RC_DROPIT,
+	RC_REPLY,
+	RC_DOIT,
+	RC_INTR
+};
+
+/*
+ * Cache types.
+ * We may want to add more types one day, e.g. for diropres and
+ * attrstat replies. Using cache entries with fixed length instead
+ * of buffer pointers may be more efficient.
+ */
+enum {
+	RC_NOCACHE,
+	RC_REPLSTAT,
+	RC_REPLBUFF,
+};
+
+/*
+ * If requests are retransmitted within this interval, they're dropped.
+ */
+#define RC_DELAY		(HZ/5)
+
+int	drc_reply_cache_init(struct drc_cache *);
+void	drc_reply_cache_shutdown(struct drc_cache *);
+int	drc_cache_lookup(struct svc_rqst *, int, struct drc_cache *);
+void	drc_cache_update(struct svc_rqst *, int, __be32 *, struct drc_cache *);
+
+#endif /* DRC_H */
net/sunrpc/Makefile +1 -1
index 9d2fca5..b3e20e4 100644
@@ -12,7 +12,7 @@ sunrpc-y := clnt.o xprt.o socklib.o xprtsock.o sched.o \
 	    svc.o svcsock.o svcauth.o svcauth_unix.o \
 	    addr.o rpcb_clnt.o timer.o xdr.o \
 	    sunrpc_syms.o cache.o rpc_pipe.o \
-	    svc_xprt.o
+	    svc_xprt.o drc.o
 sunrpc-$(CONFIG_NFS_V4_1) += backchannel_rqst.o bc_svc.o
 sunrpc-$(CONFIG_PROC_FS) += stats.o
 sunrpc-$(CONFIG_SYSCTL) += sysctl.o
net/sunrpc/drc.c new file +326 -0

326 lines not on this page · raw message

Signature · 1 line
1.6.2
author
#2 lore raw

PATCH 2/3 Add DRC for NLM using sunrpc's common DRC

In reply to Mi Jinlong

When use sunrpc's common DRC:
At first : define a global variable to record the cache with struct drc_cache; At second: initialize the variable when NLM module be load at init_nlm() by

             call drc_reply_cache_init();
  At third : the variable will be used to record the request by a new function
             nlm_dispatch();

At last : release the variable when NLM module be unload at exit_nlm() by
call drc_reply_cache_shutdown().

This patch's implemention:
1. define a global variable nlm_dc to record the request;
2. add a nlm_dispatch() function reference to nfsd_dispatch(), nlm_dispatch

     will be used to process NLM's request;
  3. add ".vs_dispatch = nlm_dispatch" to nlmsvc_version1, nlmsvc_version3,
     and nlmsvc_version4, which cause the request be processd by nlm_dispatch;

4. add cache type to nlmsvc_procedures4[], only "lock" is RC_REPLSTAT,others are RC_NOCACHE.

---
 fs/lockd/svc.c              |   66 +++++++++++++++++++++++++++++++++++++++++++
 fs/lockd/svc4proc.c         |   52 +++++++++++++++++----------------
 include/linux/lockd/lockd.h |    6 ++++
 3 files changed, 99 insertions(+), 25 deletions(-)
fs/lockd/svc.c +66 -0
index 1a54ae1..66d97ce 100644
@@ -32,6 +32,7 @@
 #include <linux/sunrpc/stats.h>
 #include <linux/sunrpc/clnt.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/sunrpc/svcsock.h>
 #include <net/ip.h>
 #include <linux/lockd/lockd.h>
@@ -76,6 +77,9 @@ static const int		nlm_port_min = 0, nlm_port_max = 65535;
 static struct ctl_table_header * nlm_sysctl_table;
 #endif
 
+/* DRC for NLM */
+struct drc_cache nlm_dc;
+
 static unsigned long get_lockd_grace_period(void)
 {
 	/* Note: nlm_timeout should always be nonzero */
@@ -363,6 +367,63 @@ out:
 }
 EXPORT_SYMBOL_GPL(lockd_down);
 
+/*
+ * DRC for NLM
+ */
+int
+nlm_dispatch(struct svc_rqst *rqstp, u32 *statp)
+{
+	struct svc_procedure *proc;
+	kxdrproc_t	   xdr;
+
+	dprintk("nlm_dispatch: vers %d proc %d\n",
+			rqstp->rq_vers, rqstp->rq_proc);
+	proc = rqstp->rq_procinfo;
+
+	/* Check whether we have this call in the cache. */
+	switch (drc_cache_lookup(rqstp, proc->pc_cachetype, &nlm_dc)) {
+	case RC_INTR:
+	case RC_DROPIT:
+		return 0;
+	case RC_REPLY:
+		return 1;
+	case RC_DOIT:;
+		/* do it */
+	}
+
+	/* Decode arguments */
+	xdr = proc->pc_decode;
+	if (xdr && !xdr(rqstp, (u32*)rqstp->rq_arg.head[0].iov_base,
+			rqstp->rq_argp)) {
+		dprintk("nlm: failed to decode arguments!\n");
+		drc_cache_update(rqstp, RC_NOCACHE, NULL, &nlm_dc);
+		*statp = rpc_garbage_args;
+		return 1;
+	}
+
+	*statp = proc->pc_func(rqstp, rqstp->rq_argp, rqstp->rq_resp);
+
+	if (*statp == rpc_drop_reply) {
+		dprintk("nlm: Dropping request due to malloc failure!\n");
+		drc_cache_update(rqstp, RC_NOCACHE, NULL, &nlm_dc);
+		return 0;
+	}
+
+	if (*statp ==rpc_success && (xdr = proc->pc_encode) && !xdr(rqstp,
+	    (rqstp->rq_res.head[0].iov_base + rqstp->rq_res.head[0].iov_len),
+	     rqstp->rq_resp)) {
+	     dprintk("nlm: failed to encode result!\n");
+	     drc_cache_update(rqstp, RC_NOCACHE, NULL, &nlm_dc);
+	     *statp = rpc_system_err;
+	     return 1;
+	}
+
+	/* Store reply in cache */
+	drc_cache_update(rqstp, proc->pc_cachetype, statp + 1, &nlm_dc);
+	return 1;
+
+}
+
 #ifdef CONFIG_SYSCTL
 
 /*
@@ -525,6 +586,7 @@ module_param(nlm_max_connections, uint, 0644);
 
 static int __init init_nlm(void)
 {
+	drc_reply_cache_init(&nlm_dc);
 #ifdef CONFIG_SYSCTL
 	nlm_sysctl_table = register_sysctl_table(nlm_sysctl_root);
 	return nlm_sysctl_table ? 0 : -ENOMEM;
@@ -536,6 +598,7 @@ static int __init init_nlm(void)
 static void __exit exit_nlm(void)
 {
 	/* FIXME: delete all NLM clients */
+	drc_reply_cache_shutdown(&nlm_dc);
 	nlm_shutdown_hosts();
 #ifdef CONFIG_SYSCTL
 	unregister_sysctl_table(nlm_sysctl_table);
@@ -552,12 +615,14 @@ static struct svc_version	nlmsvc_version1 = {
 		.vs_vers	= 1,
 		.vs_nproc	= 17,
 		.vs_proc	= nlmsvc_procedures,
+		.vs_dispatch	= nlm_dispatch,
 		.vs_xdrsize	= NLMSVC_XDRSIZE,
 };
 static struct svc_version	nlmsvc_version3 = {
 		.vs_vers	= 3,
 		.vs_nproc	= 24,
 		.vs_proc	= nlmsvc_procedures,
+		.vs_dispatch    = nlm_dispatch,
 		.vs_xdrsize	= NLMSVC_XDRSIZE,
 };
 #ifdef CONFIG_LOCKD_V4
@@ -565,6 +630,7 @@ static struct svc_version	nlmsvc_version4 = {
 		.vs_vers	= 4,
 		.vs_nproc	= 24,
 		.vs_proc	= nlmsvc_procedures4,
+		.vs_dispatch    = nlm_dispatch,
 		.vs_xdrsize	= NLMSVC_XDRSIZE,
 };
 #endif
fs/lockd/svc4proc.c +27 -25

76 lines not on this page · raw message

include/linux/lockd/lockd.h +6 -0
index a34dea4..7bfea5c 100644
@@ -198,6 +198,12 @@ extern int			nsm_use_hostnames;
 extern u32			nsm_local_state;
 
 /*
+ * DRC for NLM
+ */
+
+int nlm_dispatch(struct svc_rqst *rqstp, u32 *stap);
+
+/*
  * Lockd client functions
  */
 struct nlm_rqst * nlm_alloc_call(struct nlm_host *host);
Signature · 1 line
1.6.2
author
#3 lore raw

PATCH 3/3 Modify nfs's DRC to use sunrpc's common DRC

In reply to Mi Jinlong

1. define a global variable nfsd_dc to record the request;
2. modify the nfsd_dispatch() to using sunrpc's DRC;
3. delete file "fs/nfsd/nfscache.c" which will not be used;
4. modify file "include/linux/nfsd/cache.h", only hold the declared of function nfsd4_set_statp();
5. modify those file which include "include/linux/nfsd/cache.h" to include "linux/sunrpc/drc.h", because they are same almost;
6. delete rchits, rcmisses, rcnocache from "struct nfsd_stats", which be save at "struct drc_cache".

---
 arch/parisc/kernel/sys_parisc32.c |    2 +-
 arch/s390/kernel/compat_linux.c   |    2 +-
 arch/sparc/kernel/sys_sparc32.c   |    2 +-
 fs/nfsd/Makefile                  |    2 +-
 fs/nfsd/nfs2acl.c                 |    2 +-
 fs/nfsd/nfs3acl.c                 |    2 +-
 fs/nfsd/nfs3proc.c                |    2 +-
 fs/nfsd/nfs4proc.c                |    2 +-
 fs/nfsd/nfs4state.c               |    2 +-
 fs/nfsd/nfscache.c                |  330 -------------------------------------
 fs/nfsd/nfsctl.c                  |   10 +-
 fs/nfsd/nfsproc.c                 |    2 +-
 fs/nfsd/nfssvc.c                  |   13 +-
 fs/nfsd/stats.c                   |    9 +-
 include/linux/nfsd/cache.h        |   62 -------
 include/linux/nfsd/stats.h        |    3 -
 16 files changed, 30 insertions(+), 417 deletions(-)
 delete mode 100644 fs/nfsd/nfscache.c
arch/parisc/kernel/sys_parisc32.c +1 -1
index 561388b..3c7498e 100644
@@ -29,8 +29,8 @@
 #include <linux/nfs_fs.h>
 #include <linux/ncp_fs.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 #include <linux/nfsd/syscall.h>
 #include <linux/poll.h>
arch/s390/kernel/compat_linux.c +1 -1
index 0debcec..d441e48 100644
@@ -35,8 +35,8 @@
 #include <linux/quota.h>
 #include <linux/module.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 #include <linux/nfsd/syscall.h>
 #include <linux/poll.h>
arch/sparc/kernel/sys_sparc32.c +1 -1
index 04e28b2..dca7244 100644
@@ -27,8 +27,8 @@
 #include <linux/quota.h>
 #include <linux/module.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 #include <linux/nfsd/syscall.h>
 #include <linux/poll.h>
fs/nfsd/Makefile +1 -1
index 9b118ee..fdddb23 100644
@@ -5,7 +5,7 @@
 obj-$(CONFIG_NFSD)	+= nfsd.o
 
 nfsd-y 			:= nfssvc.o nfsctl.o nfsproc.o nfsfh.o vfs.o \
-			   export.o auth.o lockd.o nfscache.o nfsxdr.o stats.o
+			   export.o auth.o lockd.o nfsxdr.o stats.o
 nfsd-$(CONFIG_NFSD_V2_ACL) += nfs2acl.o
 nfsd-$(CONFIG_NFSD_V3)	+= nfs3proc.o nfs3xdr.o
 nfsd-$(CONFIG_NFSD_V3_ACL) += nfs3acl.o
fs/nfsd/nfs2acl.c +1 -1
index 4e3219e..06c7293 100644
@@ -7,9 +7,9 @@
  */
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfs.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 #include <linux/nfsd/xdr3.h>
 #include <linux/posix_acl.h>
fs/nfsd/nfs3acl.c +1 -1
index 9981dbb..f6527d3 100644
@@ -7,9 +7,9 @@
  */
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfs3.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr3.h>
 #include <linux/posix_acl.h>
 #include <linux/nfsacl.h>
fs/nfsd/nfs3proc.c +1 -1
index a713c41..932a343 100644
@@ -21,8 +21,8 @@
 #include <linux/magic.h>
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr3.h>
 #include <linux/nfs3.h>
 
fs/nfsd/nfs4proc.c +1 -1
index bebc0c2..5ad2212 100644
@@ -41,8 +41,8 @@
 #include <linux/file.h>
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfs4.h>
 #include <linux/nfsd/state.h>
 #include <linux/nfsd/xdr4.h>
fs/nfsd/nfs4state.c +1 -1
index 2153f9b..efde9c8 100644
@@ -39,8 +39,8 @@
 #include <linux/slab.h>
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/file.h>
 #include <linux/mount.h>
 #include <linux/workqueue.h>
fs/nfsd/nfscache.c deleted +0 -330

330 lines not on this page · raw message

fs/nfsd/nfsctl.c +6 -4
index 5c01fc1..100bb9f 100644
@@ -31,9 +31,9 @@
 #include <linux/nfsd_idmap.h>
 #include <linux/lockd/bind.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/sunrpc/svcsock.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 #include <linux/nfsd/syscall.h>
 #include <linux/lockd/lockd.h>
@@ -42,6 +42,8 @@
 #include <asm/uaccess.h>
 #include <net/ipv6.h>
 
+struct drc_cache nfsd_dc;
+
 /*
  *	We have a single directory with 9 nodes in it.
  */
@@ -1392,7 +1394,7 @@ static int __init init_nfsd(void)
 	if (retval)
 		return retval;
 	nfsd_stat_init();	/* Statistics */
-	retval = nfsd_reply_cache_init();
+	retval = drc_reply_cache_init(&nfsd_dc);
 	if (retval)
 		goto out_free_stat;
 	retval = nfsd_export_init();
@@ -1418,7 +1420,7 @@ out_free_lockd:
 	nfsd_lockd_shutdown();
 	nfsd_export_shutdown();
 out_free_cache:
-	nfsd_reply_cache_shutdown();
+	drc_reply_cache_shutdown(&nfsd_dc);
 out_free_stat:
 	nfsd_stat_shutdown();
 	nfsd4_free_slabs();
@@ -1428,7 +1430,7 @@ out_free_stat:
 static void __exit exit_nfsd(void)
 {
 	nfsd_export_shutdown();
-	nfsd_reply_cache_shutdown();
+	drc_reply_cache_shutdown(&nfsd_dc);
 	remove_proc_entry("fs/nfs/exports", NULL);
 	remove_proc_entry("fs/nfs", NULL);
 	nfsd_stat_shutdown();
fs/nfsd/nfsproc.c +1 -1
index 0eb9c82..4be2996 100644
@@ -21,8 +21,8 @@
 
 #include <linux/sunrpc/clnt.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/nfsd/nfsd.h>
-#include <linux/nfsd/cache.h>
 #include <linux/nfsd/xdr.h>
 
 typedef struct svc_rqst	svc_rqst;
fs/nfsd/nfssvc.c +8 -5
index 67ea83e..6206018 100644
@@ -26,6 +26,7 @@
 #include <linux/sunrpc/types.h>
 #include <linux/sunrpc/stats.h>
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/sunrpc/svcsock.h>
 #include <linux/sunrpc/cache.h>
 #include <linux/nfsd/nfsd.h>
@@ -42,6 +43,8 @@ extern struct svc_program	nfsd_program;
 static int			nfsd(void *vrqstp);
 struct timeval			nfssvc_boot;
 
+extern struct drc_cache nfsd_dc;
+
 /*
  * nfsd_mutex protects nfsd_serv -- both the pointer itself and the members
  * of the svc_serv struct. In particular, ->sv_nrthreads but also to some
@@ -554,7 +557,7 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 	proc = rqstp->rq_procinfo;
 
 	/* Check whether we have this call in the cache. */
-	switch (nfsd_cache_lookup(rqstp, proc->pc_cachetype)) {
+	switch (drc_cache_lookup(rqstp, proc->pc_cachetype, &nfsd_dc)) {
 	case RC_INTR:
 	case RC_DROPIT:
 		return 0;
@@ -569,7 +572,7 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 	if (xdr && !xdr(rqstp, (__be32*)rqstp->rq_arg.head[0].iov_base,
 			rqstp->rq_argp)) {
 		dprintk("nfsd: failed to decode arguments!\n");
-		nfsd_cache_update(rqstp, RC_NOCACHE, NULL);
+		drc_cache_update(rqstp, RC_NOCACHE, NULL, &nfsd_dc);
 		*statp = rpc_garbage_args;
 		return 1;
 	}
@@ -586,7 +589,7 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 	nfserr = map_new_errors(rqstp->rq_vers, nfserr);
 	if (nfserr == nfserr_dropit) {
 		dprintk("nfsd: Dropping request; may be revisited later\n");
-		nfsd_cache_update(rqstp, RC_NOCACHE, NULL);
+		drc_cache_update(rqstp, RC_NOCACHE, NULL, &nfsd_dc);
 		return 0;
 	}
 
@@ -602,14 +605,14 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 				rqstp->rq_resp)) {
 			/* Failed to encode result. Release cache entry */
 			dprintk("nfsd: failed to encode result!\n");
-			nfsd_cache_update(rqstp, RC_NOCACHE, NULL);
+			drc_cache_update(rqstp, RC_NOCACHE, NULL, &nfsd_dc);
 			*statp = rpc_system_err;
 			return 1;
 		}
 	}
 
 	/* Store reply in cache. */
-	nfsd_cache_update(rqstp, proc->pc_cachetype, statp + 1);
+	drc_cache_update(rqstp, proc->pc_cachetype, statp + 1, &nfsd_dc);
 	return 1;
 }
 
fs/nfsd/stats.c +6 -3
index 71944cd..c671a1e 100644
@@ -31,10 +31,13 @@
 #include <linux/module.h>
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/drc.h>
 #include <linux/sunrpc/stats.h>
 #include <linux/nfsd/nfsd.h>
 #include <linux/nfsd/stats.h>
 
+extern struct drc_cache nfsd_dc;
+
 struct nfsd_stats	nfsdstats;
 struct svc_stat		nfsd_svcstats = {
 	.program	= &nfsd_program,
@@ -45,9 +48,9 @@ static int nfsd_proc_show(struct seq_file *seq, void *v)
 	int i;
 
 	seq_printf(seq, "rc %u %u %u\nfh %u %u %u %u %u\nio %u %u\n",
-		      nfsdstats.rchits,
-		      nfsdstats.rcmisses,
-		      nfsdstats.rcnocache,
+		      nfsd_dc.rchits,
+		      nfsd_dc.rcmisses,
+		      nfsd_dc.rcnocache,
 		      nfsdstats.fh_stale,
 		      nfsdstats.fh_lookup,
 		      nfsdstats.fh_anon,
include/linux/nfsd/cache.h +0 -62

68 lines not on this page · raw message

include/linux/nfsd/stats.h +0 -3
index 2693ef6..7c5bf0e 100644
@@ -17,9 +17,6 @@
 #ifdef __KERNEL__
 
 struct nfsd_stats {
-	unsigned int	rchits;		/* repcache hits */
-	unsigned int	rcmisses;	/* repcache hits */
-	unsigned int	rcnocache;	/* uncached reqs */
 	unsigned int	fh_stale;	/* FH stale error */
 	unsigned int	fh_lookup;	/* dentry cached */
 	unsigned int	fh_anon;	/* anon file dentry returned */
Signature · 1 line
1.6.2
#4 lore raw
In reply to Mi Jinlong
Quoted 35 lines · Mi Jinlong

Hi Trond et al:

When i test the nfslock of NFSv3 at RHEL5.3GA (kernel: 2.6.18-128.el5), NLM server can not process client's retransmited request correctly.

Steps of my test likes followed:
client server
| |
step1 | open file |
open |------------------------------->|
| ok |
|<-------------------------------|
| | step2
| -> | <- service nfslock stop
| |
step3 | WL1: write lock request{0, 0} |
fcntl |------------------------------->|
| |
| WL1_re: WL1 retransmit |
|------------------------------->|
| |
| WL1.reply ENOLCK |
|<-------------------------------|
| | step4
| -> | <- service nfslock start
| |
step5 | WL2: write lock request{0, 0} |
fcntl |------------------------------->|
| |
| WL1_re.reply OK |
|<-------------------------------|
| WL2.reply EBLOCKD |
|<-------------------------------|
V V

Quoted 4 lines

Client can not acquire for write lock any more after step4.

Reason:
Server reply ENOLCK for WL1 to client because nfslock service stoped,

I don't completely understand that part: is it because the monitor call to statd fails?

Quoted 6 lines

but it can not distinguish retransmited request with normal request, so it reply OK for WL1_re to client after nfslock service start. But fcntl client called will return when it receive WL1.reply, WL1_re.reply will be droped at Client. So that, the lock between client and server is different. When client send a some lock request to server again, it will get a EBLOCKD error from Server.

But, OK, I get the idea, and generic DRC code shared by nfsd and nlm makes sense to me; thanks for looking into this.

--b.

author
#5 lore raw

Hi Bruce

Thanks for your answer.

J. Bruce Fields:

Quoted 3 lines · Mi Jinlong
Quoted 2 lines

Hi Trond et al:

..snip..

Quoted 4 lines
Quoted 1 line

Server reply ENOLCK for WL1 to client because nfslock service stoped,

I don't completely understand that part: is it because the monitor call to statd fails?

yes.

Quoted 10 lines
Quoted 7 lines

but it can not distinguish retransmited request with normal request, so it reply OK for WL1_re to client after nfslock service start. But fcntl client called will return when it receive WL1.reply, WL1_re.reply will be droped at Client. So that, the lock between client and server is different. When client send a some lock request to server again, it will get a EBLOCKD error from Server.

But, OK, I get the idea, and generic DRC code shared by nfsd and nlm makes sense to me; thanks for looking into this.

In my patch, the generic DRC code as a part of sunrpc, is it ok? or we should put it at fs/nfs_common?

-----
Regards
Mi Jinlong

#6 lore raw
Quoted 26 lines · Mi Jinlong

Hi Bruce

Thanks for your answer.

J. Bruce Fields:

Quoted 3 lines · Mi Jinlong
Quoted 2 lines

Hi Trond et al:

..snip..

Quoted 4 lines
Quoted 1 line

Server reply ENOLCK for WL1 to client because nfslock service stoped,

I don't completely understand that part: is it because the monitor call to statd fails?

yes.

Quoted 8 lines
Quoted 7 lines

but it can not distinguish retransmited request with normal request, so it reply OK for WL1_re to client after nfslock service start. But fcntl client called will return when it receive WL1.reply, WL1_re.reply will be droped at Client. So that, the lock between client and server is different. When client send a some lock request to server again, it will get a EBLOCKD error from Server.

Quoted 5 lines
Quoted 2 lines

But, OK, I get the idea, and generic DRC code shared by nfsd and nlm makes sense to me; thanks for looking into this.

In my patch, the generic DRC code as a part of sunrpc, is it ok? or we should put it at fs/nfs_common?

Sunrpc makes sense to me; this should be purely rpc-level code without any special knowledge of nfs.

--b.