nfsd: support for lifting grace period early
Subsystems (4)
v4 cover nfsd: support for lifting grace period early
v4:
- rename NFSDCLTRACK_RENAME_COMPLETE to NFSDCLTRACK_CLIENT_HAS_SESSION
- add patch to skip multiple create upcalls in v4.0 case
- reorder patches with less controversial ones near the front
v3:
- only accept Y/y/1 in new procfile writes
- turn minorversion env var into a "reclaim complete" boolean
- serialize nfsdcltrack upcalls for a client
- reduce duplicate upcalls via NFSD4_CLIENT_STABLE flag
- don't allow reclaims after RECLAIM_COMPLETE
v2:
- move grace period handling into its own module
This is v4 of the series. The only real change from the last one is to rename the environment var to NFSDCLTRACK_CLIENT_HAS_SESSION, and to add in a patch to prevent multiple upcalls in the case of a v4.0 client.
I'll also be posting a respin of the userland patches which have a more substantial set of changes.
Original cover letter follows:
One of the huge annoyances in dealing with knfsd is the 90s grace period that's imposed when the server reboots. This is not just an annoyance, but means a siginificant amount of "downtime" in many production environments.
This patchset aimed at reducing this pain. It adds a couple of /proc knobs that tell the lockd and nfsd lock managers to lift the grace period.
It also changes the UMH upcalls to pass a little bit of extra info in the form of environment variables so that the upcall program can determine whether there are still any clients that may be in the process of reclaiming.
There are also a couple of cleanup patches in here that are not strictly required. In particular, making a separate grace.ko module doesn't have to be done, but I think it's a good idea.
Jeff Layton (9):
lockd: move lockd's grace period handling into its own module
nfsd: remove redundant boot_time parm from grace_done client tracking
op
nfsd: reject reclaim request when client has already sent
RECLAIM_COMPLETE
lockd: add a /proc/fs/lockd/nlm_end_grace file
nfsd: add a v4_end_grace file to /proc/fs/nfsd
nfsd: pass extra info in env vars to upcalls to allow for early grace
period end
nfsd: serialize nfsdcltrack upcalls for a particular client
nfsd: set and test NFSD4_CLIENT_STABLE bit to reduce nfsdcltrack
upcalls
nfsd: skip subsequent UMH "create" operations after the first one for
v4.0 clients
fs/Kconfig | 6 +- fs/lockd/Makefile | 3 +- fs/lockd/grace.c | 65 ------------------ fs/lockd/netns.h | 1 - fs/lockd/procfs.c | 92 ++++++++++++++++++++++++++ fs/lockd/procfs.h | 28 ++++++++ fs/lockd/svc.c | 10 ++- fs/nfs_common/Makefile | 3 +- fs/nfs_common/grace.c | 113 +++++++++++++++++++++++++++++++ fs/nfsd/Kconfig | 1 + fs/nfsd/nfs4recover.c | 172 +++++++++++++++++++++++++++++++++++++++++------- fs/nfsd/nfs4state.c | 11 ++-- fs/nfsd/nfsctl.c | 45 +++++++++++++ fs/nfsd/state.h | 6 +- include/linux/proc_fs.h | 2 + 15 files changed, 461 insertions(+), 97 deletions(-) delete mode 100644 fs/lockd/grace.c create mode 100644 fs/lockd/procfs.c create mode 100644 fs/lockd/procfs.h create mode 100644 fs/nfs_common/grace.c
Signature · 1 line
1.9.3
PATCH v4 1/9 lockd: move lockd's grace period handling into its own module
Currently, all of the grace period handling is part of lockd. Eventually though we'd like to be able to build v4-only servers, at which point we'll need to put all of this elsewhere.
Move the code itself into fs/nfs_common and have it build a grace.ko module. Then, rejigger the Kconfig options so that both nfsd and lockd enable it automatically.
--- fs/Kconfig | 6 ++- fs/lockd/Makefile | 2 +- fs/lockd/grace.c | 65 ---------------------------- fs/lockd/netns.h | 1 - fs/lockd/svc.c | 1 - fs/nfs_common/Makefile | 3 +- fs/nfs_common/grace.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/Kconfig | 1 + include/linux/proc_fs.h | 2 + 9 files changed, 124 insertions(+), 70 deletions(-) delete mode 100644 fs/lockd/grace.c create mode 100644 fs/nfs_common/grace.c
fs/Kconfig +5 -1
@@ -233,9 +233,13 @@ if NETWORK_FILESYSTEMS source "fs/nfs/Kconfig" source "fs/nfsd/Kconfig" +config GRACE_PERIOD + tristate + config LOCKD tristate depends on FILE_LOCKING + select GRACE_PERIOD config LOCKD_V4 bool
@@ -249,7 +253,7 @@ config NFS_ACL_SUPPORT config NFS_COMMON bool - depends on NFSD || NFS_FS + depends on NFSD || NFS_FS || LOCKD default y source "net/sunrpc/Kconfig"
fs/lockd/Makefile +1 -1
@@ -5,6 +5,6 @@ obj-$(CONFIG_LOCKD) += lockd.o lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \ - svcshare.o svcproc.o svcsubs.o mon.o xdr.o grace.o + svcshare.o svcproc.o svcsubs.o mon.o xdr.o lockd-objs-$(CONFIG_LOCKD_V4) += clnt4xdr.o xdr4.o svc4proc.o lockd-objs := $(lockd-objs-y)
fs/lockd/grace.c deleted +0 -65
65 lines not on this page · raw message
fs/lockd/netns.h +0 -1
@@ -11,7 +11,6 @@ struct lockd_net {struct delayed_work grace_period_end;struct lock_manager lockd_manager; -struct list_head grace_list;spinlock_t nsm_clnt_lock;unsigned int nsm_users;
fs/lockd/svc.c +0 -1
@@ -586,7 +586,6 @@ static int lockd_init_net(struct net *net)struct lockd_net *ln =net_generic (net, lockd_net_id);INIT_DELAYED_WORK (&ln->grace_period_end, grace_ender); -INIT_LIST_HEAD (&ln->grace_list);spin_lock_init (&ln->nsm_clnt_lock);return 0 ; }
fs/nfs_common/Makefile +2 -1
@@ -3,5 +3,6 @@ # obj-$(CONFIG_NFS_ACL_SUPPORT) += nfs_acl.o - nfs_acl-objs := nfsacl.o + +obj-$(CONFIG_GRACE_PERIOD) += grace.o
fs/nfs_common/grace.c new file +113 -0
113 lines not on this page · raw message
fs/nfsd/Kconfig +1 -0
@@ -71,6 +71,7 @@ config NFSD_V4 select FS_POSIX_ACL select SUNRPC_GSS select CRYPTO + select GRACE_PERIOD help This option enables support in your system's NFS server for version 4 of the NFS protocol (RFC 3530).
include/linux/proc_fs.h +2 -0
@@ -74,6 +74,8 @@ static inline int remove_proc_subtree(const char *name, struct proc_dir_entry *p#endif /* CONFIG_PROC_FS */ +struct net ; +static inline struct proc_dir_entry *proc_net_mkdir (struct net *net,const char *name,struct proc_dir_entry *parent) {
Signature · 1 line
1.9.3
Re: [PATCH v4 1/9] lockd: move lockd's grace period handling into its own module
Quoted 150 lines · On Mon, 15 Sep 2014 09:14:02 -0400 Jeff Layton <[email protected]> wrote:
Currently, all of the grace period handling is part of lockd. Eventually though we'd like to be able to build v4-only servers, at which point we'll need to put all of this elsewhere.
Move the code itself into
fs/nfs_commonand have it build a grace.ko module. Then, rejigger theKconfigoptions so that both nfsd and lockd enable it automatically.Signed-off-by Jeff Layton <[email protected]>--- fs/Kconfig | 6 ++- fs/lockd/Makefile | 2 +- fs/lockd/grace.c | 65 ---------------------------- fs/lockd/netns.h | 1 - fs/lockd/svc.c | 1 - fs/nfs_common/Makefile | 3 +- fs/nfs_common/grace.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/Kconfig | 1 + include/linux/proc_fs.h | 2 + 9 files changed, 124 insertions(+), 70 deletions(-) delete mode 100644 fs/lockd/grace.c create mode 100644 fs/nfs_common/grace.cfs/Kconfig +5
-1@@ -233,9 +233,13 @@ if NETWORK_FILESYSTEMS source "fs/nfs/Kconfig" source "fs/nfsd/Kconfig" +config GRACE_PERIOD + tristate + config LOCKD tristate depends on FILE_LOCKING + select GRACE_PERIOD config LOCKD_V4 bool@@ -249,7 +253,7 @@ config NFS_ACL_SUPPORT config NFS_COMMON bool - depends on NFSD || NFS_FS + depends on NFSD || NFS_FS || LOCKD default y source "net/sunrpc/Kconfig"fs/lockd/Makefile +1
-1@@ -5,6 +5,6 @@ obj-$(CONFIG_LOCKD) += lockd.o lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \ - svcshare.o svcproc.o svcsubs.o mon.o xdr.o grace.o + svcshare.o svcproc.o svcsubs.o mon.o xdr.o lockd-objs-$(CONFIG_LOCKD_V4) += clnt4xdr.o xdr4.o svc4proc.o lockd-objs := $(lockd-objs-y)fs/lockd/grace.c deleted +0
-65@@ -1,65 +0,0 @@ -/* -* Common code for control of lockd and nfsv4 grace periods. -*/ - -#include <linux/module.h> -#include <linux/lockd/bind.h> -#include <net/net_namespace.h> - -#include "netns.h" - -static DEFINE_SPINLOCK (grace_lock); - -/** -* locks_start_grace -* @lm: who this grace period is for -* -* A grace period is a period during which locks should not be given -* out. Currently grace periods are only enforced by the two lock -* managers (lockd and nfsd), using the locks_in_grace() function to -* check when they are in a grace period. -* -* This function is called to start a grace period. -*/ -void locks_start_grace (struct net *net,struct lock_manager *lm) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -spin_lock (&grace_lock); -list_add (&lm->list, &ln->grace_list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_start_grace); - -/** -* locks_end_grace -* @lm: who this grace period is for -* -* Call this function to state that the given lock manager is ready to -* resume regular locking. The grace period will not end until all lock -* managers that called locks_start_grace() also call locks_end_grace(). -* Note that callers count on it being safe to call this more than once, -* and the second call should be a no-op. -*/ -void locks_end_grace (struct lock_manager *lm) -{ -spin_lock (&grace_lock); -list_del_init (&lm->list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_end_grace); - -/** -* locks_in_grace -* -* Lock managers call this function to determine when it is OK for them -* to answer ordinary lock requests, and when they should accept only -* lock reclaims. -*/ -int locks_in_grace (struct net *net) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -return !list_empty (&ln->grace_list); -} -EXPORT_SYMBOL_GPL (locks_in_grace);fs/lockd/netns.h +0
-1@@ -11,7 +11,6 @@ struct lockd_net {struct delayed_work grace_period_end;struct lock_manager lockd_manager; -struct list_head grace_list;spinlock_t nsm_clnt_lock;unsigned int nsm_users;fs/lockd/svc.c +0
-0
fs/lockd/svc.c static int lockd_init_net(struct net *net) · line 589
Quoted 4 lines
fs/lockd/svc.c +0
-1@@ -586,7 +586,6 @@ static int lockd_init_net(struct net *net)struct lockd_net *ln =net_generic (net, lockd_net_id);INIT_DELAYED_WORK (&ln->grace_period_end, grace_ender); -INIT_LIST_HEAD (&ln->grace_list);
Ahh, just found a bug. Now that we can initialize the grace_list independently of the lockd_manager, we must ensure that we initialize the lockd_manager's list_head in lockd_init_net. So, we need to add this in here:
INIT_LIST_HEAD(&ln->lockd_manager.list);
...or we can end up oopsing if sm-notify runs and writes to the nlm_grace_end file before lockd comes up.
I've fixed this in my tree, but I'll wait to resend just yet to see whether there are any more comments on the rest of the set.
Quoted 159 lines
fs/lockd/svc.c +0
-0spin_lock_init (&ln->nsm_clnt_lock);return 0 ; }fs/nfs_common/Makefile +2
-1@@ -3,5 +3,6 @@ # obj-$(CONFIG_NFS_ACL_SUPPORT) += nfs_acl.o - nfs_acl-objs := nfsacl.o + +obj-$(CONFIG_GRACE_PERIOD) += grace.ofs/nfs_common/grace.c new file +113
-0@@ -0,0 +1,113 @@ +/* +* Common code for control of lockd and nfsv4 grace periods. +* +* Transplanted from lockd code +*/ + +#include <linux/module.h> +#include <net/net_namespace.h> +#include <net/netns/generic.h> +#include <linux/fs.h> + +static int grace_net_id; +static DEFINE_SPINLOCK (grace_lock); + +/** +* locks_start_grace +* @net: net namespace that this lock manager belongs to +* @lm: who this grace period is for +* +* A grace period is a period during which locks should not be given +* out. Currently grace periods are only enforced by the two lock +* managers (lockd and nfsd), using the locks_in_grace() function to +* check when they are in a grace period. +* +* This function is called to start a grace period. +*/ +void +locks_start_grace (struct net *net,struct lock_manager *lm) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +spin_lock (&grace_lock); +list_add (&lm->list, grace_list); +spin_unlock (&grace_lock); +} +EXPORT_SYMBOL_GPL (locks_start_grace); + +/** +* locks_end_grace +* @net: net namespace that this lock manager belongs to +* @lm: who this grace period is for +* +* Call this function to state that the given lock manager is ready to +* resume regular locking. The grace period will not end until all lock +* managers that called locks_start_grace() also call locks_end_grace(). +* Note that callers count on it being safe to call this more than once, +* and the second call should be a no-op. +*/ +void +locks_end_grace (struct lock_manager *lm) +{ +spin_lock (&grace_lock); +list_del_init (&lm->list); +spin_unlock (&grace_lock); +} +EXPORT_SYMBOL_GPL (locks_end_grace); + +/** +* locks_in_grace +* +* Lock managers call this function to determine when it is OK for them +* to answer ordinary lock requests, and when they should accept only +* lock reclaims. +*/ +int +locks_in_grace (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +return !list_empty (grace_list); +} +EXPORT_SYMBOL_GPL (locks_in_grace); + +static int __net_init +grace_init_net (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +INIT_LIST_HEAD (grace_list); +return 0 ; +} + +static void __net_exit +grace_exit_net (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +BUG_ON (!list_empty (grace_list)); +} + +static struct pernet_operations grace_net_ops = { + .init = grace_init_net, + .exit = grace_exit_net, + .id = &grace_net_id, + .size =sizeof (struct list_head ), +}; + +static int __init +init_grace (void ) +{ +return register_pernet_subsys (&grace_net_ops); +} + +static void __exit +exit_grace (void ) +{ +unregister_pernet_subsys (&grace_net_ops); +} + +MODULE_AUTHOR ("Jeff Layton <[email protected]>" ); +MODULE_LICENSE ("GPL" ); +module_init (init_grace) +module_exit (exit_grace)fs/nfsd/Kconfig +1
-0@@ -71,6 +71,7 @@ config NFSD_V4 select FS_POSIX_ACL select SUNRPC_GSS select CRYPTO + select GRACE_PERIOD help This option enables support in your system's NFS server for version 4 of the NFS protocol (RFC 3530).include/linux/proc_fs.h +2
-0@@ -74,6 +74,8 @@ static inline int remove_proc_subtree(const char *name, struct proc_dir_entry *p#endif /* CONFIG_PROC_FS */ +struct net ; +static inline struct proc_dir_entry *proc_net_mkdir (struct net *net,const char *name,struct proc_dir_entry *parent) {
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH v4 1/9] lockd: move lockd's grace period handling into its own module
Quoted 163 lines · Jeff Layton
Quoted 150 lines · On Mon, 15 Sep 2014 09:14:02 -0400 Jeff Layton <[email protected]> wrote:
Currently, all of the grace period handling is part of lockd. Eventually though we'd like to be able to build v4-only servers, at which point we'll need to put all of this elsewhere.
Move the code itself into
fs/nfs_commonand have it build a grace.ko module. Then, rejigger theKconfigoptions so that both nfsd and lockd enable it automatically.Signed-off-by Jeff Layton <[email protected]>--- fs/Kconfig | 6 ++- fs/lockd/Makefile | 2 +- fs/lockd/grace.c | 65 ---------------------------- fs/lockd/netns.h | 1 - fs/lockd/svc.c | 1 - fs/nfs_common/Makefile | 3 +- fs/nfs_common/grace.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/Kconfig | 1 + include/linux/proc_fs.h | 2 + 9 files changed, 124 insertions(+), 70 deletions(-) delete mode 100644 fs/lockd/grace.c create mode 100644 fs/nfs_common/grace.cfs/Kconfig +5
-1@@ -233,9 +233,13 @@ if NETWORK_FILESYSTEMS source "fs/nfs/Kconfig" source "fs/nfsd/Kconfig" +config GRACE_PERIOD + tristate + config LOCKD tristate depends on FILE_LOCKING + select GRACE_PERIOD config LOCKD_V4 bool@@ -249,7 +253,7 @@ config NFS_ACL_SUPPORT config NFS_COMMON bool - depends on NFSD || NFS_FS + depends on NFSD || NFS_FS || LOCKD default y source "net/sunrpc/Kconfig"fs/lockd/Makefile +1
-1@@ -5,6 +5,6 @@ obj-$(CONFIG_LOCKD) += lockd.o lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \ - svcshare.o svcproc.o svcsubs.o mon.o xdr.o grace.o + svcshare.o svcproc.o svcsubs.o mon.o xdr.o lockd-objs-$(CONFIG_LOCKD_V4) += clnt4xdr.o xdr4.o svc4proc.o lockd-objs := $(lockd-objs-y)fs/lockd/grace.c deleted +0
-65@@ -1,65 +0,0 @@ -/* -* Common code for control of lockd and nfsv4 grace periods. -*/ - -#include <linux/module.h> -#include <linux/lockd/bind.h> -#include <net/net_namespace.h> - -#include "netns.h" - -static DEFINE_SPINLOCK (grace_lock); - -/** -* locks_start_grace -* @lm: who this grace period is for -* -* A grace period is a period during which locks should not be given -* out. Currently grace periods are only enforced by the two lock -* managers (lockd and nfsd), using the locks_in_grace() function to -* check when they are in a grace period. -* -* This function is called to start a grace period. -*/ -void locks_start_grace (struct net *net,struct lock_manager *lm) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -spin_lock (&grace_lock); -list_add (&lm->list, &ln->grace_list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_start_grace); - -/** -* locks_end_grace -* @lm: who this grace period is for -* -* Call this function to state that the given lock manager is ready to -* resume regular locking. The grace period will not end until all lock -* managers that called locks_start_grace() also call locks_end_grace(). -* Note that callers count on it being safe to call this more than once, -* and the second call should be a no-op. -*/ -void locks_end_grace (struct lock_manager *lm) -{ -spin_lock (&grace_lock); -list_del_init (&lm->list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_end_grace); - -/** -* locks_in_grace -* -* Lock managers call this function to determine when it is OK for them -* to answer ordinary lock requests, and when they should accept only -* lock reclaims. -*/ -int locks_in_grace (struct net *net) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -return !list_empty (&ln->grace_list); -} -EXPORT_SYMBOL_GPL (locks_in_grace);fs/lockd/netns.h +0
-1@@ -11,7 +11,6 @@ struct lockd_net {struct delayed_work grace_period_end;struct lock_manager lockd_manager; -struct list_head grace_list;spinlock_t nsm_clnt_lock;unsigned int nsm_users;fs/lockd/svc.c +0
-0fs/lockd/svc.c static int lockd_init_net(struct net *net) · line 589
Quoted 4 lines
fs/lockd/svc.c +0
-1@@ -586,7 +586,6 @@ static int lockd_init_net(struct net *net)struct lockd_net *ln =net_generic (net, lockd_net_id);INIT_DELAYED_WORK (&ln->grace_period_end, grace_ender); -INIT_LIST_HEAD (&ln->grace_list);Ahh, just found a bug. Now that we can initialize the
grace_listindependently of thelockd_manager, we must ensure that we initialize the lockd_manager'slist_headinlockd_init_net. So, we need to add this in here:
Quoted 7 lines
INIT_LIST_HEAD(&ln->lockd_manager.list);...or we can end up oopsing if sm-notify runs and writes to the
nlm_grace_endfile before lockd comes up.I've fixed this in my tree, but I'll wait to resend just yet to see whether there are any more comments on the rest of the set.
I don't have any more comments. I can fix this up if this is all there is to do, or you can resend or give me a git url.--b.
Quoted 164 lines
Quoted 160 lines
fs/lockd/svc.c +0
-0spin_lock_init (&ln->nsm_clnt_lock);return 0 ; }fs/nfs_common/Makefile +2
-1@@ -3,5 +3,6 @@ # obj-$(CONFIG_NFS_ACL_SUPPORT) += nfs_acl.o - nfs_acl-objs := nfsacl.o + +obj-$(CONFIG_GRACE_PERIOD) += grace.ofs/nfs_common/grace.c new file +113
-0@@ -0,0 +1,113 @@ +/* +* Common code for control of lockd and nfsv4 grace periods. +* +* Transplanted from lockd code +*/ + +#include <linux/module.h> +#include <net/net_namespace.h> +#include <net/netns/generic.h> +#include <linux/fs.h> + +static int grace_net_id; +static DEFINE_SPINLOCK (grace_lock); + +/** +* locks_start_grace +* @net: net namespace that this lock manager belongs to +* @lm: who this grace period is for +* +* A grace period is a period during which locks should not be given +* out. Currently grace periods are only enforced by the two lock +* managers (lockd and nfsd), using the locks_in_grace() function to +* check when they are in a grace period. +* +* This function is called to start a grace period. +*/ +void +locks_start_grace (struct net *net,struct lock_manager *lm) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +spin_lock (&grace_lock); +list_add (&lm->list, grace_list); +spin_unlock (&grace_lock); +} +EXPORT_SYMBOL_GPL (locks_start_grace); + +/** +* locks_end_grace +* @net: net namespace that this lock manager belongs to +* @lm: who this grace period is for +* +* Call this function to state that the given lock manager is ready to +* resume regular locking. The grace period will not end until all lock +* managers that called locks_start_grace() also call locks_end_grace(). +* Note that callers count on it being safe to call this more than once, +* and the second call should be a no-op. +*/ +void +locks_end_grace (struct lock_manager *lm) +{ +spin_lock (&grace_lock); +list_del_init (&lm->list); +spin_unlock (&grace_lock); +} +EXPORT_SYMBOL_GPL (locks_end_grace); + +/** +* locks_in_grace +* +* Lock managers call this function to determine when it is OK for them +* to answer ordinary lock requests, and when they should accept only +* lock reclaims. +*/ +int +locks_in_grace (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +return !list_empty (grace_list); +} +EXPORT_SYMBOL_GPL (locks_in_grace); + +static int __net_init +grace_init_net (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +INIT_LIST_HEAD (grace_list); +return 0 ; +} + +static void __net_exit +grace_exit_net (struct net *net) +{ +struct list_head *grace_list =net_generic (net, grace_net_id); + +BUG_ON (!list_empty (grace_list)); +} + +static struct pernet_operations grace_net_ops = { + .init = grace_init_net, + .exit = grace_exit_net, + .id = &grace_net_id, + .size =sizeof (struct list_head ), +}; + +static int __init +init_grace (void ) +{ +return register_pernet_subsys (&grace_net_ops); +} + +static void __exit +exit_grace (void ) +{ +unregister_pernet_subsys (&grace_net_ops); +} + +MODULE_AUTHOR ("Jeff Layton <[email protected]>" ); +MODULE_LICENSE ("GPL" ); +module_init (init_grace) +module_exit (exit_grace)fs/nfsd/Kconfig +1
-0@@ -71,6 +71,7 @@ config NFSD_V4 select FS_POSIX_ACL select SUNRPC_GSS select CRYPTO + select GRACE_PERIOD help This option enables support in your system's NFS server for version 4 of the NFS protocol (RFC 3530).include/linux/proc_fs.h +2
-0@@ -74,6 +74,8 @@ static inline int remove_proc_subtree(const char *name, struct proc_dir_entry *p#endif /* CONFIG_PROC_FS */ +struct net ; +static inline struct proc_dir_entry *proc_net_mkdir (struct net *net,const char *name,struct proc_dir_entry *parent) {Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH v4 1/9] lockd: move lockd's grace period handling into its own module
Quoted 166 lines · On Wed, 17 Sep 2014 15:36:43 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 165 lines · Jeff Layton
Quoted 150 lines · On Mon, 15 Sep 2014 09:14:02 -0400 Jeff Layton <[email protected]> wrote:
Currently, all of the grace period handling is part of lockd. Eventually though we'd like to be able to build v4-only servers, at which point we'll need to put all of this elsewhere.
Move the code itself into
fs/nfs_commonand have it build a grace.ko module. Then, rejigger theKconfigoptions so that both nfsd and lockd enable it automatically.Signed-off-by Jeff Layton <[email protected]>--- fs/Kconfig | 6 ++- fs/lockd/Makefile | 2 +- fs/lockd/grace.c | 65 ---------------------------- fs/lockd/netns.h | 1 - fs/lockd/svc.c | 1 - fs/nfs_common/Makefile | 3 +- fs/nfs_common/grace.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/Kconfig | 1 + include/linux/proc_fs.h | 2 + 9 files changed, 124 insertions(+), 70 deletions(-) delete mode 100644 fs/lockd/grace.c create mode 100644 fs/nfs_common/grace.cfs/Kconfig +5
-1@@ -233,9 +233,13 @@ if NETWORK_FILESYSTEMS source "fs/nfs/Kconfig" source "fs/nfsd/Kconfig" +config GRACE_PERIOD + tristate + config LOCKD tristate depends on FILE_LOCKING + select GRACE_PERIOD config LOCKD_V4 bool@@ -249,7 +253,7 @@ config NFS_ACL_SUPPORT config NFS_COMMON bool - depends on NFSD || NFS_FS + depends on NFSD || NFS_FS || LOCKD default y source "net/sunrpc/Kconfig"fs/lockd/Makefile +1
-1@@ -5,6 +5,6 @@ obj-$(CONFIG_LOCKD) += lockd.o lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \ - svcshare.o svcproc.o svcsubs.o mon.o xdr.o grace.o + svcshare.o svcproc.o svcsubs.o mon.o xdr.o lockd-objs-$(CONFIG_LOCKD_V4) += clnt4xdr.o xdr4.o svc4proc.o lockd-objs := $(lockd-objs-y)fs/lockd/grace.c deleted +0
-65@@ -1,65 +0,0 @@ -/* -* Common code for control of lockd and nfsv4 grace periods. -*/ - -#include <linux/module.h> -#include <linux/lockd/bind.h> -#include <net/net_namespace.h> - -#include "netns.h" - -static DEFINE_SPINLOCK (grace_lock); - -/** -* locks_start_grace -* @lm: who this grace period is for -* -* A grace period is a period during which locks should not be given -* out. Currently grace periods are only enforced by the two lock -* managers (lockd and nfsd), using the locks_in_grace() function to -* check when they are in a grace period. -* -* This function is called to start a grace period. -*/ -void locks_start_grace (struct net *net,struct lock_manager *lm) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -spin_lock (&grace_lock); -list_add (&lm->list, &ln->grace_list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_start_grace); - -/** -* locks_end_grace -* @lm: who this grace period is for -* -* Call this function to state that the given lock manager is ready to -* resume regular locking. The grace period will not end until all lock -* managers that called locks_start_grace() also call locks_end_grace(). -* Note that callers count on it being safe to call this more than once, -* and the second call should be a no-op. -*/ -void locks_end_grace (struct lock_manager *lm) -{ -spin_lock (&grace_lock); -list_del_init (&lm->list); -spin_unlock (&grace_lock); -} -EXPORT_SYMBOL_GPL (locks_end_grace); - -/** -* locks_in_grace -* -* Lock managers call this function to determine when it is OK for them -* to answer ordinary lock requests, and when they should accept only -* lock reclaims. -*/ -int locks_in_grace (struct net *net) -{ -struct lockd_net *ln =net_generic (net, lockd_net_id); - -return !list_empty (&ln->grace_list); -} -EXPORT_SYMBOL_GPL (locks_in_grace);fs/lockd/netns.h +0
-1@@ -11,7 +11,6 @@ struct lockd_net {struct delayed_work grace_period_end;struct lock_manager lockd_manager; -struct list_head grace_list;spinlock_t nsm_clnt_lock;unsigned int nsm_users;fs/lockd/svc.c +0
-0fs/lockd/svc.c static int lockd_init_net(struct net *net) · line 589
Quoted 4 lines
fs/lockd/svc.c +0
-1@@ -586,7 +586,6 @@ static int lockd_init_net(struct net *net)struct lockd_net *ln =net_generic (net, lockd_net_id);INIT_DELAYED_WORK (&ln->grace_period_end, grace_ender); -INIT_LIST_HEAD (&ln->grace_list);Ahh, just found a bug. Now that we can initialize the
grace_listindependently of thelockd_manager, we must ensure that we initialize the lockd_manager'slist_headinlockd_init_net. So, we need to add this in here:INIT_LIST_HEAD(&ln->lockd_manager.list);
Quoted 9 lines
Quoted 5 lines
...or we can end up oopsing if sm-notify runs and writes to the
nlm_grace_endfile before lockd comes up.I've fixed this in my tree, but I'll wait to resend just yet to see whether there are any more comments on the rest of the set.
I don't have any more comments. I can fix this up if this is all there is to do, or you can resend or give me a git url.--b.
That's all there is that needs fixing, AFAIK.
The fix is in my tree at:
git://git.samba.org/jlayton/linux.git
in my nfsd-3.18 branch. Thanks for reviewing and merging it.
The only remaining bit is to make sure that Steve D. (or anyone else for that matter) has no problem with the userland piece.
Steve, does the v4 userland set look OK to you?
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>
PATCH v4 2/9 nfsd: remove redundant boot_time parm from grace_done client tracking op
In reply to Jeff LaytonSince it's stored in nfsd_net, we don't need to pass it in separately.
--- fs/nfsd/nfs4recover.c | 17 ++++++++--------- fs/nfsd/nfs4state.c | 2 +- fs/nfsd/state.h | 2 +- 3 files changed, 10 insertions(+), 11 deletions(-)
fs/nfsd/nfs4recover.c +8 -9
@@ -58,7 +58,7 @@ struct nfsd4_client_tracking_ops {void (*create)(struct nfs4_client *);void (*remove)(struct nfs4_client *);int (*check)(struct nfs4_client *); -void (*grace_done)(struct nfsd_net *,time_t ); +void (*grace_done)(struct nfsd_net *); };/* Globals */
@@ -392,7 +392,7 @@ purge_old(struct dentry *parent, struct dentry *child, struct nfsd_net *nn) }static void -nfsd4_recdir_purge_old (struct nfsd_net *nn,time_t boot_time) +nfsd4_recdir_purge_old (struct nfsd_net *nn) {int status;
@@ -1016,7 +1016,7 @@ nfsd4_cld_check(struct nfs4_client *clp) }static void -nfsd4_cld_grace_done (struct nfsd_net *nn,time_t boot_time) +nfsd4_cld_grace_done (struct nfsd_net *nn) {int ret;struct cld_upcall *cup;
@@ -1029,7 +1029,7 @@ nfsd4_cld_grace_done(struct nfsd_net *nn, time_t boot_time) } cup->cu_msg.cm_cmd = Cld_GraceDone; - cup->cu_msg.cm_u.cm_gracetime = (int64_t )boot_time; + cup->cu_msg.cm_u.cm_gracetime = (int64_t )nn->boot_time; ret =cld_pipe_upcall (cn->cn_pipe, &cup->cu_msg);if (!ret) ret = cup->cu_msg.cm_status;
@@ -1245,13 +1245,12 @@ nfsd4_umh_cltrack_check(struct nfs4_client *clp) }static void -nfsd4_umh_cltrack_grace_done (struct nfsd_net __attribute__((unused)) *nn, -time_t boot_time) +nfsd4_umh_cltrack_grace_done (struct nfsd_net *nn) {char *legacy;char timestr[22 ];/* FIXME: better way to determine max size? */ -sprintf (timestr,"%ld" , boot_time); +sprintf (timestr,"%ld" , nn->boot_time); legacy =nfsd4_cltrack_legacy_topdir ();nfsd4_umh_cltrack_upcall ("gracedone" , timestr, legacy);kfree (legacy);
@@ -1356,10 +1355,10 @@ nfsd4_client_record_check(struct nfs4_client *clp) }void -nfsd4_record_grace_done (struct nfsd_net *nn,time_t boot_time) +nfsd4_record_grace_done (struct nfsd_net *nn) {if (nn->client_tracking_ops) - nn->client_tracking_ops->grace_done (nn, boot_time); + nn->client_tracking_ops->grace_done (nn); } static int
fs/nfsd/nfs4state.c +1 -1
@@ -4122,7 +4122,7 @@ nfsd4_end_grace(struct nfsd_net *nn)dprintk ("NFSD: end of grace period\n" ); nn->grace_ended = true; -nfsd4_record_grace_done (nn, nn->boot_time); +nfsd4_record_grace_done (nn);locks_end_grace (&nn->nfsd4_manager);/* * Now that every NFSv4 client has had the chance to recover and
fs/nfsd/state.h +1 -1
@@ -550,7 +550,7 @@ extern void nfsd4_client_tracking_exit(struct net *net);extern void nfsd4_client_record_create (struct nfs4_client *clp);extern void nfsd4_client_record_remove (struct nfs4_client *clp);extern int nfsd4_client_record_check (struct nfs4_client *clp); -extern void nfsd4_record_grace_done (struct nfsd_net *nn,time_t boot_time); +extern void nfsd4_record_grace_done (struct nfsd_net *nn);/* nfs fault injection functions */ #ifdef CONFIG_NFSD_FAULT_INJECTION
Signature · 1 line
1.9.3
PATCH v4 3/9 nfsd: reject reclaim request when client has already sent RECLAIM_COMPLETE
In reply to Jeff LaytonAs stated in RFC 5661, section 18.51.3:
Once a RECLAIM_COMPLETE is done, there can be no further reclaim operations for locks whose scope is defined as having completed recovery. Once the client sends RECLAIM_COMPLETE, the server will not allow the client to do subsequent reclaims of locking state for that scope and, if these are attempted, will return NFS4ERR_NO_GRACE.
Ensure that we enforce that requirement.
--- fs/nfsd/nfs4state.c | 3 +++ 1 file changed, 3 insertions(+)
fs/nfsd/nfs4state.c +3 -0
@@ -5664,6 +5664,9 @@ nfs4_check_open_reclaim(clientid_t *clid,if (status)return nfserr_reclaim_bad; +if (test_bit (NFSD4_CLIENT_RECLAIM_COMPLETE , &cstate->clp->cl_flags)) +return nfserr_no_grace; +if (nfsd4_client_record_check (cstate->clp))return nfserr_reclaim_bad;
Signature · 1 line
1.9.3
PATCH v4 4/9 lockd: add a /proc/fs/lockd/nlm_end_grace file
In reply to Jeff LaytonAdd a new procfile that will allow a (privileged) userland process to end the NLM grace period early. The basic idea here will be to have sm-notify write to this file, if it sent out no NOTIFY requests when it runs. In that situation, we can generally expect that there will be no reclaim requests so the grace period can be lifted early.
--- fs/lockd/Makefile | 1 + fs/lockd/procfs.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/lockd/procfs.h | 28 +++++++++++++++++ fs/lockd/svc.c | 9 ++++++ 4 files changed, 130 insertions(+) create mode 100644 fs/lockd/procfs.c create mode 100644 fs/lockd/procfs.h
fs/lockd/Makefile +1 -0
@@ -7,4 +7,5 @@ obj-$(CONFIG_LOCKD) += lockd.o lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \ svcshare.o svcproc.o svcsubs.o mon.o xdr.o lockd-objs-$(CONFIG_LOCKD_V4) += clnt4xdr.o xdr4.o svc4proc.o +lockd-objs-$(CONFIG_PROC_FS) += procfs.o lockd-objs := $(lockd-objs-y)
fs/lockd/procfs.c new file +92 -0
92 lines not on this page · raw message
fs/lockd/procfs.h new file +28 -0
@@ -0,0 +1,28 @@ +/* +* Procfs support for lockd +* +* Copyright (c) 2014 Jeff Layton <[email protected]> +*/ +#ifndef _LOCKD_PROCFS_H +#define _LOCKD_PROCFS_H + +#include <linux/kconfig.h> + +#if IS_ENABLED (CONFIG_PROC_FS ) +int lockd_create_procfs (void ); +void lockd_remove_procfs (void ); +#else +static inline int +lockd_create_procfs (void ) +{ +return 0 ; +} + +static inline void +lockd_remove_procfs (void ) +{ +return ; +} +#endif /* IS_ENABLED(CONFIG_PROC_FS) */ + +#endif /* _LOCKD_PROCFS_H */
fs/lockd/svc.c +9 -0
@@ -36,6 +36,7 @@#include <linux/nfs.h> #include "netns.h" +#include "procfs.h" #define NLMDBG_FACILITY NLMDBG_SVC#define LOCKD_BUFSIZE (1024 + NLMSVC_XDRSIZE)
@@ -619,8 +620,15 @@ static int __init init_nlm(void) err =register_pernet_subsys (&lockd_net_ops);if (err) goto err_pernet; + + err =lockd_create_procfs (); +if (err) + goto err_procfs; +return 0 ; +err_procfs: +unregister_pernet_subsys (&lockd_net_ops); err_pernet: #ifdef CONFIG_SYSCTLunregister_sysctl_table (nlm_sysctl_table );
@@ -633,6 +641,7 @@ static void __exit exit_nlm(void) {/* FIXME: delete all NLM clients */ nlm_shutdown_hosts (); +lockd_remove_procfs ();unregister_pernet_subsys (&lockd_net_ops);#ifdef CONFIG_SYSCTL unregister_sysctl_table (nlm_sysctl_table);
Signature · 1 line
1.9.3
PATCH v4 5/9 nfsd: add a v4_end_grace file to /proc/fs/nfsd
In reply to Jeff LaytonAllow a privileged userland process to end the v4 grace period early. Writing "Y", "y", or "1" to the file will cause the v4 grace period to be lifted. The basic idea with this will be to allow the userland client tracking program to lift the grace period once it knows that no more clients will be reclaiming state.
--- fs/nfsd/nfs4state.c | 2 +- fs/nfsd/nfsctl.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ fs/nfsd/state.h | 3 +++ 3 files changed, 49 insertions(+), 1 deletion(-)
fs/nfsd/nfs4state.c +1 -1
@@ -4113,7 +4113,7 @@ out:return status; } -static void +void nfsd4_end_grace (struct nfsd_net *nn) {/* do nothing if grace period already ended */
fs/nfsd/nfsctl.c +45 -0
75 lines not on this page · raw message
fs/nfsd/state.h +3 -0
@@ -544,6 +544,9 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name, struct nfsd_net *nn);extern bool nfs4_has_reclaimed_state (const char *name,struct nfsd_net *nn); +/* grace period management */ +void nfsd4_end_grace (struct nfsd_net *nn); +/* nfs4recover operations */ extern int nfsd4_client_tracking_init (struct net *net);extern void nfsd4_client_tracking_exit (struct net *net);
Signature · 1 line
1.9.3
PATCH v4 6/9 nfsd: pass extra info in env vars to upcalls to allow for early grace period end
In reply to Jeff LaytonIn order to support lifting the grace period early, we must tell nfsdcltrack what sort of client the "create" upcall is for. We can't reliably tell if a v4.0 client has completed reclaiming, so we can only lift the grace period once all the v4.1+ clients have issued a RECLAIM_COMPLETE and if there are no v4.0 clients.
Also, in order to lift the grace period, we have to tell userland when the grace period started so that it can tell whether a RECLAIM_COMPLETE has been issued for each client since then.
Since this is all optional info, we pass it along in environment variables to the "init" and "create" upcalls. By doing this, we don't need to revise the upcall format. The UMH upcall can simply make use of this info if it happens to be present. If it's not then it can just avoid lifting the grace period early.
--- fs/nfsd/nfs4recover.c | 96 ++++++++++++++++++++++++++++++++++++++++++++------- fs/nfsd/nfs4state.c | 4 +-- 2 files changed, 85 insertions(+), 15 deletions(-)
fs/nfsd/nfs4recover.c +83 -13
@@ -1062,6 +1062,8 @@ MODULE_PARM_DESC(cltrack_legacy_disable,#define LEGACY_TOPDIR_ENV_PREFIX "NFSDCLTRACK_LEGACY_TOPDIR="#define LEGACY_RECDIR_ENV_PREFIX "NFSDCLTRACK_LEGACY_RECDIR=" +#define HAS_SESSION_ENV_PREFIX "NFSDCLTRACK_CLIENT_HAS_SESSION=" +#define GRACE_START_ENV_PREFIX "NFSDCLTRACK_GRACE_START="static char *nfsd4_cltrack_legacy_topdir (void )
@@ -1126,10 +1128,60 @@ nfsd4_cltrack_legacy_recdir(const struct xdr_netobj *name)return result; } +static char * +nfsd4_cltrack_client_has_session (struct nfs4_client *clp) +{ +int copied; +size_t len; +char *result; + +/* prefix + Y/N character + terminating NULL */ + len =strlen (HAS_SESSION_ENV_PREFIX ) +1 +1 ; + + result =kmalloc (len,GFP_KERNEL ); +if (!result) +return result; + + copied =snprintf (result, len,HAS_SESSION_ENV_PREFIX "%c" , + clp->cl_minorversion ?'Y' :'N' ); +if (copied >= len) { +/* just return nothing if output was truncated */ +kfree (result); +return NULL ; + } + +return result; +} + +static char * +nfsd4_cltrack_grace_start (time_t grace_start) +{ +int copied; +size_t len; +char *result; + +/* prefix + max width of int64_t string + terminating NULL */ + len =strlen (GRACE_START_ENV_PREFIX ) +22 +1 ; + + result =kmalloc (len,GFP_KERNEL ); +if (!result) +return result; + + copied =snprintf (result, len,GRACE_START_ENV_PREFIX "%ld" , + grace_start); +if (copied >= len) { +/* just return nothing if output was truncated */ +kfree (result); +return NULL ; + } + +return result; +} +static int -nfsd4_umh_cltrack_upcall (char *cmd,char *arg,char *legacy) +nfsd4_umh_cltrack_upcall (char *cmd,char *arg,char *env0,char *env1) { -char *envp[2 ]; +char *envp[3 ];char *argv[4 ];int ret;
@@ -1140,10 +1192,12 @@ nfsd4_umh_cltrack_upcall(char *cmd, char *arg, char *legacy)dprintk ("%s: cmd: %s\n" , __func__, cmd);dprintk ("%s: arg: %s\n" , __func__, arg ? arg :"(null)" ); -dprintk ("%s: legacy: %s\n" , __func__, legacy ? legacy :"(null)" ); +dprintk ("%s: env0: %s\n" , __func__, env0 ? env0 :"(null)" ); +dprintk ("%s: env1: %s\n" , __func__, env1 ? env1 :"(null)" ); - envp[0 ] = legacy; - envp[1 ] =NULL ; + envp[0 ] = env0; + envp[1 ] = env1; + envp[2 ] =NULL ; argv[0 ] = (char *)cltrack_prog; argv[1 ] = cmd;
@@ -1187,28 +1241,40 @@ bin_to_hex_dup(const unsigned char *src, int srclen) }static int -nfsd4_umh_cltrack_init (struct net __attribute__((unused)) *net) +nfsd4_umh_cltrack_init (struct net *net) { +int ret; +struct nfsd_net *nn =net_generic (net, nfsd_net_id); +char *grace_start =nfsd4_cltrack_grace_start (nn->boot_time); +/* XXX: The usermode helper s not working in container yet. */ if (net != &init_net) {WARN (1 ,KERN_ERR "NFSD: attempt to initialize umh client " "tracking in a container!\n" );return -EINVAL ; } -return nfsd4_umh_cltrack_upcall ("init" ,NULL ,NULL ); + + ret =nfsd4_umh_cltrack_upcall ("init" ,NULL , grace_start,NULL ); +kfree (grace_start); +return ret; }static void nfsd4_umh_cltrack_create (struct nfs4_client *clp) { -char *hexid; +char *hexid, *has_session, *grace_start; +struct nfsd_net *nn =net_generic (clp->net, nfsd_net_id); hexid =bin_to_hex_dup (clp->cl_name.data, clp->cl_name.len);if (!hexid) {dprintk ("%s: can't allocate memory for upcall!\n" , __func__);return ; } -nfsd4_umh_cltrack_upcall ("create" , hexid,NULL ); + has_session =nfsd4_cltrack_client_has_session (clp); + grace_start =nfsd4_cltrack_grace_start (nn->boot_time); +nfsd4_umh_cltrack_upcall ("create" , hexid, has_session, grace_start); +kfree (has_session); +kfree (grace_start);kfree (hexid); }
@@ -1222,7 +1288,7 @@ nfsd4_umh_cltrack_remove(struct nfs4_client *clp)dprintk ("%s: can't allocate memory for upcall!\n" , __func__);return ; } -nfsd4_umh_cltrack_upcall ("remove" , hexid,NULL ); +nfsd4_umh_cltrack_upcall ("remove" , hexid,NULL ,NULL );kfree (hexid); }
@@ -1230,17 +1296,21 @@ static int nfsd4_umh_cltrack_check(struct nfs4_client *clp) {int ret; -char *hexid, *legacy; +char *hexid, *has_session, *legacy; hexid =bin_to_hex_dup (clp->cl_name.data, clp->cl_name.len);if (!hexid) {dprintk ("%s: can't allocate memory for upcall!\n" , __func__);return -ENOMEM ; } + + has_session =nfsd4_cltrack_client_has_session (clp); legacy =nfsd4_cltrack_legacy_recdir (&clp->cl_name); - ret =nfsd4_umh_cltrack_upcall ("check" , hexid, legacy); + ret =nfsd4_umh_cltrack_upcall ("check" , hexid, has_session, legacy); +kfree (has_session);kfree (legacy);kfree (hexid); +return ret; }
@@ -1252,7 +1322,7 @@ nfsd4_umh_cltrack_grace_done(struct nfsd_net *nn)sprintf (timestr,"%ld" , nn->boot_time); legacy =nfsd4_cltrack_legacy_topdir (); -nfsd4_umh_cltrack_upcall ("gracedone" , timestr, legacy); +nfsd4_umh_cltrack_upcall ("gracedone" , timestr, legacy,NULL );kfree (legacy); }
fs/nfsd/nfs4state.c +2 -2
@@ -6364,10 +6364,10 @@ nfs4_state_start_net(struct net *net) ret =nfs4_state_create_net (net);if (ret)return ret; -nfsd4_client_tracking_init (net); nn->boot_time =get_seconds (); -locks_start_grace (net, &nn->nfsd4_manager); nn->grace_ended = false; +locks_start_grace (net, &nn->nfsd4_manager); +nfsd4_client_tracking_init (net);printk (KERN_INFO "NFSD: starting %ld-second grace period (net %p)\n" , nn->nfsd4_grace, net);queue_delayed_work (laundry_wq, &nn->laundromat_work, nn->nfsd4_grace *HZ );
Signature · 1 line
1.9.3
PATCH v4 7/9 nfsd: serialize nfsdcltrack upcalls for a particular client
In reply to Jeff LaytonIn a later patch, we want to add a flag that will allow us to reduce the need for upcalls. In order to handle that correctly, we'll need to ensure that racing upcalls for the same client can't occur. In practice it should be rare for this to occur with a well-behaved client, but it is possible.
Convert one of the bits in the cl_flags field to be an upcall bitlock, and use it to ensure that upcalls for the same client are serialized.
--- fs/nfsd/nfs4recover.c | 29 +++++++++++++++++++++++++++++ fs/nfsd/state.h | 1 + 2 files changed, 30 insertions(+)
fs/nfsd/nfs4recover.c +29 -0
@@ -1260,6 +1260,22 @@ nfsd4_umh_cltrack_init(struct net *net) }static void +nfsd4_cltrack_upcall_lock (struct nfs4_client *clp) +{ +wait_on_bit_lock (&clp->cl_flags,NFSD4_CLIENT_UPCALL_LOCK , +TASK_UNINTERRUPTIBLE ); +} + +static void +nfsd4_cltrack_upcall_unlock (struct nfs4_client *clp) +{ +smp_mb__before_atomic (); +clear_bit (NFSD4_CLIENT_UPCALL_LOCK , &clp->cl_flags); +smp_mb__after_atomic (); +wake_up_bit (&clp->cl_flags,NFSD4_CLIENT_UPCALL_LOCK ); +} + +static void nfsd4_umh_cltrack_create (struct nfs4_client *clp) {char *hexid, *has_session, *grace_start;
@@ -1270,9 +1286,14 @@ nfsd4_umh_cltrack_create(struct nfs4_client *clp)dprintk ("%s: can't allocate memory for upcall!\n" , __func__);return ; } + has_session =nfsd4_cltrack_client_has_session (clp); grace_start =nfsd4_cltrack_grace_start (nn->boot_time); + +nfsd4_cltrack_upcall_lock (clp);nfsd4_umh_cltrack_upcall ("create" , hexid, has_session, grace_start); +nfsd4_cltrack_upcall_unlock (clp); +kfree (has_session);kfree (grace_start);kfree (hexid);
@@ -1288,7 +1309,11 @@ nfsd4_umh_cltrack_remove(struct nfs4_client *clp)dprintk ("%s: can't allocate memory for upcall!\n" , __func__);return ; } + +nfsd4_cltrack_upcall_lock (clp);nfsd4_umh_cltrack_upcall ("remove" , hexid,NULL ,NULL ); +nfsd4_cltrack_upcall_unlock (clp); +kfree (hexid); }
@@ -1306,7 +1331,11 @@ nfsd4_umh_cltrack_check(struct nfs4_client *clp) has_session =nfsd4_cltrack_client_has_session (clp); legacy =nfsd4_cltrack_legacy_recdir (&clp->cl_name); + +nfsd4_cltrack_upcall_lock (clp); ret =nfsd4_umh_cltrack_upcall ("check" , hexid, has_session, legacy); +nfsd4_cltrack_upcall_unlock (clp); +kfree (has_session);kfree (legacy);kfree (hexid);
fs/nfsd/state.h +1 -0
@@ -306,6 +306,7 @@ struct nfs4_client {#define NFSD4_CLIENT_STABLE (2)/* client on stable storage */ #define NFSD4_CLIENT_RECLAIM_COMPLETE (3)/* reclaim_complete done */ #define NFSD4_CLIENT_CONFIRMED (4)/* client is confirmed */ +#define NFSD4_CLIENT_UPCALL_LOCK (5)/* upcall serialization */ #define NFSD4_CLIENT_CB_FLAG_MASK (1 << NFSD4_CLIENT_CB_UPDATE | \ 1 << NFSD4_CLIENT_CB_KILL)unsigned long cl_flags;
Signature · 1 line
1.9.3
PATCH v4 8/9 nfsd: set and test NFSD4_CLIENT_STABLE bit to reduce nfsdcltrack upcalls
In reply to Jeff LaytonThe nfsdcltrack upcall doesn't utilize the NFSD4_CLIENT_STABLE flag, which basically results in an upcall every time we call into the client tracking ops.
Change it to set this bit on a successful "check" or "create" request, and clear it on a "remove" request. Also, check to see if that bit is set before upcalling on a "check" or "remove" request, and skip upcalling appropriately, depending on its state.
--- fs/nfsd/nfs4recover.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
fs/nfsd/nfs4recover.c +18 -4
@@ -1291,7 +1291,8 @@ nfsd4_umh_cltrack_create(struct nfs4_client *clp) grace_start =nfsd4_cltrack_grace_start (nn->boot_time);nfsd4_cltrack_upcall_lock (clp); -nfsd4_umh_cltrack_upcall ("create" , hexid, has_session, grace_start); +if (!nfsd4_umh_cltrack_upcall ("create" , hexid, has_session, grace_start)) +set_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags);nfsd4_cltrack_upcall_unlock (clp);kfree (has_session);
@@ -1304,6 +1305,9 @@ nfsd4_umh_cltrack_remove(struct nfs4_client *clp) {char *hexid; +if (!test_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags)) +return ; + hexid =bin_to_hex_dup (clp->cl_name.data, clp->cl_name.len);if (!hexid) {dprintk ("%s: can't allocate memory for upcall!\n" , __func__);
@@ -1311,7 +1315,9 @@ nfsd4_umh_cltrack_remove(struct nfs4_client *clp) }nfsd4_cltrack_upcall_lock (clp); -nfsd4_umh_cltrack_upcall ("remove" , hexid,NULL ,NULL ); +if (test_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags) && +nfsd4_umh_cltrack_upcall ("remove" , hexid,NULL ,NULL ) ==0 ) +clear_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags);nfsd4_cltrack_upcall_unlock (clp);kfree (hexid);
@@ -1323,6 +1329,9 @@ nfsd4_umh_cltrack_check(struct nfs4_client *clp)int ret;char *hexid, *has_session, *legacy; +if (test_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags)) +return 0 ; + hexid =bin_to_hex_dup (clp->cl_name.data, clp->cl_name.len);if (!hexid) {dprintk ("%s: can't allocate memory for upcall!\n" , __func__);
@@ -1333,9 +1342,14 @@ nfsd4_umh_cltrack_check(struct nfs4_client *clp) legacy =nfsd4_cltrack_legacy_recdir (&clp->cl_name);nfsd4_cltrack_upcall_lock (clp); - ret =nfsd4_umh_cltrack_upcall ("check" , hexid, has_session, legacy); +if (test_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags)) { + ret =0 ; + }else { + ret =nfsd4_umh_cltrack_upcall ("check" , hexid, has_session, legacy); +if (ret ==0 ) +set_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags); + }nfsd4_cltrack_upcall_unlock (clp); -kfree (has_session);kfree (legacy);kfree (hexid);
Signature · 1 line
1.9.3
PATCH v4 9/9 nfsd: skip subsequent UMH "create" operations after the first one for v4.0 clients
In reply to Jeff LaytonIn the case of v4.0 clients, we may call into the "create" client tracking operation multiple times (once for each openowner). Upcalling for each one of those is wasteful and slow however. We can skip doing further "create" operations after the first one if we know that one has already been done.
v4.1+ clients generally only call into this function once (on RECLAIM_COMPLETE), and we can't skip upcalling on the create even if the STABLE bit is set. Doing so would make it impossible for nfsdcltrack to lift the grace period early since the timestamp has a different meaning in the case where the client is expected to issue a RECLAIM_COMPLETE.
--- fs/nfsd/nfs4recover.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
fs/nfsd/nfs4recover.c +16 -0
@@ -1281,6 +1281,22 @@ nfsd4_umh_cltrack_create(struct nfs4_client *clp)char *hexid, *has_session, *grace_start;struct nfsd_net *nn =net_generic (clp->net, nfsd_net_id); +/* +* With v4.0 clients, there's little difference in outcome between a +* create and check operation, and we can end up calling into this +* function multiple times per client (once for each openowner). So, +* for v4.0 clients skip upcalling once the client has been recorded +* on stable storage. +* +* For v4.1+ clients, the outcome of the two operations is different, +* so we must ensure that we upcall for the create operation. v4.1+ +* clients call this on RECLAIM_COMPLETE though, so we should only end +* up doing a single create upcall per client. +*/ +if (clp->cl_minorversion ==0 && +test_bit (NFSD4_CLIENT_STABLE , &clp->cl_flags)) +return ; + hexid =bin_to_hex_dup (clp->cl_name.data, clp->cl_name.len);if (!hexid) {dprintk ("%s: can't allocate memory for upcall!\n" , __func__);
Signature · 1 line
1.9.3