Basic NFSv4 migration support
Subsystems (3)
cover Basic NFSv4 migration support
Here are the remaining patches required to support basic forms of NFSv4 migration. These apply on and have been tested with 3.12-rc5.
NFSv4.1 migration support remains provisional.
There are still some nits to work out about what happens to an nfs_client once all its nfs_server's are migrated away.
---
Chuck Lever (20):
SUNRPC: Modify synopsis of rpc_client_register()
NFS: Add nfs4_update_server
NFS: Add functions to swap transports during migration recovery
NFS: Introduce a vector of migration recovery ops
NFS: Export _nfs_display_fhandle()
NFS: Add method to retrieve fs_locations during migration recovery
NFS: Add a super_block backpointer to the nfs_server struct
NFS: Add basic migration support to state manager thread
NFS: Re-use exit code in nfs4_async_handle_error()
NFS: Rename "stateid_invalid" label
NFS: Add migration recovery callouts in nfs4proc.c
NFS: Handle NFS4ERR_MOVED during delegation recall
NFS: Add method to detect whether an FSID is still on the server
NFS: Support NFS4ERR_LEASE_MOVED recovery in state manager
NFS: Implement support for NFS4ERR_LEASE_MOVED
NFS: Migration support for RELEASE_LOCKOWNER
NFS: Handle NFS4ERR_LEASE_MOVED during async RENEW
NFS: Handle SEQ4_STATUS_LEASE_MOVED
NFS: Set EXCHGID4_FLAG_SUPP_MOVED_MIGR
NFS: Fix possible endless state recovery wait
Trond Myklebust (1):
SUNRPC: Add a helper to switch the transport of an rpc_clnt
fs/nfs/Kconfig | 11 + fs/nfs/client.c | 3 fs/nfs/inode.c | 2 fs/nfs/internal.h | 3 fs/nfs/nfs4_fs.h | 16 ++ fs/nfs/nfs4client.c | 111 +++++++++++++ fs/nfs/nfs4namespace.c | 101 ++++++++++++ fs/nfs/nfs4proc.c | 372 +++++++++++++++++++++++++++++++++++++++++-- fs/nfs/nfs4state.c | 249 ++++++++++++++++++++++++++++- fs/nfs/nfs4xdr.c | 113 ++++++++++++- fs/nfs/super.c | 1 include/linux/nfs4.h | 1 include/linux/nfs_fs_sb.h | 8 + include/linux/nfs_xdr.h | 17 ++ include/linux/sunrpc/clnt.h | 4 net/sunrpc/clnt.c | 120 ++++++++++++-- 16 files changed, 1081 insertions(+), 51 deletions(-)
Signature · 1 line
Chuck Lever
PATCH 1/21 SUNRPC: Modify synopsis of rpc_client_register()
The rpc_client_register() helper was added in commit e73f4cc0, "SUNRPC: split client creation routine into setup and registration," Mon Jun 24 11:52:52 2013. In a subsequent patch, I'd like to invoke rpc_client_register() from a context where a struct rpc_create_args is not available.
--- net/sunrpc/clnt.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
net/sunrpc/clnt.c +7 -6
@@ -272,12 +272,13 @@ static void rpc_clnt_set_nodename(struct rpc_clnt *clnt, const char *nodename)memcpy (clnt->cl_nodename, nodename, clnt->cl_nodelen); } -static int rpc_client_register (const struct rpc_create_args *args, -struct rpc_clnt *clnt) +static int rpc_client_register (struct rpc_clnt *clnt, +rpc_authflavor_t pseudoflavor, +const char *client_name) {struct rpc_auth_create_args auth_args = { - .pseudoflavor = args->authflavor, - .target_name = args->client_name, + .pseudoflavor = pseudoflavor, + .target_name = client_name, };struct rpc_auth *auth;struct net *net =rpc_net_ns (clnt);
@@ -298,7 +299,7 @@ static int rpc_client_register(const struct rpc_create_args *args, auth =rpcauth_create (&auth_args, clnt);if (IS_ERR (auth)) {dprintk ("RPC: Couldn't create auth handle (flavor %u)\n" , - args->authflavor); + pseudoflavor); err =PTR_ERR (auth); goto err_auth; }
@@ -398,7 +399,7 @@ static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args,/* save the nodename */ rpc_clnt_set_nodename (clnt,utsname ()->nodename); - err =rpc_client_register (args, clnt); + err =rpc_client_register (clnt, args->authflavor, args->client_name);if (err) goto out_no_path;if (parent)
PATCH 2/21 SUNRPC: Add a helper to switch the transport of an rpc_clnt
In reply to Chuck LeverFrom: Trond Myklebust <[email protected]>
Add an RPC client API to redirect an rpc_clnt's transport from a source server to a destination server during a migration event.
--- include/linux/sunrpc/clnt.h | 4 ++ net/sunrpc/clnt.c | 107 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 104 insertions(+), 7 deletions(-)
include/linux/sunrpc/clnt.h +4 -0
@@ -134,6 +134,10 @@ void rpc_task_reset_client(struct rpc_task *task, struct rpc_clnt *clnt);struct rpc_clnt *rpc_clone_client (struct rpc_clnt *);struct rpc_clnt *rpc_clone_client_set_auth (struct rpc_clnt *,rpc_authflavor_t ); +int rpc_switch_client_transport (struct rpc_clnt *, +struct xprt_create *, +const struct rpc_timeout *); +void rpc_shutdown_client (struct rpc_clnt *);void rpc_release_client (struct rpc_clnt *);void rpc_task_release_client (struct rpc_task *);
net/sunrpc/clnt.c +100 -7
148 lines not on this page · raw message
PATCH 3/21 NFS: Add nfs4_update_server
In reply to Chuck LeverNew function nfs4_update_server() moves an nfs_server to a different nfs_client. This is done as part of migration recovery.
Though it may be appealing to think of them as the same thing, migration recovery is not the same as following a referral.
For a referral, the client has not descended into the file system
yet: it has no nfs_server, no super block, no inodes or open state. It is enough to simply instantiate the nfs_server and super block, and perform a referral mount.
For a migration, however, we have all of those things already, and they have to be moved to a different nfs_client. No local namespace changes are needed here.
--- fs/nfs/client.c | 3 + fs/nfs/internal.h | 3 + fs/nfs/nfs4client.c | 108 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 1 deletion(-)
fs/nfs/client.c +2 -1
@@ -943,7 +943,7 @@ void nfs_server_insert_lists(struct nfs_server *server) }EXPORT_SYMBOL_GPL (nfs_server_insert_lists); -static void nfs_server_remove_lists (struct nfs_server *server) +void nfs_server_remove_lists (struct nfs_server *server) {struct nfs_client *clp = server->nfs_client;struct nfs_net *nn;
@@ -960,6 +960,7 @@ static void nfs_server_remove_lists(struct nfs_server *server)synchronize_rcu (); } +EXPORT_SYMBOL_GPL (nfs_server_remove_lists);/* * Allocate and initialise a server record
fs/nfs/internal.h +3 -0
@@ -154,6 +154,7 @@ struct nfs_client *nfs_get_client(const struct nfs_client_initdata *, rpc_authflavor_t);int nfs_probe_fsinfo (struct nfs_server *server,struct nfs_fh *,struct nfs_fattr *);void nfs_server_insert_lists (struct nfs_server *); +void nfs_server_remove_lists (struct nfs_server *);void nfs_init_timeout_values (struct rpc_timeout *,int ,unsigned int ,unsigned int );int nfs_init_server_rpcclient (struct nfs_server *,const struct rpc_timeout *t,rpc_authflavor_t );
@@ -174,6 +175,8 @@ extern struct nfs_server *nfs4_create_server( struct nfs_subversion *);extern struct nfs_server *nfs4_create_referral_server (struct nfs_clone_mount *,struct nfs_fh *); +extern int nfs4_update_server (struct nfs_server *server,const char *hostname, +struct sockaddr *sap,size_t salen);extern void nfs_free_server (struct nfs_server *server); extern struct nfs_server *nfs_clone_server(struct nfs_server *, struct nfs_fh *,
fs/nfs/nfs4client.c +108 -0
111 lines not on this page · raw message
PATCH 4/21 NFS: Add functions to swap transports during migration recovery
In reply to Chuck LeverIntroduce functions that can walk through an array of returned fs_locations information and connect a transport to one of the destination servers listed therein.
Note that NFS minor version 1 introduces "fs_locations_info" which extends the locations array sorting criteria available to clients. This is not supported yet.
--- fs/nfs/nfs4_fs.h | 2 + fs/nfs/nfs4namespace.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+)
fs/nfs/nfs4_fs.h +2 -0
@@ -217,6 +217,8 @@ rpc_authflavor_t nfs_find_best_sec(struct nfs4_secinfo_flavors *);struct rpc_clnt *nfs4_create_sec_client (struct rpc_clnt *,struct inode *,struct qstr *);struct vfsmount *nfs4_submount (struct nfs_server *,struct dentry *,struct nfs_fh *,struct nfs_fattr *); +int nfs4_replace_transport (struct nfs_server *server, +const struct nfs4_fs_locations *locations);/* nfs4proc.c */ extern int nfs4_proc_setclientid (struct nfs_client *,u32 ,unsigned short ,struct rpc_cred *,struct nfs4_setclientid_res *);
fs/nfs/nfs4namespace.c +101 -0
104 lines not on this page · raw message
PATCH 5/21 NFS: Introduce a vector of migration recovery ops
In reply to Chuck LeverThe differences between minor version 0 and minor version 1 migration will be abstracted by the addition of a set of migration recovery ops.
--- fs/nfs/nfs4_fs.h | 4 ++++ fs/nfs/nfs4proc.c | 10 ++++++++++ 2 files changed, 14 insertions(+)
fs/nfs/nfs4_fs.h +4 -0
@@ -50,6 +50,7 @@ struct nfs4_minor_version_ops {const struct nfs4_state_recovery_ops *reboot_recovery_ops;const struct nfs4_state_recovery_ops *nograce_recovery_ops;const struct nfs4_state_maintenance_ops *state_renewal_ops; +const struct nfs4_mig_recovery_ops *mig_recovery_ops; };#define NFS_SEQID_CONFIRMED 1
@@ -203,6 +204,9 @@ struct nfs4_state_maintenance_ops {int (*renew_lease)(struct nfs_client *,struct rpc_cred *); }; +struct nfs4_mig_recovery_ops { +}; +extern const struct dentry_operations nfs4_dentry_operations;/* dir.c */
fs/nfs/nfs4proc.c +10 -0
@@ -7887,6 +7887,14 @@ static const struct nfs4_state_maintenance_ops nfs41_state_renewal_ops = { };#endif +static const struct nfs4_mig_recovery_ops nfs40_mig_recovery_ops = { +}; + +#if defined(CONFIG_NFS_V4_1 ) +static const struct nfs4_mig_recovery_ops nfs41_mig_recovery_ops = { +}; +#endif /* CONFIG_NFS_V4_1 */ +static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = { .minor_version =0 , .init_caps =NFS_CAP_READDIRPLUS
@@ -7902,6 +7910,7 @@ static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = { .reboot_recovery_ops = &nfs40_reboot_recovery_ops, .nograce_recovery_ops = &nfs40_nograce_recovery_ops, .state_renewal_ops = &nfs40_state_renewal_ops, + .mig_recovery_ops = &nfs40_mig_recovery_ops, };#if defined(CONFIG_NFS_V4_1 )
@@ -7922,6 +7931,7 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = { .reboot_recovery_ops = &nfs41_reboot_recovery_ops, .nograce_recovery_ops = &nfs41_nograce_recovery_ops, .state_renewal_ops = &nfs41_state_renewal_ops, + .mig_recovery_ops = &nfs41_mig_recovery_ops, };#endif
PATCH 6/21 NFS: Export _nfs_display_fhandle()
In reply to Chuck LeverAllow code in nfsv4.ko to use _nfs_display_fhandle().
--- fs/nfs/inode.c | 2 ++ fs/nfs/nfs4client.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
fs/nfs/inode.c +2 -0
@@ -1209,6 +1209,7 @@ u32 _nfs_display_fhandle_hash(const struct nfs_fh *fh)* not on the result */ return nfs_fhandle_hash (fh); } +EXPORT_SYMBOL_GPL (_nfs_display_fhandle_hash);/* * _nfs_display_fhandle - display an NFS file handle on the console
@@ -1253,6 +1254,7 @@ void _nfs_display_fhandle(const struct nfs_fh *fh, const char *caption) } } } +EXPORT_SYMBOL_GPL (_nfs_display_fhandle);#endif /**
fs/nfs/nfs4client.c +1 -1
@@ -924,7 +924,7 @@ static int nfs4_server_common_setup(struct nfs_server *server,dprintk ("Server FSID: %llx:%llx\n" , (unsigned long long ) server->fsid.major, (unsigned long long ) server->fsid.minor); -dprintk ("Mount FH: %d\n" , mntfh->size); +nfs_display_fhandle (mntfh,"Pseudo-fs root FH" );nfs4_session_set_rwsize (server);
PATCH 7/21 NFS: Add method to retrieve fs_locations during migration recovery
In reply to Chuck LeverThe nfs4_proc_fs_locations() function is invoked during referral processing to perform a GETATTR(fs_locations) on an object's parent directory in order to discover the target of the referral. It performs a LOOKUP in the compound, so the client needs to know the parent's file handle a priori.
Unfortunately this function is not adequate for handling migration recovery. We need to probe fs_locations information on an FSID, but there's no parent directory available for many operations that can return NFS4ERR_MOVED.
Another subtlety: recovering from NFS4ERR_LEASE_MOVED is a process of walking over a list of known FSIDs that reside on the server, and probing whether they have migrated. Once the server has detected that the client has probed all migrated file systems, it stops returning NFS4ERR_LEASE_MOVED.
A minor version zero server needs to know what client ID is requesting fs_locations information so it can clear the flag that forces it to continue returning NFS4ERR_LEASE_MOVED. This flag is set per client ID and per FSID. However, the client ID is not an argument of either the PUTFH or GETATTR operations. Later minor versions have client ID information embedded in the compound's SEQUENCE operation.
Therefore, by convention, minor version zero clients send a RENEW operation in the same compound as the GETATTR(fs_locations), since RENEW's one argument is a clientid4. This allows a minor version zero server to identify correctly the client that is probing for a migration.
--- fs/nfs/nfs4_fs.h | 4 + fs/nfs/nfs4proc.c | 153 +++++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4xdr.c | 46 +++++++++++--- include/linux/nfs_xdr.h | 4 + 4 files changed, 196 insertions(+), 11 deletions(-)
fs/nfs/nfs4_fs.h +4 -0
@@ -205,6 +205,8 @@ struct nfs4_state_maintenance_ops { };struct nfs4_mig_recovery_ops { +int (*get_locations)(struct inode *,struct nfs4_fs_locations *, +struct page *,struct rpc_cred *); };extern const struct dentry_operations nfs4_dentry_operations;
@@ -237,6 +239,8 @@ extern int nfs4_do_close(struct nfs4_state *state, gfp_t gfp_mask, int wait);extern int nfs4_server_capabilities (struct nfs_server *server,struct nfs_fh *fhandle);extern int nfs4_proc_fs_locations (struct rpc_clnt *,struct inode *,const struct qstr *,struct nfs4_fs_locations *,struct page *); +extern int nfs4_proc_get_locations (struct inode *,struct nfs4_fs_locations *, +struct page *page,struct rpc_cred *);extern struct rpc_clnt *nfs4_proc_lookup_mountpoint (struct inode *,struct qstr *,struct nfs_fh *,struct nfs_fattr *);extern int nfs4_proc_secinfo (struct inode *,const struct qstr *,struct nfs4_secinfo_flavors *);
fs/nfs/nfs4proc.c +153 -0
169 lines not on this page · raw message
fs/nfs/nfs4xdr.c +35 -11
72 lines not on this page · raw message
include/linux/nfs_xdr.h +4 -0
@@ -1053,14 +1053,18 @@ struct nfs4_fs_locations {struct nfs4_fs_locations_arg {struct nfs4_sequence_args seq_args;const struct nfs_fh *dir_fh; +const struct nfs_fh *fh;const struct qstr *name;struct page *page;const u32 *bitmask; +clientid4 clientid; +unsigned char migration:1 , renew:1 ; };struct nfs4_fs_locations_res {struct nfs4_sequence_res seq_res;struct nfs4_fs_locations *fs_locations; +unsigned char migration:1 , renew:1 ; };struct nfs4_secinfo4 {
PATCH 8/21 NFS: Add a super_block backpointer to the nfs_server struct
In reply to Chuck LeverNFS_SB() returns the pointer to an nfs_server struct, given a pointer to a super_block. But we have no way to go back the other way.
Add a super_block backpointer field so that, given an nfs_server struct, it is easy to get to the filesystem's root dentry.
--- fs/nfs/super.c | 1 + include/linux/nfs_fs_sb.h | 1 + 2 files changed, 2 insertions(+)
fs/nfs/super.c +1 -0
@@ -2530,6 +2530,7 @@ struct dentry *nfs_fs_mount_common(struct nfs_server *server, mntroot =ERR_PTR (error); goto error_splat_bdi; } + server->super = s; }if (!s->s_root) {
include/linux/nfs_fs_sb.h +1 -0
@@ -147,6 +147,7 @@ struct nfs_server {__u64 maxfilesize;/* maximum file size */ struct timespec time_delta;/* smallest time granularity */ unsigned long mount_time;/* when this fs was mounted */ +struct super_block *super;/* VFS super block */ dev_t s_dev;/* superblock dev numbers */ #ifdef CONFIG_NFS_FSCACHE
PATCH 9/21 NFS: Add basic migration support to state manager thread
In reply to Chuck LeverMigration recovery and state recovery must be serialized, so handle both in the state manager thread.
--- fs/nfs/nfs4_fs.h | 2 + fs/nfs/nfs4client.c | 1 fs/nfs/nfs4state.c | 161 ++++++++++++++++++++++++++++++++++++++++++++- include/linux/nfs_fs_sb.h | 7 ++ 4 files changed, 168 insertions(+), 3 deletions(-)
fs/nfs/nfs4_fs.h +2 -0
@@ -29,6 +29,7 @@ enum nfs4_client_state {NFS4CLNT_SERVER_SCOPE_MISMATCH ,NFS4CLNT_PURGE_STATE ,NFS4CLNT_BIND_CONN_TO_SESSION , + NFS4CLNT_MOVED, };#define NFS4_RENEW_TIMEOUT 0x01
@@ -421,6 +422,7 @@ extern int nfs4_client_recover_expired_lease(struct nfs_client *clp);extern void nfs4_schedule_state_manager (struct nfs_client *);extern void nfs4_schedule_path_down_recovery (struct nfs_client *clp);extern int nfs4_schedule_stateid_recovery (const struct nfs_server *,struct nfs4_state *); +extern int nfs4_schedule_migration_recovery (const struct nfs_server *);extern void nfs41_handle_sequence_flag_errors (struct nfs_client *clp,u32 flags);extern void nfs41_handle_server_scope (struct nfs_client *,struct nfs41_server_scope **);
fs/nfs/nfs4client.c +1 -0
@@ -197,6 +197,7 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init) clp->cl_state =1 <<NFS4CLNT_LEASE_EXPIRED ; clp->cl_minorversion = cl_init->minorversion; clp->cl_mvops = nfs_v4_minor_ops[cl_init->minorversion]; + clp->cl_mig_gen =1 ;return clp; error:
fs/nfs/nfs4state.c +158 -3
191 lines not on this page · raw message
include/linux/nfs_fs_sb.h +7 -0
@@ -78,6 +78,7 @@ struct nfs_client {char cl_ipaddr[48 ];u32 cl_cb_ident;/* v4.0 callback identifier */ const struct nfs4_minor_version_ops *cl_mvops; +unsigned long cl_mig_gen;/* NFSv4.0 transport blocking */ struct nfs4_slot_table *cl_slot_tbl;
@@ -188,6 +189,12 @@ struct nfs_server {struct list_head state_owners_lru;struct list_head layouts;struct list_head delegations; + +unsigned long mig_gen; +unsigned long mig_status; +#define NFS_MIG_IN_TRANSITION (1) +#define NFS_MIG_FAILED (2) +void (*destroy)(struct nfs_server *);atomic_t active;/* Keep trace of any activity to this server */
PATCH 10/21 NFS: Re-use exit code in nfs4_async_handle_error()
In reply to Chuck LeverClean up.
--- fs/nfs/nfs4proc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
fs/nfs/nfs4proc.c +3 -6
@@ -4757,19 +4757,15 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,dprintk ("%s ERROR %d, Reset session\n" , __func__, task->tk_status);nfs4_schedule_session_recovery (clp->cl_session, task->tk_status); - task->tk_status =0 ; -return -EAGAIN ; + goto restart_call;#endif /* CONFIG_NFS_V4_1 */ case -NFS4ERR_DELAY :nfs_inc_server_stats (server,NFSIOS_DELAY );case -NFS4ERR_GRACE :rpc_delay (task,NFS4_POLL_RETRY_MAX ); - task->tk_status =0 ; -return -EAGAIN ;case -NFS4ERR_RETRY_UNCACHED_REP :case -NFS4ERR_OLD_STATEID : - task->tk_status =0 ; -return -EAGAIN ; + goto restart_call; } task->tk_status =nfs4_map_errors (task->tk_status);return 0 ;
@@ -4780,6 +4776,7 @@ wait_on_recovery:rpc_sleep_on (&clp->cl_rpcwaitq, task,NULL );if (test_bit (NFS4CLNT_MANAGER_RUNNING , &clp->cl_state) ==0 )rpc_wake_up_queued_task (&clp->cl_rpcwaitq, task); +restart_call: task->tk_status =0 ;return -EAGAIN ; }
PATCH 11/21 NFS: Rename "stateid_invalid" label
In reply to Chuck LeverI'm going to use this exit label also for migration recovery failures.
--- fs/nfs/nfs4proc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/nfs/nfs4proc.c +3 -3
@@ -4735,12 +4735,12 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,if (state ==NULL )break ;if (nfs4_schedule_stateid_recovery (server, state) <0 ) - goto stateid_invalid; + goto recovery_failed; goto wait_on_recovery;case -NFS4ERR_EXPIRED :if (state !=NULL ) {if (nfs4_schedule_stateid_recovery (server, state) <0 ) - goto stateid_invalid; + goto recovery_failed; }case -NFS4ERR_STALE_STATEID :case -NFS4ERR_STALE_CLIENTID :
@@ -4769,7 +4769,7 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server, } task->tk_status =nfs4_map_errors (task->tk_status);return 0 ; -stateid_invalid: +recovery_failed: task->tk_status = -EIO ;return 0 ; wait_on_recovery:
PATCH 12/21 NFS: Add migration recovery callouts in nfs4proc.c
In reply to Chuck LeverWhen a server returns NFS4ERR_MOVED, trigger the new migration recovery logic in the state manager.
--- fs/nfs/nfs4proc.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +20 -2
@@ -384,6 +384,11 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struccase -NFS4ERR_STALE_CLIENTID :nfs4_schedule_lease_recovery (clp); goto wait_on_recovery; +case -NFS4ERR_MOVED : + ret =nfs4_schedule_migration_recovery (server); +if (ret <0 ) +break ; + goto wait_on_recovery;#if defined(CONFIG_NFS_V4_1 )case -NFS4ERR_BADSESSION :case -NFS4ERR_BADSLOT :
@@ -431,6 +436,8 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, strucreturn nfs4_map_errors (ret); wait_on_recovery: ret =nfs4_wait_clnt_recover (clp); +if (test_bit (NFS_MIG_FAILED , &server->mig_status)) +return -EIO ;if (ret ==0 ) exception->retry =1 ;return ret;
@@ -2981,11 +2988,16 @@ static int nfs4_get_referral(struct rpc_clnt *client, struct inode *dir, status =nfs4_proc_fs_locations (client, dir, name, locations, page);if (status !=0 ) goto out; -/* Make sure server returned a different fsid for the referral */ + +/* +* If the fsid didn't change, this is a migration event, not a +* referral. Cause us to drop into the exception handler, which +* will kick off migration recovery. +*/ if (nfs_fsid_equal (&NFS_SERVER (dir)->fsid, &locations->fattr.fsid)) {dprintk ("%s: server did not return a different fsid for" " a referral at %s\n" , __func__, name->name); - status = -EIO ; + status = -NFS4ERR_MOVED ; goto out; }/* Fixup attributes for the nfs_lookup() call to nfs_fhget() */
@@ -4746,6 +4758,10 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,case -NFS4ERR_STALE_CLIENTID :nfs4_schedule_lease_recovery (clp); goto wait_on_recovery; +case -NFS4ERR_MOVED : +if (nfs4_schedule_migration_recovery (server) <0 ) + goto recovery_failed; + goto wait_on_recovery;#if defined(CONFIG_NFS_V4_1 )case -NFS4ERR_BADSESSION :case -NFS4ERR_BADSLOT :
@@ -4776,6 +4792,8 @@ wait_on_recovery:rpc_sleep_on (&clp->cl_rpcwaitq, task,NULL );if (test_bit (NFS4CLNT_MANAGER_RUNNING , &clp->cl_state) ==0 )rpc_wake_up_queued_task (&clp->cl_rpcwaitq, task); +if (test_bit (NFS_MIG_FAILED , &server->mig_status)) + goto recovery_failed; restart_call: task->tk_status =0 ;return -EAGAIN ;
PATCH 13/21 NFS: Handle NFS4ERR_MOVED during delegation recall
In reply to Chuck LeverWhen a server returns NFS4ERR_MOVED during a delegation recall, trigger the new migration recovery logic in the state manager.
--- fs/nfs/nfs4proc.c | 3 +++ 1 file changed, 3 insertions(+)
fs/nfs/nfs4proc.c +3 -0
@@ -1582,6 +1582,9 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct/* Don't recall a delegation if it was lost */ nfs4_schedule_lease_recovery (server->nfs_client);return -EAGAIN ; +case -NFS4ERR_MOVED : +nfs4_schedule_migration_recovery (server); +return -EAGAIN ;case -NFS4ERR_DELEG_REVOKED :case -NFS4ERR_ADMIN_REVOKED :case -NFS4ERR_BAD_STATEID :
PATCH 14/21 NFS: Add method to detect whether an FSID is still on the server
In reply to Chuck LeverIntroduce a mechanism for probing a server to determine if an FSID is present or absent.
The on-the-wire compound is different between minor version 0 and 1. Minor version 0 appends a RENEW operation to identify which client ID is probing. Minor version 1 has a SEQUENCE operation in the compound which effectively carries the same information.
--- fs/nfs/nfs4_fs.h | 2 + fs/nfs/nfs4proc.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4xdr.c | 67 ++++++++++++++++++++++++- include/linux/nfs4.h | 1 include/linux/nfs_xdr.h | 13 +++++ 5 files changed, 209 insertions(+), 2 deletions(-)
fs/nfs/nfs4_fs.h +2 -0
@@ -208,6 +208,7 @@ struct nfs4_state_maintenance_ops {struct nfs4_mig_recovery_ops {int (*get_locations)(struct inode *,struct nfs4_fs_locations *,struct page *,struct rpc_cred *); +int (*fsid_present)(struct inode *,struct rpc_cred *); };extern const struct dentry_operations nfs4_dentry_operations;
@@ -242,6 +243,7 @@ extern int nfs4_proc_fs_locations(struct rpc_clnt *, struct inode *, const struc struct nfs4_fs_locations *, struct page *);extern int nfs4_proc_get_locations (struct inode *,struct nfs4_fs_locations *,struct page *page,struct rpc_cred *); +extern int nfs4_proc_fsid_present (struct inode *,struct rpc_cred *);extern struct rpc_clnt *nfs4_proc_lookup_mountpoint (struct inode *,struct qstr *,struct nfs_fh *,struct nfs_fattr *);extern int nfs4_proc_secinfo (struct inode *,const struct qstr *,struct nfs4_secinfo_flavors *);
fs/nfs/nfs4proc.c +128 -0
145 lines not on this page · raw message
fs/nfs/nfs4xdr.c +65 -2
100 lines not on this page · raw message
include/linux/nfs4.h +1 -0
@@ -460,6 +460,7 @@ enum {NFSPROC4_CLNT_FS_LOCATIONS ,NFSPROC4_CLNT_RELEASE_LOCKOWNER ,NFSPROC4_CLNT_SECINFO , +NFSPROC4_CLNT_FSID_PRESENT ,/* nfs41 */ NFSPROC4_CLNT_EXCHANGE_ID,
include/linux/nfs_xdr.h +13 -0
@@ -1088,6 +1088,19 @@ struct nfs4_secinfo_res {struct nfs4_secinfo_flavors *flavors; }; +struct nfs4_fsid_present_arg { +struct nfs4_sequence_args seq_args; +const struct nfs_fh *fh; +clientid4 clientid; +unsigned char renew:1 ; +}; + +struct nfs4_fsid_present_res { +struct nfs4_sequence_res seq_res; +struct nfs_fh *fh; +unsigned char renew:1 ; +}; +#endif /* CONFIG_NFS_V4 */ struct nfstime4 {
PATCH 15/21 NFS: Support NFS4ERR_LEASE_MOVED recovery in state manager
In reply to Chuck LeverA migration on the FSID in play for the current NFS operation is reported via the error status code NFS4ERR_MOVED.
"Lease moved" means that a migration has occurred on some other FSID than the one for the current operation. It's a signal that the client should take action immediately to handle a migration that it may not have noticed otherwise. This is so that the client's lease does not expire unnoticed on the destination server.
In NFSv4.0, a moved lease is reported with the NFS4ERR_LEASE_MOVED error status code.
To recover from NFS4ERR_LEASE_MOVED, check each FSID for that server to see if it is still present. Invoke nfs4_try_migration() if the FSID is no longer present on the server.
--- fs/nfs/nfs4_fs.h | 2 + fs/nfs/nfs4state.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-)
fs/nfs/nfs4_fs.h +2 -0
@@ -30,6 +30,7 @@ enum nfs4_client_state {NFS4CLNT_PURGE_STATE ,NFS4CLNT_BIND_CONN_TO_SESSION ,NFS4CLNT_MOVED , + NFS4CLNT_LEASE_MOVED, };#define NFS4_RENEW_TIMEOUT 0x01
@@ -425,6 +426,7 @@ extern void nfs4_schedule_state_manager(struct nfs_client *);extern void nfs4_schedule_path_down_recovery (struct nfs_client *clp);extern int nfs4_schedule_stateid_recovery (const struct nfs_server *,struct nfs4_state *);extern int nfs4_schedule_migration_recovery (const struct nfs_server *); +extern void nfs4_schedule_lease_moved_recovery (struct nfs_client *);extern void nfs41_handle_sequence_flag_errors (struct nfs_client *clp,u32 flags);extern void nfs41_handle_server_scope (struct nfs_client *,struct nfs41_server_scope **);
fs/nfs/nfs4state.c +72 -1
97 lines not on this page · raw message
PATCH 16/21 NFS: Implement support for NFS4ERR_LEASE_MOVED
In reply to Chuck LeverTrigger lease-moved recovery when a request returns NFS4ERR_LEASE_MOVED.
--- fs/nfs/nfs4proc.c | 9 +++++++++ 1 file changed, 9 insertions(+)
fs/nfs/nfs4proc.c +9 -0
@@ -389,6 +389,9 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, strucif (ret <0 )break ; goto wait_on_recovery; +case -NFS4ERR_LEASE_MOVED : +nfs4_schedule_lease_moved_recovery (clp); + goto wait_on_recovery;#if defined(CONFIG_NFS_V4_1 )case -NFS4ERR_BADSESSION :case -NFS4ERR_BADSLOT :
@@ -1585,6 +1588,9 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, structcase -NFS4ERR_MOVED :nfs4_schedule_migration_recovery (server);return -EAGAIN ; +case -NFS4ERR_LEASE_MOVED : +nfs4_schedule_lease_moved_recovery (server->nfs_client); +return -EAGAIN ;case -NFS4ERR_DELEG_REVOKED :case -NFS4ERR_ADMIN_REVOKED :case -NFS4ERR_BAD_STATEID :
@@ -4765,6 +4771,9 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,if (nfs4_schedule_migration_recovery (server) <0 ) goto recovery_failed; goto wait_on_recovery; +case -NFS4ERR_LEASE_MOVED : +nfs4_schedule_lease_moved_recovery (clp); + goto wait_on_recovery;#if defined(CONFIG_NFS_V4_1 )case -NFS4ERR_BADSESSION :case -NFS4ERR_BADSLOT :
PATCH 17/21 NFS: Migration support for RELEASE_LOCKOWNER
In reply to Chuck LeverCurrently the Linux NFS client ignores the operation status code for the RELEASE_LOCKOWNER operation. Like NFSv3's UMNT operation, RELEASE_LOCKOWNER is a courtesy to help servers manage their resources, and the outcome is not consequential for the client.
During a migration, a server may report NFS4ERR_LEASE_MOVED, in which case the client really should retry, since typically LEASE_MOVED has nothing to do with the current operation, but does prevent it from going forward.
Also, it's important for a client to respond as soon as possible to a moved lease condition, since the client's lease could expire on the destination without further action by the client.
NFS4ERR_DELAY is not included in the list of valid status codes for RELEASE_LOCKOWNER in RFC 3530bis. However, rfc3530-migration-update does permit migration-capable servers to return DELAY to clients, but only in the context of an ongoing migration. In this case the server has frozen lock state in preparation for migration, and a client retry would help the destination server purge unneeded state once migration recovery is complete.
Interestly, NFS4ERR_MOVED is not valid for RELEASE_LOCKOWNER, even though lock owners can be migrated with Transparent State Migration.
Note that RFC 3530bis section 9.5 includes RELEASE_LOCKOWNER in the list of operations that renew a client's lease on the server if they succeed. Now that our client pays attention to the operation's status code, we can note that renewal appropriately.
--- fs/nfs/nfs4proc.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
fs/nfs/nfs4proc.c +16 -0
@@ -5806,6 +5806,7 @@ struct nfs_release_lockowner_data {struct nfs_release_lockowner_args args;struct nfs4_sequence_args seq_args;struct nfs4_sequence_res seq_res; +unsigned long timestamp; };static void nfs4_release_lockowner_prepare (struct rpc_task *task,void *calldata)
@@ -5813,12 +5814,27 @@ static void nfs4_release_lockowner_prepare(struct rpc_task *task, void *calldatastruct nfs_release_lockowner_data *data = calldata;nfs40_setup_sequence (data->server, &data->seq_args, &data->seq_res, task); + data->timestamp = jiffies; }static void nfs4_release_lockowner_done (struct rpc_task *task,void *calldata) {struct nfs_release_lockowner_data *data = calldata; +struct nfs_server *server = data->server; +nfs40_sequence_done (task, &data->seq_res); + +switch (task->tk_status) { +case 0 : +renew_lease (server, data->timestamp); +break ; +case -NFS4ERR_STALE_CLIENTID : +case -NFS4ERR_EXPIRED : +case -NFS4ERR_LEASE_MOVED : +case -NFS4ERR_DELAY : +if (nfs4_async_handle_error (task, server,NULL ) == -EAGAIN ) +rpc_restart_call_prepare (task); + } }static void nfs4_release_lockowner_release (void *calldata)
PATCH 18/21 NFS: Handle NFS4ERR_LEASE_MOVED during async RENEW
In reply to Chuck LeverWith NFSv4 minor version 0, the asynchronous lease RENEW heartbeat can return NFS4ERR_LEASE_MOVED. Error recovery logic for async RENEW is a separate code path from the generic NFS proc paths, so it must be updated to handle NFS4ERR_LEASE_MOVED as well.
--- fs/nfs/nfs4proc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/nfs/nfs4proc.c +7 -1
@@ -4242,7 +4242,13 @@ static void nfs4_renew_done(struct rpc_task *task, void *calldata)unsigned long timestamp = data->timestamp;trace_nfs4_renew_async (clp, task->tk_status); -if (task->tk_status <0 ) { +switch (task->tk_status) { +case 0 : +break ; +case -NFS4ERR_LEASE_MOVED : +nfs4_schedule_lease_moved_recovery (clp); +break ; +default :/* Unless we're shutting down, schedule state recovery! */ if (test_bit (NFS_CS_RENEWD , &clp->cl_res_state) ==0 )return ;
PATCH 19/21 NFS: Handle SEQ4_STATUS_LEASE_MOVED
In reply to Chuck LeverWith the advent of NFSv4 sessions in NFSv4.1 and following, a "lease moved" condition is reported differently than it is in NFSv4.0.
NFSv4 minor version 0 servers return an error status code, NFS4ERR_LEASE_MOVED, to signal that a lease has moved. This error causes the whole compound operation to fail. Normal compounds against this server continue to fail until the client performs migration recovery on the migrated share.
Minor version 1 and later servers assert a bit flag in the reply to a compound's SEQUENCE operation to signal LEASE_MOVED. This is not a fatal condition: operations against this server continue normally. The server asserts this flag until the client performs migration recovery on the migrated share.
Note that servers MUST NOT return NFS4ERR_LEASE_MOVED to NFSv4 clients not using NFSv4.0.
After the server asserts any of the sr_status_flags in the SEQUENCE operation in a typical compound, our client initiates standard lease recovery. For NFSv4.1+, a stand-alone SEQUENCE operation is performed to discover what recovery is needed.
If SEQ4_STATUS_LEASE_MOVED is asserted in this stand-alone SEQUENCE operation, our client attempts to discover which FSIDs have been migrated, and then performs migration recovery on each.
--- fs/nfs/nfs4state.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/nfs/nfs4state.c +3 -2
@@ -2230,9 +2230,10 @@ void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags)nfs41_handle_server_reboot (clp);if (flags & (SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED |SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED | -SEQ4_STATUS_ADMIN_STATE_REVOKED | -SEQ4_STATUS_LEASE_MOVED )) +SEQ4_STATUS_ADMIN_STATE_REVOKED ))nfs41_handle_state_revoked (clp); +if (flags &SEQ4_STATUS_LEASE_MOVED ) +nfs4_schedule_lease_moved_recovery (clp);if (flags &SEQ4_STATUS_RECALLABLE_STATE_REVOKED )nfs41_handle_recallable_state_revoked (clp);if (flags &SEQ4_STATUS_BACKCHANNEL_FAULT )
PATCH 20/21 NFS: Set EXCHGID4_FLAG_SUPP_MOVED_MIGR
In reply to Chuck LeverBroadly speaking, v4.1 migration is untested. There are no servers in the wild that support NFSv4.1 migration. However, as server implementations become available, we do want to enable testing by developers, while leaving it disabled for environments for which broken migration support would be an unpleasant surprise.
--- fs/nfs/Kconfig | 11 +++++++++++ fs/nfs/nfs4proc.c | 8 +++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
fs/nfs/Kconfig +11 -0
@@ -140,6 +140,17 @@ config NFS_V4_1_IMPLEMENTATION_ID_DOMAIN If the NFS client is unchanged from the upstream kernel, this option should be set to the default "kernel.org". +config NFS_V4_1_MIGRATION + bool "NFSv4.1 client support for migration" + depends on NFS_V4_1 + default n + help + This option makes the NFS client advertise to NFSv4.1 servers that + it can support NFSv4 migration. + + The NFSv4.1 pieces of the Linux NFSv4 migration implementation are + still experimental. If you are not an NFSv4 developer, say N here. + config NFS_V4_SECURITY_LABEL bool depends on NFS_V4_2 && SECURITY
fs/nfs/nfs4proc.c +7 -1
@@ -6602,8 +6602,14 @@ static int _nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred,struct nfs41_exchange_id_args args = { .verifier = &verifier, .client = clp, +#ifdef CONFIG_NFS_V4_1_MIGRATION .flags =EXCHGID4_FLAG_SUPP_MOVED_REFER | -EXCHGID4_FLAG_BIND_PRINC_STATEID , +EXCHGID4_FLAG_BIND_PRINC_STATEID | +EXCHGID4_FLAG_SUPP_MOVED_MIGR , +#else + .flags =EXCHGID4_FLAG_SUPP_MOVED_REFER | +EXCHGID4_FLAG_BIND_PRINC_STATEID , +#endif };struct nfs41_exchange_id_res res = {0
PATCH 21/21 NFS: Fix possible endless state recovery wait
In reply to Chuck LeverIn nfs4_wait_clnt_recover(), hold a reference to the clp being waited on. The state manager can reduce clp->cl_count to 1, in which case the nfs_put_client() in nfs4_run_state_manager() can free *clp before wait_on_bit() returns and allows nfs4_wait_clnt_recover() to run again.
The behavior at that point is non-deterministic. If the waited-on bit still happens to be zero, wait_on_bit() will wake the waiter as expected. If the bit is set again (say, if the memory was poisoned when freed) wait_on_bit() can leave the waiter asleep.
This is a narrow fix which ensures the safety of accessing *clp in nfs4_wait_clnt_recover(), but does not address the continued use of a possibly freed *clp after nfs4_wait_clnt_recover() returns (see nfs_end_delegation_return(), for example).
--- fs/nfs/nfs4state.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
fs/nfs/nfs4state.c +6 -4
@@ -1255,14 +1255,16 @@ int nfs4_wait_clnt_recover(struct nfs_client *clp)might_sleep (); +atomic_inc (&clp->cl_count); res =wait_on_bit (&clp->cl_state,NFS4CLNT_MANAGER_RUNNING , nfs_wait_bit_killable,TASK_KILLABLE );if (res) -return res; - + goto out;if (clp->cl_cons_state <0 ) -return clp->cl_cons_state; -return 0 ; + res = clp->cl_cons_state; +out: +nfs_put_client (clp); +return res; }int nfs4_client_recover_expired_lease (struct nfs_client *clp)