a simple and scalable pNFS block layout server V2
Subsystems (8)
Christoph Hellwig
reviewerThis series adds support for the pNFS operations in NFS v4.1, as well as a block layout driver that can export block based filesystems that implement a few additional export operations. Support for XFS is provided in this series, but other filesystems could be added easily.
The core pNFS code of course owns its heritage to the existing Linux pNFS server prototype, but except for a few bits and pieces in the XDR path nothing is left from it.
The design of this new pNFS server is fairly different from the old one - while the old one implemented very little semantics in nfsd and left almost everything to filesystems my implementation implements as much as possible in common nfsd code, then dispatches to a layout driver that still is part of nfsd and only then calls into the filesystem, thus keeping it free from intimate pNFS knowledge.
This version of the code has been rebased to the locks-for-3.20 tree which adds a new lock context structure to the inode. For now we still (ab)use the lease list like in the last version. Adding a pNFS-specific list would duplicate a lot of code without much benefit. But during this research I came up with way to associate a nfs4_file with a struct file at open time which should allow to greatly simplify the pNFS and delegation code. So stay tuned for some patches in this area!
For now this also doesn't take errata 3901 for rfc 5661 into account yet and sticks to the verified errata. The changes in 3901 seem useful in the longer run one verified and will be implemented eventually. Note that the only existing pNFS block client (Linux) would not benefit from the longer layout lifetimes anyway.
More details are document in the individual patch descriptions and code comments.
This code is also available from:
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-2
Changes since V1:
- rebased to the locks-for-3.20 tree
- fixed a memory leak in the nfsd4_encode_getdeviceinfo error path
- removed the always one lg_roc field
- added a nopnfs export option
- use a synchronous transaction in layoutget to simplify recovery
- various XFS fixes pointed out by Dave
- various documentation typo fixes
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-1-git-send-email-hch-jcswGhMUV9g@public.gmane.org
PATCH 1/20 nfs: add LAYOUT_TYPE_MAX enum value
This gives us a nice upper bound for later use in nfѕd.
--- include/linux/nfs4.h | 1 + 1 file changed, 1 insertion(+)
include/linux/nfs4.h +1 -0
@@ -516,6 +516,7 @@ enum pnfs_layouttype {LAYOUT_NFSV4_1_FILES =1 ,LAYOUT_OSD2_OBJECTS =2 ,LAYOUT_BLOCK_VOLUME =3 , + LAYOUT_TYPE_MAX };/* used for both layout return and recall */
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 2/20 fs: track fl_owner for leases
In reply to a message not in the archiveJust like for other lock types we should allow different owners to have a read lease on a file. Currently this can't happen, but with the addition of pNFS layout leases we'll need this feature.
--- fs/locks.c | 12 +++++++----- fs/nfsd/nfs4state.c | 2 +- 2 files changed, 8 insertions(+), 6 deletions(-)
fs/locks.c +7 -5
@@ -1661,7 +1661,8 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr*/ error = -EAGAIN ;list_for_each_entry (fl, &ctx->flc_lease, fl_list) { -if (fl->fl_file == filp) { +if (fl->fl_file == filp && + fl->fl_owner == lease->fl_owner) { my_fl = fl;continue ; }
@@ -1721,7 +1722,7 @@ out:return error; } -static int generic_delete_lease (struct file *filp) +static int generic_delete_lease (struct file *filp,void *owner) {int error = -EAGAIN ;struct file_lock *fl, *victim =NULL ;
@@ -1737,7 +1738,8 @@ static int generic_delete_lease(struct file *filp)spin_lock (&ctx->flc_lock);list_for_each_entry (fl, &ctx->flc_lease, fl_list) { -if (fl->fl_file == filp) { +if (fl->fl_file == filp && + fl->fl_owner == owner) { victim = fl;break ; }
@@ -1778,7 +1780,7 @@ int generic_setlease(struct file *filp, long arg, struct file_lock **flp,switch (arg) {case F_UNLCK : -return generic_delete_lease (filp); +return generic_delete_lease (filp, *priv);case F_RDLCK :case F_WRLCK :if (!(*flp)->fl_lmops->lm_break) {
@@ -1857,7 +1859,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)int fcntl_setlease (unsigned int fd,struct file *filp,long arg) {if (arg ==F_UNLCK ) -return vfs_setlease (filp,F_UNLCK ,NULL ,NULL ); +return vfs_setlease (filp,F_UNLCK ,NULL , (void **)&filp);return do_fcntl_add_lease (fd, filp, arg); }
fs/nfsd/nfs4state.c +1 -1
@@ -693,7 +693,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp)spin_unlock (&fp->fi_lock);if (filp) { -vfs_setlease (filp,F_UNLCK ,NULL ,NULL ); +vfs_setlease (filp,F_UNLCK ,NULL , (void **)&fp);fput (filp); } }
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/20 fs: add FL_LAYOUT lease type
In reply to a message not in the archiveThis (ab-)uses the file locking code to allow filesystems to recall outstanding pNFS layouts on a file. This new lease type is similar but not quite the same as FL_DELEG. A FL_LAYOUT lease can always be granted, an a per-filesystem lock (XFS iolock for the initial implementation) ensures not FL_LAYOUT leases granted when we would need to recall them.
Also included are changes that allow multiple outstanding read leases of different types on the same file as long as they have a differnt owner. This wasn't a problem until now as nfsd never set FL_LEASE leases, and no one else used FL_DELEG leases, but given that nfsd will also issues FL_LAYOUT leases we will have to handle it now.
--- fs/locks.c | 14 ++++++++++---- include/linux/fs.h | 16 ++++++++++++++++ 2 files changed, 26 insertions(+), 4 deletions(-)
fs/locks.c +10 -4
@@ -137,7 +137,7 @@#define IS_POSIX (fl) (fl->fl_flags & FL_POSIX)#define IS_FLOCK (fl) (fl->fl_flags & FL_FLOCK) -#define IS_LEASE (fl) (fl->fl_flags & (FL_LEASE|FL_DELEG)) +#define IS_LEASE (fl) (fl->fl_flags & (FL_LEASE|FL_DELEG|FL_LAYOUT))#define IS_OFDLCK (fl) (fl->fl_flags & FL_OFDLCK)static bool lease_breaking (struct file_lock *fl)
@@ -1371,6 +1371,8 @@ static void time_out_leases(struct inode *inode, struct list_head *dispose)static bool leases_conflict (struct file_lock *lease,struct file_lock *breaker) { +if ((breaker->fl_flags &FL_LAYOUT ) != (lease->fl_flags &FL_LAYOUT )) +return false;if ((breaker->fl_flags &FL_DELEG ) && (lease->fl_flags &FL_LEASE ))return false;return locks_conflict (breaker, lease);
@@ -1594,11 +1596,14 @@ int fcntl_getlease(struct file *filp)* conflict with the lease we're trying to set. */ static int -check_conflicting_open (const struct dentry *dentry,const long arg) +check_conflicting_open (const struct dentry *dentry,const long arg,int flags) {int ret =0 ;struct inode *inode = dentry->d_inode; +if (flags &FL_LAYOUT ) +return 0 ; +if ((arg ==F_RDLCK ) && (atomic_read (&inode->i_writecount) >0 ))return -EAGAIN ;
@@ -1647,7 +1652,7 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **prspin_lock (&ctx->flc_lock);time_out_leases (inode, &dispose); - error =check_conflicting_open (dentry, arg); + error =check_conflicting_open (dentry, arg, lease->fl_flags);if (error) goto out;
@@ -1703,7 +1708,7 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr* precedes these checks. */ smp_mb (); - error =check_conflicting_open (dentry, arg); + error =check_conflicting_open (dentry, arg, lease->fl_flags);if (error) {locks_unlink_lock_ctx (lease, &ctx->flc_lease_cnt); goto out;
@@ -1787,6 +1792,7 @@ int generic_setlease(struct file *filp, long arg, struct file_lock **flp,WARN_ON_ONCE (1 );return -ENOLCK ; } +return generic_add_lease (filp, arg, flp, priv);default :return -EINVAL ;
include/linux/fs.h +16 -0
@@ -875,6 +875,7 @@ static inline struct file *get_file(struct file *f)#define FL_DOWNGRADE_PENDING 256/* Lease is being downgraded */ #define FL_UNLOCK_PENDING 512/* Lease is being broken */ #define FL_OFDLCK 1024/* lock is "owned" by struct file */ +#define FL_LAYOUT 2048/* outstanding pNFS layout */ /* * Special return value from posix_lock_file() and vfs_lock_file() for
@@ -2036,6 +2037,16 @@ static inline int break_deleg_wait(struct inode **delegated_inode) return ret; } +static inline int break_layout (struct inode *inode,bool wait) +{ +smp_mb (); +if (inode->i_flctx && !list_empty_careful (&inode->i_flctx->flc_lease)) +return __break_lease (inode, + wait ?O_WRONLY :O_WRONLY |O_NONBLOCK , +FL_LAYOUT ); +return 0 ; +} +#else /* !CONFIG_FILE_LOCKING */ static inline int locks_mandatory_locked (struct file *file) {
@@ -2091,6 +2102,11 @@ static inline int break_deleg_wait(struct inode **delegated_inode) return 0; } +static inline int break_layout (struct inode *inode,bool wait) +{ +return 0 ; +} +#endif /* CONFIG_FILE_LOCKING */ /* fs/open.c */
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 03/20] fs: add FL_LAYOUT lease type
Quoted 12 lines · On Thu, 22 Jan 2015 12:09:49 +0100 Christoph Hellwig <[email protected]> wrote:
This (ab-)uses the file locking code to allow filesystems to recall outstanding pNFS layouts on a file. This new lease type is similar but not quite the same as
FL_DELEG. AFL_LAYOUTlease can always be granted, an a per-filesystem lock (XFS iolock for the initial implementation) ensures notFL_LAYOUTleases granted when we would need to recall them.Also included are changes that allow multiple outstanding read leases of different types on the same file as long as they have a differnt owner. This wasn't a problem until now as nfsd never set
FL_LEASEleases, and no one else usedFL_DELEGleases, but given that nfsd will also issuesFL_LAYOUTleases we will have to handle it now.
Quoted 6 lines
Signed-off-by Christoph Hellwig <[email protected]>--- fs/locks.c | 14 ++++++++++---- include/linux/fs.h | 16 ++++++++++++++++ 2 files changed, 26 insertions(+), 4 deletions(-)
Abuse indeed. :)
I'd probably prefer to move this to a separate list within the i_flctx instead of overloading the lease stuff, but I'm not religious about it.
I can live with this for now, and if and when we get around to representing different types of file locks with different types of structures, we can do that conversion then.
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-4-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 03/20] fs: add FL_LAYOUT lease type
Quoted 5 lines · Christoph Hellwig
This (ab-)uses the file locking code to allow filesystems to recall outstanding pNFS layouts on a file. This new lease type is similar but not quite the same as
FL_DELEG. AFL_LAYOUTlease can always be granted, an a per-filesystem lock (XFS iolock for the initial implementation) ensures notFL_LAYOUTleases granted when we would need to recall them.
So when there's a conflicting operation it's xfs's responsibility to call break_layout and wait for the recall?
(And what roughly is the set of conflicting operations?)
--b.
Quoted 120 lines
Also included are changes that allow multiple outstanding read leases of different types on the same file as long as they have a differnt owner. This wasn't a problem until now as nfsd never set
FL_LEASEleases, and no one else usedFL_DELEGleases, but given that nfsd will also issuesFL_LAYOUTleases we will have to handle it now.Signed-off-by Christoph Hellwig <[email protected]>--- fs/locks.c | 14 ++++++++++---- include/linux/fs.h | 16 ++++++++++++++++ 2 files changed, 26 insertions(+), 4 deletions(-)fs/locks.c +10
-4@@ -137,7 +137,7 @@#define IS_POSIX (fl) (fl->fl_flags & FL_POSIX)#define IS_FLOCK (fl) (fl->fl_flags & FL_FLOCK) -#define IS_LEASE (fl) (fl->fl_flags & (FL_LEASE|FL_DELEG)) +#define IS_LEASE (fl) (fl->fl_flags & (FL_LEASE|FL_DELEG|FL_LAYOUT))#define IS_OFDLCK (fl) (fl->fl_flags & FL_OFDLCK)static bool lease_breaking (struct file_lock *fl)@@ -1371,6 +1371,8 @@ static void time_out_leases(struct inode *inode, struct list_head *dispose)static bool leases_conflict (struct file_lock *lease,struct file_lock *breaker) { +if ((breaker->fl_flags &FL_LAYOUT ) != (lease->fl_flags &FL_LAYOUT )) +return false;if ((breaker->fl_flags &FL_DELEG ) && (lease->fl_flags &FL_LEASE ))return false;return locks_conflict (breaker, lease);@@ -1594,11 +1596,14 @@ int fcntl_getlease(struct file *filp)* conflict with the lease we're trying to set. */ static int -check_conflicting_open (const struct dentry *dentry,const long arg) +check_conflicting_open (const struct dentry *dentry,const long arg,int flags) {int ret =0 ;struct inode *inode = dentry->d_inode; +if (flags &FL_LAYOUT ) +return 0 ; +if ((arg ==F_RDLCK ) && (atomic_read (&inode->i_writecount) >0 ))return -EAGAIN ;@@ -1647,7 +1652,7 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **prspin_lock (&ctx->flc_lock);time_out_leases (inode, &dispose); - error =check_conflicting_open (dentry, arg); + error =check_conflicting_open (dentry, arg, lease->fl_flags);if (error) goto out;@@ -1703,7 +1708,7 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr* precedes these checks. */ smp_mb (); - error =check_conflicting_open (dentry, arg); + error =check_conflicting_open (dentry, arg, lease->fl_flags);if (error) {locks_unlink_lock_ctx (lease, &ctx->flc_lease_cnt); goto out;@@ -1787,6 +1792,7 @@ int generic_setlease(struct file *filp, long arg, struct file_lock **flp,WARN_ON_ONCE (1 );return -ENOLCK ; } +return generic_add_lease (filp, arg, flp, priv);default :return -EINVAL ;include/linux/fs.h +16
-0@@ -875,6 +875,7 @@ static inline struct file *get_file(struct file *f)#define FL_DOWNGRADE_PENDING 256/* Lease is being downgraded */ #define FL_UNLOCK_PENDING 512/* Lease is being broken */ #define FL_OFDLCK 1024/* lock is "owned" by struct file */ +#define FL_LAYOUT 2048/* outstanding pNFS layout */ /* * Special return value from posix_lock_file() and vfs_lock_file() for @@ -2036,6 +2037,16 @@ static inline int break_deleg_wait(struct inode **delegated_inode) return ret; } +static inline int break_layout (struct inode *inode,bool wait) +{ +smp_mb (); +if (inode->i_flctx && !list_empty_careful (&inode->i_flctx->flc_lease)) +return __break_lease (inode, + wait ?O_WRONLY :O_WRONLY |O_NONBLOCK , +FL_LAYOUT ); +return 0 ; +} +#else /* !CONFIG_FILE_LOCKING */ static inline int locks_mandatory_locked (struct file *file) {@@ -2091,6 +2102,11 @@ static inline int break_deleg_wait(struct inode **delegated_inode) return 0; } +static inline int break_layout (struct inode *inode,bool wait) +{ +return 0 ; +} +#endif /* CONFIG_FILE_LOCKING */ /* fs/open.c */ Signature · 1 line
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 03/20] fs: add FL_LAYOUT lease type
Quoted 11 lines · J. Bruce Fields
Quoted 5 lines · Christoph Hellwig
This (ab-)uses the file locking code to allow filesystems to recall outstanding pNFS layouts on a file. This new lease type is similar but not quite the same as
FL_DELEG. AFL_LAYOUTlease can always be granted, an a per-filesystem lock (XFS iolock for the initial implementation) ensures notFL_LAYOUTleases granted when we would need to recall them.So when there's a conflicting operation it's xfs's responsibility to call
break_layoutand wait for the recall?(And what roughly is the set of conflicting operations?)
The last patch in the series has a comment explaining this.
There's two categories:
a) operations that need to be protected to maintain filesystem integrity:
truncate, hole punch, collapse range, a well as any new operation that can deallocate blocks.
b) operations that write data locally, and we want to provide a best effort attempt at data cohrency between local use and pFNS clients. This covers writes in all variants, and should apply to shared mmap writes, but sleeping in the page fault handler is too hard to bother with for this sort of best effort coherency.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 6/20 nfsd: add fh_fsid_match helper
In reply to a message not in the archiveAdd a helper to check that the fsid parts of two file handles match.
--- fs/nfsd/nfsfh.h | 9 +++++++++ 1 file changed, 9 insertions(+)
fs/nfsd/nfsfh.h +9 -0
@@ -196,6 +196,15 @@ static inline bool fh_match(struct knfsd_fh *fh1, struct knfsd_fh *fh2)return true; } +static inline bool fh_fsid_match (struct knfsd_fh *fh1,struct knfsd_fh *fh2) +{ +if (fh1->fh_fsid_type != fh2->fh_fsid_type) +return false; +if (memcmp (fh1->fh_fsid, fh2->fh_fsid,key_len (fh1->fh_fsid_type) !=0 )) +return false; +return true; +} +#ifdef CONFIG_NFSD_V3 /* * The wcc data stored in current_fh should be cleared
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 7/20 nfsd: make lookup/alloc/unhash_stid available outside nfs4state.c
In reply to a message not in the archive--- fs/nfsd/nfs4state.c | 8 ++++---- fs/nfsd/state.h | 6 ++++++ 2 files changed, 10 insertions(+), 4 deletions(-)
fs/nfsd/nfs4state.c +4 -4
@@ -486,7 +486,7 @@ static void nfs4_file_put_access(struct nfs4_file *fp, u32 access)__nfs4_file_put_access (fp,O_RDONLY ); } -static struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl, +struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl,struct kmem_cache *slab) {struct nfs4_stid *stid;
@@ -690,7 +690,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp) } } -static void unhash_stid (struct nfs4_stid *s) +void nfs4_unhash_stid (struct nfs4_stid *s) { s->sc_type =0 ; }
@@ -998,7 +998,7 @@ static void unhash_lock_stateid(struct nfs4_ol_stateid *stp)list_del_init (&stp->st_locks);unhash_ol_stateid (stp); -unhash_stid (&stp->st_stid); +nfs4_unhash_stid (&stp->st_stid); }static void release_lock_stateid (struct nfs4_ol_stateid *stp)
@@ -4438,7 +4438,7 @@ out_unlock:return status; } -static __be32 +__be32 nfsd4_lookup_stateid (struct nfsd4_compound_state *cstate,stateid_t *stateid,unsigned char typemask,struct nfs4_stid **s,struct nfsd_net *nn)
fs/nfsd/state.h +6 -0
@@ -545,6 +545,12 @@ struct nfsd_net;extern __be32 nfs4_preprocess_stateid_op (struct net *net,struct nfsd4_compound_state *cstate,stateid_t *stateid,int flags,struct file **filp); +__be32 nfsd4_lookup_stateid (struct nfsd4_compound_state *cstate, +stateid_t *stateid,unsigned char typemask, +struct nfs4_stid **s,struct nfsd_net *nn); +struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl, +struct kmem_cache *slab); +void nfs4_unhash_stid (struct nfs4_stid *s);void nfs4_put_stid (struct nfs4_stid *s);void nfs4_remove_reclaim_record (struct nfs4_client_reclaim *,struct nfsd_net *);extern void nfs4_release_reclaim (struct nfsd_net *);
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 8/20 nfsd: make find/get/put file available outside nfs4state.c
In reply to a message not in the archive--- fs/nfsd/nfs4state.c | 10 ++-------- fs/nfsd/state.h | 7 +++++++ 2 files changed, 9 insertions(+), 8 deletions(-)
fs/nfsd/nfs4state.c +2 -8
@@ -282,7 +282,7 @@ static void nfsd4_free_file_rcu(struct rcu_head *rcu)kmem_cache_free (file_slab, fp); } -static inline void +void put_nfs4_file (struct nfs4_file *fi) {might_lock (&state_lock);
@@ -295,12 +295,6 @@ put_nfs4_file(struct nfs4_file *fi) } } -static inline void -get_nfs4_file (struct nfs4_file *fi) -{ -atomic_inc (&fi->fi_ref); -} -static struct file *__nfs4_get_fd (struct nfs4_file *f,int oflag) {
@@ -3300,7 +3294,7 @@ find_file_locked(struct knfsd_fh *fh, unsigned int hashval)return NULL ; } -static struct nfs4_file * +struct nfs4_file *find_file (struct knfsd_fh *fh) {struct nfs4_file *fp;
fs/nfsd/state.h +7 -0
@@ -573,6 +573,13 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name, struct nfsd_net *nn);extern bool nfs4_has_reclaimed_state (const char *name,struct nfsd_net *nn); +struct nfs4_file *find_file (struct knfsd_fh *fh); +void put_nfs4_file (struct nfs4_file *fi); +static inline void get_nfs4_file (struct nfs4_file *fi) +{ +atomic_inc (&fi->fi_ref); +} +/* grace period management */ void nfsd4_end_grace (struct nfsd_net *nn);
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 10/20 nfsd: implement pNFS operations
In reply to a message not in the archiveAdd support for the GETDEVICEINFO, LAYOUTGET, LAYOUTCOMMIT and LAYOUTRETURN NFSv4.1 operations, as well as backing code to manage outstanding layouts and devices.
Layout management is very straight forward, with a nfs4_layout_stateid structure that extends nfs4_stid to manage layout stateids as the top-level structure. It is linked into the nfs4_file and nfs4_client structures like the other stateids, and contains a linked list of layouts that hang of the stateid. The actual layout operations are implemented in layout drivers that are not part of this commit, but will be added later.
The worst part of this commit is the management of the pNFS device IDs, which suffers from a specification that is not sanely implementable due to the fact that the device-IDs are global and not bound to an export, and have a small enough size so that we can't store the fsid portion of a file handle, and must never be reused. As we still do need perform all export authentication and validation checks on a device ID passed to GETDEVICEINFO we are caught between a rock and a hard place. To work around this issue we add a new hash that maps from a 64-bit integer to a fsid so that we can look up the export to authenticate against it, a 32-bit integer as a generation that we can bump when changing the device, and a currently unused 32-bit integer that could be used in the future to handle more than a single device per export. Entries in this hash table are never deleted as we can't reuse the ids anyway, and would have a severe lifetime problem anyway as Linux export structures are temporary structures that can go away under load.
Parts of the XDR data, structures and marshaling/unmarshaling code, as well as many concepts are derived from the old pNFS server implementation from Andy Adamson, Benny Halevy, Dean Hildebrand, Marc Eshel, Fred Isaman, Mike Sager, Ricardo Labiaga and many others.
--- fs/nfsd/Kconfig | 10 + fs/nfsd/Makefile | 1 + fs/nfsd/export.c | 8 + fs/nfsd/export.h | 2 + fs/nfsd/nfs4layouts.c | 488 +++++++++++++++++++++++++++++++++++++++ fs/nfsd/nfs4proc.c | 266 +++++++++++++++++++++ fs/nfsd/nfs4state.c | 16 +- fs/nfsd/nfs4xdr.c | 312 +++++++++++++++++++++++++ fs/nfsd/nfsctl.c | 9 +- fs/nfsd/nfsd.h | 16 +- fs/nfsd/pnfs.h | 80 +++++++ fs/nfsd/state.h | 21 ++ fs/nfsd/xdr4.h | 59 +++++ include/linux/nfs4.h | 1 + include/uapi/linux/nfsd/debug.h | 1 + include/uapi/linux/nfsd/export.h | 4 +- 16 files changed, 1289 insertions(+), 5 deletions(-) create mode 100644 fs/nfsd/nfs4layouts.c create mode 100644 fs/nfsd/pnfs.h
fs/nfsd/Kconfig +10 -0
@@ -82,6 +82,16 @@ config NFSD_V4 If unsure, say N. +config NFSD_PNFS + bool "NFSv4.1 server support for Parallel NFS (pNFS)" + depends on NFSD_V4 + help + This option enables support for the parallel NFS features of the + minor version 1 of the NFSv4 protocol (RFC5661) in the kernel's NFS + server. + + If unsure, say N. + config NFSD_V4_SECURITY_LABEL bool "Provide Security Label support for NFSv4 server" depends on NFSD_V4 && SECURITY
fs/nfsd/Makefile +1 -0
@@ -12,3 +12,4 @@ nfsd-$(CONFIG_NFSD_V3) += nfs3proc.o nfs3xdr.o nfsd-$(CONFIG_NFSD_V3_ACL) += nfs3acl.o nfsd-$(CONFIG_NFSD_V4) += nfs4proc.o nfs4xdr.o nfs4state.o nfs4idmap.o \ nfs4acl.o nfs4callback.o nfs4recover.o +nfsd-$(CONFIG_NFSD_PNFS) += nfs4layouts.o
fs/nfsd/export.c +8 -0
@@ -20,6 +20,7 @@#include "nfsd.h" #include "nfsfh.h" #include "netns.h" +#include "pnfs.h" #define NFSDDBG_FACILITY NFSDDBG_EXPORT
@@ -545,6 +546,7 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen) exp.ex_client = dom; exp.cd = cd; + exp.ex_devid_map =NULL ;/* expiry */ err = -EINVAL ;
@@ -621,6 +623,8 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen)if (!gid_valid (exp.ex_anon_gid)) goto out4; err =0 ; + +nfsd4_setup_layout_type (&exp); } expp =svc_export_lookup (&exp);
@@ -703,6 +707,7 @@ static void svc_export_init(struct cache_head *cnew, struct cache_head *citem) new->ex_fslocs.locations =NULL ; new->ex_fslocs.locations_count =0 ; new->ex_fslocs.migrated =0 ; + new->ex_layout_type =0 ; new->ex_uuid =NULL ; new->cd = item->cd; }
@@ -717,6 +722,8 @@ static void export_update(struct cache_head *cnew, struct cache_head *citem) new->ex_anon_uid = item->ex_anon_uid; new->ex_anon_gid = item->ex_anon_gid; new->ex_fsid = item->ex_fsid; + new->ex_devid_map = item->ex_devid_map; + item->ex_devid_map =NULL ; new->ex_uuid = item->ex_uuid; item->ex_uuid =NULL ; new->ex_fslocs.locations = item->ex_fslocs.locations;
@@ -725,6 +732,7 @@ static void export_update(struct cache_head *cnew, struct cache_head *citem) item->ex_fslocs.locations_count =0 ; new->ex_fslocs.migrated = item->ex_fslocs.migrated; item->ex_fslocs.migrated =0 ; + new->ex_layout_type = item->ex_layout_type; new->ex_nflavors = item->ex_nflavors;for (i =0 ; i <MAX_SECINFO_LIST ; i++) { new->ex_flavors[i] = item->ex_flavors[i];
fs/nfsd/export.h +2 -0
@@ -56,6 +56,8 @@ struct svc_export {struct nfsd4_fs_locations ex_fslocs;uint32_t ex_nflavors;struct exp_flavor_info ex_flavors[MAX_SECINFO_LIST ]; +enum pnfs_layouttype ex_layout_type; +struct nfsd4_deviceid_map *ex_devid_map;struct cache_detail *cd; };
fs/nfsd/nfs4layouts.c new file +488 -0
488 lines not on this page · raw message
fs/nfsd/nfs4proc.c +266 -0
284 lines not on this page · raw message
fs/nfsd/nfs4state.c +15 -1
@@ -48,6 +48,7 @@#include "current_stateid.h" #include "netns.h" +#include "pnfs.h" #define NFSDDBG_FACILITY NFSDDBG_PROC
@@ -1544,6 +1545,9 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)INIT_LIST_HEAD (&clp->cl_lru);INIT_LIST_HEAD (&clp->cl_callbacks);INIT_LIST_HEAD (&clp->cl_revoked); +#ifdef CONFIG_NFSD_PNFS +INIT_LIST_HEAD (&clp->cl_lo_states); +#endif spin_lock_init (&clp->cl_lock);rpc_init_wait_queue (&clp->cl_cb_waitq,"Backchannel slot table" );return clp;
@@ -1648,6 +1652,7 @@ __destroy_client(struct nfs4_client *clp)nfs4_get_stateowner (&oo->oo_owner);release_openowner (oo); } +nfsd4_return_all_client_layouts (clp);nfsd4_shutdown_callback (clp);if (clp->cl_cb_conn.cb_xprt)svc_xprt_put (clp->cl_cb_conn.cb_xprt);
@@ -2131,8 +2136,11 @@ nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp,static void nfsd4_set_ex_flags (struct nfs4_client *new,struct nfsd4_exchange_id *clid) { -/* pNFS is not supported */ +#ifdef CONFIG_NFSD_PNFS + new->cl_exchange_flags |=EXCHGID4_FLAG_USE_PNFS_MDS ; +#else new->cl_exchange_flags |=EXCHGID4_FLAG_USE_NON_PNFS ; +#endif /* Referrals are supported, Migration is not. */ new->cl_exchange_flags |=EXCHGID4_FLAG_SUPP_MOVED_REFER ;
@@ -3060,6 +3068,9 @@ static void nfsd4_init_file(struct knfsd_fh *fh, unsigned int hashval, fp->fi_share_deny =0 ;memset (fp->fi_fds,0 ,sizeof (fp->fi_fds));memset (fp->fi_access,0 ,sizeof (fp->fi_access)); +#ifdef CONFIG_NFSD_PNFS +INIT_LIST_HEAD (&fp->fi_lo_states); +#endif hlist_add_head_rcu (&fp->fi_hash, &file_hashtbl[hashval]); }
@@ -4846,6 +4857,9 @@ nfsd4_close(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,update_stateid (&stp->st_stid.sc_stateid);memcpy (&close->cl_stateid, &stp->st_stid.sc_stateid,sizeof (stateid_t)); +nfsd4_return_all_file_layouts (stp->st_stateowner->so_client, + stp->st_stid.sc_file); +nfsd4_close_open_stateid (stp);/* put reference from nfs4_preprocess_seqid_op */
fs/nfsd/nfs4xdr.c +312 -0
358 lines not on this page · raw message
fs/nfsd/nfsctl.c +8 -1
@@ -21,6 +21,7 @@#include "cache.h" #include "state.h" #include "netns.h" +#include "pnfs.h" /* * We have a single directory with several nodes in it.
@@ -1258,9 +1259,12 @@ static int __init init_nfsd(void) retval =nfsd4_init_slabs ();if (retval) goto out_unregister_pernet; - retval =nfsd_fault_inject_init ();/* nfsd fault injection controls */ + retval =nfsd4_init_pnfs ();if (retval) goto out_free_slabs; + retval =nfsd_fault_inject_init ();/* nfsd fault injection controls */ +if (retval) + goto out_exit_pnfs;nfsd_stat_init ();/* Statistics */ retval =nfsd_reply_cache_init ();if (retval)
@@ -1282,6 +1286,8 @@ out_free_lockd: out_free_stat:nfsd_stat_shutdown ();nfsd_fault_inject_cleanup (); +out_exit_pnfs: +nfsd4_exit_pnfs (); out_free_slabs:nfsd4_free_slabs (); out_unregister_pernet:
@@ -1299,6 +1305,7 @@ static void __exit exit_nfsd(void)nfsd_stat_shutdown ();nfsd_lockd_shutdown ();nfsd4_free_slabs (); +nfsd4_exit_pnfs ();nfsd_fault_inject_cleanup ();unregister_filesystem (&nfsd_fs_type);unregister_pernet_subsys (&nfsd_net_ops);
fs/nfsd/nfsd.h +14 -2
@@ -325,15 +325,27 @@ void nfsd_lockd_shutdown(void);#define NFSD4_SUPPORTED_ATTRS_WORD2 0 +/* 4.1 */ +#ifdef CONFIG_NFSD_PNFS +#define PNFSD_SUPPORTED_ATTRS_WORD1 FATTR4_WORD1_FS_LAYOUT_TYPES +#define PNFSD_SUPPORTED_ATTRS_WORD2 \ +(FATTR4_WORD2_LAYOUT_BLKSIZE | FATTR4_WORD2_LAYOUT_TYPES) +#else +#define PNFSD_SUPPORTED_ATTRS_WORD1 0 +#define PNFSD_SUPPORTED_ATTRS_WORD2 0 +#endif /* CONFIG_NFSD_PNFS */ +#define NFSD4_1_SUPPORTED_ATTRS_WORD0 \ NFSD4_SUPPORTED_ATTRS_WORD0#define NFSD4_1_SUPPORTED_ATTRS_WORD1 \ - NFSD4_SUPPORTED_ATTRS_WORD1 + (NFSD4_SUPPORTED_ATTRS_WORD1 | PNFSD_SUPPORTED_ATTRS_WORD1)#define NFSD4_1_SUPPORTED_ATTRS_WORD2 \ - (NFSD4_SUPPORTED_ATTRS_WORD2 | FATTR4_WORD2_SUPPATTR_EXCLCREAT) + (NFSD4_SUPPORTED_ATTRS_WORD2 | PNFSD_SUPPORTED_ATTRS_WORD2 | \ + FATTR4_WORD2_SUPPATTR_EXCLCREAT) +/* 4.2 */ #ifdef CONFIG_NFSD_V4_SECURITY_LABEL #define NFSD4_2_SECURITY_ATTRS FATTR4_WORD2_SECURITY_LABEL#else
fs/nfsd/pnfs.h new file +80 -0
80 lines not on this page · raw message
fs/nfsd/state.h +21 -0
@@ -92,6 +92,7 @@ struct nfs4_stid {/* For a deleg stateid kept around only to process free_stateid's: */ #define NFS4_REVOKED_DELEG_STID 16#define NFS4_CLOSED_DELEG_STID 32 +#define NFS4_LAYOUT_STID 64unsigned char sc_type;stateid_t sc_stateid;struct nfs4_client *sc_client;
@@ -297,6 +298,9 @@ struct nfs4_client {struct list_head cl_delegations;struct list_head cl_revoked;/* unacknowledged, revoked 4.1 state */ struct list_head cl_lru;/* tail queue */ +#ifdef CONFIG_NFSD_PNFS +struct list_head cl_lo_states;/* outstanding layout states */ +#endif struct xdr_netobj cl_name;/* id generated by client */ nfs4_verifier cl_verifier;/* generated by client */ time_t cl_time;/* time of last lease renewal */
@@ -496,6 +500,9 @@ struct nfs4_file {atomic_t fi_delegees;struct knfsd_fh fi_fhandle;bool fi_had_conflict; +#ifdef CONFIG_NFSD_PNFS +struct list_head fi_lo_states; +#endif };/*
@@ -528,6 +535,20 @@ static inline struct nfs4_ol_stateid *openlockstateid(struct nfs4_stid *s) return container_of(s, struct nfs4_ol_stateid, st_stid); } +struct nfs4_layout_stateid { +struct nfs4_stid ls_stid; +struct list_head ls_perclnt; +struct list_head ls_perfile; +spinlock_t ls_lock; +struct list_head ls_layouts; +u32 ls_layout_type; +}; + +static inline struct nfs4_layout_stateid *layoutstateid (struct nfs4_stid *s) +{ + return container_of(s, struct nfs4_layout_stateid, ls_stid); +} +/* flags for preprocess_seqid_op() */ #define RD_STATE 0x00000010#define WR_STATE 0x00000020
fs/nfsd/xdr4.h +59 -0
71 lines not on this page · raw message
include/linux/nfs4.h +1 -0
@@ -411,6 +411,7 @@ enum lock_type4 {#define FATTR4_WORD1_TIME_MODIFY_SET (1UL << 22)#define FATTR4_WORD1_MOUNTED_ON_FILEID (1UL << 23)#define FATTR4_WORD1_FS_LAYOUT_TYPES (1UL << 30) +#define FATTR4_WORD2_LAYOUT_TYPES (1UL << 0)#define FATTR4_WORD2_LAYOUT_BLKSIZE (1UL << 1)#define FATTR4_WORD2_MDSTHRESHOLD (1UL << 4)#define FATTR4_WORD2_SECURITY_LABEL (1UL << 16)
include/uapi/linux/nfsd/debug.h +1 -0
@@ -32,6 +32,7 @@#define NFSDDBG_REPCACHE 0x0080#define NFSDDBG_XDR 0x0100#define NFSDDBG_LOCKD 0x0200 +#define NFSDDBG_PNFS 0x0400#define NFSDDBG_ALL 0x7FFF#define NFSDDBG_NOCHANGE 0xFFFF
include/uapi/linux/nfsd/export.h +3 -1
@@ -47,8 +47,10 @@* exported filesystem. */ #define NFSEXP_V4ROOT 0x10000 +#define NFSEXP_NOPNFS 0x20000 +/* All flags that we claim to support. (Note we don't support NOACL.) */ -#define NFSEXP_ALLFLAGS 0x1FE7F +#define NFSEXP_ALLFLAGS 0x3FE7F/* The flags that may vary depending on security flavor: */ #define NFSEXP_SECINFO_FLAGS (NFSEXP_READONLY | NFSEXP_ROOTSQUASH \
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-11-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 12 lines · Christoph Hellwig
Add support for the GETDEVICEINFO, LAYOUTGET, LAYOUTCOMMIT and LAYOUTRETURN NFSv4.1 operations, as well as backing code to manage outstanding layouts and devices.
Layout management is very straight forward, with a
nfs4_layout_stateidstructure that extendsnfs4_stidto manage layout stateids as the top-level structure. It is linked into thenfs4_fileandnfs4_clientstructures like the other stateids, and contains a linked list of layouts that hang of the stateid. The actual layout operations are implemented in layout drivers that are not part of this commit, but will be added later.
Quoted 15 lines
The worst part of this commit is the management of the pNFS device IDs, which suffers from a specification that is not sanely implementable due to the fact that the device-IDs are global and not bound to an export, and have a small enough size so that we can't store the fsid portion of a file handle, and must never be reused. As we still do need perform all export authentication and validation checks on a device ID passed to GETDEVICEINFO we are caught between a rock and a hard place. To work around this issue we add a new hash that maps from a 64-bit integer to a fsid so that we can look up the export to authenticate against it, a 32-bit integer as a generation that we can bump when changing the device, and a currently unused 32-bit integer that could be used in the future to handle more than a single device per export. Entries in this hash table are never deleted as we can't reuse the ids anyway, and would have a severe lifetime problem anyway as Linux export structures are temporary structures that can go away under load.
Looks to me like that works.
...
Quoted 6 lines
fs/nfsd/nfs4layouts.c new file +0
-0@@ -0,0 +1,488 @@
...
Quoted 27 lines
fs/nfsd/nfs4layouts.c new file +27
-0+__be32 +nfsd4_preprocess_layout_stateid (struct svc_rqst *rqstp, +struct nfsd4_compound_state *cstate,stateid_t *stateid, +bool create,u32 layout_type,struct nfs4_layout_stateid **lsp) +{ +struct nfs4_layout_stateid *ls; +struct nfs4_stid *stid; +unsigned char typemask =NFS4_LAYOUT_STID ; +__be32 status; + +if (create) + typemask |= (NFS4_OPEN_STID |NFS4_LOCK_STID |NFS4_DELEG_STID ); + + status =nfsd4_lookup_stateid (cstate, stateid, typemask, &stid, +net_generic (SVC_NET (rqstp), nfsd_net_id)); +if (status) + goto out; + +if (!fh_match (&cstate->current_fh.fh_handle, + &stid->sc_file->fi_fhandle)) { + status = nfserr_bad_stateid; + goto out_put_stid; + } + +if (stid->sc_type !=NFS4_LAYOUT_STID ) { + ls =nfsd4_alloc_layout_stateid (cstate, stid, layout_type); +nfs4_put_stid (stid);
fs/nfsd/nfs4layouts.c line 205
Quoted 10 lines
fs/nfsd/nfs4layouts.c new file +10
-0+ + status = nfserr_jukebox; +if (!ls) + goto out; + }else { + ls = container_of(stid, struct nfs4_layout_stateid, ls_stid); + + status = nfserr_bad_stateid; +if (stateid->si_generation > stid->sc_stateid.si_generation) + goto out_put_stid;
Is there no old_stateid case for layout stateids? And is there any chance of wraparound? (I was comparing to check_stateid_generation and expecting the only difference to be the handling of the generation-zero case.)
Quoted 8 lines
fs/nfsd/nfs4layouts.c new file +8
-0+if (layout_type != ls->ls_layout_type) + goto out_put_stid; + } + + *lsp = ls; +return 0 ; + +out_put_stid:
fs/nfsd/nfs4layouts.c line 223
Quoted 10 lines
fs/nfsd/nfs4layouts.c new file +10
-0+nfs4_put_stid (stid); +out: +return status; +} + +static inline u64 +layout_end (struct nfsd4_layout_seg *seg) +{ +u64 end = seg->offset + seg->length; +return end >= seg->offset ? seg->length :NFS4_MAX_UINT64 ;
Shouldn't that be
return end >= seg->offset ? end : NFS_MAX_UINT64;
?
fs/nfsd/nfs4layouts.c line 230
Quoted 7 lines
fs/nfsd/nfs4layouts.c new file +7
-0+} + +static void +layout_update_len (struct nfsd4_layout_seg *lo,u64 end) +{ +if (end ==NFS4_MAX_UINT64 ) + lo->length =NFS4_MAX_UINT64 ;
Is this case necessary?
Quoted 3 lines
fs/nfsd/nfs4layouts.c new file +3
-0+else + lo->length = end - lo->offset; +}
...
Quoted 4 lines
fs/nfsd/nfs4proc.c +0
-0
...
Quoted 12 lines
fs/nfsd/nfs4proc.c +8
-0@@ -1966,6 +2213,25 @@ static struct nfsd4_operation nfsd4_ops[] = { .op_get_currentstateid = (stateid_getter)nfsd4_get_freestateid, .op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize, }, +#ifdef CONFIG_NFSD_PNFS + [OP_GETDEVICEINFO] = { + .op_func = (nfsd4op_func)nfsd4_getdeviceinfo, + .op_flags = ALLOWED_WITHOUT_FH, + .op_name = "OP_GETDEVICEINFO", + }, + [OP_LAYOUTGET] = { + .op_func = (nfsd4op_func)nfsd4_layoutget,
fs/nfsd/nfs4proc.c static struct nfsd4_operation nfsd4_ops[] = { · line 2233
Quoted 10 lines
fs/nfsd/nfs4proc.c +10
-0+ .op_name = "OP_LAYOUTGET", + }, + [OP_LAYOUTCOMMIT] = { + .op_func = (nfsd4op_func)nfsd4_layoutcommit, + .op_name = "OP_LAYOUTCOMMIT", + }, + [OP_LAYOUTRETURN] = { + .op_func = (nfsd4op_func)nfsd4_layoutreturn, + .op_name = "OP_LAYOUTRETURN", + },
Should any of these have OP_MODIFIES_SOMETHING set? (Basically: would we be in trouble if we succesfully completed one of these operations and then weren't able to encode the result?)
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 4 lines · J. Bruce Fields
Is there no
old_stateidcase for layout stateids? And is there any chance of wraparound? (I was comparing tocheck_stateid_generationand expecting the only difference to be the handling of the generation-zero case.)
12.5.3. explicitly mentions that LAYOUTGET and LAYOUTRETURN might be outstading and processed in parallel, and sais that pNFS operations use special stateid rules. It does not explicitly say that old stateids are ok, but the model described in there very much requires the server to not reject them.
Quoted 11 lines
fs/nfsd/nfs4layouts.c line 223
Quoted 5 lines
fs/nfsd/nfs4layouts.c new file +5
-0+static inline u64 +layout_end (struct nfsd4_layout_seg *seg) +{ +u64 end = seg->offset + seg->length; +return end >= seg->offset ? seg->length :NFS4_MAX_UINT64 ;Shouldn't that be
return end >= seg->offset ? end : NFS_MAX_UINT64;?
Yes. This is an interesting one that sneaked in, and it turns out besides dislabling layout merging it didn't have adverse effects.
Quoted 10 lines
fs/nfsd/nfs4layouts.c line 230
Quoted 7 lines
fs/nfsd/nfs4layouts.c new file +7
-0+} + +static void +layout_update_len (struct nfsd4_layout_seg *lo,u64 end) +{ +if (end ==NFS4_MAX_UINT64 ) + lo->length =NFS4_MAX_UINT64 ;Is this case necessary?
Quoted 2 lines
fs/nfsd/nfs4layouts.c line 232
Quoted 2 lines
fs/nfsd/nfs4layouts.c new file +2
-0+else + lo->length = end - lo->offset;
We use NFS4_MAX_UINT64 as a magic value for layouts until the field end, as specified in the standard. But because we do all kinds of calculations using the end value we need to propagate that magic from and to it.
Quoted 3 lines
Should any of these have
OP_MODIFIES_SOMETHINGset? (Basically: would we be in trouble if we succesfully completed one of these operations and then weren't able to encode the result?)
All but GETDEVICEINFO should get it.
I've implemented all your suggested changes and will send out and update after doing a little more testing.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 46 lines · Christoph Hellwig
Quoted 4 lines · J. Bruce Fields
Is there no
old_stateidcase for layout stateids? And is there any chance of wraparound? (I was comparing tocheck_stateid_generationand expecting the only difference to be the handling of the generation-zero case.)12.5.3. explicitly mentions that LAYOUTGET and LAYOUTRETURN might be outstading and processed in parallel, and sais that pNFS operations use special stateid rules. It does not explicitly say that old stateids are ok, but the model described in there very much requires the server to not reject them.
Quoted 11 lines
fs/nfsd/nfs4layouts.c line 223
Quoted 5 lines
fs/nfsd/nfs4layouts.c new file +5
-0+static inline u64 +layout_end (struct nfsd4_layout_seg *seg) +{ +u64 end = seg->offset + seg->length; +return end >= seg->offset ? seg->length :NFS4_MAX_UINT64 ;Shouldn't that be
return end >= seg->offset ? end : NFS_MAX_UINT64;?
Yes. This is an interesting one that sneaked in, and it turns out besides dislabling layout merging it didn't have adverse effects.
Quoted 10 lines
fs/nfsd/nfs4layouts.c line 230
Quoted 7 lines
fs/nfsd/nfs4layouts.c new file +7
-0+} + +static void +layout_update_len (struct nfsd4_layout_seg *lo,u64 end) +{ +if (end ==NFS4_MAX_UINT64 ) + lo->length =NFS4_MAX_UINT64 ;Is this case necessary?
Quoted 2 lines
fs/nfsd/nfs4layouts.c line 232
Quoted 2 lines
fs/nfsd/nfs4layouts.c new file +2
-0+else + lo->length = end - lo->offset;We use
NFS4_MAX_UINT64as a magic value for layouts until the field end, as specified in the standard. But because we do all kinds of calculations using the end value we need to propagate that magic from and to it.
Quoted 8 lines
Quoted 3 lines
Should any of these have
OP_MODIFIES_SOMETHINGset? (Basically: would we be in trouble if we succesfully completed one of these operations and then weren't able to encode the result?)All but GETDEVICEINFO should get it.
I've implemented all your suggested changes and will send out and update after doing a little more testing.
Thanks!
I didn't notice anything that looked like a big problem to me, so absent any objections I'll commit the revised versions for 3.20 once we figure out how to handle the xfs stuff.
--b.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 3 lines · J. Bruce Fields
I didn't notice anything that looked like a big problem to me, so absent any objections I'll commit the revised versions for 3.20 once we figure out how to handle the xfs stuff.
Do you want the resend to be on top of Jeffs locks tree, or do you want me to be based just on the nfsd changes, which would require a fairly trivial merge once both of the trees hit mainline?
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · J. Bruce Fields
I didn't notice anything that looked like a big problem to me, so absent any objections I'll commit the revised versions for 3.20 once we figure out how to handle the xfs stuff.
Do you want the resend to be on top of Jeffs locks tree, or do you want me to be based just on the nfsd changes, which would require a fairly trivial merge once both of the trees hit mainline?
I'm planning to pull Jeff's tree and then apply these on top. (Even if the conflict's fairly trivial I'm just happier being able to test the combination exactly as they're commited.)
I'll do that now, should be pushed out in an hour or two.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 5 lines · J. Bruce Fields
I'm planning to pull Jeff's tree and then apply these on top. (Even if the conflict's fairly trivial I'm just happier being able to test the combination exactly as they're commited.)
I'll do that now, should be pushed out in an hour or two.
I've pushed out a pnfsd-for-3.20-3 branch togit://git.infradead.org/users/hch/pnfs.git
Or do you want me to resend the whole thing?
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 10/20] nfsd: implement pNFS operations
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · J. Bruce Fields
I'm planning to pull Jeff's tree and then apply these on top. (Even if the conflict's fairly trivial I'm just happier being able to test the combination exactly as they're commited.)
I'll do that now, should be pushed out in an hour or two.
I've pushed out a pnfsd-for-3.20-3 branch to
git://git.infradead.org/users/hch/pnfs.gitOr do you want me to resend the whole thing?
No, looks fine to me as is. Thanks! I'll play around with it a little more and then push it out.
GETLAYOUT is going to be annoying if we need big layouts. We'll have to figure out what to do with that reply size estimation, and clients will have to figure out how to get the maxresponsesized_cached right, I guess? Oh well.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 11/20 nfsd: implement pNFS layout recalls
In reply to a message not in the archiveAdd support to issue layout recalls to clients. For now we only support full-file recalls to get a simple and stable implementation. This allows to embedd a nfsd4_callback structure in the layout_state and thus avoid any memory allocations under spinlocks during a recall. For normal use cases that do not intent to share a single file between multiple clients this implementation is fully sufficient.
To ensure layouts are recalled on local filesystem access each layout state registers a new FL_LAYOUT lease with the kernel file locking code, which filesystems that support pNFS exports that require recalls need to break on conflicting access patterns.
The XDR code is based on the old pNFS server implementation by Andy Adamson, Benny Halevy, Boaz Harrosh, Dean Hildebrand, Fred Isaman, Marc Eshel, Mike Sager and Ricardo Labiaga.
--- fs/nfsd/nfs4callback.c | 99 +++++++++++++++++++++++ fs/nfsd/nfs4layouts.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++++- fs/nfsd/nfs4proc.c | 4 + fs/nfsd/nfs4state.c | 1 + fs/nfsd/state.h | 6 ++ fs/nfsd/xdr4cb.h | 7 ++ 6 files changed, 330 insertions(+), 1 deletion(-)
fs/nfsd/nfs4callback.c +99 -0
@@ -546,6 +546,102 @@ out:return status; } +#ifdef CONFIG_NFSD_PNFS +/* +* CB_LAYOUTRECALL4args +* +* struct layoutrecall_file4 { +* nfs_fh4 lor_fh; +* offset4 lor_offset; +* length4 lor_length; +* stateid4 lor_stateid; +* }; +* +* union layoutrecall4 switch(layoutrecall_type4 lor_recalltype) { +* case LAYOUTRECALL4_FILE: +* layoutrecall_file4 lor_layout; +* case LAYOUTRECALL4_FSID: +* fsid4 lor_fsid; +* case LAYOUTRECALL4_ALL: +* void; +* }; +* +* struct CB_LAYOUTRECALL4args { +* layouttype4 clora_type; +* layoutiomode4 clora_iomode; +* bool clora_changed; +* layoutrecall4 clora_recall; +* }; +*/ +static void encode_cb_layout4args (struct xdr_stream *xdr, +const struct nfs4_layout_stateid *ls, +struct nfs4_cb_compound_hdr *hdr) +{ +__be32 *p; + +BUG_ON (hdr->minorversion ==0 ); + + p =xdr_reserve_space (xdr,5 *4 ); + *p++ =cpu_to_be32 (OP_CB_LAYOUTRECALL ); + *p++ =cpu_to_be32 (ls->ls_layout_type); + *p++ =cpu_to_be32 (IOMODE_ANY ); + *p++ =cpu_to_be32 (1 ); + *p =cpu_to_be32 (RETURN_FILE ); + +encode_nfs_fh4 (xdr, &ls->ls_stid.sc_file->fi_fhandle); + + p =xdr_reserve_space (xdr,2 *8 ); + p =xdr_encode_hyper (p,0 ); +xdr_encode_hyper (p,NFS4_MAX_UINT64 ); + +encode_stateid4 (xdr, &ls->ls_recall_sid); + + hdr->nops++; +} + +static void nfs4_xdr_enc_cb_layout (struct rpc_rqst *req, +struct xdr_stream *xdr, +const struct nfsd4_callback *cb) +{ +const struct nfs4_layout_stateid *ls = + container_of(cb, struct nfs4_layout_stateid, ls_recall); +struct nfs4_cb_compound_hdr hdr = { + .ident =0 , + .minorversion = cb->cb_minorversion, + }; + +encode_cb_compound4args (xdr, &hdr); +encode_cb_sequence4args (xdr, cb, &hdr); +encode_cb_layout4args (xdr, ls, &hdr); +encode_cb_nops (&hdr); +} + +static int nfs4_xdr_dec_cb_layout (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +struct nfsd4_callback *cb) +{ +struct nfs4_cb_compound_hdr hdr; +enum nfsstat4 nfserr; +int status; + + status =decode_cb_compound4res (xdr, &hdr); +if (unlikely (status)) + goto out; +if (cb) { + status =decode_cb_sequence4res (xdr, cb); +if (unlikely (status)) + goto out; + } + status =decode_cb_op_status (xdr,OP_CB_LAYOUTRECALL , &nfserr); +if (unlikely (status)) + goto out; +if (unlikely (nfserr !=NFS4_OK )) + status =nfs_cb_stat_to_errno (nfserr); +out: +return status; +} +#endif /* CONFIG_NFSD_PNFS */ +/* * RPC procedure tables */
@@ -563,6 +659,9 @@ out:static struct rpc_procinfo nfs4_cb_procedures[] = {PROC (CB_NULL ,NULL , cb_null, cb_null),PROC (CB_RECALL ,COMPOUND , cb_recall, cb_recall), +#ifdef CONFIG_NFSD_PNFS +PROC (CB_LAYOUT ,COMPOUND , cb_layout, cb_layout), +#endif };static struct rpc_version nfs_cb_version4 = {
fs/nfsd/nfs4layouts.c +213 -1
287 lines not on this page · raw message
fs/nfsd/nfs4proc.c +4 -0
@@ -1301,6 +1301,10 @@ nfsd4_layoutget(struct svc_rqst *rqstp,if (nfserr) goto out; + nfserr = nfserr_recallconflict; +if (atomic_read (&ls->ls_stid.sc_file->fi_lo_recalls)) + goto out_put_stid; + nfserr = ops->proc_layoutget (current_fh->fh_dentry->d_inode, current_fh, lgp);if (nfserr)
fs/nfsd/nfs4state.c +1 -0
@@ -3070,6 +3070,7 @@ static void nfsd4_init_file(struct knfsd_fh *fh, unsigned int hashval,memset (fp->fi_access,0 ,sizeof (fp->fi_access));#ifdef CONFIG_NFSD_PNFS INIT_LIST_HEAD (&fp->fi_lo_states); +atomic_set (&fp->fi_lo_recalls,0 );#endif hlist_add_head_rcu (&fp->fi_hash, &file_hashtbl[hashval]); }
fs/nfsd/state.h +6 -0
@@ -502,6 +502,7 @@ struct nfs4_file {bool fi_had_conflict;#ifdef CONFIG_NFSD_PNFS struct list_head fi_lo_states; +atomic_t fi_lo_recalls;#endif };
@@ -542,6 +543,10 @@ struct nfs4_layout_stateid {spinlock_t ls_lock;struct list_head ls_layouts;u32 ls_layout_type; +struct file *ls_file; +struct nfsd4_callback ls_recall; +stateid_t ls_recall_sid; +bool ls_recalled; };static inline struct nfs4_layout_stateid *layoutstateid (struct nfs4_stid *s)
@@ -556,6 +561,7 @@ static inline struct nfs4_layout_stateid *layoutstateid(struct nfs4_stid *s)enum nfsd4_cb_op {NFSPROC4_CLNT_CB_NULL =0 ,NFSPROC4_CLNT_CB_RECALL , +NFSPROC4_CLNT_CB_LAYOUT ,NFSPROC4_CLNT_CB_SEQUENCE , };
fs/nfsd/xdr4cb.h +7 -0
@@ -21,3 +21,10 @@#define NFS4_dec_cb_recall_sz (cb_compound_dec_hdr_sz + \ cb_sequence_dec_sz + \ op_dec_sz) +#define NFS4_enc_cb_layout_sz (cb_compound_enc_hdr_sz + \ + cb_sequence_enc_sz + \ + 1 + 3 + \ + enc_nfs4_fh_sz + 4) +#define NFS4_dec_cb_layout_sz (cb_compound_dec_hdr_sz + \ + cb_sequence_dec_sz + \ + op_dec_sz)
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 14/20 exportfs: add methods for block layout exports
In reply to a message not in the archiveAdd three methods to allow exporting pnfs block layout volumes:
- get_uuid: get a filesystem unique signature exposed to clients
- map_blocks: map and if nessecary allocate blocks for a layout
- commit_blocks: commit blocks in a layout once the client is done with them
For now we stick the external pnfs block layout interfaces into s_export_op to avoid mixing them up with the internal interface between the NFS server and the layout drivers. Once we've fully internalized the latter interface we can redecide if these methods should stay in s_export_ops.
--- include/linux/exportfs.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
include/linux/exportfs.h +21 -0
@@ -4,6 +4,7 @@#include <linux/types.h> struct dentry ; +struct iattr ;struct inode ;struct super_block ;struct vfsmount ;
@@ -180,6 +181,19 @@ struct fid {* get_name is not (which is possibly inconsistent) */ +/* types of block ranges for multipage write mappings. */ +#define IOMAP_HOLE 0x01/* no blocks allocated, need allocation */ +#define IOMAP_DELALLOC 0x02/* delayed allocation blocks */ +#define IOMAP_MAPPED 0x03/* blocks allocated @blkno */ +#define IOMAP_UNWRITTEN 0x04/* blocks allocated @blkno in unwritten state */ + +struct iomap { +sector_t blkno;/* first sector of mapping */ +loff_t offset;/* file offset of mapping, bytes */ +u64 length;/* length of mapping, bytes */ +int type;/* type of mapping */ +}; +struct export_operations {int (*encode_fh)(struct inode *inode,__u32 *fh,int *max_len,struct inode *parent);
@@ -191,6 +205,13 @@ struct export_operations { struct dentry *child);struct dentry * (*get_parent)(struct dentry *child);int (*commit_metadata)(struct inode *inode); + +int (*get_uuid)(struct super_block *sb,u8 *buf,u32 *len,u64 *offset); +int (*map_blocks)(struct inode *inode,loff_t offset, +u64 len,struct iomap *iomap, +bool write,u32 *device_generation); +int (*commit_blocks)(struct inode *inode,struct iomap *iomaps, +int nr_iomaps,struct iattr *iattr); }; extern int exportfs_encode_inode_fh(struct inode *inode, struct fid *fid,
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 16/20 xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
In reply to a message not in the archiveThe code is already ready for it, and the pnfs layout commit code expects to be able to pass a larger than 32-bit argument.
--- fs/xfs/xfs_iomap.c | 2 +- fs/xfs/xfs_iomap.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
fs/xfs/xfs_iomap.c +1 -1
@@ -802,7 +802,7 @@ intxfs_iomap_write_unwritten ( xfs_inode_t *ip, xfs_off_t offset, - size_t count) + xfs_off_t count) {xfs_mount_t *mp = ip->i_mount;xfs_fileoff_t offset_fsb;
fs/xfs/xfs_iomap.h +1 -1
@@ -27,6 +27,6 @@ int xfs_iomap_write_delay(struct xfs_inode *, xfs_off_t, size_t, struct xfs_bmbt_irec *);int xfs_iomap_write_allocate (struct xfs_inode *,xfs_off_t ,struct xfs_bmbt_irec *); -int xfs_iomap_write_unwritten (struct xfs_inode *,xfs_off_t ,size_t ); +int xfs_iomap_write_unwritten (struct xfs_inode *,xfs_off_t ,xfs_off_t );#endif /* __XFS_IOMAP_H__*/
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Who can give us ACKs on these last five fs/xfs patches? (And is it going to cause trouble if they go in through the nfsd tree?)
--b.
Quoted 36 lines · Christoph Hellwig
The code is already ready for it, and the pnfs layout commit code expects to be able to pass a larger than 32-bit argument.
Signed-off-by Christoph Hellwig <[email protected]>--- fs/xfs/xfs_iomap.c | 2 +- fs/xfs/xfs_iomap.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)fs/xfs/xfs_iomap.c +1
-1@@ -802,7 +802,7 @@ intxfs_iomap_write_unwritten ( xfs_inode_t *ip, xfs_off_t offset, - size_t count) + xfs_off_t count) {xfs_mount_t *mp = ip->i_mount;xfs_fileoff_t offset_fsb;fs/xfs/xfs_iomap.h +1
-1@@ -27,6 +27,6 @@ int xfs_iomap_write_delay(struct xfs_inode *, xfs_off_t, size_t, struct xfs_bmbt_irec *);int xfs_iomap_write_allocate (struct xfs_inode *,xfs_off_t ,struct xfs_bmbt_irec *); -int xfs_iomap_write_unwritten (struct xfs_inode *,xfs_off_t ,size_t ); +int xfs_iomap_write_unwritten (struct xfs_inode *,xfs_off_t ,xfs_off_t );#endif /* __XFS_IOMAP_H__*/ Signature · 1 line
1.9.1
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Christoph Hellwig
reviewerRe: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 2 lines · J. Bruce Fields
Who can give us ACKs on these last five fs/xfs patches? (And is it going to cause trouble if they go in through the nfsd tree?)
We'd need ACKs from Dave. He already has pulled in two patches so we might run into some conflicts. Maybe the best idea is to add the exportfs patch to both the XFS and nfsd trees, so each of them can pull in the rest? Or we could commit the two XFS preparation patches to both tree and get something that compiles and works in the nfsd tree.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 11 lines · Christoph Hellwig
Quoted 2 lines · J. Bruce Fields
Who can give us ACKs on these last five fs/xfs patches? (And is it going to cause trouble if they go in through the nfsd tree?)
We'd need ACKs from Dave. He already has pulled in two patches so we might run into some conflicts. Maybe the best idea is to add the exportfs patch to both the XFS and nfsd trees, so each of them can pull in the rest? Or we could commit the two XFS preparation patches to both tree and get something that compiles and works in the nfsd tree.
This patch has already been committed to the XFS repo.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 6 lines · Dave Chinner
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines · J. Bruce Fields
Who can give us ACKs on these last five fs/xfs patches? (And is it going to cause trouble if they go in through the nfsd tree?)
Quoted 8 lines
Quoted 6 lines
We'd need ACKs from Dave. He already has pulled in two patches so we might run into some conflicts. Maybe the best idea is to add the exportfs patch to both the XFS and nfsd trees, so each of them can pull in the rest? Or we could commit the two XFS preparation patches to both tree and get something that compiles and works in the nfsd tree.
This patch has already been committed to the XFS repo.
And it looks like I missed the sync transaction on growfs patch, too, so I'll commit that one later today.
As to the pNFSD specific changes, I haven't really looked them over in any great detail yet. My main concern is that there are no specific regression tests for this yet, I'm not sure how we go about verifying it actually works properly and we don't inadvertantly break it in the future. Christoph?
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 16 lines · Dave Chinner
Quoted 6 lines · Dave Chinner
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines · J. Bruce Fields
Who can give us ACKs on these last five fs/xfs patches? (And is it going to cause trouble if they go in through the nfsd tree?)
Quoted 8 lines
Quoted 6 lines
We'd need ACKs from Dave. He already has pulled in two patches so we might run into some conflicts. Maybe the best idea is to add the exportfs patch to both the XFS and nfsd trees, so each of them can pull in the rest? Or we could commit the two XFS preparation patches to both tree and get something that compiles and works in the nfsd tree.
This patch has already been committed to the XFS repo.
Quoted 8 lines
And it looks like I missed the sync transaction on growfs patch, too, so I'll commit that one later today.
As to the pNFSD specific changes, I haven't really looked them over in any great detail yet. My main concern is that there are no specific regression tests for this yet, I'm not sure how we go about verifying it actually works properly and we don't inadvertantly break it in the future. Christoph?
Previously: http://lkml.kernel.org/r/[email protected]
> - any advice on testing? Is there was some simple > virtual setup that would allow any loser with no special > hardware (e.g., me) to check whether they've broken the > block server?
Run two kvm VMs that share the same disk. Create an XFS filesystem on the MDS, and export it. If the client has blkmapd running (on Debian it needs to be started manually) it will use pNFS for accessing the filesystem. Verify that using the per-operation counters in /proc/self/mounstats. Repeat with additional clients as nessecary.
Alternatively set up a simple iSCSI target using tgt or lio and connect to it from multiple clients.
Which sounds reasonable to me, but I haven't tried to incorporate this into my regression testing yet.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Christoph Hellwig
reviewerRe: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 14 lines · J. Bruce Fields
Previously: http://lkml.kernel.org/r/[email protected]
> - any advice on testing? Is there was some simple > virtual setup that would allow any loser with no special > hardware (e.g., me) to check whether they've broken the > block server?Run two kvm VMs that share the same disk. Create an XFS filesystem on the MDS, and export it. If the client has blkmapd running (on Debian it needs to be started manually) it will use pNFS for accessing the filesystem. Verify that using the per-operation counters in
/proc/self/mounstats. Repeat with additional clients as nessecary.
Quoted 5 lines
Alternatively set up a simple iSCSI target using tgt or lio and connect to it from multiple clients.
Which sounds reasonable to me, but I haven't tried to incorporate this into my regression testing yet.
Additionally I can offer the following script to generate recalls, which don't really happen during normal operation. I don't really know how to write a proper testcase that coordinates access to the exported filesystem and nfs unless it runs locally on the same node, though. It would need some higher level, network aware test harness:
----- snip -----
#!/bin/sh
set +x
# wait for grace period touch /mnt/nfs1/foo
dd if=/dev/zero of=/mnt/nfs1/foo bs=128M count=32 conv=fdatasync oflag=direct &
sleep 2
echo "" > /mnt/test/foo && echo "recall done"
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 18 lines · Christoph Hellwig
Quoted 17 lines · J. Bruce Fields
Previously: http://lkml.kernel.org/r/[email protected]
> - any advice on testing? Is there was some simple > virtual setup that would allow any loser with no special > hardware (e.g., me) to check whether they've broken the > block server?Run two kvm VMs that share the same disk. Create an XFS filesystem on the MDS, and export it. If the client has blkmapd running (on Debian it needs to be started manually) it will use pNFS for accessing the filesystem. Verify that using the per-operation counters in
/proc/self/mounstats. Repeat with additional clients as nessecary.Alternatively set up a simple iSCSI target using tgt or lio and connect to it from multiple clients.
Quoted 8 lines
Quoted 2 lines
Which sounds reasonable to me, but I haven't tried to incorporate this into my regression testing yet.
Additionally I can offer the following script to generate recalls, which don't really happen during normal operation. I don't really know how to write a proper testcase that coordinates access to the exported filesystem and nfs unless it runs locally on the same node, though. It would need some higher level, network aware test harness:
Thanks. I got as far as doing a quick manual test with vm's sharing a "disk":
[root@f21-2]# mount -overs=4.1 f21-1:/exports/xfs-pnfs /mnt/ [root@f21-2]# echo "hello world" >/mnt/testfile [root@f21-2]# grep LAYOUTGET /proc/self/mountstats LAYOUTGET: 1 1 0 236 196 0 4 4
I haven't tried to set up automated testing with recalls, but that shouldn't be hard.
--b.
Quoted 6 lines
----- snip -----
#!/bin/shset +x
Quoted 8 lines
# wait for grace period touch /mnt/nfs1/foodd if=/dev/zero of=/mnt/nfs1/foo bs=128M count=32 conv=fdatasync oflag=direct &
sleep 2
echo "" >
/mnt/test/foo&& echo "recall done"
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Christoph Hellwig
Quoted 19 lines · J. Bruce Fields
Previously: http://lkml.kernel.org/r/[email protected]
> - any advice on testing? Is there was some simple > virtual setup that would allow any loser with no special > hardware (e.g., me) to check whether they've broken the > block server?Run two kvm VMs that share the same disk. Create an XFS filesystem on the MDS, and export it. If the client has blkmapd running (on Debian it needs to be started manually) it will use pNFS for accessing the filesystem. Verify that using the per-operation counters in
/proc/self/mounstats. Repeat with additional clients as nessecary.Alternatively set up a simple iSCSI target using tgt or lio and connect to it from multiple clients.
Which sounds reasonable to me, but I haven't tried to incorporate this into my regression testing yet.
Quoted 8 lines
Quoted 5 lines
Additionally I can offer the following script to generate recalls, which don't really happen during normal operation. I don't really know how to write a proper testcase that coordinates access to the exported filesystem and nfs unless it runs locally on the same node, though. It would need some higher level, network aware test harness:
Thanks. I got as far as doing a quick manual test with vm's sharing a "disk":
Oh, also I forgot, on fedora:
[root@f21-2]# systemctl enable nfs-blkmap.target
Quoted 24 lines
[root@f21-2]# mount -overs=4.1 f21-1:/exports/xfs-pnfs /mnt/ [root@f21-2]# echo "hello world" >/mnt/testfile [root@f21-2]# grep LAYOUTGET /proc/self/mountstats LAYOUTGET: 1 1 0 236 196 0 4 4I haven't tried to set up automated testing with recalls, but that shouldn't be hard.
--b.
Quoted 14 lines
----- snip -----
#!/bin/shset +x
# wait for grace period touch /mnt/nfs1/foodd if=/dev/zero of=/mnt/nfs1/foo bs=128M count=32 conv=fdatasync oflag=direct &
sleep 2
echo "" >
/mnt/test/foo&& echo "recall done"
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
In reply to Dave ChinnerQuoted 5 lines · Dave Chinner
As to the pNFSD specific changes, I haven't really looked them over in any great detail yet. My main concern is that there are no specific regression tests for this yet, I'm not sure how we go about verifying it actually works properly and we don't inadvertantly break it in the future. Christoph?
Any chance you could review them this week so we can get them merged in time for 3.20? In the worst case Bruce will have to pull the xfs tree into the nfsd tree so that we have all the dependencies available.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 16/20] xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Dave Chinner
As to the pNFSD specific changes, I haven't really looked them over in any great detail yet. My main concern is that there are no specific regression tests for this yet, I'm not sure how we go about verifying it actually works properly and we don't inadvertantly break it in the future. Christoph?
Any chance you could review them this week so we can get them merged in time for 3.20? In the worst case Bruce will have to pull the xfs tree into the nfsd tree so that we have all the dependencies available.
I'm working my way through them. I'm just about to pull in the growfs transaction patch (missed it last time around), and I try to have a decent look over the other two patches later today.
I'm not sure I have any bandwidth to test them yet, but perhaps if I add a one-time message "Experimental feature in use" when the code is first executed then it will be OK to merge (i.e. process similar to delayed logging and CRC introduction). Once we've got more confidence that it's all working properly, then we can remove the experimental tag from it. Does that sound like a reasonable approach to take?
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 17/20 xfs: update the superblock using a synchronous transaction in growfs
In reply to a message not in the archiveGrowfs updates the secondary superblocks using synchronous unlogged buffer writes after committing the updates to the primary superblock.
Mark the transaction to the primary superblock as synchronous so that we guarantee it is committed to disk before we update the secondary superblocks.
--- fs/xfs/xfs_fsops.c | 1 + 1 file changed, 1 insertion(+)
fs/xfs/xfs_fsops.c +1 -0
@@ -488,6 +488,7 @@ xfs_growfs_data_private(xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , nfree);if (dpct)xfs_trans_mod_sb (tp,XFS_TRANS_SB_IMAXPCT , dpct); +xfs_trans_set_sync (tp); error =xfs_trans_commit (tp,0 );if (error)return error;
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 19/20 xfs: implement pNFS export operations
In reply to a message not in the archiveAdd operations to export pNFS block layouts from an XFS filesystem. See the previous commit adding the operations for an explanation of them.
--- fs/xfs/Makefile | 1 + fs/xfs/xfs_export.c | 6 ++ fs/xfs/xfs_fsops.c | 2 + fs/xfs/xfs_iops.c | 2 +- fs/xfs/xfs_iops.h | 1 + fs/xfs/xfs_mount.h | 11 +++ fs/xfs/xfs_pnfs.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_pnfs.h | 11 +++ 8 files changed, 276 insertions(+), 1 deletion(-) create mode 100644 fs/xfs/xfs_pnfs.c create mode 100644 fs/xfs/xfs_pnfs.h
fs/xfs/Makefile +1 -0
@@ -121,3 +121,4 @@ xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o xfs-$(CONFIG_PROC_FS) += xfs_stats.o xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o +xfs-$(CONFIG_NFSD_PNFS) += xfs_pnfs.o
fs/xfs/xfs_export.c +6 -0
@@ -30,6 +30,7 @@#include "xfs_trace.h" #include "xfs_icache.h" #include "xfs_log.h" +#include "xfs_pnfs.h" /* * Note that we only accept fileids which are long enough rather than allow
@@ -245,4 +246,9 @@ const struct export_operations xfs_export_operations = { .fh_to_parent = xfs_fs_fh_to_parent, .get_parent = xfs_fs_get_parent, .commit_metadata = xfs_fs_nfs_commit_metadata, +#ifdef CONFIG_NFSD_PNFS + .get_uuid = xfs_fs_get_uuid, + .map_blocks = xfs_fs_map_blocks, + .commit_blocks = xfs_fs_commit_blocks, +#endif };
fs/xfs/xfs_fsops.c +2 -0
@@ -602,6 +602,8 @@ xfs_growfs_data(if (!mutex_trylock (&mp->m_growlock))return -EWOULDBLOCK ; error =xfs_growfs_data_private (mp, in); +if (!error) + mp->m_generation++;mutex_unlock (&mp->m_growlock);return error; }
fs/xfs/xfs_iops.c +1 -1
@@ -496,7 +496,7 @@ xfs_setattr_mode( inode->i_mode |= mode & ~S_IFMT ; } -static void +void xfs_setattr_time (struct xfs_inode *ip,struct iattr *iattr)
fs/xfs/xfs_iops.h +1 -0
@@ -32,6 +32,7 @@ extern void xfs_setup_inode(struct xfs_inode *);*/ #define XFS_ATTR_NOACL 0x01/* Don't call posix_acl_chmod */ +extern void xfs_setattr_time (struct xfs_inode *ip,struct iattr *iattr);extern int xfs_setattr_nonsize (struct xfs_inode *ip,struct iattr *vap,int flags);extern int xfs_setattr_size (struct xfs_inode *ip,struct iattr *vap);
fs/xfs/xfs_mount.h +11 -0
@@ -175,6 +175,17 @@ typedef struct xfs_mount {struct workqueue_struct *m_reclaim_workqueue;struct workqueue_struct *m_log_workqueue;struct workqueue_struct *m_eofblocks_workqueue; + +/* +* Generation of the filesysyem layout. This is incremented by each +* growfs, and used by the pNFS server to ensure the client updates +* its view of the block device once it gets a layout that might +* reference the newly added blocks. Does not need to be persistent +* as long as we only allow file system size increments, but if we +* ever support shrinks it would have to be persisted in addition +* to various other kinds of pain inflicted on the pNFS server. +*/ +__uint32_t m_generation; } xfs_mount_t;/*
fs/xfs/xfs_pnfs.c new file +243 -0
243 lines not on this page · raw message
fs/xfs/xfs_pnfs.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef _XFS_PNFS_H +#define _XFS_PNFS_H 1 + +#ifdef CONFIG_NFSD_PNFS +int xfs_fs_get_uuid (struct super_block *sb,u8 *buf,u32 *len,u64 *offset); +int xfs_fs_map_blocks (struct inode *inode,loff_t offset,u64 length, +struct iomap *iomap,bool write,u32 *device_generation); +int xfs_fs_commit_blocks (struct inode *inode,struct iomap *maps,int nr_maps, +struct iattr *iattr); +#endif /* CONFIG_NFSD_PNFS */ +#endif /* _XFS_PNFS_H */
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-20-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 4 lines · Christoph Hellwig
Add operations to export pNFS block layouts from an XFS filesystem. See the previous commit adding the operations for an explanation of them.
Signed-off-by Christoph Hellwig <[email protected]>
Note that I haven't applied this patch, or attempted to compile it yet....
fs/xfs/Makefile xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o · line 124
Quoted 9 lines
fs/xfs/Makefile +1
-0@@ -121,3 +121,4 @@ xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o xfs-$(CONFIG_PROC_FS) += xfs_stats.o xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o +xfs-$(CONFIG_NFSD_PNFS) += xfs_pnfs.o
... because I'll have to jump through hoops to get it to compile, I think.
fs/xfs/xfs_fsops.c xfs_growfs_data( · line 609
Quoted 13 lines
fs/xfs/xfs_fsops.c +2
-0@@ -602,6 +602,8 @@ xfs_growfs_data(if (!mutex_trylock (&mp->m_growlock))return -EWOULDBLOCK ; error =xfs_growfs_data_private (mp, in); +if (!error) + mp->m_generation++;mutex_unlock (&mp->m_growlock);return error; }
Even on error I think we should bump this. Errors can come from secondary superblock updates after the filesystem has been grown, hence an error is not a reliable indication of whether the layout has changed or not.
Quoted 5 lines
fs/xfs/xfs_fsops.c +5
-0+int +xfs_fs_get_uuid( + struct super_block *sb, + u8 *buf, + u32 *len,
fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+ u64 *offset) +{ +struct xfs_mount *mp =XFS_M (sb); + +if (*len <sizeof (uuid_t)) +return -EINVAL ; + +memcpy (buf, &mp->m_sb.sb_uuid,sizeof (uuid_t)); + *len =sizeof (uuid_t); + *offset = offsetof(struct xfs_dsb , sb_uuid);
What purpose does the offset serve here? I can't tell from the usage in the PNFS code. Can we ignore offset - as it seems entirely arbitrary - and still have this work? Either way, comment please.
Quoted 4 lines
fs/xfs/xfs_fsops.c +4
-0+static void +xfs_bmbt_to_iomap( + struct xfs_inode *ip, + struct iomap *iomap,
fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+ struct xfs_bmbt_irec *imap) +{ +struct xfs_mount *mp = ip->i_mount; + +if (imap->br_startblock ==HOLESTARTBLOCK ) { + iomap->blkno =-1 ; + iomap->type =IOMAP_HOLE ; + }else if (imap->br_startblock ==DELAYSTARTBLOCK ) { + iomap->blkno =-1 ; + iomap->type =IOMAP_DELALLOC ;
I'd like to see a IOMAP_NULL_BLOCK define here for the -1 value, say:
#define IOMAP_NULL_BLOCK -1LL
Quoted 13 lines
fs/xfs/xfs_fsops.c +13
-0+int +xfs_fs_map_blocks ( +struct inode *inode, +loff_t offset, +u64 length, +struct iomap *iomap, +bool write, +u32 *device_generation) +{ +struct xfs_inode *ip =XFS_I (inode); +struct xfs_mount *mp = ip->i_mount; +struct xfs_bmbt_irec imap; +xfs_fileoff_t offset_fsb, end_fsb;
fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+loff_t limit; +int bmapi_flags =XFS_BMAPI_ENTIRE ; +int nimaps =1 ; +uint lock_flags; +int error =0 ; + +if (XFS_FORCED_SHUTDOWN (mp)) +return -EIO ; +if (XFS_IS_REALTIME_INODE (ip)) +return -ENXIO ;
OK, so we are not mapping realtime inodes here. Any specific reason? FWIW, that also means you can use XFS_FSB_TO_DADDR() in the iomap mapping as xfs_fsb_to_db() is only needed if we might be mapping realtime extents...
Quoted 5 lines
fs/xfs/xfs_fsops.c +5
-0+for (i =0 ; i < nr_maps; i++) { +u64 start, length, end; + + start = maps[i].offset; +if (start > size)
fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+continue ; + + end = start + maps[i].length; +if (end > size) + end = size; + + length = end - start; +if (!length) +continue ; +
^^^^^
Stray whitespace
Quoted 6 lines
fs/xfs/xfs_fsops.c +6
-0+ tp =xfs_trans_alloc (mp,XFS_TRANS_SETATTR_NOT_SIZE ); + error =xfs_trans_reserve (tp, &M_RES (mp)->tr_ichange,0 ,0 ); +if (error) + goto out_drop_iolock; + +xfs_ilock (ip,XFS_ILOCK_EXCL );
fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+xfs_trans_ijoin (tp, ip,XFS_ILOCK_EXCL ); +xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); + +xfs_setattr_time (ip, iattr); +if (iattr->ia_valid &ATTR_SIZE ) { +if (iattr->ia_size >i_size_read (inode)) { +i_size_write (inode, iattr->ia_size); + ip->i_d.di_size = iattr->ia_size; + } + }
The concern I have about this is that extending the file size can expose uninitialised blocks beyond the old EOF. That can happen if delayed allocation has previously been done on the file and we haven't trimmed the excess beyond EOF back yet. I know the pnfs server is not aimed at mixed usage, but it still makes me uncomfortable in the case where you have normal NFS and PNFS clients accessing the same files...
fs/xfs/xfs_fsops.c
Quoted 2 lines
fs/xfs/xfs_fsops.c +2
-0+xfs_trans_set_sync (tp); + error =xfs_trans_commit (tp,0 );
I just had a thought about these sync transctions - could the NFS server handle persistence of the maps via ->commit_metadata?
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 9 lines · Dave Chinner
fs/xfs/Makefile xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o · line 124
Quoted 6 lines
+++ b/fs/xfs/Makefile
fs/xfs/Makefile +1
-0@@ -121,3 +121,4 @@ xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o xfs-$(CONFIG_PROC_FS) += xfs_stats.o xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o +xfs-$(CONFIG_NFSD_PNFS) += xfs_pnfs.o... because I'll have to jump through hoops to get it to compile, I think.
Just get the whole git tree from
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-3
Quoted 10 lines
Quoted 10 lines
fs/xfs/xfs_fsops.c +2
-0@@ -602,6 +602,8 @@ xfs_growfs_data(if (!mutex_trylock (&mp->m_growlock))return -EWOULDBLOCK ; error =xfs_growfs_data_private (mp, in); +if (!error) + mp->m_generation++;
Quoted 8 lines
fs/xfs/xfs_fsops.c xfs_growfs_data( · line 609
Quoted 3 lines
fs/xfs/xfs_fsops.c +0
-0mutex_unlock (&mp->m_growlock);return error; }Even on error I think we should bump this. Errors can come from secondary superblock updates after the filesystem has been grown, hence an error is not a reliable indication of whether the layout has changed or not.
Ok.
Quoted 12 lines
Quoted 12 lines
fs/xfs/xfs_fsops.c +11
-0+int +xfs_fs_get_uuid ( +struct super_block *sb, +u8 *buf, +u32 *len, +u64 *offset) +{ +struct xfs_mount *mp =XFS_M (sb); + +if (*len <sizeof (uuid_t)) +return -EINVAL ;
Quoted 8 lines
fs/xfs/xfs_fsops.c
Quoted 4 lines
fs/xfs/xfs_fsops.c +4
-0+ +memcpy (buf, &mp->m_sb.sb_uuid,sizeof (uuid_t)); + *len =sizeof (uuid_t); + *offset = offsetof(struct xfs_dsb , sb_uuid);What purpose does the offset serve here? I can't tell from the usage in the PNFS code. Can we ignore offset - as it seems entirely arbitrary - and still have this work? Either way, comment please.
The get_uuid methods gets content and location of the uuid so that the client can find the disks. The offset simply is part of the wire protocol.
Quoted 4 lines
I'd like to see a
IOMAP_NULL_BLOCKdefine here for the -1 value, say:#define
IOMAP_NULL_BLOCK-1LL
Ok.
Quoted 1 line
OK, so we are not mapping realtime inodes here. Any specific reason?
The realtime device doesn't have a way for the client to find it, as it doesn't have its own superblockuuids.
Quoted 3 lines
FWIW, that also means you can use
XFS_FSB_TO_DADDR()in the iomap mapping asxfs_fsb_to_db()is only needed if we might be mapping realtime extents...
ok.
Quoted 17 lines
Quoted 6 lines
fs/xfs/xfs_fsops.c +6
-0+ tp =xfs_trans_alloc (mp,XFS_TRANS_SETATTR_NOT_SIZE ); + error =xfs_trans_reserve (tp, &M_RES (mp)->tr_ichange,0 ,0 ); +if (error) + goto out_drop_iolock; + +xfs_ilock (ip,XFS_ILOCK_EXCL );fs/xfs/xfs_fsops.c
Quoted 10 lines
fs/xfs/xfs_fsops.c +10
-0+xfs_trans_ijoin (tp, ip,XFS_ILOCK_EXCL ); +xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); + +xfs_setattr_time (ip, iattr); +if (iattr->ia_valid &ATTR_SIZE ) { +if (iattr->ia_size >i_size_read (inode)) { +i_size_write (inode, iattr->ia_size); + ip->i_d.di_size = iattr->ia_size; + } + }
Quoted 7 lines
The concern I have about this is that extending the file size can expose uninitialised blocks beyond the old EOF. That can happen if delayed allocation has previously been done on the file and we haven't trimmed the excess beyond EOF back yet. I know the pnfs server is not aimed at mixed usage, but it still makes me uncomfortable in the case where you have normal NFS and PNFS clients accessing the same files...
The protocol only allows to commit to a size that we previously returned a layout for, which means we already have allocated space for it at same time. For robustness reasons a sanity check might make sense, though.
Quoted 5 lines
fs/xfs/xfs_fsops.c
Quoted 2 lines
fs/xfs/xfs_fsops.c +2
-0+xfs_trans_set_sync (tp); + error =xfs_trans_commit (tp,0 );I just had a thought about these sync transctions - could the NFS server handle persistence of the maps via ->commit_metadata?
It probably could, but that wouldn't buy us anything over just committing the transaction synchronously.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 19/20] xfs: implement pNFS export operations
I've updated the patch and pushed out a new pnfsd-for-3.20-4 branch.
The changes relative to the old one are below:
fs/xfs/xfs_fsops.c +6 -2
@@ -602,8 +602,12 @@ xfs_growfs_data(if (!mutex_trylock (&mp->m_growlock))return -EWOULDBLOCK ; error =xfs_growfs_data_private (mp, in); -if (!error) - mp->m_generation++; +/* +* Increment the generation unconditionally, the error could be from +* updating the secondary superblocks, in which case the new size +* is live already. +*/ + mp->m_generation++;mutex_unlock (&mp->m_growlock);return error; }
fs/xfs/xfs_pnfs.c +58 -9
132 lines not on this page · raw message
include/linux/exportfs.h +2 -0
@@ -187,6 +187,8 @@ struct fid {#define IOMAP_MAPPED 0x03/* blocks allocated @blkno */ #define IOMAP_UNWRITTEN 0x04/* blocks allocated @blkno in unwritten state */ +#define IOMAP_NULL_BLOCK -1LL/* blkno is not valid */ +struct iomap {sector_t blkno;/* first sector of mapping */ loff_t offset;/* file offset of mapping, bytes */
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dave Chinner
Acked-byRe: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 3 lines · Christoph Hellwig
I've updated the patch and pushed out a new pnfsd-for-3.20-4 branch.
The changes relative to the old one are below:
Hi Christoph, with these changes I think this is fine to be merged with the experimental tag attached to it
I'm expecting the merge window to open on Monday so it's kinda late to be adding new stuff to the XFS tree and co-ordinating it with the NFS tree merge - how were you planning to get this to merged?
I've already merged all but the two pNFSD support patches, so there's some duplicate commits in your pnfsd-for-3.20-4 branch. i.e. these commits in your tree:
b8d5187 xfs: factor out a xfs_update_prealloc_flags() helper 6d5ca2a xfs: update the superblock using a synchronous transaction in growfs e3ea93e xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
are already merged into the xfs for-next branch as:
8add71c xfs: factor out a xfs_update_prealloc_flags() helper f8079b8 xfs: growfs should use synchronous transactions d32057f xfs: pass a 64-bit count argument to xfs_iomap_write_unwritten
A straight merge from your tree ends up with both sets of commits in the history. So a rebase on your side, or me pulling them into the XFS tree is probably required to keep the history clean.
I didn't really want to add any more to the XFS tree this close to the merge window opening, but I've already got a regression fix that needs to be added, so perhaps I'll delay sending Linus a pull request for a week and just merge all of these XFS changes directly.
What do you think?
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 32 lines · Dave Chinner
Quoted 3 lines · Christoph Hellwig
I've updated the patch and pushed out a new pnfsd-for-3.20-4 branch.
The changes relative to the old one are below:
Hi Christoph, with these changes I think this is fine to be merged with the experimental tag attached to it
Acked-by Dave Chinner <[email protected]>I'm expecting the merge window to open on Monday so it's kinda late to be adding new stuff to the XFS tree and co-ordinating it with the NFS tree merge - how were you planning to get this to merged?
I've already merged all but the two pNFSD support patches, so there's some duplicate commits in your pnfsd-for-3.20-4 branch. i.e. these commits in your tree:
b8d5187 xfs: factor out a
xfs_update_prealloc_flags()helper 6d5ca2a xfs: update the superblock using a synchronous transaction in growfs e3ea93e xfs: pass a 64-bit count argument toxfs_iomap_write_unwrittenare already merged into the xfs for-next branch as:
8add71c xfs: factor out a
xfs_update_prealloc_flags()helper f8079b8 xfs: growfs should use synchronous transactions d32057f xfs: pass a 64-bit count argument toxfs_iomap_write_unwrittenA straight merge from your tree ends up with both sets of commits in the history. So a rebase on your side, or me pulling them into the XFS tree is probably required to keep the history clean.
Quoted 6 lines
I didn't really want to add any more to the XFS tree this close to the merge window opening, but I've already got a regression fix that needs to be added, so perhaps I'll delay sending Linus a pull request for a week and just merge all of these XFS changes directly.
What do you think?
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
--b.
Christoph Hellwig
reviewerRe: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 17 lines · On Sun, 8 Feb 2015 14:34:35 +0100 Christoph Hellwig…
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Quoted 7 lines
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
FWIW, I plan to send a pull request for the locking changes as soon as the merge window opens. Hopefully that won't be an issue for long...
Signature · 5 lines
Jeff Layton <jeff.layton-7I+n7zu2hftEKMMhf/[email protected]> -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20150208090942.51e99687-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 24 lines · Jeff Layton
Quoted 21 lines · On Sun, 8 Feb 2015 14:34:35 +0100 Christoph Hellwig…
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Quoted 6 lines
Quoted 3 lines
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
FWIW, I plan to send a pull request for the locking changes as soon as the merge window opens. Hopefully that won't be an issue for long...
This includes Christoph's branch (all but the final xfs commits):
git://linux-nfs.org/~bfields/linux.git for-3.20
That's what I intend to submit. Hope that's OK. Then it's up to Dave whether he wants to pull that in and include the xfs patches.
Worst case we end up with not-yet-usable pnfs code in 3.20, which wouldn't be ideal but shouldn't cause any serious problem either.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 32 lines · J. Bruce Fields
Quoted 24 lines · Jeff Layton
Quoted 21 lines · On Sun, 8 Feb 2015 14:34:35 +0100 Christoph Hellwig…
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Quoted 6 lines
Quoted 3 lines
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
FWIW, I plan to send a pull request for the locking changes as soon as the merge window opens. Hopefully that won't be an issue for long...
Quoted 6 lines
This includes Christoph's branch (all but the final xfs commits):
git://linux-nfs.org/~bfields/linux.git for-3.20That's what I intend to submit. Hope that's OK. Then it's up to Dave whether he wants to pull that in and include the xfs patches.
I'm about to send a pull request to Linus for the current XFS tree. Once that is merged, I'll pull in the remaining xfs-pnfs patches and send another pull request to Linus after the NFS tree is merged.
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 35 lines · Dave Chinner
Quoted 34 lines · J. Bruce Fields
Quoted 24 lines · Jeff Layton
Quoted 21 lines · On Sun, 8 Feb 2015 14:34:35 +0100 Christoph Hellwig…
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Quoted 6 lines
Quoted 3 lines
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
FWIW, I plan to send a pull request for the locking changes as soon as the merge window opens. Hopefully that won't be an issue for long...
This includes Christoph's branch (all but the final xfs commits):
Quoted 8 lines
Quoted 4 lines
git://linux-nfs.org/~bfields/linux.git for-3.20That's what I intend to submit. Hope that's OK. Then it's up to Dave whether he wants to pull that in and include the xfs patches.
I'm about to send a pull request to Linus for the current XFS tree. Once that is merged, I'll pull in the remaining xfs-pnfs patches and send another pull request to Linus after the NFS tree is merged.
Sounds good, thanks. The nfsd tree's merged now so it should be good to go if you haven't found any show-stoppers.
--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 41 lines · J. Bruce Fields
Quoted 40 lines · Dave Chinner
Quoted 38 lines · J. Bruce Fields
Quoted 24 lines · Jeff Layton
Quoted 21 lines · On Sun, 8 Feb 2015 14:34:35 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 9 lines · J. Bruce Fields
You'd basically just be pulling my tree (Christoph's is just my nfsd tree with his patches on top, and I've been testing with exactly that locally, just putting off pushing it out till we decide this.)
So anyway, fine with me if you want to just pull that into the xfs tree. Mine's ready whenever, so if I send my pull pretty soon after the merge window and you send it a little later then we still keep the property that Linus's merge still has a diffstat only in our respective areas.
Quoted 6 lines
(OK, it's a little more complicated because I've got the same arrangement with jlayton, so the order is jlayton's lock pull, then my nfsd pull, then your xfs pull. Is this getting too complicated? jlayton and I are both ready to so and I think it'd work.)
I'm also fine with duplicating those few patches, or whatever.
Maybe the better idea is to pull the xfs tree in the nfsd tree, but that would require Dave sending an early pull request so that the nfsd pull doesn't get delayed.
Quoted 6 lines
Quoted 3 lines
Or we just defer the pnfsd merge. While I tried to get it in in time for 3.20 all the delays during review mean we're really late no and should punt it to 3.21.
FWIW, I plan to send a pull request for the locking changes as soon as the merge window opens. Hopefully that won't be an issue for long...
This includes Christoph's branch (all but the final xfs commits):
git://linux-nfs.org/~bfields/linux.git for-3.20That's what I intend to submit. Hope that's OK. Then it's up to Dave whether he wants to pull that in and include the xfs patches.
Quoted 6 lines
Quoted 3 lines
I'm about to send a pull request to Linus for the current XFS tree. Once that is merged, I'll pull in the remaining xfs-pnfs patches and send another pull request to Linus after the NFS tree is merged.
Sounds good, thanks. The nfsd tree's merged now so it should be good to go if you haven't found any show-stoppers.
Thanks Bruce. I might have to build a merged tree because one of the changes from the review modified a header file introduced in the NFS tree.
I'll see how it goes, and see if I can avoid doing something that will make Linus yell at me :P
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 19/20] xfs: implement pNFS export operations
Hi Dave,
Quoted 7 lines · On Fri, 13 Feb 2015 12:54:22 +1100 Dave Chinner <[email protected]> wrote:
Thanks Bruce. I might have to build a merged tree because one of the changes from the review modified a header file introduced in the NFS tree.
I'll see how it goes, and see if I can avoid doing something that will make Linus yell at me :P
If its a syntactic conflict with an obvious resolution, just leave it (or maybe mention it in the pull request). It gives him something to do so he feels useful. :-)
/me notes that I did not see a conflict while merging the xfs tree in linux-next today ...
Signature · 2 lines
Cheers, Stephen Rothwell [email protected]
Re: [PATCH 19/20] xfs: implement pNFS export operations
Quoted 8 lines · Stephen Rothwell
Hi Dave,
Quoted 5 lines · On Fri, 13 Feb 2015 12:54:22 +1100 Dave Chinner <[email protected]> wrote:
Thanks Bruce. I might have to build a merged tree because one of the changes from the review modified a header file introduced in the NFS tree.
Quoted 6 lines
Quoted 2 lines
I'll see how it goes, and see if I can avoid doing something that will make Linus yell at me :P
If its a syntactic conflict with an obvious resolution, just leave it (or maybe mention it in the pull request). It gives him something to do so he feels useful. :-)
Heh.
It's not actually a conflict, though. The issue is that some of the review comments on the XFS side resulted in changes files introduced from the NFS tree. So the XFS side can't be merged without the NFS changes being present....
Quoted 2 lines
/me notes that I did not see a conflict while merging the xfs tree in linux-next today ...
Because I haven't pushed them yet ;)
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 20/20 xfs: recall pNFS layouts on conflicting access
In reply to a message not in the archiveRecall all outstanding pNFS layouts and truncates, writes and similar extent list modifying operations.
--- fs/xfs/xfs_file.c | 14 ++++++++++++-- fs/xfs/xfs_ioctl.c | 9 +++++++-- fs/xfs/xfs_iops.c | 11 ++++++++--- fs/xfs/xfs_pnfs.c | 31 +++++++++++++++++++++++++++++++ fs/xfs/xfs_pnfs.h | 7 +++++++ 5 files changed, 65 insertions(+), 7 deletions(-)
fs/xfs/xfs_file.c +12 -2
@@ -36,6 +36,7 @@#include "xfs_trace.h" #include "xfs_log.h" #include "xfs_icache.h" +#include "xfs_pnfs.h" #include <linux/aio.h> #include <linux/dcache.h>
@@ -554,6 +555,10 @@ restart:if (error)return error; + error =xfs_break_layouts (inode, iolock); +if (error) +return error; +/* * If the offset is beyond the size of the file, we need to zero any * blocks that fall between the existing EOF and the start of this
@@ -822,6 +827,7 @@ xfs_file_fallocate(struct xfs_inode *ip =XFS_I (inode);long error;enum xfs_prealloc_flags flags =0 ; +uint iolock =XFS_IOLOCK_EXCL ;loff_t new_size =0 ;if (!S_ISREG (inode->i_mode))
@@ -830,7 +836,11 @@ xfs_file_fallocate( FALLOC_FL_COLLAPSE_RANGE | FALLOC_FL_ZERO_RANGE))return -EOPNOTSUPP ; -xfs_ilock (ip,XFS_IOLOCK_EXCL ); +xfs_ilock (ip, iolock); + error =xfs_break_layouts (inode, &iolock); +if (error) + goto out_unlock; +if (mode &FALLOC_FL_PUNCH_HOLE ) { error =xfs_free_file_space (ip, offset, len);if (error)
@@ -894,7 +904,7 @@ xfs_file_fallocate( } out_unlock: -xfs_iunlock (ip,XFS_IOLOCK_EXCL ); +xfs_iunlock (ip, iolock);return error; }
fs/xfs/xfs_ioctl.c +7 -2
@@ -39,6 +39,7 @@#include "xfs_icache.h" #include "xfs_symlink.h" #include "xfs_trans.h" +#include "xfs_pnfs.h" #include <linux/capability.h> #include <linux/dcache.h>
@@ -608,6 +609,7 @@ xfs_ioc_space( {struct iattr iattr;enum xfs_prealloc_flags flags =0 ; +uint iolock =XFS_IOLOCK_EXCL ;int error;/*
@@ -636,7 +638,10 @@ xfs_ioc_space(if (error)return error; -xfs_ilock (ip,XFS_IOLOCK_EXCL ); +xfs_ilock (ip, iolock); + error =xfs_break_layouts (inode, &iolock); +if (error) + goto out_unlock;switch (bf->l_whence) {case 0 :/*SEEK_SET*/
@@ -725,7 +730,7 @@ xfs_ioc_space( error =xfs_update_prealloc_flags (ip, flags); out_unlock: -xfs_iunlock (ip,XFS_IOLOCK_EXCL ); +xfs_iunlock (ip, iolock);mnt_drop_write_file (filp);return error; }
fs/xfs/xfs_iops.c +8 -3
@@ -37,6 +37,7 @@#include "xfs_da_btree.h" #include "xfs_dir2.h" #include "xfs_trans_space.h" +#include "xfs_pnfs.h" #include <linux/capability.h> #include <linux/xattr.h>
@@ -970,9 +971,13 @@ xfs_vn_setattr(int error;if (iattr->ia_valid &ATTR_SIZE ) { -xfs_ilock (ip,XFS_IOLOCK_EXCL ); - error =xfs_setattr_size (ip, iattr); -xfs_iunlock (ip,XFS_IOLOCK_EXCL ); +uint iolock =XFS_IOLOCK_EXCL ; + +xfs_ilock (ip, iolock); + error =xfs_break_layouts (dentry->d_inode, &iolock); +if (!error) + error =xfs_setattr_size (ip, iattr); +xfs_iunlock (ip, iolock); }else { error =xfs_setattr_nonsize (ip, iattr,0 ); }
fs/xfs/xfs_pnfs.c +31 -0
@@ -17,6 +17,37 @@#include "xfs_shared.h" #include "xfs_pnfs.h" +/* +* Ensure that we do not have any outstanding pNFS layouts that can be +* used by clients to directly read from or write to this inode. +* This must be called before every operation that can remove blocks +* from the extent map. Additionally we call it during the write +* operation, where aren't concerned about exposing unallocated blocks +* but just want to provide basic synchronization between a local +* writer and pNFS clients. mmap writes would also benefit from +* this sort of synchronization, but due to the tricky locking +* rules in the page fault path we don't bother. +*/ +int +xfs_break_layouts ( +struct inode *inode, +uint *iolock) +{ +struct xfs_inode *ip =XFS_I (inode); +int error; + +ASSERT (xfs_isilocked (ip,XFS_IOLOCK_SHARED |XFS_IOLOCK_EXCL )); + +while ((error =break_layout (inode, false) == -EWOULDBLOCK )) { +xfs_iunlock (ip, *iolock); + error =break_layout (inode, true); + *iolock =XFS_IOLOCK_EXCL ; +xfs_ilock (ip, *iolock); + } + +return error; +} + int xfs_fs_get_uuid( struct super_block *sb,
fs/xfs/xfs_pnfs.h +7 -0
@@ -7,5 +7,12 @@ int xfs_fs_map_blocks(struct inode *inode, loff_t offset, u64 length, struct iomap *iomap, bool write, u32 *device_generation);int xfs_fs_commit_blocks (struct inode *inode,struct iomap *maps,int nr_maps,struct iattr *iattr); + +int xfs_break_layouts (struct inode *inode,uint *iolock); +#else +static inline int xfs_break_layouts (struct inode *inode,uint *iolock) +{ +return 0 ; +}#endif /* CONFIG_NFSD_PNFS */ #endif /* _XFS_PNFS_H */
Signature · 6 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-21-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 20/20] xfs: recall pNFS layouts on conflicting access
Quoted 4 lines · Christoph Hellwig
Recall all outstanding pNFS layouts and truncates, writes and similar extent list modifying operations.
Signed-off-by Christoph Hellwig <[email protected]>
This looks fine, assuming break_layout() doesn't require any other VFS inode locks to be held for serialisation.
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Christoph Hellwig
This series adds support for the pNFS operations in NFS v4.1, as well as a block layout driver that can export block based filesystems that implement a few additional export operations. Support for XFS is provided in this series, but other filesystems could be added easily.
The core pNFS code of course owns its heritage to the existing Linux pNFS server prototype, but except for a few bits and pieces in the XDR path nothing is left from it.
The design of this new pNFS server is fairly different from the old one - while the old one implemented very little semantics in nfsd and left almost everything to filesystems my implementation implements as much as possible in common nfsd code, then dispatches to a layout driver that still is part of nfsd and only then calls into the filesystem, thus keeping it free from intimate pNFS knowledge.
Quoted 8 lines
This version of the code has been rebased to the locks-for-3.20 tree which adds a new lock context structure to the inode. For now we still (ab)use the lease list like in the last version. Adding a pNFS-specific list would duplicate a lot of code without much benefit. But during this research I came up with way to associate a
nfs4_filewith a struct file at open time which should allow to greatly simplify the pNFS and delegation code. So stay tuned for some patches in this area!
Good, that could definitely use simplification.--b.
Quoted 14 lines
For now this also doesn't take errata 3901 for rfc 5661 into account yet and sticks to the verified errata. The changes in 3901 seem useful in the longer run one verified and will be implemented eventually. Note that the only existing pNFS block client (Linux) would not benefit from the longer layout lifetimes anyway.
More details are document in the individual patch descriptions and code comments.
This code is also available from:
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-2
Quoted 8 lines
Changes since V1:
- rebased to the locks-for-3.20 tree
- fixed a memory leak in thenfsd4_encode_getdeviceinfoerror path
- removed the always onelg_rocfield
- added a nopnfs export option
- use a synchronous transaction in layoutget to simplify recovery
- various XFS fixes pointed out by Dave
- various documentation typo fixes
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Christoph Hellwig
This series adds support for the pNFS operations in NFS v4.1, as well as a block layout driver that can export block based filesystems that implement a few additional export operations. Support for XFS is provided in this series, but other filesystems could be added easily.
The core pNFS code of course owns its heritage to the existing Linux pNFS server prototype, but except for a few bits and pieces in the XDR path nothing is left from it.
The design of this new pNFS server is fairly different from the old one - while the old one implemented very little semantics in nfsd and left almost everything to filesystems my implementation implements as much as possible in common nfsd code, then dispatches to a layout driver that still is part of nfsd and only then calls into the filesystem, thus keeping it free from intimate pNFS knowledge.
Quoted 2 lines
This version of the code has been rebased to the locks-for-3.20 tree which adds a new lock context structure to the inode.
By the way, Jeff, do you consider the lock changes are ready? Would you mind if I pulled or cherry-picked them into the nfsd tree just to simplify merging this stuff for 3.20? (Assuming it's going to be ready.)
--b.
Quoted 21 lines
For now
we still (ab)use the lease list like in the last version. Adding a pNFS-specific list would duplicate a lot of code without much benefit. But during this research I came up with way to associate anfs4_filewith a struct file at open time which should allow to greatly simplify the pNFS and delegation code. So stay tuned for some patches in this area!For now this also doesn't take errata 3901 for rfc 5661 into account yet and sticks to the verified errata. The changes in 3901 seem useful in the longer run one verified and will be implemented eventually. Note that the only existing pNFS block client (Linux) would not benefit from the longer layout lifetimes anyway.
More details are document in the individual patch descriptions and code comments.
This code is also available from:
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-2
Quoted 8 lines
Changes since V1:
- rebased to the locks-for-3.20 tree
- fixed a memory leak in thenfsd4_encode_getdeviceinfoerror path
- removed the always onelg_rocfield
- added a nopnfs export option
- use a synchronous transaction in layoutget to simplify recovery
- various XFS fixes pointed out by Dave
- various documentation typo fixes
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Christoph Hellwig
reviewerQuoted 4 lines · J. Bruce Fields
By the way, Jeff, do you consider the lock changes are ready? Would you mind if I pulled or cherry-picked them into the nfsd tree just to simplify merging this stuff for 3.20? (Assuming it's going to be ready.)
I did the rebase because he mentioned he wanted to get it into 3.20.
Note that the conflicts were fairly trivial, so it's something we could just ignore and let Linus handle the merge in the worst case.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 20 lines · On Thu, 22 Jan 2015 15:06:18 -0500 "J. Bruce Fields"…
Quoted 16 lines · Christoph Hellwig
This series adds support for the pNFS operations in NFS v4.1, as well as a block layout driver that can export block based filesystems that implement a few additional export operations. Support for XFS is provided in this series, but other filesystems could be added easily.
The core pNFS code of course owns its heritage to the existing Linux pNFS server prototype, but except for a few bits and pieces in the XDR path nothing is left from it.
The design of this new pNFS server is fairly different from the old one - while the old one implemented very little semantics in nfsd and left almost everything to filesystems my implementation implements as much as possible in common nfsd code, then dispatches to a layout driver that still is part of nfsd and only then calls into the filesystem, thus keeping it free from intimate pNFS knowledge.
Quoted 2 lines
This version of the code has been rebased to the locks-for-3.20 tree which adds a new lock context structure to the inode.
Quoted 7 lines
By the way, Jeff, do you consider the lock changes are ready? Would you mind if I pulled or cherry-picked them into the nfsd tree just to simplify merging this stuff for 3.20? (Assuming it's going to be ready.)
--b.
Sure, that'd be fine.
The latest version of that set seems to be OK (tip commit == 8116bf4cb62d33) and has been sitting in linux-next for a while now. I'll let you know if I get any reports of problems in that area between now and the merge window though.
Thanks,
Jeff
Quoted 29 lines
Quoted 29 lines
For now
we still (ab)use the lease list like in the last version. Adding a pNFS-specific list would duplicate a lot of code without much benefit. But during this research I came up with way to associate anfs4_filewith a struct file at open time which should allow to greatly simplify the pNFS and delegation code. So stay tuned for some patches in this area!For now this also doesn't take errata 3901 for rfc 5661 into account yet and sticks to the verified errata. The changes in 3901 seem useful in the longer run one verified and will be implemented eventually. Note that the only existing pNFS block client (Linux) would not benefit from the longer layout lifetimes anyway.
More details are document in the individual patch descriptions and code comments.
This code is also available from:
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-2Changes since V1:
- rebased to the locks-for-3.20 tree
- fixed a memory leak in thenfsd4_encode_getdeviceinfoerror path
- removed the always onelg_rocfield
- added a nopnfs export option
- use a synchronous transaction in layoutget to simplify recovery
- various XFS fixes pointed out by Dave
- various documentation typo fixes
Signature · 5 lines
Jeff Layton <jlayton-7I+n7zu2hftEKMMhf/[email protected]> -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Christoph Hellwig
reviewerPATCH 4/20 nfsd: factor out a helper to decode nfstime4 values
In reply to Christoph Hellwig--- fs/nfsd/nfs4xdr.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)
fs/nfsd/nfs4xdr.c +26 -17
@@ -234,6 +234,26 @@ static char *savemem(struct nfsd4_compoundargs *argp, __be32 *p, int nbytes)return ret; } +/* +* We require the high 32 bits of 'seconds' to be 0, and +* we ignore all 32 bits of 'nseconds'. +*/ +static __be32 +nfsd4_decode_time (struct nfsd4_compoundargs *argp,struct timespec *tv) +{ +DECODE_HEAD ; +u64 sec; + +READ_BUF (12 ); + p =xdr_decode_hyper (p, &sec); + tv->tv_sec = sec; + tv->tv_nsec =be32_to_cpup (p++); +if (tv->tv_nsec >= (u32 )1000000000 ) +return nfserr_inval; + +DECODE_TAIL ; +} +static __be32 nfsd4_decode_bitmap (struct nfsd4_compoundargs *argp,u32 *bmval) {
@@ -267,7 +287,6 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, {int expected_len, len =0 ;u32 dummy32; -u64 sec;char *buf;DECODE_HEAD ;
@@ -358,15 +377,10 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, dummy32 =be32_to_cpup (p++);switch (dummy32) {case NFS4_SET_TO_CLIENT_TIME : -/* We require the high 32 bits of 'seconds' to be 0, and we ignore -all 32 bits of 'nseconds'. */ -READ_BUF (12 ); len +=12 ; - p =xdr_decode_hyper (p, &sec); - iattr->ia_atime.tv_sec = (time_t )sec; - iattr->ia_atime.tv_nsec =be32_to_cpup (p++); -if (iattr->ia_atime.tv_nsec >= (u32 )1000000000 ) -return nfserr_inval; + status =nfsd4_decode_time (argp, &iattr->ia_atime); +if (status) +return status; iattr->ia_valid |= (ATTR_ATIME |ATTR_ATIME_SET );break ;case NFS4_SET_TO_SERVER_TIME :
@@ -382,15 +396,10 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, dummy32 =be32_to_cpup (p++);switch (dummy32) {case NFS4_SET_TO_CLIENT_TIME : -/* We require the high 32 bits of 'seconds' to be 0, and we ignore -all 32 bits of 'nseconds'. */ -READ_BUF (12 ); len +=12 ; - p =xdr_decode_hyper (p, &sec); - iattr->ia_mtime.tv_sec = sec; - iattr->ia_mtime.tv_nsec =be32_to_cpup (p++); -if (iattr->ia_mtime.tv_nsec >= (u32 )1000000000 ) -return nfserr_inval; + status =nfsd4_decode_time (argp, &iattr->ia_mtime); +if (status) +return status; iattr->ia_valid |= (ATTR_MTIME |ATTR_MTIME_SET );break ;case NFS4_SET_TO_SERVER_TIME :
Signature · 1 line
1.9.1
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-5-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 04/20] nfsd: factor out a helper to decode nfstime4 values
I'll go ahead and take this one.--b.
Quoted 84 lines · Christoph Hellwig
Signed-off-by Christoph Hellwig <[email protected]>--- fs/nfsd/nfs4xdr.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)fs/nfsd/nfs4xdr.c +26
-17@@ -234,6 +234,26 @@ static char *savemem(struct nfsd4_compoundargs *argp, __be32 *p, int nbytes)return ret; } +/* +* We require the high 32 bits of 'seconds' to be 0, and +* we ignore all 32 bits of 'nseconds'. +*/ +static __be32 +nfsd4_decode_time (struct nfsd4_compoundargs *argp,struct timespec *tv) +{ +DECODE_HEAD ; +u64 sec; + +READ_BUF (12 ); + p =xdr_decode_hyper (p, &sec); + tv->tv_sec = sec; + tv->tv_nsec =be32_to_cpup (p++); +if (tv->tv_nsec >= (u32 )1000000000 ) +return nfserr_inval; + +DECODE_TAIL ; +} +static __be32 nfsd4_decode_bitmap (struct nfsd4_compoundargs *argp,u32 *bmval) {@@ -267,7 +287,6 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, {int expected_len, len =0 ;u32 dummy32; -u64 sec;char *buf;DECODE_HEAD ;@@ -358,15 +377,10 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, dummy32 =be32_to_cpup (p++);switch (dummy32) {case NFS4_SET_TO_CLIENT_TIME : -/* We require the high 32 bits of 'seconds' to be 0, and we ignore -all 32 bits of 'nseconds'. */ -READ_BUF (12 ); len +=12 ; - p =xdr_decode_hyper (p, &sec); - iattr->ia_atime.tv_sec = (time_t )sec; - iattr->ia_atime.tv_nsec =be32_to_cpup (p++); -if (iattr->ia_atime.tv_nsec >= (u32 )1000000000 ) -return nfserr_inval; + status =nfsd4_decode_time (argp, &iattr->ia_atime); +if (status) +return status; iattr->ia_valid |= (ATTR_ATIME |ATTR_ATIME_SET );break ;case NFS4_SET_TO_SERVER_TIME :@@ -382,15 +396,10 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, dummy32 =be32_to_cpup (p++);switch (dummy32) {case NFS4_SET_TO_CLIENT_TIME : -/* We require the high 32 bits of 'seconds' to be 0, and we ignore -all 32 bits of 'nseconds'. */ -READ_BUF (12 ); len +=12 ; - p =xdr_decode_hyper (p, &sec); - iattr->ia_mtime.tv_sec = sec; - iattr->ia_mtime.tv_nsec =be32_to_cpup (p++); -if (iattr->ia_mtime.tv_nsec >= (u32 )1000000000 ) -return nfserr_inval; + status =nfsd4_decode_time (argp, &iattr->ia_mtime); +if (status) +return status; iattr->ia_valid |= (ATTR_MTIME |ATTR_MTIME_SET );break ;case NFS4_SET_TO_SERVER_TIME :Signature · 1 line
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Christoph Hellwig
reviewerPATCH 5/20 nfsd: move nfsd_fh_match to nfsfh.h
In reply to Christoph HellwigThe pnfs code will need it too. Also remove the nfsd_ prefix to match the other filehandle helpers in that file.
--- fs/nfsd/nfs4state.c | 12 ++---------- fs/nfsd/nfsfh.h | 9 +++++++++ 2 files changed, 11 insertions(+), 10 deletions(-)
fs/nfsd/nfs4state.c +2 -10
@@ -408,14 +408,6 @@ static unsigned int file_hashval(struct knfsd_fh *fh)return nfsd_fh_hashval (fh) & (FILE_HASH_SIZE -1 ); } -static bool nfsd_fh_match (struct knfsd_fh *fh1,struct knfsd_fh *fh2) -{ -return fh1->fh_size == fh2->fh_size && - !memcmp (fh1->fh_base.fh_pad, - fh2->fh_base.fh_pad, - fh1->fh_size); -} -static struct hlist_head file_hashtbl[FILE_HASH_SIZE ]; static void
@@ -3300,7 +3292,7 @@ find_file_locked(struct knfsd_fh *fh, unsigned int hashval)struct nfs4_file *fp;hlist_for_each_entry_rcu (fp, &file_hashtbl[hashval], fi_hash) { -if (nfsd_fh_match (&fp->fi_fhandle, fh)) { +if (fh_match (&fp->fi_fhandle, fh)) {if (atomic_inc_not_zero (&fp->fi_ref))return fp; }
@@ -4295,7 +4287,7 @@ laundromat_main(struct work_struct *laundry)static inline __be32 nfs4_check_fh (struct svc_fh *fhp,struct nfs4_ol_stateid *stp) { -if (!nfsd_fh_match (&fhp->fh_handle, &stp->st_stid.sc_file->fi_fhandle)) +if (!fh_match (&fhp->fh_handle, &stp->st_stid.sc_file->fi_fhandle))return nfserr_bad_stateid;return nfs_ok; }
fs/nfsd/nfsfh.h +9 -0
@@ -187,6 +187,15 @@ fh_init(struct svc_fh *fhp, int maxsize)return fhp; } +static inline bool fh_match (struct knfsd_fh *fh1,struct knfsd_fh *fh2) +{ +if (fh1->fh_size != fh2->fh_size) +return false; +if (memcmp (fh1->fh_base.fh_pad, fh2->fh_base.fh_pad, fh1->fh_size) !=0 ) +return false; +return true; +} +#ifdef CONFIG_NFSD_V3 /* * The wcc data stored in current_fh should be cleared
Signature · 6 lines
1.9.1 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Christoph Hellwig
reviewerPATCH 9/20 nfsd: make find_any_file available outside nfs4state.c
In reply to Christoph Hellwig--- fs/nfsd/nfs4state.c | 2 +- fs/nfsd/state.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
fs/nfsd/nfs4state.c +1 -1
@@ -352,7 +352,7 @@ find_readable_file(struct nfs4_file *f)return ret; } -static struct file * +struct file *find_any_file (struct nfs4_file *f) {struct file *ret;
fs/nfsd/state.h +1 -0
@@ -579,6 +579,7 @@ static inline void get_nfs4_file(struct nfs4_file *fi) {atomic_inc (&fi->fi_ref); } +struct file *find_any_file (struct nfs4_file *f);/* grace period management */ void nfsd4_end_grace (struct nfsd_net *nn);
Signature · 1 line
1.9.1
Christoph Hellwig
reviewerPATCH 12/20 nfsd: update documentation for pNFS support
In reply to Christoph Hellwig--- Documentation/filesystems/nfs/nfs41-server.txt | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-)
Documentation/filesystems/nfs/nfs41-server.txt +8 -15
@@ -24,11 +24,6 @@ focuses on the mandatory-to-implement NFSv4.1 Sessions, providing "exactly once" semantics and better control and throttling of the resources allocated for each client. -Other NFSv4.1 features, Parallel NFS operations in particular, -are still under development out of tree. -See http://wiki.linux-nfs.org/wiki/index.php/PNFS_prototype_design -for more information. - The table below, taken from the NFSv4.1 document, lists the operations that are mandatory to implement (REQ), optional (OPT), and NFSv4.0 operations that are required not to implement (MNI)
@@ -43,9 +38,7 @@ The OPTIONAL features identified and their abbreviations are as follows: The following abbreviations indicate the linux server implementation status. I Implemented NFSv4.1 operations. NS Not Supported. - NS* unimplemented optional feature. - P pNFS features implemented out of tree. - PNS pNFS features that are not supported yet (out of tree). + NS* Unimplemented optional feature. Operations
@@ -70,13 +63,13 @@ I | DESTROY_SESSION | REQ | | Section 18.37 | I | EXCHANGE_ID | REQ | | Section 18.35 | I | FREE_STATEID | REQ | | Section 18.38 | | GETATTR | REQ | | Section 18.7 | -P | GETDEVICEINFO | OPT | pNFS (REQ) | Section 18.40 | -P | GETDEVICELIST | OPT | pNFS (OPT) | Section 18.41 | +I | GETDEVICEINFO | OPT | pNFS (REQ) | Section 18.40 | +NS*| GETDEVICELIST | OPT | pNFS (OPT) | Section 18.41 | | GETFH | REQ | | Section 18.8 | NS*| GET_DIR_DELEGATION | OPT | DDELG (REQ) | Section 18.39 | -P | LAYOUTCOMMIT | OPT | pNFS (REQ) | Section 18.42 | -P | LAYOUTGET | OPT | pNFS (REQ) | Section 18.43 | -P | LAYOUTRETURN | OPT | pNFS (REQ) | Section 18.44 | +I | LAYOUTCOMMIT | OPT | pNFS (REQ) | Section 18.42 | +I | LAYOUTGET | OPT | pNFS (REQ) | Section 18.43 | +I | LAYOUTRETURN | OPT | pNFS (REQ) | Section 18.44 | | LINK | OPT | | Section 18.9 | | LOCK | REQ | | Section 18.10 | | LOCKT | REQ | | Section 18.11 |
@@ -122,9 +115,9 @@ Callback Operations | | MNI | or OPT) | | +-------------------------+-----------+-------------+---------------+ | CB_GETATTR | OPT | FDELG (REQ) | Section 20.1 | -P | CB_LAYOUTRECALL | OPT | pNFS (REQ) | Section 20.3 | +I | CB_LAYOUTRECALL | OPT | pNFS (REQ) | Section 20.3 | NS*| CB_NOTIFY | OPT | DDELG (REQ) | Section 20.4 | -P | CB_NOTIFY_DEVICEID | OPT | pNFS (OPT) | Section 20.12 | +NS*| CB_NOTIFY_DEVICEID | OPT | pNFS (OPT) | Section 20.12 | NS*| CB_NOTIFY_LOCK | OPT | | Section 20.11 | NS*| CB_PUSH_DELEG | OPT | FDELG (OPT) | Section 20.5 | | CB_RECALL | OPT | FDELG, | Section 20.2 |
Signature · 1 line
1.9.1
Christoph Hellwig
reviewerPATCH 13/20 nfsd: add trace events
In reply to Christoph HellwigFor now just a few simple events to trace the layout stateid lifetime, but these already were enough to find several bugs in the Linux client layout stateid handling.
--- fs/nfsd/Makefile | 7 ++++++- fs/nfsd/nfs4layouts.c | 16 ++++++++++++++- fs/nfsd/nfs4proc.c | 6 +++++- fs/nfsd/trace.c | 5 +++++ fs/nfsd/trace.h | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 85 insertions(+), 3 deletions(-) create mode 100644 fs/nfsd/trace.c create mode 100644 fs/nfsd/trace.h
fs/nfsd/Makefile +6 -1
@@ -2,9 +2,14 @@ # Makefile for the Linux nfs server # +ccflags-y += -I$(src) # needed for trace events + obj-$(CONFIG_NFSD) += nfsd.o -nfsd-y := nfssvc.o nfsctl.o nfsproc.o nfsfh.o vfs.o \ +# this one should be compiled first, as the tracing macros can easily blow up +nfsd-y += trace.o + +nfsd-y += nfssvc.o nfsctl.o nfsproc.o nfsfh.o vfs.o \ export.o auth.o lockd.o nfscache.o nfsxdr.o stats.o nfsd-$(CONFIG_NFSD_FAULT_INJECTION) += fault_inject.o nfsd-$(CONFIG_NFSD_V2_ACL) += nfs2acl.o
fs/nfsd/nfs4layouts.c +15 -1
@@ -9,6 +9,7 @@#include "pnfs.h" #include "netns.h" +#include "trace.h" #define NFSDDBG_FACILITY NFSDDBG_PNFS
@@ -126,6 +127,8 @@ nfsd4_free_layout_stateid(struct nfs4_stid *stid)struct nfs4_client *clp = ls->ls_stid.sc_client;struct nfs4_file *fp = ls->ls_stid.sc_file; +trace_layoutstate_free (&ls->ls_stid.sc_stateid); +spin_lock (&clp->cl_lock);list_del_init (&ls->ls_perclnt);spin_unlock (&clp->cl_lock);
@@ -216,6 +219,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,list_add (&ls->ls_perfile, &fp->fi_lo_states);spin_unlock (&fp->fi_lock); +trace_layoutstate_alloc (&ls->ls_stid.sc_stateid);return ls; }
@@ -281,6 +285,8 @@ nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)if (list_empty (&ls->ls_layouts)) goto out_unlock; +trace_layout_recall (&ls->ls_stid.sc_stateid); +atomic_inc (&ls->ls_stid.sc_count);update_stateid (&ls->ls_stid.sc_stateid);memcpy (&ls->ls_recall_sid, &ls->ls_stid.sc_stateid,sizeof (stateid_t));
@@ -455,8 +461,10 @@ nfsd4_return_file_layouts(struct svc_rqst *rqstp, nfserr =nfsd4_preprocess_layout_stateid (rqstp, cstate, &lrp->lr_sid, false, lrp->lr_layout_type, &ls); -if (nfserr) +if (nfserr) { +trace_layout_return_lookup_fail (&lrp->lr_sid);return nfserr; + }spin_lock (&ls->ls_lock);list_for_each_entry_safe (lp, n, &ls->ls_layouts, lo_perstate) {
@@ -473,6 +481,7 @@ nfsd4_return_file_layouts(struct svc_rqst *rqstp, } lrp->lrs_present =1 ; }else { +trace_layoutstate_unhash (&ls->ls_stid.sc_stateid);nfs4_unhash_stid (&ls->ls_stid); lrp->lrs_present =0 ; }
@@ -571,6 +580,8 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls)rpc_ntop ((struct sockaddr *)&clp->cl_addr, addr_str,sizeof (addr_str)); +nfsd4_cb_layout_fail (ls); +printk (KERN_WARNING "nfsd: client %s failed to respond to layout recall. " " Fencing..\n" , addr_str);
@@ -598,6 +609,7 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)case 0 :return 1 ;case -NFS4ERR_NOMATCHING_LAYOUT : +trace_layout_recall_done (&ls->ls_stid.sc_stateid); task->tk_status =0 ;return 1 ;case -NFS4ERR_DELAY :
@@ -625,6 +637,8 @@ nfsd4_cb_layout_release(struct nfsd4_callback *cb) container_of(cb, struct nfs4_layout_stateid, ls_recall);LIST_HEAD (reaplist); +trace_layout_recall_release (&ls->ls_stid.sc_stateid); +nfsd4_return_all_layouts (ls, &reaplist);nfsd4_free_layouts (&reaplist);nfs4_put_stid (&ls->ls_stid);
fs/nfsd/nfs4proc.c +5 -1
@@ -44,6 +44,7 @@#include "netns.h" #include "acl.h" #include "pnfs.h" +#include "trace.h" #ifdef CONFIG_NFSD_V4_SECURITY_LABEL #include <linux/security.h>
@@ -1298,8 +1299,10 @@ nfsd4_layoutget(struct svc_rqst *rqstp, nfserr =nfsd4_preprocess_layout_stateid (rqstp, cstate, &lgp->lg_sid, true, lgp->lg_layout_type, &ls); -if (nfserr) +if (nfserr) { +trace_layout_get_lookup_fail (&lgp->lg_sid); goto out; + } nfserr = nfserr_recallconflict;if (atomic_read (&ls->ls_stid.sc_file->fi_lo_recalls))
@@ -1359,6 +1362,7 @@ nfsd4_layoutcommit(struct svc_rqst *rqstp, false, lcp->lc_layout_type, &ls);if (nfserr) { +trace_layout_commit_lookup_fail (&lcp->lc_sid);/* fixup error code as per RFC5661 */ if (nfserr == nfserr_bad_stateid) nfserr = nfserr_badlayout;
fs/nfsd/trace.c new file +5 -0
@@ -0,0 +1,5 @@ + +#include "state.h" + +#define CREATE_TRACE_POINTS +#include "trace.h"
fs/nfsd/trace.h new file +54 -0
54 lines not on this page · raw message
Signature · 1 line
1.9.1
Christoph Hellwig
reviewerPATCH 15/20 nfsd: pNFS block layout driver
In reply to Christoph HellwigAdd a small shim between core nfsd and filesystems to translate the somewhat cumbersome pNFS data structures and semantics to something more palatable for Linux filesystems.
Thanks to Rick McNeal for the old prototype pNFS blocklayout server code, which gave a lot of inspiration to this version even if no code is left from it.
--- .../filesystems/nfs/pnfs-block-server.txt | 37 ++++ fs/nfsd/Makefile | 2 +- fs/nfsd/blocklayout.c | 190 +++++++++++++++++++++ fs/nfsd/blocklayoutxdr.c | 157 +++++++++++++++++ fs/nfsd/blocklayoutxdr.h | 62 +++++++ fs/nfsd/nfs4layouts.c | 8 + fs/nfsd/pnfs.h | 1 + 7 files changed, 456 insertions(+), 1 deletion(-) create mode 100644 Documentation/filesystems/nfs/pnfs-block-server.txt create mode 100644 fs/nfsd/blocklayout.c create mode 100644 fs/nfsd/blocklayoutxdr.c create mode 100644 fs/nfsd/blocklayoutxdr.h
Documentation/filesystems/nfs/pnfs-block-server.txt new file +37 -0
@@ -0,0 +1,37 @@ +pNFS block layout server user guide + +The Linux NFS server now supports the pNFS block layout extension. In this +case the NFS server acts as Metadata Server (MDS) for pNFS, which in addition +to handling all the metadata access to the NFS export also hands out layouts +to the clients to directly access the underlying block devices that are +shared with the client. + +To use pNFS block layouts with with the Linux NFS server the exported file +system needs to support the pNFS block layouts (currently just XFS), and the +file system must sit on shared storage (typically iSCSI) that is accessible +to the clients in addition to the MDS. As of now the file system needs to +sit directly on the exported volume, striping or concatenation of +volumes on the MDS and clients is not supported yet. + +On the server, pNFS block volume support is automatically if the file system +support it. On the client make sure the kernel has the CONFIG_PNFS_BLOCK +option enabled, the blkmapd daemon from nfs-utils is running, and the +file system is mounted using the NFSv4.1 protocol version (mount -o vers=4.1). + +If the nfsd server needs to fence a non-responding client it calls +/sbin/nfsd-recall-failed with the first argument set to the IP address of +the client, and the second argument set to the device node without the /dev +prefix for the file system to be fenced. Below is an example file that shows +how to translate the device into a serial number from SCSI EVPD 0x80: + +cat > /sbin/nfsd-recall-failed << EOF +#!/bin/sh + +CLIENT="$1" +DEV="/dev/$2" +EVPD=`sg_inq --page=0x80 ${DEV} | \ + grep "Unit serial number:" | \ + awk -F ': ' '{print $2}'` + +echo "fencing client ${CLIENT} serial ${EVPD}" >> /var/log/pnfsd-fence.log +EOF
fs/nfsd/Makefile +1 -1
@@ -17,4 +17,4 @@ nfsd-$(CONFIG_NFSD_V3) += nfs3proc.o nfs3xdr.o nfsd-$(CONFIG_NFSD_V3_ACL) += nfs3acl.o nfsd-$(CONFIG_NFSD_V4) += nfs4proc.o nfs4xdr.o nfs4state.o nfs4idmap.o \ nfs4acl.o nfs4callback.o nfs4recover.o -nfsd-$(CONFIG_NFSD_PNFS) += nfs4layouts.o +nfsd-$(CONFIG_NFSD_PNFS) += nfs4layouts.o blocklayout.o blocklayoutxdr.o
fs/nfsd/blocklayout.c new file +190 -0
190 lines not on this page · raw message
fs/nfsd/blocklayoutxdr.c new file +157 -0
157 lines not on this page · raw message
fs/nfsd/blocklayoutxdr.h new file +62 -0
62 lines not on this page · raw message
fs/nfsd/nfs4layouts.c +8 -0
@@ -26,6 +26,7 @@ static struct nfsd4_callback_ops nfsd4_cb_layout_ops;static const struct lock_manager_operations nfsd4_layouts_lm_ops;const struct nfsd4_layout_ops *nfsd4_layout_ops[LAYOUT_TYPE_MAX ] = { + [LAYOUT_BLOCK_VOLUME ] = &bl_layout_ops, };/* pNFS device ID to export fsid mapping */
@@ -116,8 +117,15 @@ nfsd4_set_deviceid(struct nfsd4_deviceid *id, const struct svc_fh *fhp,void nfsd4_setup_layout_type (struct svc_export *exp) { +struct super_block *sb = exp->ex_path.mnt->mnt_sb; +if (exp->ex_flags &NFSEXP_NOPNFS )return ; + +if (sb->s_export_op->get_uuid && + sb->s_export_op->map_blocks && + sb->s_export_op->commit_blocks) + exp->ex_layout_type =LAYOUT_BLOCK_VOLUME ; } static void
fs/nfsd/pnfs.h +1 -0
@@ -34,6 +34,7 @@ struct nfsd4_layout_ops { };extern const struct nfsd4_layout_ops *nfsd4_layout_ops[]; +extern const struct nfsd4_layout_ops bl_layout_ops; __be32 nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, stateid_t *stateid,
Signature · 1 line
1.9.1
Christoph Hellwig
reviewerPATCH 18/20 xfs: factor out a xfs_update_prealloc_flags() helper
In reply to Christoph HellwigThis logic is duplicated in xfs_file_fallocate and xfs_ioc_space, and we'll need another copy of it for pNFS block support.
--- fs/xfs/xfs_file.c | 64 +++++++++++++++++++++++++++++++++++------------------- fs/xfs/xfs_inode.h | 9 ++++++++ fs/xfs/xfs_ioctl.c | 50 ++++++++++-------------------------------- 3 files changed, 63 insertions(+), 60 deletions(-)
fs/xfs/xfs_file.c +42 -22
@@ -127,6 +127,42 @@ xfs_iozero(return (-status); } +int +xfs_update_prealloc_flags ( +struct xfs_inode *ip, +enum xfs_prealloc_flags flags) +{ +struct xfs_trans *tp; +int error; + + tp =xfs_trans_alloc (ip->i_mount,XFS_TRANS_WRITEID ); + error =xfs_trans_reserve (tp, &M_RES (ip->i_mount)->tr_writeid,0 ,0 ); +if (error) { +xfs_trans_cancel (tp,0 ); +return error; + } + +xfs_ilock (ip,XFS_ILOCK_EXCL ); +xfs_trans_ijoin (tp, ip,XFS_ILOCK_EXCL ); + +if (!(flags &XFS_PREALLOC_INVISIBLE )) { + ip->i_d.di_mode &= ~S_ISUID ; +if (ip->i_d.di_mode &S_IXGRP ) + ip->i_d.di_mode &= ~S_ISGID ; +xfs_trans_ichgtime (tp, ip,XFS_ICHGTIME_MOD |XFS_ICHGTIME_CHG ); + } + +if (flags &XFS_PREALLOC_SET ) + ip->i_d.di_flags |=XFS_DIFLAG_PREALLOC ; +if (flags &XFS_PREALLOC_CLEAR ) + ip->i_d.di_flags &= ~XFS_DIFLAG_PREALLOC ; + +xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); +if (flags &XFS_PREALLOC_SYNC ) +xfs_trans_set_sync (tp); +return xfs_trans_commit (tp,0 ); +} +/* * Fsync operations on directories are much simpler than on regular files, * as there is no file data to flush, and thus also no need for explicit
@@ -784,8 +820,8 @@ xfs_file_fallocate( {struct inode *inode =file_inode (file);struct xfs_inode *ip =XFS_I (inode); -struct xfs_trans *tp;long error; +enum xfs_prealloc_flags flags =0 ;loff_t new_size =0 ;if (!S_ISREG (inode->i_mode))
@@ -822,6 +858,8 @@ xfs_file_fallocate(if (error) goto out_unlock; }else { + flags |=XFS_PREALLOC_SET ; +if (!(mode &FALLOC_FL_KEEP_SIZE ) && offset + len >i_size_read (inode)) { new_size = offset + len;
@@ -839,28 +877,10 @@ xfs_file_fallocate( goto out_unlock; } - tp =xfs_trans_alloc (ip->i_mount,XFS_TRANS_WRITEID ); - error =xfs_trans_reserve (tp, &M_RES (ip->i_mount)->tr_writeid,0 ,0 ); -if (error) { -xfs_trans_cancel (tp,0 ); - goto out_unlock; - } - -xfs_ilock (ip,XFS_ILOCK_EXCL ); -xfs_trans_ijoin (tp, ip,XFS_ILOCK_EXCL ); - ip->i_d.di_mode &= ~S_ISUID ; -if (ip->i_d.di_mode &S_IXGRP ) - ip->i_d.di_mode &= ~S_ISGID ; - -if (!(mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_COLLAPSE_RANGE ))) - ip->i_d.di_flags |=XFS_DIFLAG_PREALLOC ; - -xfs_trans_ichgtime (tp, ip,XFS_ICHGTIME_MOD |XFS_ICHGTIME_CHG ); -xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); -if (file->f_flags &O_DSYNC ) -xfs_trans_set_sync (tp); - error =xfs_trans_commit (tp,0 ); + flags |=XFS_PREALLOC_SYNC ; + + error =xfs_update_prealloc_flags (ip, flags);if (error) goto out_unlock;
fs/xfs/xfs_inode.h +9 -0
@@ -377,6 +377,15 @@ int xfs_droplink(struct xfs_trans *, struct xfs_inode *);int xfs_bumplink (struct xfs_trans *,struct xfs_inode *);/* from xfs_file.c */ +enum xfs_prealloc_flags { +XFS_PREALLOC_SET = (1 <<1 ), +XFS_PREALLOC_CLEAR = (1 <<2 ), +XFS_PREALLOC_SYNC = (1 <<3 ), +XFS_PREALLOC_INVISIBLE = (1 <<4 ), +}; + +int xfs_update_prealloc_flags (struct xfs_inode *, +enum xfs_prealloc_flags );int xfs_zero_eof (struct xfs_inode *,xfs_off_t ,xfs_fsize_t );int xfs_iozero (struct xfs_inode *,loff_t ,size_t );
fs/xfs/xfs_ioctl.c +12 -38
100 lines not on this page · raw message
Signature · 1 line
1.9.1
A message that is not in the archive (sent off-list or not yet received): 1421925006-24231-19-git-send-email-hch-jcswGhMUV9g@public.gmane.org
Re: [PATCH 18/20] xfs: factor out a xfs_update_prealloc_flags() helper
Quoted 4 lines · Christoph Hellwig
This logic is duplicated in
xfs_file_fallocateandxfs_ioc_space, and we'll need another copy of it for pNFS block support.Signed-off-by Christoph Hellwig <[email protected]>
This stands alone, so I'm pulling it into the XFS tree without waiting for the rest of the patch set.
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Christoph-
Very nice, and welcome, work.
Quoted 4 lines · Christoph Hellwig
This series adds support for the pNFS operations in NFS v4.1, as well as a block layout driver that can export block based filesystems that implement a few additional export operations. Support for XFS is provided in this series, but other filesystems could be added easily.
I’m still not clear on what layout types are supported. Clearly the block layout type is supported. But you mention XFS here. Does that mean file layouts are also supported now? If so, can you briefly describe the architecture and any current limitations?
Did I miss documentation for how administrators construct layouts?
Quoted 42 lines
The core pNFS code of course owns its heritage to the existing Linux pNFS server prototype, but except for a few bits and pieces in the XDR path nothing is left from it.
The design of this new pNFS server is fairly different from the old one - while the old one implemented very little semantics in nfsd and left almost everything to filesystems my implementation implements as much as possible in common nfsd code, then dispatches to a layout driver that still is part of nfsd and only then calls into the filesystem, thus keeping it free from intimate pNFS knowledge.
This version of the code has been rebased to the locks-for-3.20 tree which adds a new lock context structure to the inode. For now we still (ab)use the lease list like in the last version. Adding a pNFS-specific list would duplicate a lot of code without much benefit. But during this research I came up with way to associate a
nfs4_filewith a struct file at open time which should allow to greatly simplify the pNFS and delegation code. So stay tuned for some patches in this area!For now this also doesn't take errata 3901 for rfc 5661 into account yet and sticks to the verified errata. The changes in 3901 seem useful in the longer run one verified and will be implemented eventually. Note that the only existing pNFS block client (Linux) would not benefit from the longer layout lifetimes anyway.
More details are document in the individual patch descriptions and code comments.
This code is also available from:
git://git.infradead.org/users/hch/pnfs.git pnfsd-for-3.20-2Changes since V1:
- rebased to the locks-for-3.20 tree
- fixed a memory leak in thenfsd4_encode_getdeviceinfoerror path
- removed the always onelg_rocfield
- added a nopnfs export option
- use a synchronous transaction in layoutget to simplify recovery
- various XFS fixes pointed out by Dave
- various documentation typo fixes
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Christoph Hellwig
reviewerQuoted 6 lines · Chuck Lever
I’m still not clear on what layout types are supported. Clearly the block layout type is supported. But you mention XFS here. Does that mean file layouts are also supported now? If so, can you briefly describe the architecture and any current limitations?
Did I miss documentation for how administrators construct layouts?
Only block layouts are supporte at the moment. They are automatically
enabled when an XFS filesystem is exported. There is no requirement
for the administrator to construct in layout, the only requirement for working block layouts is shared storage that is accessible to the NFS Server / MDS and the clients that wish to use pNFS‥
If you are interested in file layout support porting over the existing file layout drivers from the old pNFS server tree should only be a moderate effort, and at least the lexp code would be grately simplified by the common recall infrastructure in this series. --
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html