Fix NFS commit to DS
Subsystems (2)
[email protected]
maintainercover Fix NFS commit to DS
From: Trond Myklebust <[email protected]>
The commit to DS code is currently broken in the face of layout segments that do not cover the full file case. In particular, if the layout is a flexfile mirrored layout, we see that if two layout segments have differing numbers of mirrors, or just have different ordering of mirrors, then the current code can lead to data corruption and even kernel Oopses on the client.
The root cause of these problems is the fact that we have one array of commit lists per struct pnfs_ds_commit_info, and that once that array is set up, it cannot change. This despite the fact that layout segments can be recalled and changed on the fly.
This patch set attempts to fix that problem by changing the array of commit lists into a list of lists. It does so in a way that ensures 1-1 correspondence between a layout segment and a set of commits for the case of buffered writes, and a similar setup for a collection of O_DIRECT writes.
Since most of this work is acting on code that is currently common to both the files layout and the flexfiles layout, I've fixed up both layout types in the same way.
Trond Myklebust (22):
pNFS/flexfiles: Simplify allocation of the mirror array NFS/pNFS: Refactor pnfs_generic_commit_pagelist()
pNFS: Add a helper to allocate the array of buckets NFSv4/pnfs: Support a list of commit arrays in struct
pnfs_ds_commit_info
NFSv4/pNFS: Scan the full list of commit arrays when committing
pNFS: Support per-layout segment commits in
pnfs_generic_recover_commit_reqs()
pNFS: Support per-layout segment commits in
pnfs_generic_commit_pagelist()
NFS/pNFS: Allow O_DIRECT to release the DS commitinfo
NFS: commit errors should be fatal
NFS: Fix O_DIRECT commit verifier handling
NFS/pNFS: Support commit arrays in
nfs_clear_pnfs_ds_commit_verifiers()
pNFS: Add infrastructure for cleaning up per-layout commit structures
pNFS: Enable per-layout segment commit structures NFS/pNFS: Add a helper pnfs_generic_search_commit_reqs()
NFS: Remove bucket array from struct pnfs_ds_commit_info NFS/pNFS: Clean up pNFS commit operations
NFS/pNFS: Simplify bucket layout segment reference counting NFS/pNFS: Fix pnfs_layout_mark_request_commit() invalid layout segment
handling
pNFS/flexfile: Don't merge layout segments if the mirrors don't match pNFS/flexfiles: Check the layout segment range before doing I/O pNFS/flexfiles: remove requirement for whole file layouts pNFS/flexfiles: Specify the layout segment range in LAYOUTGET
fs/nfs/direct.c | 174 +++------ fs/nfs/filelayout/filelayout.c | 163 +++----- fs/nfs/flexfilelayout/flexfilelayout.c | 203 ++++------ fs/nfs/flexfilelayout/flexfilelayout.h | 2 +- fs/nfs/internal.h | 26 +- fs/nfs/pnfs.c | 4 +- fs/nfs/pnfs.h | 127 ++++-- fs/nfs/pnfs_nfs.c | 514 ++++++++++++++++++------- fs/nfs/write.c | 16 +- include/linux/nfs_xdr.h | 32 +- 10 files changed, 699 insertions(+), 562 deletions(-)
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 1/22 pNFS/flexfiles: Simplify allocation of the mirror array
From: Trond Myklebust <[email protected]>
Just allocate the array at the end of the layout segment structure, instead of allocating it as a separate array of pointers.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 21 +++++---------------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 +- 2 files changed, 6 insertions(+), 17 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +5 -16
@@ -248,18 +248,10 @@ static void ff_layout_put_mirror(struct nfs4_ff_layout_mirror *mirror)static void ff_layout_free_mirror_array (struct nfs4_ff_layout_segment *fls) { -int i; +u32 i; -if (fls->mirror_array) { -for (i =0 ; i < fls->mirror_array_cnt; i++) { -/* normally mirror_ds is freed in -* .free_deviceid_node but we still do it here -* for .alloc_lseg error path */ -ff_layout_put_mirror (fls->mirror_array[i]); - } -kfree (fls->mirror_array); - fls->mirror_array =NULL ; - } +for (i =0 ; i < fls->mirror_array_cnt; i++) +ff_layout_put_mirror (fls->mirror_array[i]); }static int ff_layout_check_layout (struct nfs4_layoutget_res *lgr)
@@ -400,16 +392,13 @@ ff_layout_alloc_lseg(struct pnfs_layout_hdr *lh, goto out_err_free; rc = -ENOMEM ; - fls =kzalloc (sizeof (*fls), gfp_flags); + fls =kzalloc (struct_size (fls, mirror_array, mirror_array_cnt), + gfp_flags);if (!fls) goto out_err_free; fls->mirror_array_cnt = mirror_array_cnt; fls->stripe_unit = stripe_unit; - fls->mirror_array =kcalloc (fls->mirror_array_cnt, -sizeof (fls->mirror_array[0 ]), gfp_flags); -if (fls->mirror_array ==NULL ) - goto out_err_free;for (i =0 ; i < fls->mirror_array_cnt; i++) {struct nfs4_ff_layout_mirror *mirror;
fs/nfs/flexfilelayout/flexfilelayout.h +1 -1
@@ -99,7 +99,7 @@ struct nfs4_ff_layout_segment {u64 stripe_unit;u32 flags;u32 mirror_array_cnt; -struct nfs4_ff_layout_mirror **mirror_array; +struct nfs4_ff_layout_mirror *mirror_array[]; };struct nfs4_flexfile_layout {
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 2/22 NFS/pNFS: Refactor pnfs_generic_commit_pagelist()
From: Trond Myklebust <[email protected]>
Refactor pnfs_generic_commit_pagelist() to simplify the conversion to layout segment based commit lists.
--- fs/nfs/pnfs_nfs.c | 164 +++++++++++++++++++--------------------------- fs/nfs/write.c | 13 ++-- 2 files changed, 76 insertions(+), 101 deletions(-)
fs/nfs/pnfs_nfs.c +67 -97
@@ -156,103 +156,86 @@ void pnfs_generic_recover_commit_reqs(struct list_head *dst, }EXPORT_SYMBOL_GPL (pnfs_generic_recover_commit_reqs); -static void pnfs_generic_retry_commit (struct nfs_commit_info *cinfo,int idx) +static struct pnfs_layout_segment * +pnfs_bucket_get_committing (struct list_head *head, +struct pnfs_commit_bucket *bucket, +struct nfs_commit_info *cinfo) { -struct pnfs_ds_commit_info *fl_cinfo = cinfo->ds; -struct pnfs_commit_bucket *bucket;struct pnfs_layout_segment *freeme;struct list_head *pos; + +list_for_each (pos, &bucket->committing) + cinfo->ds->ncommitting--; +list_splice_init (&bucket->committing, head); + freeme = bucket->clseg; + bucket->clseg =NULL ; +return freeme; +} + +static struct nfs_commit_data * +pnfs_bucket_fetch_commitdata (struct pnfs_commit_bucket *bucket, +struct nfs_commit_info *cinfo) +{ +struct nfs_commit_data *data =nfs_commitdata_alloc (false); + +if (!data) +return NULL ; + data->lseg =pnfs_bucket_get_committing (&data->pages, bucket, cinfo); +return data; +} + +static void pnfs_generic_retry_commit (struct pnfs_commit_bucket *buckets, +unsigned int nbuckets, +struct nfs_commit_info *cinfo, +unsigned int idx) +{ +struct pnfs_commit_bucket *bucket; +struct pnfs_layout_segment *freeme;LIST_HEAD (pages); -int i; -mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); -for (i = idx; i < fl_cinfo->nbuckets; i++) { - bucket = &fl_cinfo->buckets[i]; +for (bucket = buckets; idx < nbuckets; bucket++, idx++) {if (list_empty (&bucket->committing))continue ; - freeme = bucket->clseg; - bucket->clseg =NULL ; -list_for_each (pos, &bucket->committing) - cinfo->ds->ncommitting--; -list_splice_init (&bucket->committing, &pages); +mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); + freeme =pnfs_bucket_get_committing (&pages, bucket, cinfo);mutex_unlock (&NFS_I (cinfo->inode)->commit_mutex); -nfs_retry_commit (&pages, freeme, cinfo, i); +nfs_retry_commit (&pages, freeme, cinfo, idx);pnfs_put_lseg (freeme); -mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); } -mutex_unlock (&NFS_I (cinfo->inode)->commit_mutex); }static unsigned int -pnfs_generic_alloc_ds_commits (struct nfs_commit_info *cinfo, -struct list_head *list) +pnfs_bucket_alloc_ds_commits (struct list_head *list, +struct pnfs_commit_bucket *buckets, +unsigned int nbuckets, +struct nfs_commit_info *cinfo) { -struct pnfs_ds_commit_info *fl_cinfo;struct pnfs_commit_bucket *bucket;struct nfs_commit_data *data; -int i; +unsigned int i;unsigned int nreq =0 ; - fl_cinfo = cinfo->ds; - bucket = fl_cinfo->buckets; -for (i =0 ; i < fl_cinfo->nbuckets; i++, bucket++) { +for (i =0 , bucket = buckets; i < nbuckets; i++, bucket++) {if (list_empty (&bucket->committing))continue ; - data =nfs_commitdata_alloc (false); -if (!data) -break ; - data->ds_commit_index = i; -list_add (&data->pages, list); - nreq++; +mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); +if (!list_empty (&bucket->committing)) { + data =pnfs_bucket_fetch_commitdata (bucket, cinfo); +if (!data) + goto out_error; + data->ds_commit_index = i; +list_add_tail (&data->list, list); +atomic_inc (&cinfo->mds->rpcs_out); + nreq++; + } +mutex_unlock (&NFS_I (cinfo->inode)->commit_mutex); } - -/* Clean up on error */ -pnfs_generic_retry_commit (cinfo, i);return nreq; -} - -static inline -void pnfs_fetch_commit_bucket_list (struct list_head *pages, -struct nfs_commit_data *data, -struct nfs_commit_info *cinfo) -{ -struct pnfs_commit_bucket *bucket; -struct list_head *pos; - - bucket = &cinfo->ds->buckets[data->ds_commit_index]; -mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); -list_for_each (pos, &bucket->committing) - cinfo->ds->ncommitting--; -list_splice_init (&bucket->committing, pages); - data->lseg = bucket->clseg; - bucket->clseg =NULL ; +out_error:mutex_unlock (&NFS_I (cinfo->inode)->commit_mutex); - -} - -/* Helper function for pnfs_generic_commit_pagelist to catch an empty -* page list. This can happen when two commits race. -* -* This must be called instead of nfs_init_commit - call one or the other, but -* not both! -*/ -static bool -pnfs_generic_commit_cancel_empty_pagelist (struct list_head *pages, -struct nfs_commit_data *data, -struct nfs_commit_info *cinfo) -{ -if (list_empty (pages)) { -if (atomic_dec_and_test (&cinfo->mds->rpcs_out)) -wake_up_var (&cinfo->mds->rpcs_out); -/* don't call nfs_commitdata_release - it tries to put -* the open_context which is not acquired until nfs_init_commit -* which has not been called on @data */ -WARN_ON_ONCE (data->context); -nfs_commit_free (data); -return true; - } - -return false; +/* Clean up on error */ +pnfs_generic_retry_commit (buckets, nbuckets, cinfo, i); +return nreq; }/* This follows nfs_commit_list pretty closely */
@@ -262,6 +245,7 @@ pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages,int (*initiate_commit)(struct nfs_commit_data *data, int how)) { +struct pnfs_ds_commit_info *fl_cinfo = cinfo->ds;struct nfs_commit_data *data, *tmp;LIST_HEAD (list);unsigned int nreq =0 ;
@@ -269,40 +253,26 @@ pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages,if (!list_empty (mds_pages)) { data =nfs_commitdata_alloc (true); data->ds_commit_index =-1 ; -list_add (&data->pages, &list); +list_splice_init (mds_pages, &data->pages); +list_add_tail (&data->list, &list); +atomic_inc (&cinfo->mds->rpcs_out); nreq++; } - nreq +=pnfs_generic_alloc_ds_commits (cinfo, &list); - + nreq +=pnfs_bucket_alloc_ds_commits (&list, fl_cinfo->buckets, + fl_cinfo->nbuckets, cinfo);if (nreq ==0 ) goto out; -atomic_add (nreq, &cinfo->mds->rpcs_out); - -list_for_each_entry_safe (data, tmp, &list, pages) { -list_del_init (&data->pages); +list_for_each_entry_safe (data, tmp, &list, list) { +list_del (&data->list);if (data->ds_commit_index <0 ) { -/* another commit raced with us */ -if (pnfs_generic_commit_cancel_empty_pagelist (mds_pages, - data, cinfo)) -continue ; - -nfs_init_commit (data, mds_pages,NULL , cinfo); +nfs_init_commit (data,NULL ,NULL , cinfo);nfs_initiate_commit (NFS_CLIENT (inode), data,NFS_PROTO (data->inode), data->mds_ops, how,0 ); }else { -LIST_HEAD (pages); - -pnfs_fetch_commit_bucket_list (&pages, data, cinfo); - -/* another commit raced with us */ -if (pnfs_generic_commit_cancel_empty_pagelist (&pages, - data, cinfo)) -continue ; - -nfs_init_commit (data, &pages, data->lseg, cinfo); +nfs_init_commit (data,NULL , data->lseg, cinfo);initiate_commit (data, how); } }
fs/nfs/write.c +9 -4
@@ -1746,14 +1746,19 @@ void nfs_init_commit(struct nfs_commit_data *data,struct pnfs_layout_segment *lseg, struct nfs_commit_info *cinfo) { - struct nfs_page *first = nfs_list_entry(head->next); -struct nfs_open_context *ctx =nfs_req_openctx (first); -struct inode *inode =d_inode (ctx->dentry); + struct nfs_page *first; +struct nfs_open_context *ctx; +struct inode *inode;/* Set up the RPC argument and reply structs * NB: take care not to mess about with data->commit et al. */ -list_splice_init (head, &data->pages); +if (head) +list_splice_init (head, &data->pages); + + first =nfs_list_entry (data->pages.next); + ctx =nfs_req_openctx (first); + inode =d_inode (ctx->dentry); data->inode = inode; data->cred = ctx->cred;
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 3/22 pNFS: Add a helper to allocate the array of buckets
From: Trond Myklebust <[email protected]>
--- fs/nfs/pnfs.h | 3 +++ fs/nfs/pnfs_nfs.c | 31 +++++++++++++++++++++++++++++++ include/linux/nfs_xdr.h | 15 ++++++++++++--- 3 files changed, 46 insertions(+), 3 deletions(-)
fs/nfs/pnfs.h +3 -0
@@ -366,6 +366,9 @@ bool nfs4_test_deviceid_unavailable(struct nfs4_deviceid_node *node);void nfs4_deviceid_purge_client (const struct nfs_client *);/* pnfs_nfs.c */ +struct pnfs_commit_array *pnfs_alloc_commit_array (size_t n,gfp_t gfp_flags); +void pnfs_free_commit_array (struct pnfs_commit_array *p); +void pnfs_generic_clear_request_commit (struct nfs_page *req,struct nfs_commit_info *cinfo);void pnfs_generic_commit_release (void *calldata);
fs/nfs/pnfs_nfs.c +31 -0
@@ -87,6 +87,37 @@ pnfs_generic_clear_request_commit(struct nfs_page *req, }EXPORT_SYMBOL_GPL (pnfs_generic_clear_request_commit); +struct pnfs_commit_array * +pnfs_alloc_commit_array (size_t n,gfp_t gfp_flags) +{ +struct pnfs_commit_array *p; +struct pnfs_commit_bucket *b; + + p =kmalloc (struct_size (p, buckets, n), gfp_flags); +if (!p) +return NULL ; + p->nbuckets = n; +INIT_LIST_HEAD (&p->cinfo_list); +INIT_LIST_HEAD (&p->lseg_list); + p->lseg =NULL ; +for (b = &p->buckets[0 ]; n !=0 ; b++, n--) { +INIT_LIST_HEAD (&b->written); +INIT_LIST_HEAD (&b->committing); + b->wlseg =NULL ; + b->clseg =NULL ; + b->direct_verf.committed =NFS_INVALID_STABLE_HOW ; + } +return p; +} +EXPORT_SYMBOL_GPL (pnfs_alloc_commit_array); + +void +pnfs_free_commit_array (struct pnfs_commit_array *p) +{ +kfree_rcu (p, rcu); +} +EXPORT_SYMBOL_GPL (pnfs_free_commit_array); + static int pnfs_generic_scan_ds_commit_list(struct pnfs_commit_bucket *bucket, struct nfs_commit_info *cinfo,
include/linux/nfs_xdr.h +12 -3
@@ -1270,10 +1270,19 @@ struct pnfs_commit_bucket {struct nfs_writeverf direct_verf; }; +struct pnfs_commit_array { +struct list_head cinfo_list; +struct list_head lseg_list; +struct pnfs_layout_segment *lseg; +struct rcu_head rcu; +unsigned int nbuckets; +struct pnfs_commit_bucket buckets[]; +}; +struct pnfs_ds_commit_info { -int nwritten; -int ncommitting; -int nbuckets; +unsigned int nwritten; +unsigned int ncommitting; +unsigned int nbuckets;struct pnfs_commit_bucket *buckets; };
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 4/22 NFSv4/pnfs: Support a list of commit arrays in struct pnfs_ds_commit_info
From: Trond Myklebust <[email protected]>
When we have multiple layout segments with different lists of mirrored data, we need to track the commits on a per layout segment basis. This patch adds a list to support this tracking in struct pnfs_ds_commit_info.
--- fs/nfs/direct.c | 1 + fs/nfs/filelayout/filelayout.c | 5 ++++- fs/nfs/flexfilelayout/flexfilelayout.c | 1 + include/linux/nfs_xdr.h | 1 + 4 files changed, 7 insertions(+), 1 deletion(-)
fs/nfs/direct.c +1 -0
@@ -305,6 +305,7 @@ static inline struct nfs_direct_req *nfs_direct_req_alloc(void)kref_get (&dreq->kref);init_completion (&dreq->completion);INIT_LIST_HEAD (&dreq->mds_cinfo.list); +INIT_LIST_HEAD (&dreq->ds_cinfo.commits); dreq->verf.committed =NFS_INVALID_STABLE_HOW ;/* not set yet */ INIT_WORK (&dreq->work, nfs_direct_write_schedule_work);spin_lock_init (&dreq->lock);
fs/nfs/filelayout/filelayout.c +4 -1
@@ -1140,7 +1140,10 @@ filelayout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags)struct nfs4_filelayout *flo; flo =kzalloc (sizeof (*flo), gfp_flags); -return flo !=NULL ? &flo->generic_hdr :NULL ; +if (flo ==NULL ) +return NULL ; +INIT_LIST_HEAD (&flo->commit_info.commits); +return &flo->generic_hdr; } static void
fs/nfs/flexfilelayout/flexfilelayout.c +1 -0
@@ -48,6 +48,7 @@ ff_layout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags) ffl =kzalloc (sizeof (*ffl), gfp_flags);if (ffl) { +INIT_LIST_HEAD (&ffl->commit_info.commits);INIT_LIST_HEAD (&ffl->error_list);INIT_LIST_HEAD (&ffl->mirrors); ffl->last_report_time =ktime_get ();
include/linux/nfs_xdr.h +1 -0
@@ -1280,6 +1280,7 @@ struct pnfs_commit_array { };struct pnfs_ds_commit_info { +struct list_head commits;unsigned int nwritten;unsigned int ncommitting;unsigned int nbuckets;
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 5/22 NFSv4/pNFS: Scan the full list of commit arrays when committing
From: Trond Myklebust <[email protected]>
Add support for scanning the full list of per-layout segment commit arrays to pnfs_generic_scan_commit_lists()
--- fs/nfs/pnfs_nfs.c | 52 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 12 deletions(-)
fs/nfs/pnfs_nfs.c +40 -12
@@ -118,10 +118,14 @@ pnfs_free_commit_array(struct pnfs_commit_array *p) }EXPORT_SYMBOL_GPL (pnfs_free_commit_array); +/* +* Locks the nfs_page requests for commit and moves them to +* @bucket->committing. +*/ static int -pnfs_generic_scan_ds_commit_list (struct pnfs_commit_bucket *bucket, -struct nfs_commit_info *cinfo, -int max) +pnfs_bucket_scan_ds_commit_list (struct pnfs_commit_bucket *bucket, +struct nfs_commit_info *cinfo, +int max) {struct list_head *src = &bucket->written;struct list_head *dst = &bucket->committing;
@@ -142,20 +146,44 @@ pnfs_generic_scan_ds_commit_list(struct pnfs_commit_bucket *bucket,return ret; } +static int pnfs_bucket_scan_array (struct nfs_commit_info *cinfo, +struct pnfs_commit_bucket *buckets, +unsigned int nbuckets, +int max) +{ +unsigned int i; +int rv =0 , cnt; + +for (i =0 ; i < nbuckets && max !=0 ; i++) { + cnt =pnfs_bucket_scan_ds_commit_list (&buckets[i], cinfo, max); + rv += cnt; + max -= cnt; + } +return rv; +} +/* Move reqs from written to committing lists, returning count * of number moved. */ -int pnfs_generic_scan_commit_lists (struct nfs_commit_info *cinfo, -int max) +int pnfs_generic_scan_commit_lists (struct nfs_commit_info *cinfo,int max) { -int i, rv =0 , cnt; - -lockdep_assert_held (&NFS_I (cinfo->inode)->commit_mutex); -for (i =0 ; i < cinfo->ds->nbuckets && max !=0 ; i++) { - cnt =pnfs_generic_scan_ds_commit_list (&cinfo->ds->buckets[i], - cinfo, max); - max -= cnt; +struct pnfs_ds_commit_info *fl_cinfo = cinfo->ds; +struct pnfs_commit_array *array; +int rv =0 , cnt; + + cnt =pnfs_bucket_scan_array (cinfo, fl_cinfo->buckets, + fl_cinfo->nbuckets, max); + rv += cnt; + max -= cnt; +if (!max) +return rv; +list_for_each_entry (array, &fl_cinfo->commits, cinfo_list) { + cnt =pnfs_bucket_scan_array (cinfo, array->buckets, + array->nbuckets, max); rv += cnt; + max -= cnt; +if (!max) +break ; }return rv; }
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 6/22 pNFS: Support per-layout segment commits in pnfs_generic_recover_commit_reqs()
From: Trond Myklebust <[email protected]>
Add support for scanning the full list of per-layout segment commit arrays to pnfs_generic_recover_commit_reqs().
--- fs/nfs/pnfs_nfs.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)
fs/nfs/pnfs_nfs.c +33 -8
@@ -189,22 +189,23 @@ int pnfs_generic_scan_commit_lists(struct nfs_commit_info *cinfo, int max) }EXPORT_SYMBOL_GPL (pnfs_generic_scan_commit_lists); -/* Pull everything off the committing lists and dump into @dst. */ -void pnfs_generic_recover_commit_reqs (struct list_head *dst, -struct nfs_commit_info *cinfo) +static unsigned int +pnfs_bucket_recover_commit_reqs (struct list_head *dst, +struct pnfs_commit_bucket *buckets, +unsigned int nbuckets, +struct nfs_commit_info *cinfo) {struct pnfs_commit_bucket *b;struct pnfs_layout_segment *freeme; -int nwritten; -int i; +unsigned int nwritten, ret =0 ; +unsigned int i; -lockdep_assert_held (&NFS_I (cinfo->inode)->commit_mutex); restart: -for (i =0 , b = cinfo->ds->buckets; i < cinfo->ds->nbuckets; i++, b++) { +for (i =0 , b = buckets; i < nbuckets; i++, b++) { nwritten =nfs_scan_commit_list (&b->written, dst, cinfo,0 );if (!nwritten)continue ; - cinfo->ds->nwritten -= nwritten; + ret += nwritten;if (list_empty (&b->written)) { freeme = b->wlseg; b->wlseg =NULL ;
@@ -212,6 +213,30 @@ void pnfs_generic_recover_commit_reqs(struct list_head *dst, goto restart; } } +return ret; +} + +/* Pull everything off the committing lists and dump into @dst. */ +void pnfs_generic_recover_commit_reqs (struct list_head *dst, +struct nfs_commit_info *cinfo) +{ +struct pnfs_ds_commit_info *fl_cinfo = cinfo->ds; +struct pnfs_commit_array *array; +unsigned int nwritten; + +lockdep_assert_held (&NFS_I (cinfo->inode)->commit_mutex); + nwritten =pnfs_bucket_recover_commit_reqs (dst, + fl_cinfo->buckets, + fl_cinfo->nbuckets, + cinfo); + fl_cinfo->nwritten -= nwritten; +list_for_each_entry (array, &fl_cinfo->commits, cinfo_list) { + nwritten =pnfs_bucket_recover_commit_reqs (dst, + array->buckets, + array->nbuckets, + cinfo); + fl_cinfo->nwritten -= nwritten; + } }EXPORT_SYMBOL_GPL (pnfs_generic_recover_commit_reqs);
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 7/22 pNFS: Support per-layout segment commits in pnfs_generic_commit_pagelist()
From: Trond Myklebust <[email protected]>
Add support for scanning the full list of per-layout segment commit arrays to pnfs_generic_commit_pagelist().
--- fs/nfs/pnfs_nfs.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
fs/nfs/pnfs_nfs.c +16 -0
@@ -322,6 +322,20 @@ pnfs_bucket_alloc_ds_commits(struct list_head *list,return nreq; } +static unsigned int +pnfs_alloc_ds_commits_list (struct list_head *list, +struct pnfs_ds_commit_info *fl_cinfo, +struct nfs_commit_info *cinfo) +{ +struct pnfs_commit_array *array; +unsigned int ret =0 ; + +list_for_each_entry (array, &fl_cinfo->commits, cinfo_list) + ret +=pnfs_bucket_alloc_ds_commits (list, array->buckets, + array->nbuckets, cinfo); +return ret; +} +/* This follows nfs_commit_list pretty closely */ int pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages,
@@ -345,6 +359,8 @@ pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages, nreq +=pnfs_bucket_alloc_ds_commits (&list, fl_cinfo->buckets, fl_cinfo->nbuckets, cinfo); + + nreq +=pnfs_alloc_ds_commits_list (&list, fl_cinfo, cinfo);if (nreq ==0 ) goto out;
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 8/22 NFS/pNFS: Allow O_DIRECT to release the DS commitinfo
From: Trond Myklebust <[email protected]>
Add a pNFS callback to allow the O_DIRECT code to release the DS commitinfo when freeing the dreq.
--- fs/nfs/direct.c | 1 + fs/nfs/pnfs.h | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
fs/nfs/direct.c +1 -0
@@ -317,6 +317,7 @@ static void nfs_direct_req_free(struct kref *kref) { struct nfs_direct_req *dreq = container_of(kref, struct nfs_direct_req, kref); +pnfs_release_ds_info (&dreq->ds_cinfo, dreq->inode);nfs_free_pnfs_ds_cinfo (&dreq->ds_cinfo);if (dreq->l_ctx !=NULL )nfs_put_lock_context (dreq->l_ctx);
fs/nfs/pnfs.h +16 -0
@@ -149,6 +149,8 @@ struct pnfs_layoutdriver_type {const struct nfs_pageio_ops *pg_write_ops;struct pnfs_ds_commit_info *(*get_ds_info) (struct inode *inode); +void (*release_ds_info)(struct pnfs_ds_commit_info *, +struct inode *inode); void (*mark_request_commit) (struct nfs_page *req, struct pnfs_layout_segment *lseg, struct nfs_commit_info *cinfo,
@@ -462,6 +464,15 @@ pnfs_get_ds_info(struct inode *inode)return ld->get_ds_info (inode); } +static inline void +pnfs_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo,struct inode *inode) +{ +struct pnfs_layoutdriver_type *ld =NFS_SERVER (inode)->pnfs_curr_ld; + +if (ld !=NULL && ld->release_ds_info !=NULL ) + ld->release_ds_info (fl_cinfo, inode); +} +static inline void pnfs_generic_mark_devid_invalid (struct nfs4_deviceid_node *node) {
@@ -759,6 +770,11 @@ pnfs_get_ds_info(struct inode *inode)return NULL ; } +static inline void +pnfs_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo,struct inode *inode) +{ +} +static inline bool pnfs_mark_request_commit (struct nfs_page *req,struct pnfs_layout_segment *lseg,struct nfs_commit_info *cinfo,u32 ds_commit_idx)
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 9/22 NFS: commit errors should be fatal
From: Trond Myklebust <[email protected]>
Fix the O_DIRECT code to avoid retries if the COMMIT fails with a fatal error.
--- fs/nfs/direct.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
fs/nfs/direct.c +30 -2
@@ -94,6 +94,7 @@ struct nfs_direct_req {#define NFS_ODIRECT_RESCHED_WRITES (2)/* write verification failed */ /* for read */ #define NFS_ODIRECT_SHOULD_DIRTY (3)/* dirty user-space page after read */ +#define NFS_ODIRECT_DONE INT_MAX/* write verification failed */ struct nfs_writeverf verf;/* unstable write verifier */ };
@@ -678,8 +679,17 @@ static void nfs_direct_commit_complete(struct nfs_commit_data *data)struct nfs_page *req;int status = data->task.tk_status; +if (status <0 ) { +/* Errors in commit are fatal */ + dreq->error = status; + dreq->max_count =0 ; + dreq->count =0 ; + dreq->flags =NFS_ODIRECT_DONE ; + }else if (dreq->flags ==NFS_ODIRECT_DONE ) + status = dreq->error; +nfs_init_cinfo_from_dreq (&cinfo, dreq); -if (status <0 ||nfs_direct_cmp_commit_data_verf (dreq, data)) +if (nfs_direct_cmp_commit_data_verf (dreq, data)) dreq->flags =NFS_ODIRECT_RESCHED_WRITES ;while (!list_empty (&data->pages)) {
@@ -708,7 +718,8 @@ static void nfs_direct_resched_write(struct nfs_commit_info *cinfo,struct nfs_direct_req *dreq = cinfo->dreq;spin_lock (&dreq->lock); - dreq->flags =NFS_ODIRECT_RESCHED_WRITES ; +if (dreq->flags !=NFS_ODIRECT_DONE ) + dreq->flags =NFS_ODIRECT_RESCHED_WRITES ;spin_unlock (&dreq->lock);nfs_mark_request_commit (req,NULL , cinfo,0 ); }
@@ -731,6 +742,22 @@ static void nfs_direct_commit_schedule(struct nfs_direct_req *dreq)nfs_direct_write_reschedule (dreq); } +static void nfs_direct_write_clear_reqs (struct nfs_direct_req *dreq) +{ +struct nfs_commit_info cinfo; +struct nfs_page *req; +LIST_HEAD (reqs); + +nfs_init_cinfo_from_dreq (&cinfo, dreq); +nfs_direct_write_scan_commit_list (dreq->inode, &reqs, &cinfo); + +while (!list_empty (&reqs)) { + req =nfs_list_entry (reqs.next); +nfs_list_remove_request (req); +nfs_unlock_and_release_request (req); + } +} +static void nfs_direct_write_schedule_work (struct work_struct *work) { struct nfs_direct_req *dreq = container_of(work, struct nfs_direct_req, work);
@@ -745,6 +772,7 @@ static void nfs_direct_write_schedule_work(struct work_struct *work)nfs_direct_write_reschedule (dreq);break ;default : +nfs_direct_write_clear_reqs (dreq);nfs_zap_mapping (dreq->inode, dreq->inode->i_mapping);nfs_direct_complete (dreq); }
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 11/22 NFS/pNFS: Support commit arrays in nfs_clear_pnfs_ds_commit_verifiers()
In reply to [email protected]From: Trond Myklebust <[email protected]>
Add support for scanning the full list of per-layout segment commit arrays to nfs_clear_pnfs_ds_commit_verifiers().
--- fs/nfs/internal.h | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
fs/nfs/internal.h +16 -3
@@ -517,13 +517,26 @@ int nfs_filemap_write_and_wait_range(struct address_space *mapping, loff_t lstart, loff_t lend);#ifdef CONFIG_NFS_V4_1 +static inline void +pnfs_bucket_clear_pnfs_ds_commit_verifiers (struct pnfs_commit_bucket *buckets, +unsigned int nbuckets) +{ +unsigned int i; + +for (i =0 ; i < nbuckets; i++) + buckets[i].direct_verf.committed =NFS_INVALID_STABLE_HOW ; +}static inline void nfs_clear_pnfs_ds_commit_verifiers (struct pnfs_ds_commit_info *cinfo) { -int i; +struct pnfs_commit_array *array; + +pnfs_bucket_clear_pnfs_ds_commit_verifiers (cinfo->buckets, + cinfo->nbuckets); -for (i =0 ; i < cinfo->nbuckets; i++) - cinfo->buckets[i].direct_verf.committed =NFS_INVALID_STABLE_HOW ; +list_for_each_entry (array, &cinfo->commits, cinfo_list) +pnfs_bucket_clear_pnfs_ds_commit_verifiers (array->buckets, + array->nbuckets); }#else static inline
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 10/22 NFS: Fix O_DIRECT commit verifier handling
In reply to [email protected]From: Trond Myklebust <[email protected]>
Instead of trying to save the commit verifiers and checking them against previous writes, adopt the same strategy as for buffered writes, of just checking the verifiers at commit time.
--- fs/nfs/direct.c | 135 +++++----------------------------------------- fs/nfs/internal.h | 8 +++ fs/nfs/write.c | 3 +- 3 files changed, 22 insertions(+), 124 deletions(-)
fs/nfs/direct.c +13 -122
@@ -95,7 +95,6 @@ struct nfs_direct_req {/* for read */ #define NFS_ODIRECT_SHOULD_DIRTY (3)/* dirty user-space page after read */ #define NFS_ODIRECT_DONE INT_MAX/* write verification failed */ -struct nfs_writeverf verf;/* unstable write verifier */ };static const struct nfs_pgio_completion_ops nfs_direct_write_completion_ops;
@@ -152,106 +151,6 @@ nfs_direct_count_bytes(struct nfs_direct_req *dreq, dreq->count = dreq_len; } -/* -* nfs_direct_select_verf - select the right verifier -* @dreq - direct request possibly spanning multiple servers -* @ds_clp - nfs_client of data server or NULL if MDS / non-pnfs -* @commit_idx - commit bucket index for the DS -* -* returns the correct verifier to use given the role of the server -*/ -static struct nfs_writeverf * -nfs_direct_select_verf (struct nfs_direct_req *dreq, -struct nfs_client *ds_clp, -int commit_idx) -{ -struct nfs_writeverf *verfp = &dreq->verf; - -#ifdef CONFIG_NFS_V4_1 -/* -* pNFS is in use, use the DS verf except commit_through_mds is set -* for layout segment where nbuckets is zero. -*/ -if (ds_clp && dreq->ds_cinfo.nbuckets >0 ) { -if (commit_idx >=0 && commit_idx < dreq->ds_cinfo.nbuckets) - verfp = &dreq->ds_cinfo.buckets[commit_idx].direct_verf; -else -WARN_ON_ONCE (1 ); - } -#endif -return verfp; -} - - -/* -* nfs_direct_set_hdr_verf - set the write/commit verifier -* @dreq - direct request possibly spanning multiple servers -* @hdr - pageio header to validate against previously seen verfs -* -* Set the server's (MDS or DS) "seen" verifier -*/ -static void nfs_direct_set_hdr_verf (struct nfs_direct_req *dreq, -struct nfs_pgio_header *hdr) -{ -struct nfs_writeverf *verfp; - - verfp =nfs_direct_select_verf (dreq, hdr->ds_clp, hdr->ds_commit_idx); -WARN_ON_ONCE (verfp->committed >=0 ); -memcpy (verfp, &hdr->verf,sizeof (struct nfs_writeverf )); -WARN_ON_ONCE (verfp->committed <0 ); -} - -static int nfs_direct_cmp_verf (const struct nfs_writeverf *v1, -const struct nfs_writeverf *v2) -{ -return nfs_write_verifier_cmp (&v1->verifier, &v2->verifier); -} - -/* -* nfs_direct_cmp_hdr_verf - compare verifier for pgio header -* @dreq - direct request possibly spanning multiple servers -* @hdr - pageio header to validate against previously seen verf -* -* set the server's "seen" verf if not initialized. -* returns result of comparison between @hdr->verf and the "seen" -* verf of the server used by @hdr (DS or MDS) -*/ -static int nfs_direct_set_or_cmp_hdr_verf (struct nfs_direct_req *dreq, -struct nfs_pgio_header *hdr) -{ -struct nfs_writeverf *verfp; - - verfp =nfs_direct_select_verf (dreq, hdr->ds_clp, hdr->ds_commit_idx); -if (verfp->committed <0 ) { -nfs_direct_set_hdr_verf (dreq, hdr); -return 0 ; - } -return nfs_direct_cmp_verf (verfp, &hdr->verf); -} - -/* -* nfs_direct_cmp_commit_data_verf - compare verifier for commit data -* @dreq - direct request possibly spanning multiple servers -* @data - commit data to validate against previously seen verf -* -* returns result of comparison between @data->verf and the verf of -* the server used by @data (DS or MDS) -*/ -static int nfs_direct_cmp_commit_data_verf (struct nfs_direct_req *dreq, -struct nfs_commit_data *data) -{ -struct nfs_writeverf *verfp; - - verfp =nfs_direct_select_verf (dreq, data->ds_clp, - data->ds_commit_index); - -/* verifier not set so always fail */ -if (verfp->committed <0 || data->res.verf->committed <=NFS_UNSTABLE ) -return 1 ; - -return nfs_direct_cmp_verf (verfp, data->res.verf); -} -/** * nfs_direct_IO - NFS address space operation for direct I/O * @iocb: target I/O control block
@@ -307,7 +206,6 @@ static inline struct nfs_direct_req *nfs_direct_req_alloc(void)init_completion (&dreq->completion);INIT_LIST_HEAD (&dreq->mds_cinfo.list);INIT_LIST_HEAD (&dreq->ds_cinfo.commits); - dreq->verf.committed =NFS_INVALID_STABLE_HOW ;/* not set yet */ INIT_WORK (&dreq->work, nfs_direct_write_schedule_work);spin_lock_init (&dreq->lock);
@@ -637,7 +535,6 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq) dreq->max_count =0 ;list_for_each_entry (req, &reqs, wb_list) dreq->max_count += req->wb_bytes; - dreq->verf.committed =NFS_INVALID_STABLE_HOW ;nfs_clear_pnfs_ds_commit_verifiers (&dreq->ds_cinfo);get_dreq (dreq);
@@ -674,6 +571,7 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)static void nfs_direct_commit_complete (struct nfs_commit_data *data) { +const struct nfs_writeverf *verf = data->res.verf;struct nfs_direct_req *dreq = data->dreq;struct nfs_commit_info cinfo;struct nfs_page *req;
@@ -689,21 +587,19 @@ static void nfs_direct_commit_complete(struct nfs_commit_data *data) status = dreq->error;nfs_init_cinfo_from_dreq (&cinfo, dreq); -if (nfs_direct_cmp_commit_data_verf (dreq, data)) - dreq->flags =NFS_ODIRECT_RESCHED_WRITES ;while (!list_empty (&data->pages)) { req =nfs_list_entry (data->pages.next);nfs_list_remove_request (req); -if (dreq->flags ==NFS_ODIRECT_RESCHED_WRITES ) { +if (status >=0 && !nfs_write_match_verf (verf, req)) { + dreq->flags =NFS_ODIRECT_RESCHED_WRITES ;/* * Despite the reboot, the write was successful, * so reset wb_nio. */ req->wb_nio =0 ; -/* Note the rewrite will go through mds */ nfs_mark_request_commit (req,NULL , &cinfo,0 ); - }else + }else /* Error or match */ nfs_release_request (req);nfs_unlock_and_release_request (req); }
@@ -799,20 +695,15 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr) }nfs_direct_count_bytes (dreq, hdr); -if (hdr->good_bytes !=0 ) { -if (nfs_write_need_commit (hdr)) { -if (dreq->flags ==NFS_ODIRECT_RESCHED_WRITES ) - request_commit = true; -else if (dreq->flags ==0 ) { -nfs_direct_set_hdr_verf (dreq, hdr); - request_commit = true; - dreq->flags =NFS_ODIRECT_DO_COMMIT ; - }else if (dreq->flags ==NFS_ODIRECT_DO_COMMIT ) { - request_commit = true; -if (nfs_direct_set_or_cmp_hdr_verf (dreq, hdr)) - dreq->flags = -NFS_ODIRECT_RESCHED_WRITES ; - } +if (hdr->good_bytes !=0 &&nfs_write_need_commit (hdr)) { +switch (dreq->flags) { +case 0 : + dreq->flags =NFS_ODIRECT_DO_COMMIT ; + request_commit = true; +break ; +case NFS_ODIRECT_RESCHED_WRITES : +case NFS_ODIRECT_DO_COMMIT : + request_commit = true; } }spin_unlock (&dreq->lock);
fs/nfs/internal.h +8 -0
@@ -544,6 +544,14 @@ nfs_write_verifier_cmp(const struct nfs_write_verifier *v1,return memcmp (v1->data, v2->data,sizeof (v1->data)); } +static inline bool +nfs_write_match_verf (const struct nfs_writeverf *verf, +struct nfs_page *req) +{ +return verf->committed >NFS_UNSTABLE && + !nfs_write_verifier_cmp (&req->wb_verf, &verf->verifier); +} +/* unlink.c */ extern struct rpc_task * nfs_async_rename(struct inode *old_dir, struct inode *new_dir,
fs/nfs/write.c +1 -2
@@ -1874,8 +1874,7 @@ static void nfs_commit_release_pages(struct nfs_commit_data *data)/* Okay, COMMIT succeeded, apparently. Check the verifier * returned by the server against all stored verfs. */ -if (verf->committed >NFS_UNSTABLE && - !nfs_write_verifier_cmp (&req->wb_verf, &verf->verifier)) { +if (nfs_write_match_verf (verf, req)) {/* We have a match */ if (req->wb_page)nfs_inode_remove_request (req);
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 13/22 pNFS: Enable per-layout segment commit structures
In reply to [email protected]From: Trond Myklebust <[email protected]>
Enable adding and lookup of per-layout segment commits in filelayout and flexfilelayout.
--- fs/nfs/filelayout/filelayout.c | 21 +++++++ fs/nfs/flexfilelayout/flexfilelayout.c | 19 +++++++ fs/nfs/pnfs.h | 6 ++ fs/nfs/pnfs_nfs.c | 77 ++++++++++++++++++++++++-- 4 files changed, 117 insertions(+), 6 deletions(-)
fs/nfs/filelayout/filelayout.c +21 -0
@@ -1168,6 +1168,26 @@ filelayout_get_ds_info(struct inode *inode)return &FILELAYOUT_FROM_HDR (layout)->commit_info; } +static void +filelayout_setup_ds_info (struct pnfs_ds_commit_info *fl_cinfo, +struct pnfs_layout_segment *lseg) +{ +struct nfs4_filelayout_segment *fl =FILELAYOUT_LSEG (lseg); +struct inode *inode = lseg->pls_layout->plh_inode; +struct pnfs_commit_array *array, *new; +unsigned int size = (fl->stripe_type ==STRIPE_SPARSE ) ? + fl->dsaddr->ds_num : fl->dsaddr->stripe_count; + + new =pnfs_alloc_commit_array (size,GFP_NOIO ); +if (new) { +spin_lock (&inode->i_lock); + array =pnfs_add_commit_array (fl_cinfo, new, lseg); +spin_unlock (&inode->i_lock); +if (array != new) +pnfs_free_commit_array (new); + } +} +static void filelayout_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo,struct inode *inode)
@@ -1191,6 +1211,7 @@ static struct pnfs_layoutdriver_type filelayout_type = { .pg_read_ops = &filelayout_pg_read_ops, .pg_write_ops = &filelayout_pg_write_ops, .get_ds_info = &filelayout_get_ds_info, + .setup_ds_info = filelayout_setup_ds_info, .release_ds_info = filelayout_release_ds_info, .mark_request_commit = filelayout_mark_request_commit, .clear_request_commit = pnfs_generic_clear_request_commit,
fs/nfs/flexfilelayout/flexfilelayout.c +19 -0
@@ -2004,6 +2004,24 @@ ff_layout_get_ds_info(struct inode *inode)return &FF_LAYOUT_FROM_HDR (layout)->commit_info; } +static void +ff_layout_setup_ds_info (struct pnfs_ds_commit_info *fl_cinfo, +struct pnfs_layout_segment *lseg) +{ +struct nfs4_ff_layout_segment *flseg =FF_LAYOUT_LSEG (lseg); +struct inode *inode = lseg->pls_layout->plh_inode; +struct pnfs_commit_array *array, *new; + + new =pnfs_alloc_commit_array (flseg->mirror_array_cnt,GFP_NOIO ); +if (new) { +spin_lock (&inode->i_lock); + array =pnfs_add_commit_array (fl_cinfo, new, lseg); +spin_unlock (&inode->i_lock); +if (array != new) +pnfs_free_commit_array (new); + } +} +static void ff_layout_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo,struct inode *inode)
@@ -2513,6 +2531,7 @@ static struct pnfs_layoutdriver_type flexfilelayout_type = { .pg_read_ops = &ff_layout_pg_read_ops, .pg_write_ops = &ff_layout_pg_write_ops, .get_ds_info = ff_layout_get_ds_info, + .setup_ds_info = ff_layout_setup_ds_info, .release_ds_info = ff_layout_release_ds_info, .free_deviceid_node = ff_layout_free_deviceid_node, .mark_request_commit = pnfs_layout_mark_request_commit,
fs/nfs/pnfs.h +6 -0
@@ -150,6 +150,8 @@ struct pnfs_layoutdriver_type {const struct nfs_pageio_ops *pg_write_ops;struct pnfs_ds_commit_info *(*get_ds_info) (struct inode *inode); +void (*setup_ds_info)(struct pnfs_ds_commit_info *, +struct pnfs_layout_segment *);void (*release_ds_info)(struct pnfs_ds_commit_info *,struct inode *inode); void (*mark_request_commit) (struct nfs_page *req,
@@ -371,6 +373,10 @@ void nfs4_deviceid_purge_client(const struct nfs_client *);/* pnfs_nfs.c */ struct pnfs_commit_array *pnfs_alloc_commit_array (size_t n,gfp_t gfp_flags);void pnfs_free_commit_array (struct pnfs_commit_array *p); +struct pnfs_commit_array *pnfs_add_commit_array (struct pnfs_ds_commit_info *, +struct pnfs_commit_array *, +struct pnfs_layout_segment *); +void pnfs_generic_ds_cinfo_release_lseg (struct pnfs_ds_commit_info *fl_cinfo,struct pnfs_layout_segment *lseg);void pnfs_generic_ds_cinfo_destroy (struct pnfs_ds_commit_info *fl_cinfo);
fs/nfs/pnfs_nfs.c +71 -6
100 lines not on this page · raw message
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 14/22 NFS/pNFS: Add a helper pnfs_generic_search_commit_reqs()
From: Trond Myklebust <[email protected]>
Lift filelayout_search_commit_reqs() into the generic pnfs/nfs code, and add support for commit arrays.
--- fs/nfs/filelayout/filelayout.c | 32 +-------------------- fs/nfs/pnfs.h | 2 ++ fs/nfs/pnfs_nfs.c | 51 ++++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 31 deletions(-)
fs/nfs/filelayout/filelayout.c +1 -31
@@ -1083,36 +1083,6 @@ static int filelayout_initiate_commit(struct nfs_commit_data *data, int how)return -EAGAIN ; } -/* filelayout_search_commit_reqs - Search lists in @cinfo for the head reqest -* for @page -* @cinfo - commit info for current inode -* @page - page to search for matching head request -* -* Returns a the head request if one is found, otherwise returns NULL. -*/ -static struct nfs_page * -filelayout_search_commit_reqs (struct nfs_commit_info *cinfo,struct page *page) -{ -struct nfs_page *freq, *t; -struct pnfs_commit_bucket *b; -int i; - -/* Linearly search the commit lists for each bucket until a matching -* request is found */ -for (i =0 , b = cinfo->ds->buckets; i < cinfo->ds->nbuckets; i++, b++) { -list_for_each_entry_safe (freq, t, &b->written, wb_list) { -if (freq->wb_page == page) -return freq->wb_head; - } -list_for_each_entry_safe (freq, t, &b->committing, wb_list) { -if (freq->wb_page == page) -return freq->wb_head; - } - } - -return NULL ; -} -static int filelayout_commit_pagelist (struct inode *inode,struct list_head *mds_pages,int how,struct nfs_commit_info *cinfo)
@@ -1217,7 +1187,7 @@ static struct pnfs_layoutdriver_type filelayout_type = { .clear_request_commit = pnfs_generic_clear_request_commit, .scan_commit_lists = pnfs_generic_scan_commit_lists, .recover_commit_reqs = pnfs_generic_recover_commit_reqs, - .search_commit_reqs = filelayout_search_commit_reqs, + .search_commit_reqs = pnfs_generic_search_commit_reqs, .commit_pagelist = filelayout_commit_pagelist, .read_pagelist = filelayout_read_pagelist, .write_pagelist = filelayout_write_pagelist,
fs/nfs/pnfs.h +2 -0
@@ -388,6 +388,8 @@ void pnfs_generic_prepare_to_resend_writes(struct nfs_commit_data *data);void pnfs_generic_rw_release (void *data);void pnfs_generic_recover_commit_reqs (struct list_head *dst,struct nfs_commit_info *cinfo); +struct nfs_page *pnfs_generic_search_commit_reqs (struct nfs_commit_info *cinfo, +struct page *page); int pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages, int how,
fs/nfs/pnfs_nfs.c +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 12/22 pNFS: Add infrastructure for cleaning up per-layout commit structures
In reply to [email protected]From: Trond Myklebust <[email protected]>
Ensure that both the file and flexfiles layout types clean up when freeing the layout segments.
--- fs/nfs/filelayout/filelayout.c | 16 +++++ fs/nfs/flexfilelayout/flexfilelayout.c | 11 ++++ fs/nfs/internal.h | 4 +- fs/nfs/pnfs.c | 1 + fs/nfs/pnfs.h | 4 ++ fs/nfs/pnfs_nfs.c | 88 +++++++++++++++++++++++++- include/linux/nfs_xdr.h | 1 + 7 files changed, 121 insertions(+), 4 deletions(-)
fs/nfs/filelayout/filelayout.c +16 -0
@@ -750,11 +750,16 @@ filelayout_free_lseg(struct pnfs_layout_segment *lseg)/* This assumes a single RW lseg */ if (lseg->pls_range.iomode ==IOMODE_RW ) {struct nfs4_filelayout *flo; +struct inode *inode; flo =FILELAYOUT_FROM_HDR (lseg->pls_layout); + inode = flo->generic_hdr.plh_inode; +spin_lock (&inode->i_lock); flo->commit_info.nbuckets =0 ;kfree (flo->commit_info.buckets); flo->commit_info.buckets =NULL ; +pnfs_generic_ds_cinfo_release_lseg (&flo->commit_info, lseg); +spin_unlock (&inode->i_lock); }_filelayout_free_lseg (fl); }
@@ -1163,6 +1168,16 @@ filelayout_get_ds_info(struct inode *inode)return &FILELAYOUT_FROM_HDR (layout)->commit_info; } +static void +filelayout_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo, +struct inode *inode) +{ +spin_lock (&inode->i_lock); +pnfs_generic_ds_cinfo_destroy (fl_cinfo); +spin_unlock (&inode->i_lock); +} + +static struct pnfs_layoutdriver_type filelayout_type = { .id =LAYOUT_NFSV4_1_FILES , .name ="LAYOUT_NFSV4_1_FILES" ,
@@ -1176,6 +1191,7 @@ static struct pnfs_layoutdriver_type filelayout_type = { .pg_read_ops = &filelayout_pg_read_ops, .pg_write_ops = &filelayout_pg_write_ops, .get_ds_info = &filelayout_get_ds_info, + .release_ds_info = filelayout_release_ds_info, .mark_request_commit = filelayout_mark_request_commit, .clear_request_commit = pnfs_generic_clear_request_commit, .scan_commit_lists = pnfs_generic_scan_commit_lists,
fs/nfs/flexfilelayout/flexfilelayout.c +11 -0
@@ -580,6 +580,7 @@ ff_layout_free_lseg(struct pnfs_layout_segment *lseg)kfree (ffl->commit_info.buckets); ffl->commit_info.buckets =NULL ; } +pnfs_generic_ds_cinfo_release_lseg (&ffl->commit_info, lseg);spin_unlock (&inode->i_lock); }_ff_layout_free_lseg (fls);
@@ -2003,6 +2004,15 @@ ff_layout_get_ds_info(struct inode *inode)return &FF_LAYOUT_FROM_HDR (layout)->commit_info; } +static void +ff_layout_release_ds_info (struct pnfs_ds_commit_info *fl_cinfo, +struct inode *inode) +{ +spin_lock (&inode->i_lock); +pnfs_generic_ds_cinfo_destroy (fl_cinfo); +spin_unlock (&inode->i_lock); +} +static void ff_layout_free_deviceid_node (struct nfs4_deviceid_node *d) {
@@ -2503,6 +2513,7 @@ static struct pnfs_layoutdriver_type flexfilelayout_type = { .pg_read_ops = &ff_layout_pg_read_ops, .pg_write_ops = &ff_layout_pg_write_ops, .get_ds_info = ff_layout_get_ds_info, + .release_ds_info = ff_layout_release_ds_info, .free_deviceid_node = ff_layout_free_deviceid_node, .mark_request_commit = pnfs_layout_mark_request_commit, .clear_request_commit = pnfs_generic_clear_request_commit,
fs/nfs/internal.h +3 -1
@@ -534,9 +534,11 @@ void nfs_clear_pnfs_ds_commit_verifiers(struct pnfs_ds_commit_info *cinfo)pnfs_bucket_clear_pnfs_ds_commit_verifiers (cinfo->buckets, cinfo->nbuckets); -list_for_each_entry (array, &cinfo->commits, cinfo_list) +rcu_read_lock (); +list_for_each_entry_rcu (array, &cinfo->commits, cinfo_list)pnfs_bucket_clear_pnfs_ds_commit_verifiers (array->buckets, array->nbuckets); +rcu_read_unlock (); }#else static inline
fs/nfs/pnfs.c +1 -0
@@ -506,6 +506,7 @@ pnfs_init_lseg(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg, {INIT_LIST_HEAD (&lseg->pls_list);INIT_LIST_HEAD (&lseg->pls_lc_list); +INIT_LIST_HEAD (&lseg->pls_commits);refcount_set (&lseg->pls_refcount,1 );set_bit (NFS_LSEG_VALID , &lseg->pls_flags); lseg->pls_layout = lo;
fs/nfs/pnfs.h +4 -0
@@ -66,6 +66,7 @@ struct nfs4_pnfs_ds {struct pnfs_layout_segment {struct list_head pls_list;struct list_head pls_lc_list; +struct list_head pls_commits;struct pnfs_layout_range pls_range;refcount_t pls_refcount;u32 pls_seq;
@@ -370,6 +371,9 @@ void nfs4_deviceid_purge_client(const struct nfs_client *);/* pnfs_nfs.c */ struct pnfs_commit_array *pnfs_alloc_commit_array (size_t n,gfp_t gfp_flags);void pnfs_free_commit_array (struct pnfs_commit_array *p); +void pnfs_generic_ds_cinfo_release_lseg (struct pnfs_ds_commit_info *fl_cinfo, +struct pnfs_layout_segment *lseg); +void pnfs_generic_ds_cinfo_destroy (struct pnfs_ds_commit_info *fl_cinfo);void pnfs_generic_clear_request_commit (struct nfs_page *req,struct nfs_commit_info *cinfo);
fs/nfs/pnfs_nfs.c +85 -3
127 lines not on this page · raw message
include/linux/nfs_xdr.h +1 -0
@@ -1275,6 +1275,7 @@ struct pnfs_commit_array {struct list_head lseg_list;struct pnfs_layout_segment *lseg;struct rcu_head rcu; +refcount_t refcount;unsigned int nbuckets;struct pnfs_commit_bucket buckets[]; };
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 15/22 NFS: Remove bucket array from struct pnfs_ds_commit_info
In reply to [email protected]From: Trond Myklebust <[email protected]>
Remove the unused bucket array in struct pnfs_ds_commit_info.
--- fs/nfs/direct.c | 1 - fs/nfs/filelayout/filelayout.c | 75 +------------------------ fs/nfs/flexfilelayout/flexfilelayout.c | 76 -------------------------- fs/nfs/internal.h | 3 - fs/nfs/pnfs_nfs.c | 18 ------ include/linux/nfs_xdr.h | 13 ----- 6 files changed, 1 insertion(+), 185 deletions(-)
fs/nfs/direct.c +0 -1
@@ -217,7 +217,6 @@ static void nfs_direct_req_free(struct kref *kref) struct nfs_direct_req *dreq = container_of(kref, struct nfs_direct_req, kref);pnfs_release_ds_info (&dreq->ds_cinfo, dreq->inode); -nfs_free_pnfs_ds_cinfo (&dreq->ds_cinfo);if (dreq->l_ctx !=NULL )nfs_put_lock_context (dreq->l_ctx);if (dreq->ctx !=NULL )
fs/nfs/filelayout/filelayout.c +1 -74
99 lines not on this page · raw message
fs/nfs/flexfilelayout/flexfilelayout.c +0 -76
114 lines not on this page · raw message
fs/nfs/internal.h +0 -3
@@ -531,9 +531,6 @@ void nfs_clear_pnfs_ds_commit_verifiers(struct pnfs_ds_commit_info *cinfo) {struct pnfs_commit_array *array; -pnfs_bucket_clear_pnfs_ds_commit_verifiers (cinfo->buckets, - cinfo->nbuckets); -rcu_read_lock ();list_for_each_entry_rcu (array, &cinfo->commits, cinfo_list) pnfs_bucket_clear_pnfs_ds_commit_verifiers(array->buckets,
fs/nfs/pnfs_nfs.c +0 -18
@@ -292,12 +292,6 @@ int pnfs_generic_scan_commit_lists(struct nfs_commit_info *cinfo, int max)struct pnfs_commit_array *array;int rv =0 , cnt; - cnt =pnfs_bucket_scan_array (cinfo, fl_cinfo->buckets, - fl_cinfo->nbuckets, max); - rv += cnt; - max -= cnt; -if (!max) -return rv;rcu_read_lock ();list_for_each_entry_rcu (array, &fl_cinfo->commits, cinfo_list) {if (!array->lseg || !pnfs_get_commit_array (array))
@@ -353,11 +347,6 @@ void pnfs_generic_recover_commit_reqs(struct list_head *dst,unsigned int nwritten;lockdep_assert_held (&NFS_I (cinfo->inode)->commit_mutex); - nwritten =pnfs_bucket_recover_commit_reqs (dst, - fl_cinfo->buckets, - fl_cinfo->nbuckets, - cinfo); - fl_cinfo->nwritten -= nwritten;rcu_read_lock ();list_for_each_entry_rcu (array, &fl_cinfo->commits, cinfo_list) {if (!array->lseg || !pnfs_get_commit_array (array))
@@ -412,10 +401,6 @@ pnfs_generic_search_commit_reqs(struct nfs_commit_info *cinfo, struct page *pagestruct pnfs_commit_array *array;struct nfs_page *req; - req =pnfs_bucket_search_commit_reqs (fl_cinfo->buckets, - fl_cinfo->nbuckets, page); -if (req) -return req;list_for_each_entry (array, &fl_cinfo->commits, cinfo_list) { req =pnfs_bucket_search_commit_reqs (array->buckets, array->nbuckets, page);
@@ -550,9 +535,6 @@ pnfs_generic_commit_pagelist(struct inode *inode, struct list_head *mds_pages, nreq++; } - nreq +=pnfs_bucket_alloc_ds_commits (&list, fl_cinfo->buckets, - fl_cinfo->nbuckets, cinfo); - nreq +=pnfs_alloc_ds_commits_list (&list, fl_cinfo, cinfo);if (nreq ==0 ) goto out;
include/linux/nfs_xdr.h +0 -13
@@ -1284,8 +1284,6 @@ struct pnfs_ds_commit_info {struct list_head commits;unsigned int nwritten;unsigned int ncommitting; -unsigned int nbuckets; -struct pnfs_commit_bucket *buckets; };struct nfs41_state_protection {
@@ -1396,22 +1394,11 @@ struct nfs41_free_stateid_res { unsigned int status; }; -static inline void -nfs_free_pnfs_ds_cinfo (struct pnfs_ds_commit_info *cinfo) -{ -kfree (cinfo->buckets); -} -#else struct pnfs_ds_commit_info { }; -static inline void -nfs_free_pnfs_ds_cinfo (struct pnfs_ds_commit_info *cinfo) -{ -} -#endif /* CONFIG_NFS_V4_1 */ #ifdef CONFIG_NFS_V4_2
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 17/22 NFS/pNFS: Simplify bucket layout segment reference counting
In reply to [email protected]From: Trond Myklebust <[email protected]>
--- fs/nfs/pnfs_nfs.c | 39 ++++++++++++++++++++------------------- include/linux/nfs_xdr.h | 3 +-- 2 files changed, 21 insertions(+), 21 deletions(-)
fs/nfs/pnfs_nfs.c +20 -19
@@ -59,6 +59,17 @@ void pnfs_generic_commit_release(void *calldata) }EXPORT_SYMBOL_GPL (pnfs_generic_commit_release); +static struct pnfs_layout_segment * +pnfs_free_bucket_lseg (struct pnfs_commit_bucket *bucket) +{ +if (list_empty (&bucket->committing) &&list_empty (&bucket->written)) { +struct pnfs_layout_segment *freeme = bucket->lseg; + bucket->lseg =NULL ; +return freeme; + } +return NULL ; +} +/* The generic layer is about to remove the req from the commit list. * If this will make the bucket empty, it will need to put the lseg reference. * Note this must be called holding nfsi->commit_mutex
@@ -78,8 +89,7 @@ pnfs_generic_clear_request_commit(struct nfs_page *req, bucket =list_first_entry (&req->wb_list, struct pnfs_commit_bucket, written); - freeme = bucket->wlseg; - bucket->wlseg =NULL ; + freeme =pnfs_free_bucket_lseg (bucket); } out:nfs_request_remove_commit_list (req, cinfo);
@@ -103,8 +113,7 @@ pnfs_alloc_commit_array(size_t n, gfp_t gfp_flags)for (b = &p->buckets[0 ]; n !=0 ; b++, n--) {INIT_LIST_HEAD (&b->written);INIT_LIST_HEAD (&b->committing); - b->wlseg =NULL ; - b->clseg =NULL ; + b->lseg =NULL ; b->direct_verf.committed =NFS_INVALID_STABLE_HOW ; }return p;
@@ -246,12 +255,6 @@ pnfs_bucket_scan_ds_commit_list(struct pnfs_commit_bucket *bucket,if (ret) { cinfo->ds->nwritten -= ret; cinfo->ds->ncommitting += ret; -if (bucket->clseg ==NULL ) - bucket->clseg =pnfs_get_lseg (bucket->wlseg); -if (list_empty (src)) { -pnfs_put_lseg (bucket->wlseg); - bucket->wlseg =NULL ; - } }return ret; }
@@ -317,9 +320,8 @@ pnfs_bucket_recover_commit_reqs(struct list_head *dst,if (!nwritten)continue ; ret += nwritten; -if (list_empty (&b->written)) { - freeme = b->wlseg; - b->wlseg =NULL ; + freeme =pnfs_free_bucket_lseg (b); +if (freeme) {pnfs_put_lseg (freeme); goto restart; }
@@ -405,15 +407,12 @@ pnfs_bucket_get_committing(struct list_head *head,struct pnfs_commit_bucket *bucket, struct nfs_commit_info *cinfo) { - struct pnfs_layout_segment *freeme; struct list_head *pos;list_for_each (pos, &bucket->committing) cinfo->ds->ncommitting--;list_splice_init (&bucket->committing, head); - freeme = bucket->clseg; - bucket->clseg =NULL ; -return freeme; +return pnfs_free_bucket_lseg (bucket); } static struct nfs_commit_data *
@@ -425,6 +424,8 @@ pnfs_bucket_fetch_commitdata(struct pnfs_commit_bucket *bucket,if (!data)return NULL ; data->lseg =pnfs_bucket_get_committing (&data->pages, bucket, cinfo); +if (!data->lseg) + data->lseg =pnfs_get_lseg (bucket->lseg);return data; }
@@ -1182,8 +1183,8 @@ pnfs_layout_mark_request_commit(struct nfs_page *req,* off due to a rewrite, in which case it will be done in * pnfs_common_clear_request_commit */ -WARN_ON_ONCE (buckets[ds_commit_idx].wlseg !=NULL ); - buckets[ds_commit_idx].wlseg =pnfs_get_lseg (lseg); +if (!buckets[ds_commit_idx].lseg) + buckets[ds_commit_idx].lseg =pnfs_get_lseg (lseg); }set_bit (PG_COMMIT_TO_DS , &req->wb_flags); cinfo->ds->nwritten++;
include/linux/nfs_xdr.h +1 -2
@@ -1265,8 +1265,7 @@ struct nfstime4 {struct pnfs_commit_bucket {struct list_head written;struct list_head committing; -struct pnfs_layout_segment *wlseg; -struct pnfs_layout_segment *clseg; +struct pnfs_layout_segment *lseg;struct nfs_writeverf direct_verf; };
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 18/22 NFS/pNFS: Fix pnfs_layout_mark_request_commit() invalid layout segment handling
From: Trond Myklebust <[email protected]>
Fix up pnfs_layout_mark_request_commit() to alway reschedule the write if the layout segment is invalid. Also minor cleanup.
--- fs/nfs/pnfs_nfs.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-)
fs/nfs/pnfs_nfs.c +12 -16
@@ -1166,26 +1166,22 @@ pnfs_layout_mark_request_commit(struct nfs_page *req, {struct list_head *list;struct pnfs_commit_array *array; -struct pnfs_commit_bucket *buckets; +struct pnfs_commit_bucket *bucket;mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); array =pnfs_lookup_commit_array (cinfo->ds, lseg); -if (!array) +if (!array || !pnfs_is_valid_lseg (lseg)) goto out_resched; - buckets = array->buckets; - list = &buckets[ds_commit_idx].written; -if (list_empty (list)) { -if (!pnfs_is_valid_lseg (lseg)) - goto out_resched; -/* Non-empty buckets hold a reference on the lseg. That ref -* is normally transferred to the COMMIT call and released -* there. It could also be released if the last req is pulled -* off due to a rewrite, in which case it will be done in -* pnfs_common_clear_request_commit -*/ -if (!buckets[ds_commit_idx].lseg) - buckets[ds_commit_idx].lseg =pnfs_get_lseg (lseg); - } + bucket = &array->buckets[ds_commit_idx]; + list = &bucket->written; +/* Non-empty buckets hold a reference on the lseg. That ref +* is normally transferred to the COMMIT call and released +* there. It could also be released if the last req is pulled +* off due to a rewrite, in which case it will be done in +* pnfs_common_clear_request_commit +*/ +if (!bucket->lseg) + bucket->lseg =pnfs_get_lseg (lseg);set_bit (PG_COMMIT_TO_DS , &req->wb_flags); cinfo->ds->nwritten++;
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 19/22 pNFS/flexfile: Don't merge layout segments if the mirrors don't match
From: Trond Myklebust <[email protected]>
Check that the number of mirrors, and the mirror information matches before deciding to merge layout segments in pNFS/flexfiles.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
fs/nfs/flexfilelayout/flexfilelayout.c +19 -0
@@ -283,6 +283,23 @@ static void _ff_layout_free_lseg(struct nfs4_ff_layout_segment *fls) } } +static bool +ff_lseg_match_mirrors (struct pnfs_layout_segment *l1, +struct pnfs_layout_segment *l2) +{ +const struct nfs4_ff_layout_segment *fl1 =FF_LAYOUT_LSEG (l1); +const struct nfs4_ff_layout_segment *fl2 =FF_LAYOUT_LSEG (l1); +u32 i; + +if (fl1->mirror_array_cnt != fl2->mirror_array_cnt) +return false; +for (i =0 ; i < fl1->mirror_array_cnt; i++) { +if (fl1->mirror_array[i] != fl2->mirror_array[i]) +return false; + } +return true; +} +static bool ff_lseg_range_is_after (const struct pnfs_layout_range *l1,const struct pnfs_layout_range *l2)
@@ -318,6 +335,8 @@ ff_lseg_merge(struct pnfs_layout_segment *new, new->pls_range.length);if (new_end < old->pls_range.offset)return false; +if (!ff_lseg_match_mirrors (new, old)) +return false;/* Mergeable: copy info from 'old' to 'new' */ if (new_end < old_end)
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 16/22 NFS/pNFS: Clean up pNFS commit operations
In reply to [email protected]From: Trond Myklebust <[email protected]>
Move the pNFS commit related operations into a separate structure that can be carried by the pnfs_ds_commit_info.
--- fs/nfs/direct.c | 6 +- fs/nfs/filelayout/filelayout.c | 20 +++-- fs/nfs/flexfilelayout/flexfilelayout.c | 19 +++-- fs/nfs/pnfs.h | 111 ++++++++++++++++--------- fs/nfs/pnfs_nfs.c | 13 +-- include/linux/nfs_xdr.h | 1 + 6 files changed, 99 insertions(+), 71 deletions(-)
fs/nfs/direct.c +2 -4
@@ -511,10 +511,7 @@ nfs_direct_write_scan_commit_list(struct inode *inode, struct nfs_commit_info *cinfo) {mutex_lock (&NFS_I (cinfo->inode)->commit_mutex); -#ifdef CONFIG_NFS_V4_1 -if (cinfo->ds !=NULL && cinfo->ds->nwritten !=0 ) -NFS_SERVER (inode)->pnfs_curr_ld->recover_commit_reqs (list, cinfo); -#endif +pnfs_recover_commit_reqs (list, cinfo);nfs_scan_commit_list (&cinfo->mds->list, list, cinfo,0 );mutex_unlock (&NFS_I (cinfo->inode)->commit_mutex); }
@@ -917,6 +914,7 @@ ssize_t nfs_file_direct_write(struct kiocb *iocb, struct iov_iter *iter) dreq->l_ctx = l_ctx;if (!is_sync_kiocb (iocb)) dreq->iocb = iocb; +pnfs_init_ds_commit_info (&dreq->ds_cinfo, inode);nfs_start_io_direct (inode);
fs/nfs/filelayout/filelayout.c +12 -8
@@ -49,6 +49,7 @@ MODULE_AUTHOR("Dean Hildebrand <[email protected]>");MODULE_DESCRIPTION ("The NFSv4 file layout driver" );#define FILELAYOUT_POLL_RETRY_MAX (15*HZ) +static const struct pnfs_commit_ops filelayout_commit_ops; static loff_t filelayout_get_dense_offset(struct nfs4_filelayout_segment *flseg,
@@ -1045,6 +1046,7 @@ filelayout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags)if (flo ==NULL )return NULL ;INIT_LIST_HEAD (&flo->commit_info.commits); + flo->commit_info.ops = &filelayout_commit_ops;return &flo->generic_hdr; }
@@ -1094,6 +1096,16 @@ filelayout_release_ds_info(struct pnfs_ds_commit_info *fl_cinfo,spin_unlock (&inode->i_lock); } +static const struct pnfs_commit_ops filelayout_commit_ops = { + .setup_ds_info = filelayout_setup_ds_info, + .release_ds_info = filelayout_release_ds_info, + .mark_request_commit = filelayout_mark_request_commit, + .clear_request_commit = pnfs_generic_clear_request_commit, + .scan_commit_lists = pnfs_generic_scan_commit_lists, + .recover_commit_reqs = pnfs_generic_recover_commit_reqs, + .search_commit_reqs = pnfs_generic_search_commit_reqs, + .commit_pagelist = filelayout_commit_pagelist, +};static struct pnfs_layoutdriver_type filelayout_type = { .id =LAYOUT_NFSV4_1_FILES ,
@@ -1108,14 +1120,6 @@ static struct pnfs_layoutdriver_type filelayout_type = { .pg_read_ops = &filelayout_pg_read_ops, .pg_write_ops = &filelayout_pg_write_ops, .get_ds_info = &filelayout_get_ds_info, - .setup_ds_info = filelayout_setup_ds_info, - .release_ds_info = filelayout_release_ds_info, - .mark_request_commit = filelayout_mark_request_commit, - .clear_request_commit = pnfs_generic_clear_request_commit, - .scan_commit_lists = pnfs_generic_scan_commit_lists, - .recover_commit_reqs = pnfs_generic_recover_commit_reqs, - .search_commit_reqs = pnfs_generic_search_commit_reqs, - .commit_pagelist = filelayout_commit_pagelist, .read_pagelist = filelayout_read_pagelist, .write_pagelist = filelayout_write_pagelist, .alloc_deviceid_node = filelayout_alloc_deviceid_node,
fs/nfs/flexfilelayout/flexfilelayout.c +12 -7
@@ -32,6 +32,7 @@static unsigned short io_maxretrans; +static const struct pnfs_commit_ops ff_layout_commit_ops;static void ff_layout_read_record_layoutstats_done (struct rpc_task *task,struct nfs_pgio_header *hdr); static int ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
@@ -52,6 +53,7 @@ ff_layout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags)INIT_LIST_HEAD (&ffl->error_list);INIT_LIST_HEAD (&ffl->mirrors); ffl->last_report_time =ktime_get (); + ffl->commit_info.ops = &ff_layout_commit_ops;return &ffl->generic_hdr; }else return NULL ;
@@ -2440,6 +2442,16 @@ ff_layout_set_layoutdriver(struct nfs_server *server,return 0 ; } +static const struct pnfs_commit_ops ff_layout_commit_ops = { + .setup_ds_info = ff_layout_setup_ds_info, + .release_ds_info = ff_layout_release_ds_info, + .mark_request_commit = pnfs_layout_mark_request_commit, + .clear_request_commit = pnfs_generic_clear_request_commit, + .scan_commit_lists = pnfs_generic_scan_commit_lists, + .recover_commit_reqs = pnfs_generic_recover_commit_reqs, + .commit_pagelist = ff_layout_commit_pagelist, +}; +static struct pnfs_layoutdriver_type flexfilelayout_type = { .id =LAYOUT_FLEX_FILES , .name ="LAYOUT_FLEX_FILES" ,
@@ -2455,14 +2467,7 @@ static struct pnfs_layoutdriver_type flexfilelayout_type = { .pg_read_ops = &ff_layout_pg_read_ops, .pg_write_ops = &ff_layout_pg_write_ops, .get_ds_info = ff_layout_get_ds_info, - .setup_ds_info = ff_layout_setup_ds_info, - .release_ds_info = ff_layout_release_ds_info, .free_deviceid_node = ff_layout_free_deviceid_node, - .mark_request_commit = pnfs_layout_mark_request_commit, - .clear_request_commit = pnfs_generic_clear_request_commit, - .scan_commit_lists = pnfs_generic_scan_commit_lists, - .recover_commit_reqs = pnfs_generic_recover_commit_reqs, - .commit_pagelist = ff_layout_commit_pagelist, .read_pagelist = ff_layout_read_pagelist, .write_pagelist = ff_layout_write_pagelist, .alloc_deviceid_node = ff_layout_alloc_deviceid_node,
fs/nfs/pnfs.h +71 -40
181 lines not on this page · raw message
fs/nfs/pnfs_nfs.c +1 -12
@@ -149,17 +149,6 @@ pnfs_add_commit_array(struct pnfs_ds_commit_info *fl_cinfo, }EXPORT_SYMBOL_GPL (pnfs_add_commit_array); -static void -pnfs_setup_ds_info (struct pnfs_ds_commit_info *fl_cinfo, -struct pnfs_layout_segment *lseg) -{ -struct inode *inode = lseg->pls_layout->plh_inode; -struct pnfs_layoutdriver_type *ld =NFS_SERVER (inode)->pnfs_curr_ld; - -if (ld->setup_ds_info !=NULL ) - ld->setup_ds_info (fl_cinfo, lseg); -} -static struct pnfs_commit_array *pnfs_lookup_commit_array (struct pnfs_ds_commit_info *fl_cinfo,struct pnfs_layout_segment *lseg)
@@ -170,7 +159,7 @@ pnfs_lookup_commit_array(struct pnfs_ds_commit_info *fl_cinfo, array =pnfs_find_commit_array_by_lseg (fl_cinfo, lseg);if (!array) {rcu_read_unlock (); -pnfs_setup_ds_info (fl_cinfo, lseg); + fl_cinfo->ops->setup_ds_info (fl_cinfo, lseg);rcu_read_lock (); array =pnfs_find_commit_array_by_lseg (fl_cinfo, lseg); }
include/linux/nfs_xdr.h +1 -0
@@ -1284,6 +1284,7 @@ struct pnfs_ds_commit_info {struct list_head commits;unsigned int nwritten;unsigned int ncommitting; +const struct pnfs_commit_ops *ops; };struct nfs41_state_protection {
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 20/22 pNFS/flexfiles: Check the layout segment range before doing I/O
In reply to [email protected]From: Trond Myklebust <[email protected]>
When starting to read or write with a layout segment, check that the range matches our request.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 12 ++++++++++-- fs/nfs/pnfs.c | 3 ++- fs/nfs/pnfs.h | 1 + 3 files changed, 13 insertions(+), 3 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +10 -2
@@ -830,6 +830,14 @@ ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio, } } +static void +ff_layout_pg_check_layout (struct nfs_pageio_descriptor *pgio, +struct nfs_page *req) +{ +pnfs_generic_pg_check_layout (pgio); +pnfs_generic_pg_check_range (pgio, req); +} +static void ff_layout_pg_init_read (struct nfs_pageio_descriptor *pgio,struct nfs_page *req)
@@ -840,7 +848,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,int ds_idx; retry: -pnfs_generic_pg_check_layout (pgio); +ff_layout_pg_check_layout (pgio, req);/* Use full layout for now */ if (!pgio->pg_lseg) {ff_layout_pg_get_read (pgio, req, false);
@@ -900,7 +908,7 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,int i; retry: -pnfs_generic_pg_check_layout (pgio); +ff_layout_pg_check_layout (pgio, req);if (!pgio->pg_lseg) { pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, nfs_req_openctx(req),
fs/nfs/pnfs.c +2 -1
@@ -2611,7 +2611,7 @@ EXPORT_SYMBOL_GPL(pnfs_generic_pg_check_layout);* Check for any intersection between the request and the pgio->pg_lseg, * and if none, put this pgio->pg_lseg away. */ -static void +void pnfs_generic_pg_check_range (struct nfs_pageio_descriptor *pgio,struct nfs_page *req) {if (pgio->pg_lseg && !pnfs_lseg_request_intersecting (pgio->pg_lseg, req)) {
@@ -2619,6 +2619,7 @@ pnfs_generic_pg_check_range(struct nfs_pageio_descriptor *pgio, struct nfs_page pgio->pg_lseg =NULL ; } } +EXPORT_SYMBOL_GPL (pnfs_generic_pg_check_range);void pnfs_generic_pg_init_read (struct nfs_pageio_descriptor *pgio,struct nfs_page *req)
fs/nfs/pnfs.h +1 -0
@@ -252,6 +252,7 @@ void pnfs_put_lseg(struct pnfs_layout_segment *lseg);void set_pnfs_layoutdriver (struct nfs_server *,const struct nfs_fh *,struct nfs_fsinfo *);void unset_pnfs_layoutdriver (struct nfs_server *);void pnfs_generic_pg_check_layout (struct nfs_pageio_descriptor *pgio); +void pnfs_generic_pg_check_range (struct nfs_pageio_descriptor *pgio,struct nfs_page *req);void pnfs_generic_pg_init_read (struct nfs_pageio_descriptor *,struct nfs_page *);int pnfs_generic_pg_readpages (struct nfs_pageio_descriptor *desc); void pnfs_generic_pg_init_write(struct nfs_pageio_descriptor *pgio,
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 21/22 pNFS/flexfiles: remove requirement for whole file layouts
From: Trond Myklebust <[email protected]>
Remove the requirement that the server always sends whole file layouts.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 21 --------------------- 1 file changed, 21 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +0 -21
@@ -257,24 +257,6 @@ static void ff_layout_free_mirror_array(struct nfs4_ff_layout_segment *fls)ff_layout_put_mirror (fls->mirror_array[i]); } -static int ff_layout_check_layout (struct nfs4_layoutget_res *lgr) -{ -int ret =0 ; - -dprintk ("--> %s\n" , __func__); - -/* FIXME: remove this check when layout segment support is added */ -if (lgr->range.offset !=0 || - lgr->range.length !=NFS4_MAX_UINT64 ) { -dprintk ("%s Only whole file layouts supported. Use MDS i/o\n" , - __func__); - ret = -EINVAL ; - } - -dprintk ("--> %s returns %d\n" , __func__, ret); -return ret; -} -static void _ff_layout_free_lseg (struct nfs4_ff_layout_segment *fls) {if (fls) {
@@ -556,9 +538,6 @@ ff_layout_alloc_lseg(struct pnfs_layout_hdr *lh, out_sort_mirrors:ff_layout_sort_mirrors (fls); - rc =ff_layout_check_layout (lgr); -if (rc) - goto out_err_free; ret = &fls->generic_hdr;dprintk ("<-- %s (success)\n" , __func__); out_free_page:
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 22/22 pNFS/flexfiles: Specify the layout segment range in LAYOUTGET
From: Trond Myklebust <[email protected]>
Move from requesting only full file layout segments, to requesting layout segments that match our I/O size. This means the server is still free to return a full file layout, but we will no longer error out if it does not.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +4 -4
@@ -798,8 +798,8 @@ ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio,pnfs_put_lseg (pgio->pg_lseg); pgio->pg_lseg =pnfs_update_layout (pgio->pg_inode,nfs_req_openctx (req), -0 , -NFS4_MAX_UINT64 , +req_offset (req), + req->wb_bytes,IOMODE_READ , strict_iomode,GFP_KERNEL );
@@ -891,8 +891,8 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,if (!pgio->pg_lseg) { pgio->pg_lseg =pnfs_update_layout (pgio->pg_inode,nfs_req_openctx (req), -0 , -NFS4_MAX_UINT64 , +req_offset (req), + req->wb_bytes,IOMODE_RW , false,GFP_NOFS );
Signature · 1 line
2.25.1
kbuild test robot
build botRe: [PATCH 04/22] NFSv4/pnfs: Support a list of commit arrays in struct pnfs_ds_commit_info
In reply to [email protected]I love your patch! Yet something to improve:
Hi,
I love your patch! Yet something to improve:
[auto build test ERROR on nfs/linux-next]
[also build test ERROR on v5.6-rc7 next-20200326]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/trondmy-kernel-org/Fix-NFS-commit-to-DS/20200326-051206 base: git://git.linux-nfs.org/projects/trondmy/linux-nfs.git linux-next config: s390-randconfig-a001-20200326 (attached as .config) compiler: s390-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
fs/nfs/direct.c: In function 'nfs_direct_req_alloc':
>> fs/nfs/direct.c:308:32: error: 'struct pnfs_ds_commit_info' has no member named 'commits'
308 | INIT_LIST_HEAD(&dreq->ds_cinfo.commits);
| ^vim +308 fs/nfs/direct.c
295
296 static inline struct nfs_direct_req *nfs_direct_req_alloc(void)
297 {
298 struct nfs_direct_req *dreq;
299
300 dreq = kmem_cache_zalloc(nfs_direct_cachep, GFP_KERNEL);
301 if (!dreq)
302 return NULL;
303
304 kref_init(&dreq->kref);
305 kref_get(&dreq->kref);
306 init_completion(&dreq->completion);
307 INIT_LIST_HEAD(&dreq->mds_cinfo.list);
> 308 INIT_LIST_HEAD(&dreq->ds_cinfo.commits);
309 dreq->verf.committed = NFS_INVALID_STABLE_HOW; /* not set yet */
310 INIT_WORK(&dreq->work, nfs_direct_write_schedule_work);
311 spin_lock_init(&dreq->lock);
312
313 return dreq;
314 }
315
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kbuild test robot
build botRe: [PATCH 16/22] NFS/pNFS: Clean up pNFS commit operations
In reply to [email protected]I love your patch! Yet something to improve:
Hi,
I love your patch! Yet something to improve:
[auto build test ERROR on nfs/linux-next]
[also build test ERROR on v5.6-rc7 next-20200326]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/trondmy-kernel-org/Fix-NFS-commit-to-DS/20200326-051206 base: git://git.linux-nfs.org/projects/trondmy/linux-nfs.git linux-next config: s390-randconfig-a001-20200326 (attached as .config) compiler: s390-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from fs/nfs/nfs4xdr.c:61:
fs/nfs/pnfs.h: In function 'pnfs_init_ds_commit_info':
>> fs/nfs/pnfs.h:808:26: error: 'struct pnfs_ds_commit_info' has no member named 'commits'
808 | INIT_LIST_HEAD(&fl_cinfo->commits);
| ^~
--
In file included from fs/nfs/direct.c:61:
fs/nfs/pnfs.h: In function 'pnfs_init_ds_commit_info':
>> fs/nfs/pnfs.h:808:26: error: 'struct pnfs_ds_commit_info' has no member named 'commits'
808 | INIT_LIST_HEAD(&fl_cinfo->commits);
| ^~
fs/nfs/direct.c: In function 'nfs_direct_req_alloc':
fs/nfs/direct.c:208:32: error: 'struct pnfs_ds_commit_info' has no member named 'commits'
208 | INIT_LIST_HEAD(&dreq->ds_cinfo.commits);
| ^vim +808 fs/nfs/pnfs.h
804
805 static inline void
806 pnfs_init_ds_commit_info(struct pnfs_ds_commit_info *fl_cinfo, struct inode *inode)
807 {
> 808 INIT_LIST_HEAD(&fl_cinfo->commits);
809 }
810
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org