NFS: Fix a number of memory leaks and use-after-free
Subsystems (2)
[email protected]
maintainercover NFS: Fix a number of memory leaks and use-after-free
From: Trond Myklebust <[email protected]>
When running xfstests with wsize=1024, a number of use-after-free issues and memory leaks can currently be hit. One of the more obvious leaks is seen when the generic/013 test fails due to the presence of sillyrenamed files that never go away.
After testing with kasan enabled, and adding some debugging code to detect leaked nfs_page and nfs_direct_req structures, I found a number of issues that appear to be fixed by the following patchset.
Trond Myklebust (10):
NFS: Fix a page leak in nfs_destroy_unlinked_subrequests()
NFS: Fix races nfs_page_group_destroy() vs
nfs_destroy_unlinked_subrequests()
NFS: Fix use-after-free issues in nfs_pageio_add_request()
NFS: Fix a request reference leak in nfs_direct_write_clear_reqs()
NFS: Fix memory leaks in nfs_pageio_stop_mirroring()
NFS: Remove the redundant function nfs_pgio_has_mirroring()
NFS: Clean up nfs_lock_and_join_requests()
NFS: Reverse the submission order of requests in
__nfs_pageio_add_request()
NFS: Refactor nfs_lock_and_join_requests()
NFS: Try to join page groups before an O_DIRECT retransmission
fs/nfs/direct.c | 21 +++ fs/nfs/internal.h | 6 - fs/nfs/pagelist.c | 350 +++++++++++++++++++++++++-------------- fs/nfs/write.c | 258 ++++++++++++++--------------- include/linux/nfs_page.h | 5 + 5 files changed, 379 insertions(+), 261 deletions(-)
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 1/10 NFS: Fix a page leak in nfs_destroy_unlinked_subrequests()
From: Trond Myklebust <[email protected]>
When we detach a subrequest from the list, we must also release the reference it holds to the parent.
--- fs/nfs/write.c | 1 + 1 file changed, 1 insertion(+)
fs/nfs/write.c +1 -0
@@ -444,6 +444,7 @@ nfs_destroy_unlinked_subrequests(struct nfs_page *destroy_list, } subreq->wb_head = subreq; +nfs_release_request (old_head);if (test_and_clear_bit (PG_INODE_REF , &subreq->wb_flags)) {nfs_release_request (subreq);
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 2/10 NFS: Fix races nfs_page_group_destroy() vs nfs_destroy_unlinked_subrequests()
From: Trond Myklebust <[email protected]>
When a subrequest is being detached from the subgroup, we want to ensure that it is not holding the group lock, or in the process of waiting for the group lock.
--- fs/nfs/pagelist.c | 67 +++++++++++++++++++++++++++------------- fs/nfs/write.c | 10 ++++-- include/linux/nfs_page.h | 2 ++ 3 files changed, 55 insertions(+), 24 deletions(-)
fs/nfs/pagelist.c +45 -22
@@ -133,47 +133,70 @@ nfs_async_iocounter_wait(struct rpc_task *task, struct nfs_lock_context *l_ctx)EXPORT_SYMBOL_GPL (nfs_async_iocounter_wait);/* -* nfs_page_group_lock - lock the head of the page group -* @req - request in group that is to be locked +* nfs_page_set_headlock - set the request PG_HEADLOCK +* @req: request that is to be locked * -* this lock must be held when traversing or modifying the page -* group list +* this lock must be held when modifying req->wb_head * * return 0 on success, < 0 on error */ int -nfs_page_group_lock (struct nfs_page *req) +nfs_page_set_headlock (struct nfs_page *req) { -struct nfs_page *head = req->wb_head; - -WARN_ON_ONCE (head != head->wb_head); - -if (!test_and_set_bit (PG_HEADLOCK , &head->wb_flags)) +if (!test_and_set_bit (PG_HEADLOCK , &req->wb_flags))return 0 ; -set_bit (PG_CONTENDED1 , &head->wb_flags); +set_bit (PG_CONTENDED1 , &req->wb_flags);smp_mb__after_atomic (); -return wait_on_bit_lock (&head->wb_flags,PG_HEADLOCK , +return wait_on_bit_lock (&req->wb_flags,PG_HEADLOCK ,TASK_UNINTERRUPTIBLE ); }/* -* nfs_page_group_unlock - unlock the head of the page group -* @req - request in group that is to be unlocked +* nfs_page_clear_headlock - clear the request PG_HEADLOCK +* @req: request that is to be locked */ void -nfs_page_group_unlock (struct nfs_page *req) +nfs_page_clear_headlock (struct nfs_page *req) { -struct nfs_page *head = req->wb_head; - -WARN_ON_ONCE (head != head->wb_head); -smp_mb__before_atomic (); -clear_bit (PG_HEADLOCK , &head->wb_flags); +clear_bit (PG_HEADLOCK , &req->wb_flags);smp_mb__after_atomic (); -if (!test_bit (PG_CONTENDED1 , &head->wb_flags)) +if (!test_bit (PG_CONTENDED1 , &req->wb_flags))return ; -wake_up_bit (&head->wb_flags,PG_HEADLOCK ); +wake_up_bit (&req->wb_flags,PG_HEADLOCK ); +} + +/* +* nfs_page_group_lock - lock the head of the page group +* @req: request in group that is to be locked +* +* this lock must be held when traversing or modifying the page +* group list +* +* return 0 on success, < 0 on error +*/ +int +nfs_page_group_lock (struct nfs_page *req) +{ +int ret; + + ret =nfs_page_set_headlock (req); +if (ret || req->wb_head == req) +return ret; +return nfs_page_set_headlock (req->wb_head); +} + +/* +* nfs_page_group_unlock - unlock the head of the page group +* @req: request in group that is to be unlocked +*/ +void +nfs_page_group_unlock (struct nfs_page *req) +{ +if (req != req->wb_head) +nfs_page_clear_headlock (req->wb_head); +nfs_page_clear_headlock (req); }/*
fs/nfs/write.c +8 -2
@@ -428,22 +428,28 @@ nfs_destroy_unlinked_subrequests(struct nfs_page *destroy_list, destroy_list = (subreq->wb_this_page == old_head) ?NULL : subreq->wb_this_page; +/* Note: lock subreq in order to change subreq->wb_head */ +nfs_page_set_headlock (subreq);WARN_ON_ONCE (old_head != subreq->wb_head);/* make sure old group is not used */ subreq->wb_this_page = subreq; + subreq->wb_head = subreq;clear_bit (PG_REMOVE , &subreq->wb_flags);/* Note: races with nfs_page_group_destroy() */ if (!kref_read (&subreq->wb_kref)) {/* Check if we raced with nfs_page_group_destroy() */ -if (test_and_clear_bit (PG_TEARDOWN , &subreq->wb_flags)) +if (test_and_clear_bit (PG_TEARDOWN , &subreq->wb_flags)) { +nfs_page_clear_headlock (subreq);nfs_free_request (subreq); + }else +nfs_page_clear_headlock (subreq);continue ; } +nfs_page_clear_headlock (subreq); - subreq->wb_head = subreq;nfs_release_request (old_head);if (test_and_clear_bit (PG_INODE_REF , &subreq->wb_flags)) {
include/linux/nfs_page.h +2 -0
@@ -142,6 +142,8 @@ extern void nfs_unlock_and_release_request(struct nfs_page *);extern int nfs_page_group_lock (struct nfs_page *);extern void nfs_page_group_unlock (struct nfs_page *);extern bool nfs_page_group_sync_on_bit (struct nfs_page *,unsigned int ); +extern int nfs_page_set_headlock (struct nfs_page *req); +extern void nfs_page_clear_headlock (struct nfs_page *req);extern bool nfs_async_iocounter_wait (struct rpc_task *,struct nfs_lock_context *);/*
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 3/10 NFS: Fix use-after-free issues in nfs_pageio_add_request()
From: Trond Myklebust <[email protected]>
We need to ensure that we create the mirror requests before calling nfs_pageio_add_request_mirror() on the request we are adding. Otherwise, we can end up with a use-after-free if the call to nfs_pageio_add_request_mirror() triggers I/O.
--- fs/nfs/pagelist.c | 48 +++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-)
fs/nfs/pagelist.c +24 -24
@@ -1191,38 +1191,38 @@ int nfs_pageio_add_request(struct nfs_pageio_descriptor *desc,if (desc->pg_error <0 ) goto out_failed; -for (midx =0 ; midx < desc->pg_mirror_count; midx++) { -if (midx) { -nfs_page_group_lock (req); - -/* find the last request */ -for (lastreq = req->wb_head; - lastreq->wb_this_page != req->wb_head; - lastreq = lastreq->wb_this_page) - ; - - dupreq =nfs_create_subreq (req, lastreq, - pgbase, offset, bytes); - -nfs_page_group_unlock (req); -if (IS_ERR (dupreq)) { - desc->pg_error =PTR_ERR (dupreq); - goto out_failed; - } - }else - dupreq = req; +/* Create the mirror instances first, and fire them off */ +for (midx =1 ; midx < desc->pg_mirror_count; midx++) { +nfs_page_group_lock (req); + +/* find the last request */ +for (lastreq = req->wb_head; + lastreq->wb_this_page != req->wb_head; + lastreq = lastreq->wb_this_page) + ; + + dupreq =nfs_create_subreq (req, lastreq, + pgbase, offset, bytes); + +nfs_page_group_unlock (req); +if (IS_ERR (dupreq)) { + desc->pg_error =PTR_ERR (dupreq); + goto out_failed; + } -if (nfs_pgio_has_mirroring (desc)) - desc->pg_mirror_idx = midx; + desc->pg_mirror_idx = midx;if (!nfs_pageio_add_request_mirror (desc, dupreq)) goto out_cleanup_subreq; } + desc->pg_mirror_idx =0 ; +if (!nfs_pageio_add_request_mirror (desc, req)) + goto out_failed; +return 1 ; out_cleanup_subreq: -if (req != dupreq) -nfs_pageio_cleanup_request (desc, dupreq); +nfs_pageio_cleanup_request (desc, dupreq); out_failed:nfs_pageio_error_cleanup (desc);return 0 ;
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 4/10 NFS: Fix a request reference leak in nfs_direct_write_clear_reqs()
From: Trond Myklebust <[email protected]>
nfs_direct_write_scan_commit_list() will lock the request and bump the reference count, but we also need to account for the reference that was taken when we initially added the request to the commit list.
--- fs/nfs/direct.c | 1 + 1 file changed, 1 insertion(+)
fs/nfs/direct.c +1 -0
@@ -646,6 +646,7 @@ static void nfs_direct_write_clear_reqs(struct nfs_direct_req *dreq)while (!list_empty (&reqs)) { req =nfs_list_entry (reqs.next);nfs_list_remove_request (req); +nfs_release_request (req);nfs_unlock_and_release_request (req); } }
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 5/10 NFS: Fix memory leaks in nfs_pageio_stop_mirroring()
From: Trond Myklebust <[email protected]>
If we just set the mirror count to 1 without first clearing out the mirrors, we can leak queued up requests.
--- fs/nfs/pagelist.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
fs/nfs/pagelist.c +8 -9
@@ -900,15 +900,6 @@ static void nfs_pageio_setup_mirroring(struct nfs_pageio_descriptor *pgio, pgio->pg_mirror_count = mirror_count; } -/* -* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) -*/ -void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) -{ - pgio->pg_mirror_count =1 ; - pgio->pg_mirror_idx =0 ; -} -static void nfs_pageio_cleanup_mirroring (struct nfs_pageio_descriptor *pgio) { pgio->pg_mirror_count =1 ;
@@ -1334,6 +1325,14 @@ void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *desc, pgoff_t index) } } +/* +* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) +*/ +void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) +{ +nfs_pageio_complete (pgio); +} +int __init nfs_init_nfspagecache (void ) { nfs_page_cachep = kmem_cache_create("nfs_page",
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 6/10 NFS: Remove the redundant function nfs_pgio_has_mirroring()
From: Trond Myklebust <[email protected]>
We need to trust that desc->pg_mirror_idx is set correctly, whether or not mirroring is enabled.
--- fs/nfs/internal.h | 6 ------ fs/nfs/pagelist.c | 7 ++----- 2 files changed, 2 insertions(+), 11 deletions(-)
fs/nfs/internal.h +0 -6
@@ -274,12 +274,6 @@ void nfs_free_request(struct nfs_page *req);struct nfs_pgio_mirror *nfs_pgio_current_mirror (struct nfs_pageio_descriptor *desc); -static inline bool nfs_pgio_has_mirroring (struct nfs_pageio_descriptor *desc) -{ -WARN_ON_ONCE (desc->pg_mirror_count <1 ); -return desc->pg_mirror_count >1 ; -} -static inline bool nfs_match_open_context (const struct nfs_open_context *ctx1,const struct nfs_open_context *ctx2) {
fs/nfs/pagelist.c +2 -5
@@ -33,9 +33,7 @@ static const struct rpc_call_ops nfs_pgio_common_ops;struct nfs_pgio_mirror *nfs_pgio_current_mirror (struct nfs_pageio_descriptor *desc) { -return nfs_pgio_has_mirroring (desc) ? - &desc->pg_mirrors[desc->pg_mirror_idx] : - &desc->pg_mirrors[0 ]; +return &desc->pg_mirrors[desc->pg_mirror_idx]; }EXPORT_SYMBOL_GPL (nfs_pgio_current_mirror);
@@ -1231,8 +1229,7 @@ static void nfs_pageio_complete_mirror(struct nfs_pageio_descriptor *desc,struct nfs_pgio_mirror *mirror = &desc->pg_mirrors[mirror_idx];u32 restore_idx = desc->pg_mirror_idx; -if (nfs_pgio_has_mirroring (desc)) - desc->pg_mirror_idx = mirror_idx; + desc->pg_mirror_idx = mirror_idx;for (;;) {nfs_pageio_doio (desc);if (desc->pg_error <0 || !mirror->pg_recoalesce)
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 7/10 NFS: Clean up nfs_lock_and_join_requests()
From: Trond Myklebust <[email protected]>
Clean up nfs_lock_and_join_requests() to simplify the calculation of the range covered by the page group, taking into account the presence of mirrors.
--- fs/nfs/pagelist.c | 74 ++++++++++++++++++++++++++++++++ fs/nfs/write.c | 91 +++++++++------------------------------- include/linux/nfs_page.h | 1 + 3 files changed, 95 insertions(+), 71 deletions(-)
fs/nfs/pagelist.c +74 -0
@@ -130,6 +130,80 @@ nfs_async_iocounter_wait(struct rpc_task *task, struct nfs_lock_context *l_ctx) }EXPORT_SYMBOL_GPL (nfs_async_iocounter_wait); +/* +* nfs_unroll_locks - unlock all newly locked reqs and wait on @req +* @head: head request of page group, must be holding head lock +* @req: request that couldn't lock and needs to wait on the req bit lock +* +* This is a helper function for nfs_lock_and_join_requests +* returns 0 on success, < 0 on error. +*/ +static void +nfs_unroll_locks (struct nfs_page *head,struct nfs_page *req) +{ +struct nfs_page *tmp; + +/* relinquish all the locks successfully grabbed this run */ +for (tmp = head->wb_this_page ; tmp != req; tmp = tmp->wb_this_page) { +if (!kref_read (&tmp->wb_kref)) +continue ; +nfs_unlock_and_release_request (tmp); + } +} + +/* +* nfs_page_group_lock_subreq - try to lock a subrequest +* @head: head request of page group +* @subreq: request to lock +* +* This is a helper function for nfs_lock_and_join_requests which +* must be called with the head request and page group both locked. +* On error, it returns with the page group unlocked. +*/ +static int +nfs_page_group_lock_subreq (struct nfs_page *head,struct nfs_page *subreq) +{ +int ret; + +if (!kref_get_unless_zero (&subreq->wb_kref)) +return 0 ; +while (!nfs_lock_request (subreq)) { +nfs_page_group_unlock (head); + ret =nfs_wait_on_request (subreq); +if (!ret) + ret =nfs_page_group_lock (head); +if (ret <0 ) { +nfs_unroll_locks (head, subreq); +nfs_release_request (subreq); +return ret; + } + } +return 0 ; +} + +/* +* nfs_page_group_lock_subrequests - try to lock the subrequests +* @head: head request of page group +* +* This is a helper function for nfs_lock_and_join_requests which +* must be called with the head request and page group both locked. +* On error, it returns with the page group unlocked. +*/ +int nfs_page_group_lock_subrequests (struct nfs_page *head) +{ +struct nfs_page *subreq; +int ret; + +/* lock each request in the page group */ +for (subreq = head->wb_this_page; subreq != head; + subreq = subreq->wb_this_page) { + ret =nfs_page_group_lock_subreq (head, subreq); +if (ret <0 ) +return ret; + } +return 0 ; +} +/* * nfs_page_set_headlock - set the request PG_HEADLOCK * @req: request that is to be locked
fs/nfs/write.c +20 -71
120 lines not on this page · raw message
include/linux/nfs_page.h +1 -0
@@ -139,6 +139,7 @@ extern size_t nfs_generic_pg_test(struct nfs_pageio_descriptor *desc,extern int nfs_wait_on_request (struct nfs_page *);extern void nfs_unlock_request (struct nfs_page *req);extern void nfs_unlock_and_release_request (struct nfs_page *); +extern int nfs_page_group_lock_subrequests (struct nfs_page *head);extern int nfs_page_group_lock (struct nfs_page *);extern void nfs_page_group_unlock (struct nfs_page *);extern bool nfs_page_group_sync_on_bit (struct nfs_page *,unsigned int );
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 8/10 NFS: Reverse the submission order of requests in __nfs_pageio_add_request()
From: Trond Myklebust <[email protected]>
If we have to split the request up into subrequests, we have to submit the request pointed to by the function call parameter last, in case there is an error or other issue that causes us to exit before the last request is submitted. The reason is that the caller is expected to perform cleanup in those cases.
--- fs/nfs/pagelist.c | 133 ++++++++++++++++++++++------------------------ 1 file changed, 64 insertions(+), 69 deletions(-)
fs/nfs/pagelist.c +64 -69
@@ -454,15 +454,23 @@ nfs_create_request(struct nfs_open_context *ctx, struct page *page, }static struct nfs_page * -nfs_create_subreq (struct nfs_page *req,struct nfs_page *last, -unsigned int pgbase,unsigned int offset, +nfs_create_subreq (struct nfs_page *req, +unsigned int pgbase, +unsigned int offset,unsigned int count) { +struct nfs_page *last;struct nfs_page *ret; ret =__nfs_create_request (req->wb_lock_context, req->wb_page, pgbase, offset, count);if (!IS_ERR (ret)) { +/* find the last request */ +for (last = req->wb_head; + last->wb_this_page != req->wb_head; + last = last->wb_this_page) + ; +nfs_lock_request (ret); ret->wb_index = req->wb_index;nfs_page_group_init (ret, last);
@@ -988,7 +996,7 @@ static bool nfs_match_lock_context(const struct nfs_lock_context *l1, }/** -* nfs_can_coalesce_requests - test two requests for compatibility +* nfs_coalesce_size - test two requests for compatibility * @prev: pointer to nfs_page * @req: pointer to nfs_page * @pgio: pointer to nfs_pagio_descriptor
@@ -997,41 +1005,36 @@ static bool nfs_match_lock_context(const struct nfs_lock_context *l1,* page data area they describe is contiguous, and that their RPC * credentials, NFSv4 open state, and lockowners are the same. * -* Return 'true' if this is the case, else return 'false'. +* Returns size of the request that can be coalesced */ -static bool nfs_can_coalesce_requests (struct nfs_page *prev, +static unsigned int nfs_coalesce_size (struct nfs_page *prev,struct nfs_page *req,struct nfs_pageio_descriptor *pgio) { -size_t size;struct file_lock_context *flctx;if (prev) {if (!nfs_match_open_context (nfs_req_openctx (req),nfs_req_openctx (prev))) -return false; +return 0 ; flctx =d_inode (nfs_req_openctx (req)->dentry)->i_flctx;if (flctx !=NULL && !(list_empty_careful (&flctx->flc_posix) &&list_empty_careful (&flctx->flc_flock)) && !nfs_match_lock_context (req->wb_lock_context, prev->wb_lock_context)) -return false; +return 0 ;if (req_offset (req) !=req_offset (prev) + prev->wb_bytes) -return false; +return 0 ;if (req->wb_page == prev->wb_page) {if (req->wb_pgbase != prev->wb_pgbase + prev->wb_bytes) -return false; +return 0 ; }else {if (req->wb_pgbase !=0 || prev->wb_pgbase + prev->wb_bytes !=PAGE_SIZE ) -return false; +return 0 ; } } - size = pgio->pg_ops->pg_test (pgio, prev, req); -WARN_ON_ONCE (size > req->wb_bytes); -if (size && size < req->wb_bytes) - req->wb_bytes = size; -return size >0 ; +return pgio->pg_ops->pg_test (pgio, prev, req); }/**
@@ -1039,15 +1042,16 @@ static bool nfs_can_coalesce_requests(struct nfs_page *prev,* @desc: destination io descriptor * @req: request * -* Returns true if the request 'req' was successfully coalesced into the -* existing list of pages 'desc'. +* If the request 'req' was successfully coalesced into the existing list +* of pages 'desc', it returns the size of req. */ -static int nfs_pageio_do_add_request (struct nfs_pageio_descriptor *desc, -struct nfs_page *req) +static unsigned int +nfs_pageio_do_add_request (struct nfs_pageio_descriptor *desc, +struct nfs_page *req) {struct nfs_pgio_mirror *mirror =nfs_pgio_current_mirror (desc); -struct nfs_page *prev =NULL ; +unsigned int size;if (mirror->pg_count !=0 ) { prev =nfs_list_entry (mirror->pg_list.prev);
@@ -1067,11 +1071,12 @@ static int nfs_pageio_do_add_request(struct nfs_pageio_descriptor *desc,return 0 ; } -if (!nfs_can_coalesce_requests (prev, req, desc)) -return 0 ; + size =nfs_coalesce_size (prev, req, desc); +if (size < req->wb_bytes) +return size;nfs_list_move_request (req, &mirror->pg_list); mirror->pg_count += req->wb_bytes; -return 1 ; +return req->wb_bytes; }/*
@@ -1111,7 +1116,8 @@ nfs_pageio_cleanup_request(struct nfs_pageio_descriptor *desc,* @req: request * * This may split a request into subrequests which are all part of the -* same page group. +* same page group. If so, it will submit @req as the last one, to ensure +* the pointer to @req is still valid in case of failure. * * Returns true if the request 'req' was successfully coalesced into the * existing list of pages 'desc'.
@@ -1120,51 +1126,50 @@ static int __nfs_pageio_add_request(struct nfs_pageio_descriptor *desc, struct nfs_page *req) {struct nfs_pgio_mirror *mirror =nfs_pgio_current_mirror (desc); -struct nfs_page *subreq; -unsigned int bytes_left =0 ; -unsigned int offset, pgbase; +unsigned int size, subreq_size;nfs_page_group_lock (req); subreq = req; - bytes_left = subreq->wb_bytes; - offset = subreq->wb_offset; - pgbase = subreq->wb_pgbase; - -do { -if (!nfs_pageio_do_add_request (desc, subreq)) { -/* make sure pg_test call(s) did nothing */ -WARN_ON_ONCE (subreq->wb_bytes != bytes_left); -WARN_ON_ONCE (subreq->wb_offset != offset); -WARN_ON_ONCE (subreq->wb_pgbase != pgbase); - + subreq_size = subreq->wb_bytes; +for (;;) { + size =nfs_pageio_do_add_request (desc, subreq); +if (size == subreq_size) { +/* We successfully submitted a request */ +if (subreq == req) +break ; + req->wb_pgbase += size; + req->wb_bytes -= size; + req->wb_offset += size; + subreq_size = req->wb_bytes; + subreq = req; +continue ; + } +if (WARN_ON_ONCE (subreq != req)) { +nfs_page_group_unlock (req); +nfs_pageio_cleanup_request (desc, subreq); + subreq = req; + subreq_size = req->wb_bytes; +nfs_page_group_lock (req); + } +if (!size) { +/* Can't coalesce any more, so do I/O */ nfs_page_group_unlock (req); desc->pg_moreio =1 ;nfs_pageio_doio (desc);if (desc->pg_error <0 || mirror->pg_recoalesce) - goto out_cleanup_subreq; +return 0 ;/* retry add_request for this subreq */ nfs_page_group_lock (req);continue ; } - -/* check for buggy pg_test call(s) */ -WARN_ON_ONCE (subreq->wb_bytes + subreq->wb_pgbase >PAGE_SIZE ); -WARN_ON_ONCE (subreq->wb_bytes > bytes_left); -WARN_ON_ONCE (subreq->wb_bytes ==0 ); - - bytes_left -= subreq->wb_bytes; - offset += subreq->wb_bytes; - pgbase += subreq->wb_bytes; - -if (bytes_left) { - subreq =nfs_create_subreq (req, subreq, pgbase, - offset, bytes_left); -if (IS_ERR (subreq)) - goto err_ptr; - } - }while (bytes_left >0 ); + subreq =nfs_create_subreq (req, req->wb_pgbase, + req->wb_offset, size); +if (IS_ERR (subreq)) + goto err_ptr; + subreq_size = size; + }nfs_page_group_unlock (req);return 1 ;
@@ -1172,10 +1177,6 @@ static int __nfs_pageio_add_request(struct nfs_pageio_descriptor *desc, desc->pg_error =PTR_ERR (subreq);nfs_page_group_unlock (req);return 0 ; -out_cleanup_subreq: -if (req != subreq) -nfs_pageio_cleanup_request (desc, subreq); -return 0 ; }static int nfs_do_recoalesce (struct nfs_pageio_descriptor *desc)
@@ -1244,7 +1245,7 @@ int nfs_pageio_add_request(struct nfs_pageio_descriptor *desc, {u32 midx;unsigned int pgbase, offset, bytes; -struct nfs_page *dupreq, *lastreq; +struct nfs_page *dupreq; pgbase = req->wb_pgbase; offset = req->wb_offset;
@@ -1258,13 +1259,7 @@ int nfs_pageio_add_request(struct nfs_pageio_descriptor *desc,for (midx =1 ; midx < desc->pg_mirror_count; midx++) {nfs_page_group_lock (req); -/* find the last request */ -for (lastreq = req->wb_head; - lastreq->wb_this_page != req->wb_head; - lastreq = lastreq->wb_this_page) - ; - - dupreq =nfs_create_subreq (req, lastreq, + dupreq =nfs_create_subreq (req, pgbase, offset, bytes);nfs_page_group_unlock (req);
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 9/10 NFS: Refactor nfs_lock_and_join_requests()
From: Trond Myklebust <[email protected]>
Refactor nfs_lock_and_join_requests() in order to separate out the subrequest merging into its own function nfs_lock_and_join_group() that can be used by O_DIRECT.
--- fs/nfs/pagelist.c | 26 ++++++- fs/nfs/write.c | 164 +++++++++++++++++++++++---------------- include/linux/nfs_page.h | 1 + 3 files changed, 123 insertions(+), 68 deletions(-)
fs/nfs/pagelist.c +24 -2
@@ -130,6 +130,25 @@ nfs_async_iocounter_wait(struct rpc_task *task, struct nfs_lock_context *l_ctx) }EXPORT_SYMBOL_GPL (nfs_async_iocounter_wait); +/* +* nfs_page_lock_head_request - page lock the head of the page group +* @req: any member of the page group +*/ +struct nfs_page * +nfs_page_group_lock_head (struct nfs_page *req) +{ +struct nfs_page *head = req->wb_head; + +while (!nfs_lock_request (head)) { +int ret =nfs_wait_on_request (head); +if (ret <0 ) +return ERR_PTR (ret); + } +if (head != req) +kref_get (&head->wb_kref); +return head; +} +/* * nfs_unroll_locks - unlock all newly locked reqs and wait on @req * @head: head request of page group, must be holding head lock
@@ -186,14 +205,16 @@ nfs_page_group_lock_subreq(struct nfs_page *head, struct nfs_page *subreq)* @head: head request of page group * * This is a helper function for nfs_lock_and_join_requests which -* must be called with the head request and page group both locked. -* On error, it returns with the page group unlocked. +* must be called with the head request locked. */ int nfs_page_group_lock_subrequests (struct nfs_page *head) {struct nfs_page *subreq;int ret; + ret =nfs_page_group_lock (head); +if (ret <0 ) +return ret;/* lock each request in the page group */ for (subreq = head->wb_this_page; subreq != head; subreq = subreq->wb_this_page) {
@@ -201,6 +222,7 @@ int nfs_page_group_lock_subrequests(struct nfs_page *head)if (ret <0 )return ret; } +nfs_page_group_unlock (head);return 0 ; }
fs/nfs/write.c +98 -66
205 lines not on this page · raw message
include/linux/nfs_page.h +1 -0
@@ -139,6 +139,7 @@ extern size_t nfs_generic_pg_test(struct nfs_pageio_descriptor *desc,extern int nfs_wait_on_request (struct nfs_page *);extern void nfs_unlock_request (struct nfs_page *req);extern void nfs_unlock_and_release_request (struct nfs_page *); +extern struct nfs_page *nfs_page_group_lock_head (struct nfs_page *req);extern int nfs_page_group_lock_subrequests (struct nfs_page *head);extern int nfs_page_group_lock (struct nfs_page *);extern void nfs_page_group_unlock (struct nfs_page *);
Signature · 1 line
2.25.1
[email protected]
maintainerPATCH 10/10 NFS: Try to join page groups before an O_DIRECT retransmission
From: Trond Myklebust <[email protected]>
If we have to retransmit requests, try to join their page groups first.
--- fs/nfs/direct.c | 20 ++++++++++++++++++++ fs/nfs/write.c | 2 +- include/linux/nfs_page.h | 1 + 3 files changed, 22 insertions(+), 1 deletion(-)
fs/nfs/direct.c +20 -0
@@ -505,6 +505,24 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter)return result; } +static void +nfs_direct_join_group (struct list_head *list,struct inode *inode) +{ +struct nfs_page *req, *next; + +list_for_each_entry (req, list, wb_list) { +if (req->wb_head != req || req->wb_this_page == req) +continue ; +for (next = req->wb_this_page; + next != req->wb_head; + next = next->wb_this_page) { +nfs_list_remove_request (next); +nfs_release_request (next); + } +nfs_join_page_group (req, inode); + } +} + static void nfs_direct_write_scan_commit_list(struct inode *inode, struct list_head *list,
@@ -527,6 +545,8 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)nfs_init_cinfo_from_dreq (&cinfo, dreq);nfs_direct_write_scan_commit_list (dreq->inode, &reqs, &cinfo); +nfs_direct_join_group (&reqs, dreq->inode); + dreq->count =0 ; dreq->max_count =0 ;list_for_each_entry (req, &reqs, wb_list)
fs/nfs/write.c +1 -1
@@ -501,7 +501,7 @@ nfs_destroy_unlinked_subrequests(struct nfs_page *destroy_list,* the (former) group. All subrequests are removed from any write or commit * lists, unlinked from the group and destroyed. */ -static void +void nfs_join_page_group (struct nfs_page *head,struct inode *inode) {struct nfs_page *subreq;
include/linux/nfs_page.h +1 -0
@@ -141,6 +141,7 @@ extern void nfs_unlock_request(struct nfs_page *req);extern void nfs_unlock_and_release_request (struct nfs_page *);extern struct nfs_page *nfs_page_group_lock_head (struct nfs_page *req);extern int nfs_page_group_lock_subrequests (struct nfs_page *head); +extern void nfs_join_page_group (struct nfs_page *head,struct inode *inode);extern int nfs_page_group_lock (struct nfs_page *);extern void nfs_page_group_unlock (struct nfs_page *);extern bool nfs_page_group_sync_on_bit (struct nfs_page *,unsigned int );
Signature · 1 line
2.25.1
Re: [PATCH 05/10] NFS: Fix memory leaks in nfs_pageio_stop_mirroring()
In reply to [email protected]Hi Trond,
Quoted 32 lines · On 4/1/20 2:56 PM, [email protected] wrote:
From: Trond Myklebust <[email protected]>
If we just set the mirror count to 1 without first clearing out the mirrors, we can leak queued up requests.
Signed-off-by Trond Myklebust <[email protected]>--- fs/nfs/pagelist.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)fs/nfs/pagelist.c +0
-9@@ -900,15 +900,6 @@ static void nfs_pageio_setup_mirroring(struct nfs_pageio_descriptor *pgio, pgio->pg_mirror_count = mirror_count; } -/* -* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) -*/ -void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) -{ - pgio->pg_mirror_count =1 ; - pgio->pg_mirror_idx =0 ; -} -static void nfs_pageio_cleanup_mirroring (struct nfs_pageio_descriptor *pgio) { pgio->pg_mirror_count =1 ;@@ -1334,6 +1325,14 @@ void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *desc, pgoff_t index) }
fs/nfs/pagelist.c void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *desc, pgoff_t index) · line 1335
Quoted 10 lines
fs/nfs/pagelist.c +8
-0} +/* +* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) +*/ +void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) +{ +nfs_pageio_complete (pgio); +} +
Would it make sense to replace calls to nfs_pageio_stop_mirroring() with nfs_pageio_complete() instead?
Anna
Quoted 3 lines
fs/nfs/pagelist.c +0
-0int __init nfs_init_nfspagecache (void ) { nfs_page_cachep = kmem_cache_create("nfs_page",
Re: [PATCH 05/10] NFS: Fix memory leaks in nfs_pageio_stop_mirroring()
Quoted 41 lines · Anna Schumaker
Hi Trond,
Quoted 38 lines · On 4/1/20 2:56 PM, [email protected] wrote:
From: Trond Myklebust <[email protected]>
If we just set the mirror count to 1 without first clearing out the mirrors, we can leak queued up requests.
Signed-off-by Trond Myklebust <[email protected]>--- fs/nfs/pagelist.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)fs/nfs/pagelist.c +0
-9@@ -900,15 +900,6 @@ static void nfs_pageio_setup_mirroring(struct nfs_pageio_descriptor *pgio, pgio->pg_mirror_count = mirror_count; } -/* -* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) -*/ -void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) -{ - pgio->pg_mirror_count =1 ; - pgio->pg_mirror_idx =0 ; -} - static void nfs_pageio_cleanup_mirroring(struct
nfs_pageio_descriptor*pgio)
{fs/nfs/pagelist.c +0
-0pgio->pg_mirror_count =1 ;@@ -1334,6 +1325,14 @@ void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *desc, pgoff_t index) } }
Quoted 13 lines
fs/nfs/pagelist.c void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *desc, pgoff_t index) · line 1335
Quoted 9 lines
fs/nfs/pagelist.c +8
-0+/* +* nfs_pageio_stop_mirroring - stop using mirroring (set mirror count to 1) +*/ +void nfs_pageio_stop_mirroring (struct nfs_pageio_descriptor *pgio) +{ +nfs_pageio_complete (pgio); +} +Would it make sense to replace calls to
nfs_pageio_stop_mirroring()withnfs_pageio_complete()instead?
No. Let's keep the wrapper, since it makes the writeback code easier to read (it expresses the intent more clearly).
Quoted 5 lines
Anna
Quoted 3 lines
fs/nfs/pagelist.c +0
-0int __init nfs_init_nfspagecache (void ){
nfs_page_cachep= kmem_cache_create("nfs_page",
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]