Skip to contentSkip to conversation

alternative take on the same page merging leak fix v2

PATCH New Christoph Hellwig 3 participants 8 messages Jun 13, 2019 – Jun 14, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

#1 lore raw

PATCH 1/2 block: return from __bio_try_merge_page if merging occured in the same page

We currently have an input same_page parameter to __bio_try_merge_page to prohibit merging in the same page. The rationale for that is that some callers need to account for every page added to a bio. Instead of letting these callers call twice into the merge code to account for the new vs existing page cases, just turn the paramter into an output one that returns if a merge in the same page occured and let them act accordingly.

---
 block/bio.c         | 23 +++++++++--------------
 fs/iomap.c          | 12 ++++++++----
 fs/xfs/xfs_aops.c   | 11 ++++++++---
 include/linux/bio.h |  2 +-
 4 files changed, 26 insertions(+), 22 deletions(-)
block/bio.c +9 -14
index 683cbb40f051..59588c57694d 100644
@@ -636,7 +636,7 @@ EXPORT_SYMBOL(bio_clone_fast);
 
 static inline bool page_is_mergeable(const struct bio_vec *bv,
 		struct page *page, unsigned int len, unsigned int off,
-		bool same_page)
+		bool *same_page)
 {
 	phys_addr_t vec_end_addr = page_to_phys(bv->bv_page) +
 		bv->bv_offset + bv->bv_len - 1;
@@ -647,15 +647,9 @@ static inline bool page_is_mergeable(const struct bio_vec *bv,
 	if (xen_domain() && !xen_biovec_phys_mergeable(bv, page))
 		return false;
 
-	if ((vec_end_addr & PAGE_MASK) != page_addr) {
-		if (same_page)
-			return false;
-		if (pfn_to_page(PFN_DOWN(vec_end_addr)) + 1 != page)
-			return false;
-	}
-
-	WARN_ON_ONCE(same_page && (len + off) > PAGE_SIZE);
-
+	*same_page = ((vec_end_addr & PAGE_MASK) == page_addr);
+	if (!*same_page && pfn_to_page(PFN_DOWN(vec_end_addr)) + 1 != page)
+		return false;
 	return true;
 }
 
@@ -767,8 +761,7 @@ EXPORT_SYMBOL(bio_add_pc_page);
  * @page: start page to add
  * @len: length of the data to add
  * @off: offset of the data relative to @page
- * @same_page: if %true only merge if the new data is in the same physical
- *		page as the last segment of the bio.
+ * @same_page: return if the segment has been merged inside the same page
  *
  * Try to add the data at @page + @off to the last bvec of @bio.  This is a
  * a useful optimisation for file systems with a block size smaller than the
@@ -779,7 +772,7 @@ EXPORT_SYMBOL(bio_add_pc_page);
  * Return %true on success or %false on failure.
  */
 bool __bio_try_merge_page(struct bio *bio, struct page *page,
-		unsigned int len, unsigned int off, bool same_page)
+		unsigned int len, unsigned int off, bool *same_page)
 {
 	if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
 		return false;
@@ -837,7 +830,9 @@ EXPORT_SYMBOL_GPL(__bio_add_page);
 int bio_add_page(struct bio *bio, struct page *page,
 		 unsigned int len, unsigned int offset)
 {
-	if (!__bio_try_merge_page(bio, page, len, offset, false)) {
+	bool same_page = false;
+
+	if (!__bio_try_merge_page(bio, page, len, offset, &same_page)) {
 		if (bio_full(bio))
 			return 0;
 		__bio_add_page(bio, page, len, offset);
fs/iomap.c +8 -4
index 23ef63fd1669..12654c2e78f8 100644
@@ -287,7 +287,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 	struct iomap_readpage_ctx *ctx = data;
 	struct page *page = ctx->cur_page;
 	struct iomap_page *iop = iomap_page_create(inode, page);
-	bool is_contig = false;
+	bool same_page = false, is_contig = false;
 	loff_t orig_pos = pos;
 	unsigned poff, plen;
 	sector_t sector;
@@ -315,10 +315,14 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data,
 	 * Try to merge into a previous segment if we can.
 	 */
 	sector = iomap_sector(iomap, pos);
-	if (ctx->bio && bio_end_sector(ctx->bio) == sector) {
-		if (__bio_try_merge_page(ctx->bio, page, plen, poff, true))
-			goto done;
+	if (ctx->bio && bio_end_sector(ctx->bio) == sector)
 		is_contig = true;
+
+	if (is_contig &&
+	    __bio_try_merge_page(ctx->bio, page, plen, poff, &same_page)) {
+		if (!same_page && iop)
+			atomic_inc(&iop->read_count);
+		goto done;
 	}
 
 	/*
fs/xfs/xfs_aops.c +8 -3
index a6f0f4761a37..8da5e6637771 100644
@@ -758,6 +758,7 @@ xfs_add_to_ioend(
 	struct block_device	*bdev = xfs_find_bdev_for_inode(inode);
 	unsigned		len = i_blocksize(inode);
 	unsigned		poff = offset & (PAGE_SIZE - 1);
+	bool			merged, same_page = false;
 	sector_t		sector;
 
 	sector = xfs_fsb_to_db(ip, wpc->imap.br_startblock) +
@@ -774,9 +775,13 @@ xfs_add_to_ioend(
 				wpc->imap.br_state, offset, bdev, sector);
 	}
 
-	if (!__bio_try_merge_page(wpc->ioend->io_bio, page, len, poff, true)) {
-		if (iop)
-			atomic_inc(&iop->write_count);
+	merged = __bio_try_merge_page(wpc->ioend->io_bio, page, len, poff,
+			&same_page);
+
+	if (iop && !same_page)
+		atomic_inc(&iop->write_count);
+
+	if (!merged) {
 		if (bio_full(wpc->ioend->io_bio))
 			xfs_chain_bio(wpc->ioend, wbc, bdev, sector);
 		bio_add_page(wpc->ioend->io_bio, page, len, poff);
include/linux/bio.h +1 -1
index 0f23b5682640..f87abaa898f0 100644
@@ -423,7 +423,7 @@ extern int bio_add_page(struct bio *, struct page *, unsigned int,unsigned int);
 extern int bio_add_pc_page(struct request_queue *, struct bio *, struct page *,
 			   unsigned int, unsigned int);
 bool __bio_try_merge_page(struct bio *bio, struct page *page,
-		unsigned int len, unsigned int off, bool same_page);
+		unsigned int len, unsigned int off, bool *same_page);
 void __bio_add_page(struct bio *bio, struct page *page,
 		unsigned int len, unsigned int off);
 int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter);
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/2 block: fix page leak when merging to same page

In reply to Christoph Hellwig

When multiple iovecs reference the same page, each get_user_page call will add a reference to the page. But once we've created the bio that information gets lost and only a single reference will be dropped after I/O completion. Use the same_page information returned from __bio_try_merge_page to drop additional references to pages that were already present in the bio.

Based on a patch from Ming Lei.

Fixes 576ed913 ("block: use bio_add_page in bio_iov_iter_get_pages")
---
 block/bio.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
block/bio.c +10 -2
index 59588c57694d..35b3c568a48f 100644
@@ -895,6 +895,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;
 	struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;
 	struct page **pages = (struct page **)bv;
+	bool same_page = false;
 	ssize_t size, left;
 	unsigned len, i;
 	size_t offset;
@@ -915,8 +916,15 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 		struct page *page = pages[i];
 
 		len = min_t(size_t, PAGE_SIZE - offset, left);
-		if (WARN_ON_ONCE(bio_add_page(bio, page, len, offset) != len))
-			return -EINVAL;
+
+		if (__bio_try_merge_page(bio, page, len, offset, &same_page)) {
+			if (same_page)
+				put_page(page);
+		} else {
+			if (WARN_ON_ONCE(bio_full(bio)))
+                                return -EINVAL;
+			__bio_add_page(bio, page, len, offset);
+		}
 		offset = 0;
 	}
 
Signature · 1 line
2.20.1
maintainer
#3 lore raw
In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

Applied for 5.2, thanks.

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Quoted 5 lines
Quoted 3 lines

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

Applied for 5.2, thanks.

Hi Jens & Christoph,

Looks the following change is missed in patch 1, otherwise kernel oops is triggered during kernel booting:

block/bio.c +3 -1
index 35b3c568a48f..9ccf07c666f7 100644
@@ -706,6 +706,8 @@ static int __bio_add_pc_page(struct request_queue *q, struct bio *bio,
 		return 0;
 
 	if (bio->bi_vcnt > 0) {
+		bool same_page;
+
 		bvec = &bio->bi_io_vec[bio->bi_vcnt - 1];
 
 		if (page == bvec->bv_page &&
@@ -723,7 +725,7 @@ static int __bio_add_pc_page(struct request_queue *q, struct bio *bio,
 		if (bvec_gap_to_prev(q, bvec, offset))
 			return 0;
 
-		if (page_is_mergeable(bvec, page, len, offset, false) &&
+		if (page_is_mergeable(bvec, page, len, offset, &same_page) &&
 		    can_add_page_to_seg(q, bvec, page, len, offset)) {
 			bvec->bv_len += len;
 			goto done;

Thanks,
Ming

author Reviewed-by
#5 lore raw
Quoted 12 lines · Ming Lei
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Christoph Hellwig

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

Quoted 6 lines
Quoted 1 line

Applied for 5.2, thanks.

Hi Jens & Christoph,

Looks the following change is missed in patch 1, otherwise kernel oops is triggered during kernel booting:

Ok, true, sorry.

and you probably want to cook it up for Jens with a proper changelog.

#6 lore raw
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Ming Lei
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Quoted 5 lines
Quoted 3 lines

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

Applied for 5.2, thanks.

Hi Jens & Christoph,

Quoted 8 lines
Quoted 2 lines

Looks the following change is missed in patch 1, otherwise kernel oops is triggered during kernel booting:

Ok, true, sorry.

and you probably want to cook it up for Jens with a proper changelog.

This one can be wrapped into the patch 1 given it is just at top of for-linus, so I guess Jens may do it.

Thanks,
Ming

maintainer
#7 lore raw
In reply to Christoph Hellwig
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Ming Lei
Quoted 12 lines · Jens Axboe
Quoted 9 lines · Christoph Hellwig

Hi Jens, hi Ming,

this is the tested and split version of what I think is the better fix for the get_user_pages page leak, as it leaves the intelligence in the callers instead of in bio_try_to_merge_page.

Changes since v1:
- drop patches not required for 5.2
- added Reviewed-by tags

Applied for 5.2, thanks.

Quoted 8 lines
Quoted 4 lines

Hi Jens & Christoph,

Looks the following change is missed in patch 1, otherwise kernel oops is triggered during kernel booting:

Ok, true, sorry.

I've dropped the two patches. Please send them again, when they are tested. Not making -rc5 at this time.

Signature · 1 line
Jens Axboe