Skip to contentSkip to conversation

bcache: cached_dev_free needs to put the sb page

PATCH In review Liang Chen 4 participants 13 messages Dec 6, 2019 – Dec 9, 2019 bcache (block layer cache)

lore
Share & tools
lore

Liang Chen

author
#0 lore raw

PATCH 1/2 bcache: cached_dev_free needs to put the sb page

Same as cache device, the buffer page needs to be put while freeing cached_dev. Otherwise a page would be leaked every time a cached_dev is stopped.

---
 drivers/md/bcache/super.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/md/bcache/super.c +3 -0
index 77e9869345e7..a573ce1d85aa 100644
@@ -1275,6 +1275,9 @@ static void cached_dev_free(struct closure *cl)
 
 	mutex_unlock(&bch_register_lock);
 
+	if (dc->sb_bio.bi_inline_vecs[0].bv_page)
+		put_page(bio_first_page_all(&dc->sb_bio));
+
 	if (!IS_ERR_OR_NULL(dc->bdev))
 		blkdev_put(dc->bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
 
Signature · 1 line
2.17.0
author
#1 lore raw

PATCH 2/2 bcache: __write_super to handle page sizes other than 4k

__write_super assumes super block data starts at offset 0 of the page read in with __bread from read_super, which is not true when page size is not 4k. We encountered the issue on system with 64K page size - commonly
seen on aarch64 architecture.

Instead of making any assumption on the offset of the data within the page, this patch calls __bread again to locate the data. That should not introduce an extra io since the page has been held when it's read in from read_super, and __write_super is not on performance critical code path.

---
 drivers/md/bcache/super.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
drivers/md/bcache/super.c +27 -5
index a573ce1d85aa..a39450c9bc34 100644
@@ -207,15 +207,27 @@ static void write_bdev_super_endio(struct bio *bio)
 	closure_put(&dc->sb_write);
 }
 
-static void __write_super(struct cache_sb *sb, struct bio *bio)
+static int __write_super(struct cache_sb *sb, struct bio *bio,
+			 struct block_device *bdev)
 {
-	struct cache_sb *out = page_address(bio_first_page_all(bio));
+	struct cache_sb *out;
 	unsigned int i;
+	struct buffer_head *bh;
+
+	/*
+	 * The page is held since read_super, this __bread * should not
+	 * cause an extra io read.
+	 */
+	bh = __bread(bdev, 1, SB_SIZE);
+	if (!bh)
+		goto out_bh;
+
+	out = (struct cache_sb *) bh->b_data;
 
 	bio->bi_iter.bi_sector	= SB_SECTOR;
 	bio->bi_iter.bi_size	= SB_SIZE;
 	bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_SYNC|REQ_META);
-	bch_bio_map(bio, NULL);
+	bch_bio_map(bio, bh->b_data);
 
 	out->offset		= cpu_to_le64(sb->offset);
 	out->version		= cpu_to_le64(sb->version);
@@ -239,7 +251,14 @@ static void __write_super(struct cache_sb *sb, struct bio *bio)
 	pr_debug("ver %llu, flags %llu, seq %llu",
 		 sb->version, sb->flags, sb->seq);
 
+	/* The page will still be held without this bh.*/
+	put_bh(bh);
 	submit_bio(bio);
+	return 0;
+
+out_bh:
+	pr_err("Couldn't read super block, __write_super failed");
+	return -1;
 }
 
 static void bch_write_bdev_super_unlock(struct closure *cl)
@@ -264,7 +283,8 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent)
 
 	closure_get(cl);
 	/* I/O request sent to backing device */
-	__write_super(&dc->sb, bio);
+	if(__write_super(&dc->sb, bio, dc->bdev))
+		closure_put(cl);
 
 	closure_return_with_destructor(cl, bch_write_bdev_super_unlock);
 }
@@ -312,7 +332,9 @@ void bcache_write_super(struct cache_set *c)
 		bio->bi_private = ca;
 
 		closure_get(cl);
-		__write_super(&ca->sb, bio);
+		if(__write_super(&ca->sb, bio, ca->bdev))
+			closure_put(cl);
+
 	}
 
 	closure_return_with_destructor(cl, bcache_write_super_unlock);
Signature · 1 line
2.17.0
#2 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

Quoted 9 lines · Liang Chen

__write_super assumes super block data starts at offset 0 of the page read in with __bread from read_super, which is not true when page size is not 4k. We encountered the issue on system with 64K page size - commonly
seen on aarch64 architecture.

Instead of making any assumption on the offset of the data within the page, this patch calls __bread again to locate the data. That should not introduce an extra io since the page has been held when it's read in from read_super, and __write_super is not on performance critical code path.

No need to use buffer heads here, you can just use offset_in_page to calculate the offset. Similarly I think the read side shouldn't use buffer heads either (it is the only use of buffer heads in bcache!), a siple read_cache_page should be all that is needed.

author
#3 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

Thanks for the advise.
Yeah, calculating the offset based on the buffer size is possible. I just wanted to avoid making a dependency on some buffer head internal logic here, like the way it dividesthe page into equal sized buffers, and at the same time keep the patch less intrusive.

Quoted 16 lines · Christoph Hellwig
Quoted 9 lines · Liang Chen

__write_super assumes super block data starts at offset 0 of the page read in with __bread from read_super, which is not true when page size is not 4k. We encountered the issue on system with 64K page size - commonly
seen on aarch64 architecture.

Instead of making any assumption on the offset of the data within the page, this patch calls __bread again to locate the data. That should not introduce an extra io since the page has been held when it's read in from read_super, and __write_super is not on performance critical code path.

No need to use buffer heads here, you can just use offset_in_page to calculate the offset. Similarly I think the read side shouldn't use buffer heads either (it is the only use of buffer heads in bcache!), a siple read_cache_page should be all that is needed.

#4 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

In reply to Liang Chen
Quoted 11 lines · Liang Chen

__write_super assumes super block data starts at offset 0 of the page read in with __bread from read_super, which is not true when page size is not 4k. We encountered the issue on system with 64K page size - commonly
seen on aarch64 architecture.

Instead of making any assumption on the offset of the data within the page, this patch calls __bread again to locate the data. That should not introduce an extra io since the page has been held when it's read in from read_super, and __write_super is not on performance critical code path.

In general the patch is good for me. Just two minor requests I add them in line the email.

Thanks.

Quoted 20 lines
---
 drivers/md/bcache/super.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
drivers/md/bcache/super.c +4 -2
index a573ce1d85aa..a39450c9bc34 100644
@@ -207,15 +207,27 @@ static void write_bdev_super_endio(struct bio *bio)
 	closure_put(&dc->sb_write);
 }
 
-static void __write_super(struct cache_sb *sb, struct bio *bio)
+static int __write_super(struct cache_sb *sb, struct bio *bio,
+			 struct block_device *bdev)
 {
-	struct cache_sb *out = page_address(bio_first_page_all(bio));
+	struct cache_sb *out;
 	unsigned int i;
+	struct buffer_head *bh;

drivers/md/bcache/super.c static void write_bdev_super_endio(struct bio *bio) · line 225

Quoted 10 lines
drivers/md/bcache/super.c +10 -0
+
+	/*
+	 * The page is held since read_super, this __bread * should not
+	 * cause an extra io read.
+	 */
+	bh = __bread(bdev, 1, SB_SIZE);
+	if (!bh)
+		goto out_bh;
+
+	out = (struct cache_sb *) bh->b_data;

This is quite tricky here. Could you please to move this code piece into an inline function and add code comments to explain why a read is necessary for a write.

Quoted 35 lines
drivers/md/bcache/super.c +10 -2
 	bio->bi_iter.bi_sector	= SB_SECTOR;
 	bio->bi_iter.bi_size	= SB_SIZE;
 	bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_SYNC|REQ_META);
-	bch_bio_map(bio, NULL);
+	bch_bio_map(bio, bh->b_data);
 
 	out->offset		= cpu_to_le64(sb->offset);
 	out->version		= cpu_to_le64(sb->version);
@@ -239,7 +251,14 @@ static void __write_super(struct cache_sb *sb, struct bio *bio)
 	pr_debug("ver %llu, flags %llu, seq %llu",
 		 sb->version, sb->flags, sb->seq);
 
+	/* The page will still be held without this bh.*/
+	put_bh(bh);
 	submit_bio(bio);
+	return 0;
+
+out_bh:
+	pr_err("Couldn't read super block, __write_super failed");
+	return -1;
 }
 
 static void bch_write_bdev_super_unlock(struct closure *cl)
@@ -264,7 +283,8 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent)
 
 	closure_get(cl);
 	/* I/O request sent to backing device */
-	__write_super(&dc->sb, bio);
+	if(__write_super(&dc->sb, bio, dc->bdev))
+		closure_put(cl);
 
 	closure_return_with_destructor(cl, bch_write_bdev_super_unlock);
 }

drivers/md/bcache/super.c void bcache_write_super(struct cache_set *c) · line 335

Quoted 5 lines
drivers/md/bcache/super.c +1 -1
@@ -312,7 +332,9 @@ void bcache_write_super(struct cache_set *c)
 		bio->bi_private = ca;
 
 		closure_get(cl);
-		__write_super(&ca->sb, bio);
+		if(__write_super(&ca->sb, bio, ca->bdev))

And here, please add code comments for why closure_put() is necessary here.

Quoted 6 lines
drivers/md/bcache/super.c +2 -0
+			closure_put(cl);
+
 	}
 
 	closure_return_with_destructor(cl, bcache_write_super_unlock);
Signature · 1 line
Coly Li
author
#5 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

Sure. will do in a follow up patch.

Quoted 109 lines · Coly Li
Quoted 11 lines · Liang Chen

__write_super assumes super block data starts at offset 0 of the page read in with __bread from read_super, which is not true when page size is not 4k. We encountered the issue on system with 64K page size - commonly
seen on aarch64 architecture.

Instead of making any assumption on the offset of the data within the page, this patch calls __bread again to locate the data. That should not introduce an extra io since the page has been held when it's read in from read_super, and __write_super is not on performance critical code path.

In general the patch is good for me. Just two minor requests I add them in line the email.

Thanks.

Quoted 20 lines
---
 drivers/md/bcache/super.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
drivers/md/bcache/super.c +4 -2
index a573ce1d85aa..a39450c9bc34 100644
@@ -207,15 +207,27 @@ static void write_bdev_super_endio(struct bio *bio)
      closure_put(&dc->sb_write);
 }
 
-static void __write_super(struct cache_sb *sb, struct bio *bio)
+static int __write_super(struct cache_sb *sb, struct bio *bio,
+                      struct block_device *bdev)
 {
-     struct cache_sb *out = page_address(bio_first_page_all(bio));
+     struct cache_sb *out;
      unsigned int i;
+     struct buffer_head *bh;

drivers/md/bcache/super.c static void write_bdev_super_endio(struct bio *bio) · line 225

Quoted 10 lines
drivers/md/bcache/super.c +10 -0
+
+     /*
+      * The page is held since read_super, this __bread * should not
+      * cause an extra io read.
+      */
+     bh = __bread(bdev, 1, SB_SIZE);
+     if (!bh)
+             goto out_bh;
+
+     out = (struct cache_sb *) bh->b_data;

This is quite tricky here. Could you please to move this code piece into an inline function and add code comments to explain why a read is necessary for a write.

Quoted 35 lines
drivers/md/bcache/super.c +10 -2
      bio->bi_iter.bi_sector  = SB_SECTOR;
      bio->bi_iter.bi_size    = SB_SIZE;
      bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_SYNC|REQ_META);
-     bch_bio_map(bio, NULL);
+     bch_bio_map(bio, bh->b_data);
 
      out->offset             = cpu_to_le64(sb->offset);
      out->version            = cpu_to_le64(sb->version);
@@ -239,7 +251,14 @@ static void __write_super(struct cache_sb *sb, struct bio *bio)
      pr_debug("ver %llu, flags %llu, seq %llu",
               sb->version, sb->flags, sb->seq);
 
+     /* The page will still be held without this bh.*/
+     put_bh(bh);
      submit_bio(bio);
+     return 0;
+
+out_bh:
+     pr_err("Couldn't read super block, __write_super failed");
+     return -1;
 }
 
 static void bch_write_bdev_super_unlock(struct closure *cl)
@@ -264,7 +283,8 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent)
 
      closure_get(cl);
      /* I/O request sent to backing device */
-     __write_super(&dc->sb, bio);
+     if(__write_super(&dc->sb, bio, dc->bdev))
+             closure_put(cl);
 
      closure_return_with_destructor(cl, bch_write_bdev_super_unlock);
 }

drivers/md/bcache/super.c void bcache_write_super(struct cache_set *c) · line 335

Quoted 5 lines
drivers/md/bcache/super.c +1 -1
@@ -312,7 +332,9 @@ void bcache_write_super(struct cache_set *c)
              bio->bi_private = ca;
 
              closure_get(cl);
-             __write_super(&ca->sb, bio);
+             if(__write_super(&ca->sb, bio, ca->bdev))

And here, please add code comments for why closure_put() is necessary here.

drivers/md/bcache/super.c void bcache_write_super(struct cache_set *c) · line 340

Quoted 6 lines
drivers/md/bcache/super.c +2 -0
+                     closure_put(cl);
+
      }
 
      closure_return_with_destructor(cl, bcache_write_super_unlock);

--

Coly Li

#6 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

In reply to Coly Li
Quoted 11 lines · Coly Li
Quoted 11 lines

{

drivers/md/bcache/super.c +8 -1
-	struct cache_sb *out = page_address(bio_first_page_all(bio));
+	struct cache_sb *out;
 	unsigned int i;
+	struct buffer_head *bh;
+
+	/*
+	 * The page is held since read_super, this __bread * should not
+	 * cause an extra io read.
+	 */
+	bh = __bread(bdev, 1, SB_SIZE);
Quoted 8 lines

drivers/md/bcache/super.c static void write_bdev_super_endio(struct bio *bio) · line 225

Quoted 4 lines
drivers/md/bcache/super.c +4 -0
+	if (!bh)
+		goto out_bh;
+
+	out = (struct cache_sb *) bh->b_data;

This is quite tricky here. Could you please to move this code piece into an inline function and add code comments to explain why a read is necessary for a write.

A read is not nessecary. He only added it because he was too fearful of calculating the data offset directly. But calculating it directly is almost trivial and should just be done here. Alternatively if that is still to hard just keep a pointer to the cache_sb around, which is how most file systems do it.

#7 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

Quoted 21 lines · Christoph Hellwig
Quoted 11 lines · Coly Li
Quoted 11 lines

{

drivers/md/bcache/super.c +8 -1
-	struct cache_sb *out = page_address(bio_first_page_all(bio));
+	struct cache_sb *out;
 	unsigned int i;
+	struct buffer_head *bh;
+
+	/*
+	 * The page is held since read_super, this __bread * should not
+	 * cause an extra io read.
+	 */
+	bh = __bread(bdev, 1, SB_SIZE);
Quoted 8 lines

drivers/md/bcache/super.c static void write_bdev_super_endio(struct bio *bio) · line 225

Quoted 4 lines
drivers/md/bcache/super.c +4 -0
+	if (!bh)
+		goto out_bh;
+
+	out = (struct cache_sb *) bh->b_data;

This is quite tricky here. Could you please to move this code piece into an inline function and add code comments to explain why a read is necessary for a write.

Quoted 6 lines

A read is not nessecary. He only added it because he was too fearful of calculating the data offset directly. But calculating it directly is almost trivial and should just be done here. Alternatively if that is still to hard just keep a pointer to the cache_sb around, which is how most file systems do it.

Copied, if Liang does not have time to handle this as your suggestion, I will handle it.

Thanks for the hint.

Signature · 1 line
Coly Li
author
#8 lore raw

Re: [PATCH 2/2] [PATCH] bcache: __write_super to handle page sizes other than 4k

No problem. I will change the patch to remove this extra read. Thanks.

Quoted 37 lines · Coly Li
Quoted 21 lines · Christoph Hellwig
Quoted 11 lines · Coly Li
Quoted 11 lines

{

drivers/md/bcache/super.c +8 -1
-   struct cache_sb *out = page_address(bio_first_page_all(bio));
+   struct cache_sb *out;
    unsigned int i;
+   struct buffer_head *bh;
+
+   /*
+    * The page is held since read_super, this __bread * should not
+    * cause an extra io read.
+    */
+   bh = __bread(bdev, 1, SB_SIZE);
Quoted 8 lines

drivers/md/bcache/super.c static void write_bdev_super_endio(struct bio *bio) · line 225

Quoted 4 lines
drivers/md/bcache/super.c +4 -0
+   if (!bh)
+           goto out_bh;
+
+   out = (struct cache_sb *) bh->b_data;

This is quite tricky here. Could you please to move this code piece into an inline function and add code comments to explain why a read is necessary for a write.

Quoted 6 lines

A read is not nessecary. He only added it because he was too fearful of calculating the data offset directly. But calculating it directly is almost trivial and should just be done here. Alternatively if that is still to hard just keep a pointer to the cache_sb around, which is how most file systems do it.

Copied, if Liang does not have time to handle this as your suggestion, I will handle it.

Thanks for the hint.

--

Coly Li

#9 lore raw
In reply to Liang Chen
Quoted 14 lines · Liang Chen

Same as cache device, the buffer page needs to be put while freeing cached_dev. Otherwise a page would be leaked every time a cached_dev is stopped.

---
 drivers/md/bcache/super.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/md/bcache/super.c +0 -0
index 77e9869345e7..a573ce1d85aa 100644

drivers/md/bcache/super.c static void cached_dev_free(struct closure *cl) · line 1279

Quoted 5 lines
drivers/md/bcache/super.c +2 -0
@@ -1275,6 +1275,9 @@ static void cached_dev_free(struct closure *cl)
 
 	mutex_unlock(&bch_register_lock);
 
+	if (dc->sb_bio.bi_inline_vecs[0].bv_page)
+		put_page(bio_first_page_all(&dc->sb_bio));

Using bio_first_page_all in the put_page call, but open coding it for the check looks rather strange.

The cleanest thing here would be to just add a page pointer to the cached device structure and use that instead of all the indirections.

author
#10 lore raw

Sure. I will make a patch to clean up all the occurrences of this usages later. Thanks.

Quoted 27 lines · Christoph Hellwig
Quoted 14 lines · Liang Chen

Same as cache device, the buffer page needs to be put while freeing cached_dev. Otherwise a page would be leaked every time a cached_dev is stopped.

---
 drivers/md/bcache/super.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/md/bcache/super.c +0 -0
index 77e9869345e7..a573ce1d85aa 100644

drivers/md/bcache/super.c static void cached_dev_free(struct closure *cl) · line 1279

Quoted 5 lines
drivers/md/bcache/super.c +2 -0
@@ -1275,6 +1275,9 @@ static void cached_dev_free(struct closure *cl)
 
      mutex_unlock(&bch_register_lock);
 
+     if (dc->sb_bio.bi_inline_vecs[0].bv_page)
+             put_page(bio_first_page_all(&dc->sb_bio));

Using bio_first_page_all in the put_page call, but open coding it for the check looks rather strange.

The cleanest thing here would be to just add a page pointer to the cached device structure and use that instead of all the indirections.

#11 lore raw
In reply to Liang Chen
Quoted 5 lines · Liang Chen

Same as cache device, the buffer page needs to be put while freeing cached_dev. Otherwise a page would be leaked every time a cached_dev is stopped.

I have this in my for-test directory, thanks.

Quoted 19 lines
---
 drivers/md/bcache/super.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/md/bcache/super.c +3 -0
index 77e9869345e7..a573ce1d85aa 100644
@@ -1275,6 +1275,9 @@ static void cached_dev_free(struct closure *cl)
 
 	mutex_unlock(&bch_register_lock);
 
+	if (dc->sb_bio.bi_inline_vecs[0].bv_page)
+		put_page(bio_first_page_all(&dc->sb_bio));
+
 	if (!IS_ERR_OR_NULL(dc->bdev))
 		blkdev_put(dc->bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
 
Signature · 1 line
Coly Li
#12 lore raw
In reply to Liang Chen
Quoted 5 lines · Liang Chen

Same as cache device, the buffer page needs to be put while freeing cached_dev. Otherwise a page would be leaked every time a cached_dev is stopped.

+cc stable?

--
Eric Wheeler

Quoted 22 lines
---
 drivers/md/bcache/super.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/md/bcache/super.c +3 -0
index 77e9869345e7..a573ce1d85aa 100644
@@ -1275,6 +1275,9 @@ static void cached_dev_free(struct closure *cl)
 
 	mutex_unlock(&bch_register_lock);
 
+	if (dc->sb_bio.bi_inline_vecs[0].bv_page)
+		put_page(bio_first_page_all(&dc->sb_bio));
+
 	if (!IS_ERR_OR_NULL(dc->bdev))
 		blkdev_put(dc->bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
 
Signature · 1 line
2.17.0