Skip to contentSkip to conversation

dm: io accounting & polling improvement

PATCH In review Ming Lei 4 participants 31 messages Apr 12, 2022 – Apr 17, 2022

Subsystems (3)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover dm: io accounting & polling improvement

Hello Guys,

The 1st patch adds bdev based io accounting interface.

The 2nd ~ 5th patches improves dm's io accounting & split, meantime fixes kernel panic on dm-zone.

The other patches improves io polling & dm io reference handling.

Ming Lei (8):
block: replace disk based account with bdev's
dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct
dm: pass 'dm_io' instance to dm_io_acct directly
dm: switch to bdev based io accounting interface
dm: always setup ->orig_bio in alloc_io
dm: don't grab target io reference in dm_zone_map_bio
dm: improve target io referencing
dm: put all polled io into one single list

 block/blk-core.c              |  15 +--
 drivers/block/zram/zram_drv.c |   5 +-
 drivers/md/dm-core.h          |  17 ++-
 drivers/md/dm-zone.c          |  10 --
 drivers/md/dm.c               | 190 +++++++++++++++++++---------------
 include/linux/blkdev.h        |   7 +-
 6 files changed, 131 insertions(+), 113 deletions(-)
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author
#1 lore raw

PATCH 1/8 block: replace disk based account with bdev's

'block device' is generic type for interface, and gendisk becomes more one block layer internal type, so replace disk based account interface with bdec's.

Also add 'start_time' parameter to bdev_start_io_acct() so that we can cover device mapper's io accounting by the two bdev based interface.

---
 block/blk-core.c              | 15 ++++++++-------
 drivers/block/zram/zram_drv.c |  5 +++--
 include/linux/blkdev.h        |  7 ++++---
 3 files changed, 15 insertions(+), 12 deletions(-)
block/blk-core.c +8 -7
index 937bb6b86331..a3ae13b129ff 100644
@@ -1063,12 +1063,13 @@ unsigned long bio_start_io_acct(struct bio *bio)
 }
 EXPORT_SYMBOL_GPL(bio_start_io_acct);
 
-unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors,
-				 unsigned int op)
+unsigned long bdev_start_io_acct(struct block_device *bdev,
+				 unsigned int sectors, unsigned int op,
+				 unsigned long start_time)
 {
-	return __part_start_io_acct(disk->part0, sectors, op, jiffies);
+	return __part_start_io_acct(bdev, sectors, op, start_time);
 }
-EXPORT_SYMBOL(disk_start_io_acct);
+EXPORT_SYMBOL(bdev_start_io_acct);
 
 static void __part_end_io_acct(struct block_device *part, unsigned int op,
 			       unsigned long start_time)
@@ -1091,12 +1092,12 @@ void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time,
 }
 EXPORT_SYMBOL_GPL(bio_end_io_acct_remapped);
 
-void disk_end_io_acct(struct gendisk *disk, unsigned int op,
+void bdev_end_io_acct(struct block_device *bdev, unsigned int op,
 		      unsigned long start_time)
 {
-	__part_end_io_acct(disk->part0, op, start_time);
+	__part_end_io_acct(bdev, op, start_time);
 }
-EXPORT_SYMBOL(disk_end_io_acct);
+EXPORT_SYMBOL(bdev_end_io_acct);
 
 /**
  * blk_lld_busy - Check if underlying low-level drivers of a device are busy
drivers/block/zram/zram_drv.c +3 -2
index e9474b02012d..adb5209a556a 100644
@@ -1675,9 +1675,10 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector,
 	bv.bv_len = PAGE_SIZE;
 	bv.bv_offset = 0;
 
-	start_time = disk_start_io_acct(bdev->bd_disk, SECTORS_PER_PAGE, op);
+	start_time = bdev_start_io_acct(bdev->bd_disk->part0,
+			SECTORS_PER_PAGE, op, jiffies);
 	ret = zram_bvec_rw(zram, &bv, index, offset, op, NULL);
-	disk_end_io_acct(bdev->bd_disk, op, start_time);
+	bdev_end_io_acct(bdev->bd_disk->part0, op, start_time);
 out:
 	/*
 	 * If I/O fails, just return error(ie, non-zero) without
include/linux/blkdev.h +4 -3
index 60d016138997..f680ba6f0ab2 100644
@@ -1491,9 +1491,10 @@ static inline void blk_wake_io_task(struct task_struct *waiter)
 		wake_up_process(waiter);
 }
 
-unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors,
-		unsigned int op);
-void disk_end_io_acct(struct gendisk *disk, unsigned int op,
+unsigned long bdev_start_io_acct(struct block_device *bdev,
+				 unsigned int sectors, unsigned int op,
+				 unsigned long start_time);
+void bdev_end_io_acct(struct block_device *bdev, unsigned int op,
 		unsigned long start_time);
 
 void bio_start_io_acct_time(struct bio *bio, unsigned long start_time);
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#2 lore raw

Re: [PATCH 1/8] block: replace disk based account with bdev's

Quoted 3 lines · Ming Lei

'block device' is generic type for interface, and gendisk becomes more one block layer internal type, so replace disk based account interface with bdec's.

I can't parse this sentence.

block/blk-core.c unsigned long bio_start_io_acct(struct bio *bio) · line 1072

Quoted 9 lines
block/blk-core.c +5 -2
+unsigned long bdev_start_io_acct(struct block_device *bdev,
+				 unsigned int sectors, unsigned int op,
+				 unsigned long start_time)
 {
-	return __part_start_io_acct(disk->part0, sectors, op, jiffies);
+	return __part_start_io_acct(bdev, sectors, op, start_time);
 }
-EXPORT_SYMBOL(disk_start_io_acct);
+EXPORT_SYMBOL(bdev_start_io_acct);

Just rename __part_start_io_acct to bdev_start_io_acct instead.

block/blk-core.c void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time, · line 1100

Quoted 9 lines
block/blk-core.c +3 -3
-void disk_end_io_acct(struct gendisk *disk, unsigned int op,
+void bdev_end_io_acct(struct block_device *bdev, unsigned int op,
 		      unsigned long start_time)
 {
-	__part_end_io_acct(disk->part0, op, start_time);
+	__part_end_io_acct(bdev, op, start_time);
 }
-EXPORT_SYMBOL(disk_end_io_acct);
+EXPORT_SYMBOL(bdev_end_io_acct);

Same here.

author
#3 lore raw

PATCH 2/8 dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct

In reply to Ming Lei

dm->orig_bio is always passed to __dm_start_io_acct and dm_end_io_acct, so it isn't necessary to take one bio parameter for the two helpers.

---
 drivers/md/dm.c | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
drivers/md/dm.c +8 -11
index 3c5fad7c4ee6..62f7af815ef8 100644
@@ -526,16 +526,13 @@ static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio,
 		bio->bi_iter.bi_size = bi_size;
 }
 
-static void __dm_start_io_acct(struct dm_io *io, struct bio *bio)
+static void __dm_start_io_acct(struct dm_io *io)
 {
-	dm_io_acct(false, io->md, bio, io->start_time, &io->stats_aux);
+	dm_io_acct(false, io->md, io->orig_bio, io->start_time, &io->stats_aux);
 }
 
 static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
 {
-	/* Must account IO to DM device in terms of orig_bio */
-	struct bio *bio = io->orig_bio;
-
 	/*
 	 * Ensure IO accounting is only ever started once.
 	 * Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO.
@@ -555,12 +552,12 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
 		spin_unlock_irqrestore(&io->lock, flags);
 	}
 
-	__dm_start_io_acct(io, bio);
+	__dm_start_io_acct(io);
 }
 
-static void dm_end_io_acct(struct dm_io *io, struct bio *bio)
+static void dm_end_io_acct(struct dm_io *io)
 {
-	dm_io_acct(true, io->md, bio, io->start_time, &io->stats_aux);
+	dm_io_acct(true, io->md, io->orig_bio, io->start_time, &io->stats_aux);
 }
 
 static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
@@ -875,14 +872,14 @@ static void dm_io_complete(struct dm_io *io)
 
 	io_error = io->status;
 	if (dm_io_flagged(io, DM_IO_ACCOUNTED))
-		dm_end_io_acct(io, bio);
+		dm_end_io_acct(io);
 	else if (!io_error) {
 		/*
 		 * Must handle target that DM_MAPIO_SUBMITTED only to
 		 * then bio_endio() rather than dm_submit_bio_remap()
 		 */
-		__dm_start_io_acct(io, bio);
-		dm_end_io_acct(io, bio);
+		__dm_start_io_acct(io);
+		dm_end_io_acct(io);
 	}
 	free_io(io);
 	smp_wmb();
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author
#4 lore raw

PATCH 3/8 dm: pass 'dm_io' instance to dm_io_acct directly

In reply to Ming Lei

All the other 4 parameters are retrieved from the 'dm_io' instance, so not necessary to pass all four to dm_io_acct().

---
 drivers/md/dm.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/md/dm.c +7 -4
index 62f7af815ef8..ed85cd1165a4 100644
@@ -498,9 +498,12 @@ static bool bio_is_flush_with_data(struct bio *bio)
 	return ((bio->bi_opf & REQ_PREFLUSH) && bio->bi_iter.bi_size);
 }
 
-static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio,
-		       unsigned long start_time, struct dm_stats_aux *stats_aux)
+static void dm_io_acct(struct dm_io *io, bool end)
 {
+	struct dm_stats_aux *stats_aux = &io->stats_aux;
+	unsigned long start_time = io->start_time;
+	struct mapped_device *md = io->md;
+	struct bio *bio = io->orig_bio;
 	bool is_flush_with_data;
 	unsigned int bi_size;
 
@@ -528,7 +531,7 @@ static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio,
 
 static void __dm_start_io_acct(struct dm_io *io)
 {
-	dm_io_acct(false, io->md, io->orig_bio, io->start_time, &io->stats_aux);
+	dm_io_acct(io, false);
 }
 
 static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
@@ -557,7 +560,7 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
 
 static void dm_end_io_acct(struct dm_io *io)
 {
-	dm_io_acct(true, io->md, io->orig_bio, io->start_time, &io->stats_aux);
+	dm_io_acct(io, true);
 }
 
 static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#5 lore raw

Re: [PATCH 3/8] dm: pass 'dm_io' instance to dm_io_acct directly

Quoted 4 lines · Ming Lei

All the other 4 parameters are retrieved from the 'dm_io' instance, so not necessary to pass all four to dm_io_acct().

Yeah, commit 0ab30b4079e103 ("dm: eliminate copying of dm_io fields in dm_io_dec_pending") could've gone further to do what you've done here in this patch.

But it stopped short because of the additional "games" associated with the late assignment of io->orig_bio that is in the dm-5.19 branch.

Mike

Quoted 49 lines
---
 drivers/md/dm.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/md/dm.c +7 -4
index 62f7af815ef8..ed85cd1165a4 100644
@@ -498,9 +498,12 @@ static bool bio_is_flush_with_data(struct bio *bio)
 	return ((bio->bi_opf & REQ_PREFLUSH) && bio->bi_iter.bi_size);
 }
 
-static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio,
-		       unsigned long start_time, struct dm_stats_aux *stats_aux)
+static void dm_io_acct(struct dm_io *io, bool end)
 {
+	struct dm_stats_aux *stats_aux = &io->stats_aux;
+	unsigned long start_time = io->start_time;
+	struct mapped_device *md = io->md;
+	struct bio *bio = io->orig_bio;
 	bool is_flush_with_data;
 	unsigned int bi_size;
 
@@ -528,7 +531,7 @@ static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio,
 
 static void __dm_start_io_acct(struct dm_io *io)
 {
-	dm_io_acct(false, io->md, io->orig_bio, io->start_time, &io->stats_aux);
+	dm_io_acct(io, false);
 }
 
 static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
@@ -557,7 +560,7 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
 
 static void dm_end_io_acct(struct dm_io *io)
 {
-	dm_io_acct(true, io->md, io->orig_bio, io->start_time, &io->stats_aux);
+	dm_io_acct(io, true);
 }
 
 static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

author
#6 lore raw

Re: [PATCH 3/8] dm: pass 'dm_io' instance to dm_io_acct directly

Quoted 6 lines · Mike Snitzer
Quoted 3 lines · Ming Lei

All the other 4 parameters are retrieved from the 'dm_io' instance, so not necessary to pass all four to dm_io_acct().

Quoted 8 lines
Quoted 1 line

Yeah, commit 0ab30b4079e103 ("dm: eliminate copying of dm_io fields in dm_io_dec_pending") could've gone further to do what you've done here in this patch.

But it stopped short because of the additional "games" associated with the late assignment of io->orig_bio that is in the dm-5.19 branch.

OK, I will rebase on dm-5.19, but IMO the idea of late assignment of io->orig_bio isn't good, same with splitting one bio just for accounting, things shouldn't be so tricky.

Thanks,
Ming

author
#7 lore raw

PATCH 4/8 dm: switch to bdev based io accounting interface

In reply to Ming Lei

DM won't account sectors in flush IO, also we can retrieve sectors from 'dm_io' for avoiding to allocate & update new original bio, which will be done in the following patch.

So switch to bdev based io accounting interface.

---
 drivers/md/dm.c | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)
drivers/md/dm.c +8 -13
index ed85cd1165a4..31eacc0e93ed 100644
@@ -504,29 +504,24 @@ static void dm_io_acct(struct dm_io *io, bool end)
 	unsigned long start_time = io->start_time;
 	struct mapped_device *md = io->md;
 	struct bio *bio = io->orig_bio;
-	bool is_flush_with_data;
-	unsigned int bi_size;
+	unsigned int sectors;
 
 	/* If REQ_PREFLUSH set save any payload but do not account it */
-	is_flush_with_data = bio_is_flush_with_data(bio);
-	if (is_flush_with_data) {
-		bi_size = bio->bi_iter.bi_size;
-		bio->bi_iter.bi_size = 0;
-	}
+	if (bio_is_flush_with_data(bio))
+		sectors = 0;
+	else
+		sectors = bio_sectors(bio);
 
 	if (!end)
-		bio_start_io_acct_time(bio, start_time);
+		bdev_start_io_acct(bio->bi_bdev, sectors, bio_op(bio),
+				start_time);
 	else
-		bio_end_io_acct(bio, start_time);
+		bdev_end_io_acct(bio->bi_bdev, bio_op(bio), start_time);
 
 	if (unlikely(dm_stats_used(&md->stats)))
 		dm_stats_account_io(&md->stats, bio_data_dir(bio),
 				    bio->bi_iter.bi_sector, bio_sectors(bio),
 				    end, start_time, stats_aux);
-
-	/* Restore bio's payload so it does get accounted upon requeue */
-	if (is_flush_with_data)
-		bio->bi_iter.bi_size = bi_size;
 }
 
 static void __dm_start_io_acct(struct dm_io *io)
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author
#8 lore raw

PATCH 5/8 dm: always setup ->orig_bio in alloc_io

In reply to Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

---
 drivers/md/dm-core.h |  8 ++++++-
 drivers/md/dm.c      | 51 ++++++++++++++++++++++----------------------
 2 files changed, 32 insertions(+), 27 deletions(-)
drivers/md/dm-core.h +7 -1
index 4277853c7535..aefb080c230d 100644
@@ -247,7 +247,12 @@ struct dm_io {
 	blk_short_t flags;
 	atomic_t io_count;
 	struct mapped_device *md;
+
+	/* The three fields represent mapped part of original bio */
 	struct bio *orig_bio;
+	unsigned int sector_offset; /* offset to end of orig_bio */
+	unsigned int sectors;
+
 	blk_status_t status;
 	spinlock_t lock;
 	unsigned long start_time;
@@ -264,7 +269,8 @@ struct dm_io {
  */
 enum {
 	DM_IO_START_ACCT,
-	DM_IO_ACCOUNTED
+	DM_IO_ACCOUNTED,
+	DM_IO_SPLITTED
 };
 
 static inline bool dm_io_flagged(struct dm_io *io, unsigned int bit)
drivers/md/dm.c +25 -26

92 lines not on this page · raw message

Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#9 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

But I'm (ab)using io->orig_bio assignment to indicate to completion that it may proceed. See these dm-5.19 commits to see it imposed even further:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=311a8e6650601a79079000466db77386c5ec2abb
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=56219ebb5f5c84785aa821f755d545eae41bdb1a

And then leveraged here:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=4aa7a368370c2a172d5a0b8927c6332c4b6a3514

Could be all these dm-5.19 changes suck.. but I do know dm-zone.c is too tightly coupled to DM core. So I'll focus on that first, fix 5.18, and then circle back to "what's next?".

Mike

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#10 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 19 lines · Mike Snitzer
Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

Problem is that we need to look at the BIO op in submission AND completion path to handle zone append requests. So looking at the clone on submission is OK since its op is still the original one. But on the completion path, the clone may now be a regular write emulating a zone append op. And looking at the clone only does not allow detecting that zone append.

We could have the orig_bio op saved in dm_io to avoid completely looking at the orig_bio for detecting zone append.

Quoted 16 lines

But I'm (ab)using io->orig_bio assignment to indicate to completion that it may proceed. See these dm-5.19 commits to see it imposed even further:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=311a8e6650601a79079000466db77386c5ec2abb
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=56219ebb5f5c84785aa821f755d545eae41bdb1a

And then leveraged here:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19&id=4aa7a368370c2a172d5a0b8927c6332c4b6a3514

Could be all these dm-5.19 changes suck.. but I do know dm-zone.c is too tightly coupled to DM core. So I'll focus on that first, fix 5.18, and then circle back to "what's next?".

Mike

Signature · 7 lines
Damien Le Moal
Western Digital Research

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#11 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 27 lines · Damien Le Moal
Quoted 19 lines · Mike Snitzer
Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

Quoted 8 lines

Problem is that we need to look at the BIO op in submission AND completion path to handle zone append requests. So looking at the clone on submission is OK since its op is still the original one. But on the completion path, the clone may now be a regular write emulating a zone append op. And looking at the clone only does not allow detecting that zone append.

We could have the orig_bio op saved in dm_io to avoid completely looking at the orig_bio for detecting zone append.

Can you please try the following patch?

Really sorry for breaking dm-zone.c; please teach this man how to test the basics of all things dm-zoned (is there a testsuite in the tools or something?).

Thanks,
Mike

drivers/md/dm-zone.c +29 -22
index c1ca9be4b79e..896127366000 100644
@@ -360,16 +360,20 @@ static int dm_update_zone_wp_offset(struct mapped_device *md, unsigned int zno,
 	return 0;
 }
 
+struct orig_bio_details {
+	unsigned int op;
+	unsigned int nr_sectors;
+};
+
 /*
  * First phase of BIO mapping for targets with zone append emulation:
  * check all BIO that change a zone writer pointer and change zone
  * append operations into regular write operations.
  */
-static bool dm_zone_map_bio_begin(struct mapped_device *md,
-				  struct bio *orig_bio, struct bio *clone)
+static bool dm_zone_map_bio_begin(struct mapped_device *md, unsigned int zno,
+				  struct bio *clone)
 {
 	sector_t zsectors = blk_queue_zone_sectors(md->queue);
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/*
@@ -384,7 +388,7 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		WRITE_ONCE(md->zwp_offset[zno], zwp_offset);
 	}
 
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 		return true;
@@ -401,9 +405,8 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		 * target zone.
 		 */
 		clone->bi_opf = REQ_OP_WRITE | REQ_NOMERGE |
-			(orig_bio->bi_opf & (~REQ_OP_MASK));
-		clone->bi_iter.bi_sector =
-			orig_bio->bi_iter.bi_sector + zwp_offset;
+			(clone->bi_opf & (~REQ_OP_MASK));
+		clone->bi_iter.bi_sector += zwp_offset;
 		break;
 	default:
 		DMWARN_LIMIT("Invalid BIO operation");
@@ -423,11 +426,10 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
  * data written to a zone. Note that at this point, the remapped clone BIO
  * may already have completed, so we do not touch it.
  */
-static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
-					struct bio *orig_bio,
+static blk_status_t dm_zone_map_bio_end(struct mapped_device *md, unsigned int zno,
+					struct orig_bio_details *orig_bio_details,
 					unsigned int nr_sectors)
 {
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/* The clone BIO may already have been completed and failed */
@@ -435,7 +437,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		return BLK_STS_IOERR;
 
 	/* Update the zone wp offset */
-	switch (bio_op(orig_bio)) {
+	switch (orig_bio_details->op) {
 	case REQ_OP_ZONE_RESET:
 		WRITE_ONCE(md->zwp_offset[zno], 0);
 		return BLK_STS_OK;
@@ -452,7 +454,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		 * Check that the target did not truncate the write operation
 		 * emulating a zone append.
 		 */
-		if (nr_sectors != bio_sectors(orig_bio)) {
+		if (nr_sectors != orig_bio_details->nr_sectors) {
 			DMWARN_LIMIT("Truncated write for zone append");
 			return BLK_STS_IOERR;
 		}
@@ -488,7 +490,7 @@ static inline void dm_zone_unlock(struct request_queue *q,
 	bio_clear_flag(clone, BIO_ZONE_WRITE_LOCKED);
 }
 
-static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
+static bool dm_need_zone_wp_tracking(struct bio *clone)
 {
 	/*
 	 * Special processing is not needed for operations that do not need the
@@ -496,15 +498,15 @@ static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
 	 * zones and all operations that do not modify directly a sequential
 	 * zone write pointer.
 	 */
-	if (op_is_flush(orig_bio->bi_opf) && !bio_sectors(orig_bio))
+	if (op_is_flush(clone->bi_opf) && !bio_sectors(clone))
 		return false;
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_WRITE_ZEROES:
 	case REQ_OP_WRITE:
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 	case REQ_OP_ZONE_APPEND:
-		return bio_zone_is_seq(orig_bio);
+		return bio_zone_is_seq(clone);
 	default:
 		return false;
 	}
@@ -519,8 +521,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	struct dm_target *ti = tio->ti;
 	struct mapped_device *md = io->md;
 	struct request_queue *q = md->queue;
-	struct bio *orig_bio = io->orig_bio;
 	struct bio *clone = &tio->clone;
+	struct orig_bio_details orig_bio_details;
 	unsigned int zno;
 	blk_status_t sts;
 	int r;
@@ -529,18 +531,21 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	 * IOs that do not change a zone write pointer do not need
 	 * any additional special processing.
 	 */
-	if (!dm_need_zone_wp_tracking(orig_bio))
+	if (!dm_need_zone_wp_tracking(clone))
 		return ti->type->map(ti, clone);
 
 	/* Lock the target zone */
-	zno = bio_zone_no(orig_bio);
+	zno = bio_zone_no(clone);
 	dm_zone_lock(q, zno, clone);
 
+	orig_bio_details.nr_sectors = bio_sectors(clone);
+	orig_bio_details.op = bio_op(clone);
+
 	/*
 	 * Check that the bio and the target zone write pointer offset are
 	 * both valid, and if the bio is a zone append, remap it to a write.
 	 */
-	if (!dm_zone_map_bio_begin(md, orig_bio, clone)) {
+	if (!dm_zone_map_bio_begin(md, zno, clone)) {
 		dm_zone_unlock(q, zno, clone);
 		return DM_MAPIO_KILL;
 	}
@@ -560,7 +565,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * The target submitted the clone BIO. The target zone will
 		 * be unlocked on completion of the clone.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		break;
 	case DM_MAPIO_REMAPPED:
 		/*
@@ -568,7 +574,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * unlock the target zone here as the clone will not be
 		 * submitted.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		if (sts != BLK_STS_OK)
 			dm_zone_unlock(q, zno, clone);
 		break;

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#12 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 36 lines · Mike Snitzer
Quoted 33 lines · Damien Le Moal
Quoted 19 lines · Mike Snitzer
Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

Problem is that we need to look at the BIO op in submission AND completion path to handle zone append requests. So looking at the clone on submission is OK since its op is still the original one. But on the completion path, the clone may now be a regular write emulating a zone append op. And looking at the clone only does not allow detecting that zone append.

Quoted 4 lines
Quoted 2 lines

We could have the orig_bio op saved in dm_io to avoid completely looking at the orig_bio for detecting zone append.

Can you please try the following patch?

OK. Will do right away.

Quoted 4 lines

Really sorry for breaking dm-zone.c; please teach this man how to test the basics of all things dm-zoned (is there a testsuite in the tools or something?).

We have an internal test suite to check all things related to zone. We run that weekly on all RC releases. We did not catch the problem earlier as we do not run against for-next trees in previous cycles. We could add such runs :)

We would be happy to contribute stuff for testing. Ideally, integrating that into blktest, with a new DM group, would be nice. That was discussed in past LSF. Maybe a topic again for this year ? Beside zone stuff, I am sure we can add more DM tests (I am sure you do also have a test suite ?).

For quick tests, I generally use a zoned nullblk device. I am attaching 2 scripts which allow creating and deleting nullblk devices easily.

Quoted 169 lines

Thanks,
Mike

drivers/md/dm-zone.c +29 -22
index c1ca9be4b79e..896127366000 100644
@@ -360,16 +360,20 @@ static int dm_update_zone_wp_offset(struct mapped_device *md, unsigned int zno,
 	return 0;
 }
 
+struct orig_bio_details {
+	unsigned int op;
+	unsigned int nr_sectors;
+};
+
 /*
  * First phase of BIO mapping for targets with zone append emulation:
  * check all BIO that change a zone writer pointer and change zone
  * append operations into regular write operations.
  */
-static bool dm_zone_map_bio_begin(struct mapped_device *md,
-				  struct bio *orig_bio, struct bio *clone)
+static bool dm_zone_map_bio_begin(struct mapped_device *md, unsigned int zno,
+				  struct bio *clone)
 {
 	sector_t zsectors = blk_queue_zone_sectors(md->queue);
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/*
@@ -384,7 +388,7 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		WRITE_ONCE(md->zwp_offset[zno], zwp_offset);
 	}
 
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 		return true;
@@ -401,9 +405,8 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		 * target zone.
 		 */
 		clone->bi_opf = REQ_OP_WRITE | REQ_NOMERGE |
-			(orig_bio->bi_opf & (~REQ_OP_MASK));
-		clone->bi_iter.bi_sector =
-			orig_bio->bi_iter.bi_sector + zwp_offset;
+			(clone->bi_opf & (~REQ_OP_MASK));
+		clone->bi_iter.bi_sector += zwp_offset;
 		break;
 	default:
 		DMWARN_LIMIT("Invalid BIO operation");
@@ -423,11 +426,10 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
  * data written to a zone. Note that at this point, the remapped clone BIO
  * may already have completed, so we do not touch it.
  */
-static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
-					struct bio *orig_bio,
+static blk_status_t dm_zone_map_bio_end(struct mapped_device *md, unsigned int zno,
+					struct orig_bio_details *orig_bio_details,
 					unsigned int nr_sectors)
 {
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/* The clone BIO may already have been completed and failed */
@@ -435,7 +437,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		return BLK_STS_IOERR;
 
 	/* Update the zone wp offset */
-	switch (bio_op(orig_bio)) {
+	switch (orig_bio_details->op) {
 	case REQ_OP_ZONE_RESET:
 		WRITE_ONCE(md->zwp_offset[zno], 0);
 		return BLK_STS_OK;
@@ -452,7 +454,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		 * Check that the target did not truncate the write operation
 		 * emulating a zone append.
 		 */
-		if (nr_sectors != bio_sectors(orig_bio)) {
+		if (nr_sectors != orig_bio_details->nr_sectors) {
 			DMWARN_LIMIT("Truncated write for zone append");
 			return BLK_STS_IOERR;
 		}
@@ -488,7 +490,7 @@ static inline void dm_zone_unlock(struct request_queue *q,
 	bio_clear_flag(clone, BIO_ZONE_WRITE_LOCKED);
 }
 
-static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
+static bool dm_need_zone_wp_tracking(struct bio *clone)
 {
 	/*
 	 * Special processing is not needed for operations that do not need the
@@ -496,15 +498,15 @@ static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
 	 * zones and all operations that do not modify directly a sequential
 	 * zone write pointer.
 	 */
-	if (op_is_flush(orig_bio->bi_opf) && !bio_sectors(orig_bio))
+	if (op_is_flush(clone->bi_opf) && !bio_sectors(clone))
 		return false;
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_WRITE_ZEROES:
 	case REQ_OP_WRITE:
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 	case REQ_OP_ZONE_APPEND:
-		return bio_zone_is_seq(orig_bio);
+		return bio_zone_is_seq(clone);
 	default:
 		return false;
 	}
@@ -519,8 +521,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	struct dm_target *ti = tio->ti;
 	struct mapped_device *md = io->md;
 	struct request_queue *q = md->queue;
-	struct bio *orig_bio = io->orig_bio;
 	struct bio *clone = &tio->clone;
+	struct orig_bio_details orig_bio_details;
 	unsigned int zno;
 	blk_status_t sts;
 	int r;
@@ -529,18 +531,21 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	 * IOs that do not change a zone write pointer do not need
 	 * any additional special processing.
 	 */
-	if (!dm_need_zone_wp_tracking(orig_bio))
+	if (!dm_need_zone_wp_tracking(clone))
 		return ti->type->map(ti, clone);
 
 	/* Lock the target zone */
-	zno = bio_zone_no(orig_bio);
+	zno = bio_zone_no(clone);
 	dm_zone_lock(q, zno, clone);
 
+	orig_bio_details.nr_sectors = bio_sectors(clone);
+	orig_bio_details.op = bio_op(clone);
+
 	/*
 	 * Check that the bio and the target zone write pointer offset are
 	 * both valid, and if the bio is a zone append, remap it to a write.
 	 */
-	if (!dm_zone_map_bio_begin(md, orig_bio, clone)) {
+	if (!dm_zone_map_bio_begin(md, zno, clone)) {
 		dm_zone_unlock(q, zno, clone);
 		return DM_MAPIO_KILL;
 	}
@@ -560,7 +565,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * The target submitted the clone BIO. The target zone will
 		 * be unlocked on completion of the clone.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		break;
 	case DM_MAPIO_REMAPPED:
 		/*
@@ -568,7 +574,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * unlock the target zone here as the clone will not be
 		 * submitted.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		if (sts != BLK_STS_OK)
 			dm_zone_unlock(q, zno, clone);
 		break;
Signature · 2 lines
Damien Le Moal
Western Digital Research

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#13 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

In reply to Mike Snitzer
Quoted 36 lines · Mike Snitzer
Quoted 33 lines · Damien Le Moal
Quoted 19 lines · Mike Snitzer
Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

Problem is that we need to look at the BIO op in submission AND completion path to handle zone append requests. So looking at the clone on submission is OK since its op is still the original one. But on the completion path, the clone may now be a regular write emulating a zone append op. And looking at the clone only does not allow detecting that zone append.

Quoted 4 lines
Quoted 2 lines

We could have the orig_bio op saved in dm_io to avoid completely looking at the orig_bio for detecting zone append.

Can you please try the following patch?

This works. I tested with a zoned nullblk + dm-crypt, forcing the zone append emulation code to be used. I ran zonefs tests on top of that with no issues. I will run btrfs tests too later today to exercise things a little more.

Quoted 173 lines

Really sorry for breaking dm-zone.c; please teach this man how to test the basics of all things dm-zoned (is there a testsuite in the tools or something?).

Thanks,
Mike

drivers/md/dm-zone.c +29 -22
index c1ca9be4b79e..896127366000 100644
@@ -360,16 +360,20 @@ static int dm_update_zone_wp_offset(struct mapped_device *md, unsigned int zno,
 	return 0;
 }
 
+struct orig_bio_details {
+	unsigned int op;
+	unsigned int nr_sectors;
+};
+
 /*
  * First phase of BIO mapping for targets with zone append emulation:
  * check all BIO that change a zone writer pointer and change zone
  * append operations into regular write operations.
  */
-static bool dm_zone_map_bio_begin(struct mapped_device *md,
-				  struct bio *orig_bio, struct bio *clone)
+static bool dm_zone_map_bio_begin(struct mapped_device *md, unsigned int zno,
+				  struct bio *clone)
 {
 	sector_t zsectors = blk_queue_zone_sectors(md->queue);
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/*
@@ -384,7 +388,7 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		WRITE_ONCE(md->zwp_offset[zno], zwp_offset);
 	}
 
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 		return true;
@@ -401,9 +405,8 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
 		 * target zone.
 		 */
 		clone->bi_opf = REQ_OP_WRITE | REQ_NOMERGE |
-			(orig_bio->bi_opf & (~REQ_OP_MASK));
-		clone->bi_iter.bi_sector =
-			orig_bio->bi_iter.bi_sector + zwp_offset;
+			(clone->bi_opf & (~REQ_OP_MASK));
+		clone->bi_iter.bi_sector += zwp_offset;
 		break;
 	default:
 		DMWARN_LIMIT("Invalid BIO operation");
@@ -423,11 +426,10 @@ static bool dm_zone_map_bio_begin(struct mapped_device *md,
  * data written to a zone. Note that at this point, the remapped clone BIO
  * may already have completed, so we do not touch it.
  */
-static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
-					struct bio *orig_bio,
+static blk_status_t dm_zone_map_bio_end(struct mapped_device *md, unsigned int zno,
+					struct orig_bio_details *orig_bio_details,
 					unsigned int nr_sectors)
 {
-	unsigned int zno = bio_zone_no(orig_bio);
 	unsigned int zwp_offset = READ_ONCE(md->zwp_offset[zno]);
 
 	/* The clone BIO may already have been completed and failed */
@@ -435,7 +437,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		return BLK_STS_IOERR;
 
 	/* Update the zone wp offset */
-	switch (bio_op(orig_bio)) {
+	switch (orig_bio_details->op) {
 	case REQ_OP_ZONE_RESET:
 		WRITE_ONCE(md->zwp_offset[zno], 0);
 		return BLK_STS_OK;
@@ -452,7 +454,7 @@ static blk_status_t dm_zone_map_bio_end(struct mapped_device *md,
 		 * Check that the target did not truncate the write operation
 		 * emulating a zone append.
 		 */
-		if (nr_sectors != bio_sectors(orig_bio)) {
+		if (nr_sectors != orig_bio_details->nr_sectors) {
 			DMWARN_LIMIT("Truncated write for zone append");
 			return BLK_STS_IOERR;
 		}
@@ -488,7 +490,7 @@ static inline void dm_zone_unlock(struct request_queue *q,
 	bio_clear_flag(clone, BIO_ZONE_WRITE_LOCKED);
 }
 
-static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
+static bool dm_need_zone_wp_tracking(struct bio *clone)
 {
 	/*
 	 * Special processing is not needed for operations that do not need the
@@ -496,15 +498,15 @@ static bool dm_need_zone_wp_tracking(struct bio *orig_bio)
 	 * zones and all operations that do not modify directly a sequential
 	 * zone write pointer.
 	 */
-	if (op_is_flush(orig_bio->bi_opf) && !bio_sectors(orig_bio))
+	if (op_is_flush(clone->bi_opf) && !bio_sectors(clone))
 		return false;
-	switch (bio_op(orig_bio)) {
+	switch (bio_op(clone)) {
 	case REQ_OP_WRITE_ZEROES:
 	case REQ_OP_WRITE:
 	case REQ_OP_ZONE_RESET:
 	case REQ_OP_ZONE_FINISH:
 	case REQ_OP_ZONE_APPEND:
-		return bio_zone_is_seq(orig_bio);
+		return bio_zone_is_seq(clone);
 	default:
 		return false;
 	}
@@ -519,8 +521,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	struct dm_target *ti = tio->ti;
 	struct mapped_device *md = io->md;
 	struct request_queue *q = md->queue;
-	struct bio *orig_bio = io->orig_bio;
 	struct bio *clone = &tio->clone;
+	struct orig_bio_details orig_bio_details;
 	unsigned int zno;
 	blk_status_t sts;
 	int r;
@@ -529,18 +531,21 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 	 * IOs that do not change a zone write pointer do not need
 	 * any additional special processing.
 	 */
-	if (!dm_need_zone_wp_tracking(orig_bio))
+	if (!dm_need_zone_wp_tracking(clone))
 		return ti->type->map(ti, clone);
 
 	/* Lock the target zone */
-	zno = bio_zone_no(orig_bio);
+	zno = bio_zone_no(clone);
 	dm_zone_lock(q, zno, clone);
 
+	orig_bio_details.nr_sectors = bio_sectors(clone);
+	orig_bio_details.op = bio_op(clone);
+
 	/*
 	 * Check that the bio and the target zone write pointer offset are
 	 * both valid, and if the bio is a zone append, remap it to a write.
 	 */
-	if (!dm_zone_map_bio_begin(md, orig_bio, clone)) {
+	if (!dm_zone_map_bio_begin(md, zno, clone)) {
 		dm_zone_unlock(q, zno, clone);
 		return DM_MAPIO_KILL;
 	}
@@ -560,7 +565,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * The target submitted the clone BIO. The target zone will
 		 * be unlocked on completion of the clone.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		break;
 	case DM_MAPIO_REMAPPED:
 		/*
@@ -568,7 +574,8 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		 * unlock the target zone here as the clone will not be
 		 * submitted.
 		 */
-		sts = dm_zone_map_bio_end(md, orig_bio, *tio->len_ptr);
+		sts = dm_zone_map_bio_end(md, zno, &orig_bio_details,
+					  *tio->len_ptr);
 		if (sts != BLK_STS_OK)
 			dm_zone_unlock(q, zno, clone);
 		break;
Signature · 2 lines
Damien Le Moal
Western Digital Research
author
#14 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

In reply to Mike Snitzer
Quoted 19 lines · Mike Snitzer
Quoted 8 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Quoted 7 lines

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

You can just consider the cleanup part of this patches, :-)

1) no late assignment of ->orig_bio, and always set it in alloc_io()

2) no waiting on on ->origi_bio, especially the waiting is done in fast path of dm_submit_bio_remap().

3) no split for io accounting

Thanks,
Ming

#15 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 20 lines · Ming Lei
Quoted 19 lines · Mike Snitzer
Quoted 15 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 8 lines
Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

You can just consider the cleanup part of this patches, :-)

I will. But your following list doesn't reflect any "cleanup" that I saw in your patchset. Pretty fundamental changes that are similar, but different, to the dm-5.19 changes I've staged.

Quoted 4 lines

1) no late assignment of ->orig_bio, and always set it in alloc_io()

2) no waiting on on ->origi_bio, especially the waiting is done in fast path of dm_submit_bio_remap().

For 5.18 waiting on io->orig_bio just enables a signal that the IO was split and can be accounted.

For 5.19 I also plan on using late io->orig_bio assignment as an alternative to the full-blown refcounting currently done with io->io_count. I've yet to quantify the gains with focused testing but in theory this approach should scale better on large systems with many concurrent IO threads to the same device (RCU is primary constraint now).

I'll try to write a bpfrace script to measure how frequently "waiting on io->orig_bio" occurs for dm_submit_bio_remap() heavy usage (like dm-crypt). But I think we'll find it is very rarely, if ever, waited on in the fast path.

Quoted 1 line

3) no split for io accounting

DM's more recent approach to splitting has never been done for benefit or use of IO accounting, see this commit for its origin:
18a25da84354c6b ("dm: ensure bio submission follows a depth-first tree walk")

Not sure why you keep poking fun at DM only doing a single split when:
that is the actual design. DM splits off orig_bio then recurses to handle the remainder of the bio that wasn't issued. Storing it in io->orig_bio (previously io->bio) was always a means of reflecting things properly. And yes IO accounting is one use, the other is IO completion. But unfortunately DM's IO accounting has always been a mess ever since the above commit. Changes in 5.18 fixed that.

But again, DM's splitting has _nothing_ to do with IO accounting. Splitting only happens when needed for IO submission given constraints of DM target(s) or underlying layers.

All said, I will look closer at your entire set and see if it better to go with your approach. This patch in particular is interesting (avoids cloning and other complexity of bio_split + bio_chain):
https://patchwork.kernel.org/project/dm-devel/patch/[email protected]/

author
#16 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 51 lines · Mike Snitzer
Quoted 20 lines · Ming Lei
Quoted 19 lines · Mike Snitzer
Quoted 15 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 8 lines
Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

You can just consider the cleanup part of this patches, :-)

I will. But your following list doesn't reflect any "cleanup" that I saw in your patchset. Pretty fundamental changes that are similar, but different, to the dm-5.19 changes I've staged.

Quoted 4 lines

1) no late assignment of ->orig_bio, and always set it in alloc_io()

2) no waiting on on ->origi_bio, especially the waiting is done in fast path of dm_submit_bio_remap().

For 5.18 waiting on io->orig_bio just enables a signal that the IO was split and can be accounted.

For 5.19 I also plan on using late io->orig_bio assignment as an alternative to the full-blown refcounting currently done with io->io_count. I've yet to quantify the gains with focused testing but in theory this approach should scale better on large systems with many concurrent IO threads to the same device (RCU is primary constraint now).

Quoted 4 lines

I'll try to write a bpfrace script to measure how frequently "waiting on io->orig_bio" occurs for dm_submit_bio_remap() heavy usage (like dm-crypt). But I think we'll find it is very rarely, if ever, waited on in the fast path.

The waiting depends on CPU and device's speed, if device is quicker than CPU, the wait should be longer. Testing in one environment is usually not enough.

Quoted 15 lines
Quoted 2 lines

3) no split for io accounting

DM's more recent approach to splitting has never been done for benefit or use of IO accounting, see this commit for its origin:
18a25da84354c6b ("dm: ensure bio submission follows a depth-first tree walk")

Not sure why you keep poking fun at DM only doing a single split when:
that is the actual design. DM splits off orig_bio then recurses to handle the remainder of the bio that wasn't issued. Storing it in io->orig_bio (previously io->bio) was always a means of reflecting things properly. And yes IO accounting is one use, the other is IO completion. But unfortunately DM's IO accounting has always been a mess ever since the above commit. Changes in 5.18 fixed that.

Quoted 3 lines

But again, DM's splitting has _nothing_ to do with IO accounting. Splitting only happens when needed for IO submission given constraints of DM target(s) or underlying layers.

What I meant is that the bio returned from bio_split() is only for io accounting. Yeah, the comment said it can be for io completion too, but that is easily done without the splitted bio.

Quoted 5 lines

All said, I will look closer at your entire set and see if it better to go with your approach. This patch in particular is interesting (avoids cloning and other complexity of bio_split + bio_chain):
https://patchwork.kernel.org/project/dm-devel/patch/[email protected]/

That patch shows we can avoid the extra split, also shows that the splitted bio from bio_split() is for io accounting only.

thanks,
Ming

#17 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 84 lines · Ming Lei
Quoted 51 lines · Mike Snitzer
Quoted 20 lines · Ming Lei
Quoted 19 lines · Mike Snitzer
Quoted 15 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 8 lines
Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

You can just consider the cleanup part of this patches, :-)

I will. But your following list doesn't reflect any "cleanup" that I saw in your patchset. Pretty fundamental changes that are similar, but different, to the dm-5.19 changes I've staged.

Quoted 4 lines

1) no late assignment of ->orig_bio, and always set it in alloc_io()

2) no waiting on on ->origi_bio, especially the waiting is done in fast path of dm_submit_bio_remap().

For 5.18 waiting on io->orig_bio just enables a signal that the IO was split and can be accounted.

For 5.19 I also plan on using late io->orig_bio assignment as an alternative to the full-blown refcounting currently done with io->io_count. I've yet to quantify the gains with focused testing but in theory this approach should scale better on large systems with many concurrent IO threads to the same device (RCU is primary constraint now).

Quoted 4 lines

I'll try to write a bpfrace script to measure how frequently "waiting on io->orig_bio" occurs for dm_submit_bio_remap() heavy usage (like dm-crypt). But I think we'll find it is very rarely, if ever, waited on in the fast path.

The waiting depends on CPU and device's speed, if device is quicker than CPU, the wait should be longer. Testing in one environment is usually not enough.

Quoted 15 lines
Quoted 2 lines

3) no split for io accounting

DM's more recent approach to splitting has never been done for benefit or use of IO accounting, see this commit for its origin:
18a25da84354c6b ("dm: ensure bio submission follows a depth-first tree walk")

Not sure why you keep poking fun at DM only doing a single split when:
that is the actual design. DM splits off orig_bio then recurses to handle the remainder of the bio that wasn't issued. Storing it in io->orig_bio (previously io->bio) was always a means of reflecting things properly. And yes IO accounting is one use, the other is IO completion. But unfortunately DM's IO accounting has always been a mess ever since the above commit. Changes in 5.18 fixed that.

Quoted 3 lines

But again, DM's splitting has _nothing_ to do with IO accounting. Splitting only happens when needed for IO submission given constraints of DM target(s) or underlying layers.

What I meant is that the bio returned from bio_split() is only for io accounting. Yeah, the comment said it can be for io completion too, but that is easily done without the splitted bio.

Quoted 7 lines
Quoted 4 lines

All said, I will look closer at your entire set and see if it better to go with your approach. This patch in particular is interesting (avoids cloning and other complexity of bio_split + bio_chain):
https://patchwork.kernel.org/project/dm-devel/patch/[email protected]/

That patch shows we can avoid the extra split, also shows that the splitted bio from bio_split() is for io accounting only.

Yes I see that now. But it also served to preserve the original bio for use in completion. Not a big deal, but it did track the head of the bio_chain.

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Mike

author
#18 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 99 lines · Mike Snitzer
Quoted 84 lines · Ming Lei
Quoted 51 lines · Mike Snitzer
Quoted 20 lines · Ming Lei
Quoted 19 lines · Mike Snitzer
Quoted 15 lines · Ming Lei

The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap().

The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting.

Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more.

Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit.

Quoted 8 lines
Quoted 6 lines

You're conflating quite a few things here. DM zone really has no business accessing io->orig_bio (dm-zone.c can just as easily inspect the tio->clone, because it hasn't been remapped yet it reflects the io->origin_bio, so there is no need to look at io->orig_bio) -- but yes I clearly broke things during the 5.18 merge and it needs fixing ASAP.

You can just consider the cleanup part of this patches, :-)

I will. But your following list doesn't reflect any "cleanup" that I saw in your patchset. Pretty fundamental changes that are similar, but different, to the dm-5.19 changes I've staged.

Quoted 4 lines

1) no late assignment of ->orig_bio, and always set it in alloc_io()

2) no waiting on on ->origi_bio, especially the waiting is done in fast path of dm_submit_bio_remap().

For 5.18 waiting on io->orig_bio just enables a signal that the IO was split and can be accounted.

For 5.19 I also plan on using late io->orig_bio assignment as an alternative to the full-blown refcounting currently done with io->io_count. I've yet to quantify the gains with focused testing but in theory this approach should scale better on large systems with many concurrent IO threads to the same device (RCU is primary constraint now).

Quoted 4 lines

I'll try to write a bpfrace script to measure how frequently "waiting on io->orig_bio" occurs for dm_submit_bio_remap() heavy usage (like dm-crypt). But I think we'll find it is very rarely, if ever, waited on in the fast path.

The waiting depends on CPU and device's speed, if device is quicker than CPU, the wait should be longer. Testing in one environment is usually not enough.

Quoted 15 lines
Quoted 2 lines

3) no split for io accounting

DM's more recent approach to splitting has never been done for benefit or use of IO accounting, see this commit for its origin:
18a25da84354c6b ("dm: ensure bio submission follows a depth-first tree walk")

Not sure why you keep poking fun at DM only doing a single split when:
that is the actual design. DM splits off orig_bio then recurses to handle the remainder of the bio that wasn't issued. Storing it in io->orig_bio (previously io->bio) was always a means of reflecting things properly. And yes IO accounting is one use, the other is IO completion. But unfortunately DM's IO accounting has always been a mess ever since the above commit. Changes in 5.18 fixed that.

Quoted 3 lines

But again, DM's splitting has _nothing_ to do with IO accounting. Splitting only happens when needed for IO submission given constraints of DM target(s) or underlying layers.

What I meant is that the bio returned from bio_split() is only for io accounting. Yeah, the comment said it can be for io completion too, but that is easily done without the splitted bio.

Quoted 7 lines
Quoted 4 lines

All said, I will look closer at your entire set and see if it better to go with your approach. This patch in particular is interesting (avoids cloning and other complexity of bio_split + bio_chain):
https://patchwork.kernel.org/project/dm-devel/patch/[email protected]/

That patch shows we can avoid the extra split, also shows that the splitted bio from bio_split() is for io accounting only.

Yes I see that now. But it also served to preserve the original bio for use in completion. Not a big deal, but it did track the head of the bio_chain.

Quoted 7 lines

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 1 line
Ming
#19 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

dm_accept_partial_bio() has been around for a long time, it keeps growing BUG_ONs that are actually helpful to narrow its use to "normal IO", so it should be OK.

Running 'make check' in a built cryptsetup source tree should be a good test for DM target interface functionality.

But there aren't automated tests for IO accounting correctness yet.

Mike

author
#20 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 56 lines · Mike Snitzer
Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

Quoted 2 lines

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

->orig_bio is just for serving io accounting, but ->orig_bio isn't passed to dm_accept_partial_bio(), and not gets updated after dm_accept_partial_bio() is called.

If that is one issue, it must be one existed issue in dm io accounting since ->orig_bio isn't updated when dm_accept_partial_bio() is called.

So do we have to update it?

Quoted 7 lines

dm_accept_partial_bio() has been around for a long time, it keeps growing BUG_ONs that are actually helpful to narrow its use to "normal IO", so it should be OK.

Running 'make check' in a built cryptsetup source tree should be a good test for DM target interface functionality.

Care to share the test tree?

Quoted 2 lines

But there aren't automated tests for IO accounting correctness yet.

I did verify io accounting by running dm-thin with blk-throttle, and the observed throughput is same with expected setting. Running both small bs and large bs, so non-split and split code path are covered.

Maybe you can add this kind of test into dm io accounting automated test.

Thanks,
Ming

#21 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 60 lines · Ming Lei
Quoted 56 lines · Mike Snitzer
Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

Quoted 2 lines

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

Quoted 6 lines

->orig_bio is just for serving io accounting, but ->orig_bio isn't passed to dm_accept_partial_bio(), and not gets updated after dm_accept_partial_bio() is called.

If that is one issue, it must be one existed issue in dm io accounting since ->orig_bio isn't updated when dm_accept_partial_bio() is called.

Recall that ->orig_bio is updated after the bio_split() at the bottom of dm_split_and_process_bio().

That bio_split() is based on ci->sector_count, which is reduced as a side-effect of dm_accept_partial_bio() reducing tio->len_ptr. It is pretty circuitous so I can absolutely understand why you didn't immediately appreciate the interface. The block comment above dm_accept_partial_bio() does a pretty comprehensive job of explaining.

But basically dm_accept_partial_bio() provides DM targets access to control DM core's splitting if they find that they cannot accommodate the entirety of the clone bio that is sent to their ->map. dm_accept_partial_bio() may only ever be called from a target's ->map

Quoted 11 lines

So do we have to update it?

Quoted 7 lines

dm_accept_partial_bio() has been around for a long time, it keeps growing BUG_ONs that are actually helpful to narrow its use to "normal IO", so it should be OK.

Running 'make check' in a built cryptsetup source tree should be a good test for DM target interface functionality.

Care to share the test tree?

https://gitlab.com/cryptsetup/cryptsetup.git

Quoted 9 lines
Quoted 3 lines

But there aren't automated tests for IO accounting correctness yet.

I did verify io accounting by running dm-thin with blk-throttle, and the observed throughput is same with expected setting. Running both small bs and large bs, so non-split and split code path are covered.

Maybe you can add this kind of test into dm io accounting automated test.

Yeah, something like that would be good.

Mike

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

author
#22 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 70 lines · Mike Snitzer
Quoted 60 lines · Ming Lei
Quoted 56 lines · Mike Snitzer
Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

Quoted 2 lines

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

Quoted 6 lines

->orig_bio is just for serving io accounting, but ->orig_bio isn't passed to dm_accept_partial_bio(), and not gets updated after dm_accept_partial_bio() is called.

If that is one issue, it must be one existed issue in dm io accounting since ->orig_bio isn't updated when dm_accept_partial_bio() is called.

Quoted 8 lines

Recall that ->orig_bio is updated after the bio_split() at the bottom of dm_split_and_process_bio().

That bio_split() is based on ci->sector_count, which is reduced as a side-effect of dm_accept_partial_bio() reducing tio->len_ptr. It is pretty circuitous so I can absolutely understand why you didn't immediately appreciate the interface. The block comment above dm_accept_partial_bio() does a pretty comprehensive job of explaining.

Go it now, thanks for the explanation.

As you mentioned, it can be addressed in dm_accept_partial_bio() by updating ti->io->sectors.

Thanks,
Ming

#23 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 80 lines · Ming Lei
Quoted 70 lines · Mike Snitzer
Quoted 60 lines · Ming Lei
Quoted 56 lines · Mike Snitzer
Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

Quoted 2 lines

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

Quoted 6 lines

->orig_bio is just for serving io accounting, but ->orig_bio isn't passed to dm_accept_partial_bio(), and not gets updated after dm_accept_partial_bio() is called.

If that is one issue, it must be one existed issue in dm io accounting since ->orig_bio isn't updated when dm_accept_partial_bio() is called.

Quoted 8 lines

Recall that ->orig_bio is updated after the bio_split() at the bottom of dm_split_and_process_bio().

That bio_split() is based on ci->sector_count, which is reduced as a side-effect of dm_accept_partial_bio() reducing tio->len_ptr. It is pretty circuitous so I can absolutely understand why you didn't immediately appreciate the interface. The block comment above dm_accept_partial_bio() does a pretty comprehensive job of explaining.

Quoted 4 lines

Go it now, thanks for the explanation.

As you mentioned, it can be addressed in dm_accept_partial_bio() by updating ti->io->sectors.

Yes, I rebased your patchset ontop of dm-5.19 and fixed up your splitting like we discussed. I'll be rebasing ontop of v5.18-rc3 once it is released but please have a look at this 'dm-5.19-v2' branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19-v2

And this commit in particular:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19-v2&id=fe5a99da8b0d0518342f5cdb522a06b0f123ca09

Once I've verified with you that it looks OK I'll fold it into your commit (at the same time I rebase on v5.18-rc3 early next week).

In general this rebase sucked.. but I wanted to layer your changes ontop of earlier changes I had already staged for 5.19. I've at least verified that DM's bio polling seems to work like usual and also verified that cryptsetup's 'make check' passes.

No urgency on you reviewing before early next week. Have a great weekend.

Mike

author
#24 lore raw

Re: [PATCH 5/8] dm: always setup ->orig_bio in alloc_io

Quoted 93 lines · Mike Snitzer
Quoted 80 lines · Ming Lei
Quoted 70 lines · Mike Snitzer
Quoted 60 lines · Ming Lei
Quoted 56 lines · Mike Snitzer
Quoted 48 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

The bigger issue with this patch is that you've caused dm_submit_bio_remap() to go back to accounting the entire original bio before any split occurs. That is a problem because you'll end up accounting that bio for every split, so in split heavy workloads the IO accounting won't reflect when the IO is actually issued and we'll regress back to having very inaccurate and incorrect IO accounting for dm_submit_bio_remap() heavy targets (e.g. dm-crypt).

Good catch, but we know the length of mapped part in original bio before calling __map_bio(), so io->sectors/io->offset_sector can be setup here, something like the following delta change should address it:

drivers/md/dm.c +7 -5
index db23efd6bbf6..06b554f3104b 100644
@@ -1558,6 +1558,13 @@ static int __split_and_process_bio(struct clone_info *ci)
 
 	len = min_t(sector_t, max_io_len(ti, ci->sector), ci->sector_count);
 	clone = alloc_tio(ci, ti, 0, &len, GFP_NOIO);
+
+	if (ci->sector_count > len) {
+		/* setup the mapped part for accounting */
+		dm_io_set_flag(ci->io, DM_IO_SPLITTED);
+		ci->io->sectors = len;
+		ci->io->sector_offset = bio_end_sector(ci->bio) - ci->sector;
+	}
 	__map_bio(clone);
 
 	ci->sector += len;
@@ -1603,11 +1610,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.sector_count)
 		goto out;
 
-	/* setup the mapped part for accounting */
-	dm_io_set_flag(ci.io, DM_IO_SPLITTED);
-	ci.io->sectors = bio_sectors(bio) - ci.sector_count;
-	ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector;
-
 	bio_trim(bio, ci.io->sectors, ci.sector_count);
 	trace_block_split(bio, bio->bi_iter.bi_sector);
 	bio_inc_remaining(bio);
Signature · 2 lines
-- 
Ming

Unfortunately we do need splitting after __map_bio() because a dm target's ->map can use dm_accept_partial_bio() to further reduce a bio's mapped part.

Quoted 2 lines

But I think dm_accept_partial_bio() could be trained to update tio->io->sectors?

Quoted 6 lines

->orig_bio is just for serving io accounting, but ->orig_bio isn't passed to dm_accept_partial_bio(), and not gets updated after dm_accept_partial_bio() is called.

If that is one issue, it must be one existed issue in dm io accounting since ->orig_bio isn't updated when dm_accept_partial_bio() is called.

Quoted 8 lines

Recall that ->orig_bio is updated after the bio_split() at the bottom of dm_split_and_process_bio().

That bio_split() is based on ci->sector_count, which is reduced as a side-effect of dm_accept_partial_bio() reducing tio->len_ptr. It is pretty circuitous so I can absolutely understand why you didn't immediately appreciate the interface. The block comment above dm_accept_partial_bio() does a pretty comprehensive job of explaining.

Quoted 4 lines

Go it now, thanks for the explanation.

As you mentioned, it can be addressed in dm_accept_partial_bio() by updating ti->io->sectors.

Yes, I rebased your patchset ontop of dm-5.19 and fixed up your splitting like we discussed. I'll be rebasing ontop of v5.18-rc3 once it is released but please have a look at this 'dm-5.19-v2' branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19-v2

Quoted 5 lines

And this commit in particular:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.19-v2&id=fe5a99da8b0d0518342f5cdb522a06b0f123ca09

Once I've verified with you that it looks OK I'll fold it into your commit (at the same time I rebase on v5.18-rc3 early next week).

Hi Mike,

Your delta change looks good, thanks for fixing it!

Thanks,
Ming

author
#25 lore raw

PATCH 6/8 dm: don't grab target io reference in dm_zone_map_bio

In reply to Ming Lei

dm_zone_map_bio() is only called from __map_bio in which the io's reference is grabbed already, and the reference won't be released until the bio is submitted, so no necessary to do it dm_zone_map_bio any more.

---
 drivers/md/dm-core.h |  7 -------
 drivers/md/dm-zone.c | 10 ----------
 drivers/md/dm.c      |  7 ++++++-
 3 files changed, 6 insertions(+), 18 deletions(-)
drivers/md/dm-core.h +0 -7
index aefb080c230d..811c0ccbc63d 100644
@@ -283,13 +283,6 @@ static inline void dm_io_set_flag(struct dm_io *io, unsigned int bit)
 	io->flags |= (1U << bit);
 }
 
-static inline void dm_io_inc_pending(struct dm_io *io)
-{
-	atomic_inc(&io->io_count);
-}
-
-void dm_io_dec_pending(struct dm_io *io, blk_status_t error);
-
 static inline struct completion *dm_get_completion_from_kobject(struct kobject *kobj)
 {
 	return &container_of(kobj, struct dm_kobject_holder, kobj)->completion;
drivers/md/dm-zone.c +0 -10
index c1ca9be4b79e..85d3c158719f 100644
@@ -545,13 +545,6 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		return DM_MAPIO_KILL;
 	}
 
-	/*
-	 * The target map function may issue and complete the IO quickly.
-	 * Take an extra reference on the IO to make sure it does disappear
-	 * until we run dm_zone_map_bio_end().
-	 */
-	dm_io_inc_pending(io);
-
 	/* Let the target do its work */
 	r = ti->type->map(ti, clone);
 	switch (r) {
@@ -580,9 +573,6 @@ int dm_zone_map_bio(struct dm_target_io *tio)
 		break;
 	}
 
-	/* Drop the extra reference on the IO */
-	dm_io_dec_pending(io, sts);
-
 	if (sts != BLK_STS_OK)
 		return DM_MAPIO_KILL;
 
drivers/md/dm.c +6 -1
index df1d013fb793..3c3ba6b4e19b 100644
@@ -937,11 +937,16 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)
 		!dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO));
 }
 
+static void dm_io_inc_pending(struct dm_io *io)
+{
+	atomic_inc(&io->io_count);
+}
+
 /*
  * Decrements the number of outstanding ios that a bio has been
  * cloned into, completing the original io if necc.
  */
-void dm_io_dec_pending(struct dm_io *io, blk_status_t error)
+static void dm_io_dec_pending(struct dm_io *io, blk_status_t error)
 {
 	/* Push-back supersedes any I/O errors */
 	if (unlikely(error)) {
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author
#26 lore raw

PATCH 7/8 dm: improve target io referencing

In reply to Ming Lei

Currently target io's reference counter is grabbed before calling __map_bio(), this way isn't efficient since we can move this grabbing into alloc_io().

Meantime it becomes typical async io reference counter model: one is for submission side, the other is for completion side, and the io won't be completed until both two sides are done.

---
 drivers/md/dm.c | 51 ++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 38 insertions(+), 13 deletions(-)
drivers/md/dm.c +38 -13
index 3c3ba6b4e19b..2987f7cf7b47 100644
@@ -587,7 +587,9 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
 	io = container_of(tio, struct dm_io, tio);
 	io->magic = DM_IO_MAGIC;
 	io->status = 0;
-	atomic_set(&io->io_count, 1);
+
+	/* one is for submission, the other is for completion */
+	atomic_set(&io->io_count, 2);
 	this_cpu_inc(*md->pending_io);
 	io->orig_bio = bio;
 	io->md = md;
@@ -937,11 +939,6 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)
 		!dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO));
 }
 
-static void dm_io_inc_pending(struct dm_io *io)
-{
-	atomic_inc(&io->io_count);
-}
-
 /*
  * Decrements the number of outstanding ios that a bio has been
  * cloned into, completing the original io if necc.
@@ -1276,7 +1273,6 @@ static void __map_bio(struct bio *clone)
 	/*
 	 * Map the clone.
 	 */
-	dm_io_inc_pending(io);
 	tio->old_sector = clone->bi_iter.bi_sector;
 
 	if (unlikely(swap_bios_limit(ti, clone))) {
@@ -1358,11 +1354,12 @@ static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci,
 	}
 }
 
-static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
+static int __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
 				  unsigned num_bios, unsigned *len)
 {
 	struct bio_list blist = BIO_EMPTY_LIST;
 	struct bio *clone;
+	int ret = 0;
 
 	switch (num_bios) {
 	case 0:
@@ -1371,15 +1368,19 @@ static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
 		clone = alloc_tio(ci, ti, 0, len, GFP_NOIO);
 		dm_tio_set_flag(clone_to_tio(clone), DM_TIO_IS_DUPLICATE_BIO);
 		__map_bio(clone);
+		ret = 1;
 		break;
 	default:
 		alloc_multiple_bios(&blist, ci, ti, num_bios, len);
 		while ((clone = bio_list_pop(&blist))) {
 			dm_tio_set_flag(clone_to_tio(clone), DM_TIO_IS_DUPLICATE_BIO);
 			__map_bio(clone);
+			ret += 1;
 		}
 		break;
 	}
+
+	return ret;
 }
 
 static void __send_empty_flush(struct clone_info *ci)
@@ -1399,8 +1400,19 @@ static void __send_empty_flush(struct clone_info *ci)
 	ci->bio = &flush_bio;
 	ci->sector_count = 0;
 
-	while ((ti = dm_table_get_target(ci->map, target_nr++)))
-		__send_duplicate_bios(ci, ti, ti->num_flush_bios, NULL);
+	while ((ti = dm_table_get_target(ci->map, target_nr++))) {
+		int bios;
+
+		atomic_add(ti->num_flush_bios, &ci->io->io_count);
+		bios = __send_duplicate_bios(ci, ti, ti->num_flush_bios, NULL);
+		atomic_sub(ti->num_flush_bios - bios, &ci->io->io_count);
+	}
+
+	/*
+	 * alloc_io() takes one extra reference for submission, so the
+	 * reference won't reach 0 after the following subtraction
+	 */
+	atomic_sub(1, &ci->io->io_count);
 
 	bio_uninit(ci->bio);
 }
@@ -1409,6 +1421,7 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_target
 					unsigned num_bios)
 {
 	unsigned len;
+	int bios;
 
 	len = min_t(sector_t, ci->sector_count,
 		    max_io_len_target_boundary(ti, dm_target_offset(ti, ci->sector)));
@@ -1420,7 +1433,13 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_target
 	ci->sector += len;
 	ci->sector_count -= len;
 
-	__send_duplicate_bios(ci, ti, num_bios, &len);
+	atomic_add(num_bios, &ci->io->io_count);
+	bios = __send_duplicate_bios(ci, ti, num_bios, &len);
+	/*
+	 * alloc_io() takes one extra reference for submission, so the
+	 * reference won't reach 0 after the following subtraction
+	 */
+	atomic_sub(num_bios - bios + 1, &ci->io->io_count);
 }
 
 static bool is_abnormal_io(struct bio *bio)
@@ -1603,9 +1622,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	 * Add every dm_io instance into the hlist_head which is stored in
 	 * bio->bi_private, so that dm_poll_bio can poll them all.
 	 */
-	if (error || !ci.submit_as_polled)
+	if (error || !ci.submit_as_polled) {
+		/*
+		 * In case of submission failure, the extra reference for
+		 * submitting io isn't consumed yet
+		 */
+		if (error)
+			atomic_dec(&ci.io->io_count);
 		dm_io_dec_pending(ci.io, errno_to_blk_status(error));
-	else
+	} else
 		dm_queue_poll_io(bio, ci.io);
 }
 
Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author
#27 lore raw

PATCH 8/8 dm: put all polled io into one single list

In reply to Ming Lei

If bio_split() isn't involved, it is a bit overkill to link dm_io into hlist, given there is only single dm_io in the list, so convert to single list for holding all dm_io instances associated with this bio.

---
 drivers/md/dm-core.h |  2 +-
 drivers/md/dm.c      | 46 +++++++++++++++++++++++---------------------
 2 files changed, 25 insertions(+), 23 deletions(-)
drivers/md/dm-core.h +1 -1
index 811c0ccbc63d..7f51957913e8 100644
@@ -257,7 +257,7 @@ struct dm_io {
 	spinlock_t lock;
 	unsigned long start_time;
 	void *data;
-	struct hlist_node node;
+	struct dm_io *next;
 	struct task_struct *map_task;
 	struct dm_stats_aux stats_aux;
 	/* last member of dm_target_io is 'struct bio' */
drivers/md/dm.c +24 -22

106 lines not on this page · raw message

Signature · 6 lines
2.31.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#28 lore raw
In reply to Ming Lei
Quoted 20 lines · Ming Lei

Hello Guys,

The 1st patch adds bdev based io accounting interface.

The 2nd ~ 5th patches improves dm's io accounting & split, meantime fixes kernel panic on dm-zone.

The other patches improves io polling & dm io reference handling.

Ming Lei (8):
block: replace disk based account with bdev's
dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct
dm: pass 'dm_io' instance to dm_io_acct directly
dm: switch to bdev based io accounting interface
dm: always setup ->orig_bio in alloc_io
dm: don't grab target io reference in dm_zone_map_bio
dm: improve target io referencing
dm: put all polled io into one single list

Quoted 11 lines
 block/blk-core.c              |  15 +--
 drivers/block/zram/zram_drv.c |   5 +-
 drivers/md/dm-core.h          |  17 ++-
 drivers/md/dm-zone.c          |  10 --
 drivers/md/dm.c               | 190 +++++++++++++++++++---------------
 include/linux/blkdev.h        |   7 +-
 6 files changed, 131 insertions(+), 113 deletions(-)
Signature · 1 line
2.31.1

I'll review this closely but, a couple weeks ago, I queued up quite a lot of conflicting changes for 5.19 here:

https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#30 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Mike Snitzer

I'll review this closely but, a couple weeks ago, I queued up quite a lot of conflicting changes for 5.19 here:

https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

Can you please end them out to the device mapper list for review?

Yes, no problem. I brought my various earlier changes to the front, dropped my changes for removing dm_io refcounting for normal r/w IO (still visible in 'dm-5.19-v1'), and rebased Ming's series (now visible in 'dm-5.19-v2').

Common: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19
Old: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19-v1
New: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19-v2

I'll clean up the 'dm-5.19-v2' series further and likely post to the list on Tuesday.

I discussed with Jens and we'll both be rebasing our 5.19 trees to v5.18-rc3. dm will be rebasing on block (I can fix up Ming's patch for bdev_{start,end}_io_acct based on your review and Jens can then pick it up from the first patch in the series I post).

Mike