Skip to contentSkip to conversation

block/dm: support bio polling

PATCH Superseded Mike Snitzer 4 participants 10 messages Mar 5, 2022 – Mar 7, 2022

Subsystems (2)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v5 cover block/dm: support bio polling

Hi,

I've rebased Ming's latest [1] ontop of dm-5.18 [2] (which is based on for-5.18/block). End result available in dm-5.18-biopoll branch [3]

These changes add bio polling support to DM. Tested with linear and striped DM targets.

IOPS improvement was ~5% on my baremetal system with a single Intel Optane NVMe device (555K hipri=1 vs 525K hipri=0).

Ming has seen better improvement while testing within a VM:
dm-linear: hipri=1 vs hipri=0 15~20% iops improvement
dm-stripe: hipri=1 vs hipri=0 ~30% iops improvement

I'd like to merge these changes via the DM tree when the 5.18 merge window opens. The first block patch that adds ->poll_bio to block_device_operations will need review so that I can take it through the DM tree. Reason for going through the DM tree is there have been some fairly extensive changes queued in dm-5.18 that build on for-5.18/block. So I think it easiest to just add the block depenency via DM tree since DM is first consumer of ->poll_bio

FYI, Ming does have another DM patch [4] that looks to avoid using hlist but I only just saw it. bio_split() _is_ involved (see dm_split_and_process_bio) so I'm not exactly sure where he is going with that change. But that is DM-implementation detail that we'll sort out. Big thing is we need approval for the first block patch to go to Linus via the DM tree ;)

Thanks,
Mike

[1] https://github.com/ming1/linux/commits/my_v5.18-dm-bio-poll
[2] https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.18
[3] https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.18-biopoll
[4] https://github.com/ming1/linux/commit/c107c30e15041ac1ce672f56809961406e2a3e52

v5: remove WARN_ONs in ->poll_bio interface patch. Fixed comment typo
along the way (found while seeing how other block_device_operations are referenced in block's code comments).

Ming Lei (2):
block: add ->poll_bio to block_device_operations
dm: support bio polling

 block/blk-core.c       |  14 +++--
 block/genhd.c          |   4 ++
 drivers/md/dm-core.h   |   2 +
 drivers/md/dm-table.c  |  27 +++++++++
 drivers/md/dm.c        | 150 ++++++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/blkdev.h |   2 +
 6 files changed, 191 insertions(+), 8 deletions(-)
Signature · 1 line
2.15.0
#1 lore raw

PATCH v5 1/2 block: add ->poll_bio to block_device_operations

From: Ming Lei <[email protected]>

Prepare for supporting IO polling for bio-based driver.

Add ->poll_bio callback so that bio-based driver can provide their own logic for polling bio.

Also fix ->submit_bio_bio typo in comment block above __submit_bio_noacct.

---
 block/blk-core.c       | 14 +++++++++-----
 block/genhd.c          |  4 ++++
 include/linux/blkdev.h |  2 ++
 3 files changed, 15 insertions(+), 5 deletions(-)
block/blk-core.c +9 -5
index 94bf37f8e61d..ce08f0aa9dfc 100644
@@ -708,7 +708,7 @@ static void __submit_bio(struct bio *bio)
  *
  * bio_list_on_stack[0] contains bios submitted by the current ->submit_bio.
  * bio_list_on_stack[1] contains bios that were submitted before the current
- *	->submit_bio_bio, but that haven't been processed yet.
+ *	->submit_bio, but that haven't been processed yet.
  */
 static void __submit_bio_noacct(struct bio *bio)
 {
@@ -975,7 +975,7 @@ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags)
 {
 	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
 	blk_qc_t cookie = READ_ONCE(bio->bi_cookie);
-	int ret;
+	int ret = 0;
 
 	if (cookie == BLK_QC_T_NONE ||
 	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
@@ -985,10 +985,14 @@ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags)
 
 	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))
 		return 0;
-	if (WARN_ON_ONCE(!queue_is_mq(q)))
-		ret = 0;	/* not yet implemented, should not happen */
-	else
+	if (queue_is_mq(q)) {
 		ret = blk_mq_poll(q, cookie, iob, flags);
+	} else {
+		struct gendisk *disk = q->disk;
+
+		if (disk && disk->fops->poll_bio)
+			ret = disk->fops->poll_bio(bio, iob, flags);
+	}
 	blk_queue_exit(q);
 	return ret;
 }
block/genhd.c +4 -0
index e351fac41bf2..1ed46a6f94f5 100644
@@ -410,6 +410,10 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk,
 	struct device *ddev = disk_to_dev(disk);
 	int ret;
 
+	/* Only makes sense for bio-based to set ->poll_bio */
+	if (queue_is_mq(disk->queue) && disk->fops->poll_bio)
+		return -EINVAL;
+
 	/*
 	 * The disk queue should now be all set with enough information about
 	 * the device for the elevator code to pick an adequate default
include/linux/blkdev.h +2 -0
index f757f9c2871f..51f1b1ddbed2 100644
@@ -1455,6 +1455,8 @@ enum blk_unique_id {
 
 struct block_device_operations {
 	void (*submit_bio)(struct bio *bio);
+	int (*poll_bio)(struct bio *bio, struct io_comp_batch *iob,
+			unsigned int flags);
 	int (*open) (struct block_device *, fmode_t);
 	void (*release) (struct gendisk *, fmode_t);
 	int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int);
Signature · 1 line
2.15.0
#3 lore raw

PATCH v5 2/2 dm: support bio polling

In reply to Mike Snitzer

From: Ming Lei <[email protected]>

Support bio(REQ_POLLED) polling in the following approach:

1) only support io polling on normal READ/WRITE, and other abnormal IOs still fallback to IRQ mode, so the target io is exactly inside the dm io.

2) hold one refcnt on io->io_count after submitting this dm bio with REQ_POLLED

3) support dm native bio splitting, any dm io instance associated with current bio will be added into one list which head is bio->bi_end_io which will be recovered before ending this bio

4) implement .poll_bio() callback, call bio_poll() on the single target bio inside the dm io which is retrieved via bio->bi_bio_drv_data; call dm_io_dec_pending() after the target io is done in .poll_bio()

5) enable QUEUE_FLAG_POLL if all underlying queues enable QUEUE_FLAG_POLL, which is based on Jeffle's previous patch.

---
 drivers/md/dm-core.h  |   2 +
 drivers/md/dm-table.c |  27 +++++++++
 drivers/md/dm.c       | 150 +++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 176 insertions(+), 3 deletions(-)
drivers/md/dm-core.h +2 -0
index 8078b6c155ef..b3d1429fba83 100644
@@ -235,6 +235,8 @@ struct dm_io {
 	bool start_io_acct:1;
 	int was_accounted;
 	unsigned long start_time;
+	void *saved_bio_end_io;
+	struct hlist_node node;
 	spinlock_t endio_lock;
 	struct dm_stats_aux stats_aux;
 	/* last member of dm_target_io is 'struct bio' */
drivers/md/dm-table.c +27 -0
index f4ed756ab391..c0be4f60b427 100644
@@ -1481,6 +1481,14 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector)
 	return &t->targets[(KEYS_PER_NODE * n) + k];
 }
 
+static int device_not_poll_capable(struct dm_target *ti, struct dm_dev *dev,
+				   sector_t start, sector_t len, void *data)
+{
+	struct request_queue *q = bdev_get_queue(dev->bdev);
+
+	return !test_bit(QUEUE_FLAG_POLL, &q->queue_flags);
+}
+
 /*
  * type->iterate_devices() should be called when the sanity check needs to
  * iterate and check all underlying data devices. iterate_devices() will
@@ -1531,6 +1539,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev,
 	return 0;
 }
 
+static int dm_table_supports_poll(struct dm_table *t)
+{
+	return !dm_table_any_dev_attr(t, device_not_poll_capable, NULL);
+}
+
 /*
  * Check whether a table has no data devices attached using each
  * target's iterate_devices method.
@@ -2067,6 +2080,20 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
 	dm_update_crypto_profile(q, t);
 	disk_update_readahead(t->md->disk);
 
+	/*
+	 * Check for request-based device is left to
+	 * dm_mq_init_request_queue()->blk_mq_init_allocated_queue().
+	 *
+	 * For bio-based device, only set QUEUE_FLAG_POLL when all
+	 * underlying devices supporting polling.
+	 */
+	if (__table_type_bio_based(t->type)) {
+		if (dm_table_supports_poll(t))
+			blk_queue_flag_set(QUEUE_FLAG_POLL, q);
+		else
+			blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
+	}
+
 	return 0;
 }
 
drivers/md/dm.c +147 -3

211 lines not on this page · raw message

Signature · 1 line
2.15.0
#4 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471

Quoted 11 lines
drivers/md/dm.c +11 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

maintainer
#5 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

Quoted 9 lines · Christoph Hellwig
Quoted 9 lines
drivers/md/dm.c +9 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

If it's possible, then that would be preferable. But I don't think that's going to be easy to do...

Signature · 1 line
Jens Axboe
#6 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

Quoted 19 lines · Jens Axboe
Quoted 9 lines · Christoph Hellwig
Quoted 9 lines
drivers/md/dm.c +9 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

Quoted 2 lines

If it's possible, then that would be preferable. But I don't think that's going to be easy to do...

I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...

Thanks,
Ming

maintainer
#7 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig

drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471

Quoted 11 lines
drivers/md/dm.c +11 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

Quoted 5 lines
Quoted 2 lines

If it's possible, then that would be preferable. But I don't think that's going to be easy to do...

I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...

There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.

Maybe some commenting would help? Is bi_next available at this point?

Signature · 1 line
Jens Axboe
#8 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

Quoted 27 lines · Jens Axboe
Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig

drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471

Quoted 11 lines
drivers/md/dm.c +11 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

Quoted 5 lines
Quoted 2 lines

If it's possible, then that would be preferable. But I don't think that's going to be easy to do...

I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...

Quoted 6 lines

There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.

Maybe some commenting would help?

OK, ->bi_end_io is safe because it is only called until the bio is ended, so we can retrieve the list head and recover ->bi_end_io before polling.

Quoted 1 line

Is bi_next available at this point?

The same bio can be re-submitted to block layer because of splitting, and will be linked to current->bio_list[].

BTW, bio splitting can be very often for some dm target, that is why we don't ignore bio splitting for dm polling.

Thanks,
Ming

#9 lore raw

Re: [PATCH v5 2/2] dm: support bio polling

Quoted 33 lines · Ming Lei
Quoted 32 lines · Jens Axboe
Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig

drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471

Quoted 11 lines
drivers/md/dm.c +11 -0
+/*
+ * Reuse ->bi_end_io as hlist head for storing all dm_io instances
+ * associated with this bio, and this bio's bi_end_io has to be
+ * stored in one of 'dm_io' instance first.
+ */
+static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio)
+{
+	WARN_ON_ONCE(!(bio->bi_opf & REQ_DM_POLL_LIST));
+
+	return (struct hlist_head *)&bio->bi_end_io;
+}

So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.

Quoted 5 lines
Quoted 2 lines

If it's possible, then that would be preferable. But I don't think that's going to be easy to do...

I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...

There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.

Quoted 5 lines
Quoted 1 line

Maybe some commenting would help?

OK, ->bi_end_io is safe because it is only called until the bio is ended, so we can retrieve the list head and recover ->bi_end_io before polling.

->bi_private can be reused too, is that better?

Yeah, both belong to owner(higher level storage), then block layer can't touch them inside submit_bio_noacct(), that is also why this trick is safe.

Thanks,
Ming