Skip to contentSkip to conversation

block/dm: support bio polling

PATCH In review Ming Lei 2 participants 8 messages Jun 23, 2021 – Mar 2, 2022

Subsystems (3)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v3 cover block/dm: support bio polling

Hello Guys,

Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.

Patch 1 adds helper of blk_queue_poll().

Patch 2 adds .bio_poll() callback to block_device_operations, so bio driver can implement its own logic for io polling.

Patch 3 implements bio polling for device mapper.

V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFC

V2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add comment

Ming Lei (3):
block: add helper of blk_queue_poll
block: add ->poll_bio to block_device_operations
dm: support bio polling

 block/blk-core.c         |  18 +++---
 block/blk-sysfs.c        |   4 +-
 block/genhd.c            |   2 +
 drivers/md/dm-table.c    |  24 +++++++
 drivers/md/dm.c          | 131 ++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/core.c |   2 +-
 include/linux/blkdev.h   |   2 +
 7 files changed, 170 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v3 1/3 block: add helper of blk_queue_poll

There has been 3 users, and will be more, so add one such helper.

---
 block/blk-core.c         | 5 ++---
 block/blk-sysfs.c        | 4 ++--
 drivers/nvme/host/core.c | 2 +-
 include/linux/blkdev.h   | 1 +
 4 files changed, 6 insertions(+), 6 deletions(-)
block/blk-core.c +2 -3
index 531176578221..1e24c71c6738 100644
@@ -835,7 +835,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio)
 		}
 	}
 
-	if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
+	if (!blk_queue_poll(q))
 		bio->bi_opf &= ~REQ_POLLED;
 
 	switch (bio_op(bio)) {
@@ -1117,8 +1117,7 @@ int bio_poll(struct bio *bio, unsigned int flags)
 	blk_qc_t cookie = READ_ONCE(bio->bi_cookie);
 	int ret;
 
-	if (cookie == BLK_QC_T_NONE ||
-	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
+	if (cookie == BLK_QC_T_NONE || !blk_queue_poll(q))
 		return 0;
 
 	if (current->plug)
block/blk-sysfs.c +2 -2
index f78e73ca6091..93dcf2dfaafd 100644
@@ -422,13 +422,13 @@ static ssize_t queue_poll_delay_store(struct request_queue *q, const char *page,
 
 static ssize_t queue_poll_show(struct request_queue *q, char *page)
 {
-	return queue_var_show(test_bit(QUEUE_FLAG_POLL, &q->queue_flags), page);
+	return queue_var_show(blk_queue_poll(q), page);
 }
 
 static ssize_t queue_poll_store(struct request_queue *q, const char *page,
 				size_t count)
 {
-	if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
+	if (!blk_queue_poll(q))
 		return -EINVAL;
 	pr_info_ratelimited("writes to the poll attribute are ignored.\n");
 	pr_info_ratelimited("please use driver specific parameters instead.\n");
drivers/nvme/host/core.c +1 -1
index fe0b8da3de7f..e31c7704ef4d 100644
@@ -1025,7 +1025,7 @@ static void nvme_execute_rq_polled(struct request_queue *q,
 {
 	DECLARE_COMPLETION_ONSTACK(wait);
 
-	WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags));
+	WARN_ON_ONCE(!blk_queue_poll(q));
 
 	rq->cmd_flags |= REQ_POLLED;
 	rq->end_io_data = &wait;
include/linux/blkdev.h +1 -0
index 561b04117bd4..fc0ba0b80776 100644
@@ -677,6 +677,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
 #define blk_queue_fua(q)	test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags)
 #define blk_queue_registered(q)	test_bit(QUEUE_FLAG_REGISTERED, &(q)->queue_flags)
 #define blk_queue_nowait(q)	test_bit(QUEUE_FLAG_NOWAIT, &(q)->queue_flags)
+#define blk_queue_poll(q)	test_bit(QUEUE_FLAG_POLL, &(q)->queue_flags)
 
 extern void blk_set_pm_only(struct request_queue *q);
 extern void blk_clear_pm_only(struct request_queue *q);
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH v3 2/3 block: add ->poll_bio to block_device_operations

In reply to Ming Lei

Prepare for supporting IO polling for bio based driver.

Add ->poll_bio callback so that bio driver can provide their own logic for polling bio.

---
 block/blk-core.c       | 13 +++++++++----
 block/genhd.c          |  2 ++
 include/linux/blkdev.h |  1 +
 3 files changed, 12 insertions(+), 4 deletions(-)
block/blk-core.c +9 -4
index 1e24c71c6738..e585e549c291 100644
@@ -1113,7 +1113,8 @@ EXPORT_SYMBOL(submit_bio);
  */
 int bio_poll(struct bio *bio, unsigned int flags)
 {
-	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
+	struct gendisk *disk = bio->bi_bdev->bd_disk;
+	struct request_queue *q = disk->queue;
 	blk_qc_t cookie = READ_ONCE(bio->bi_cookie);
 	int ret;
 
@@ -1125,10 +1126,14 @@ int bio_poll(struct bio *bio, unsigned int flags)
 
 	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))
 		return 0;
-	if (WARN_ON_ONCE(!queue_is_mq(q)))
-		ret = 0;	/* not yet implemented, should not happen */
-	else
+
+	if (queue_is_mq(q))
 		ret = blk_mq_poll(q, cookie, flags);
+	else if (disk->fops->poll_bio)
+		ret = disk->fops->poll_bio(bio, flags);
+	else
+		ret = !WARN_ON_ONCE(1);
+
 	blk_queue_exit(q);
 	return ret;
 }
block/genhd.c +2 -0
index 5f5628216295..3dfb7d52e280 100644
@@ -471,6 +471,8 @@ static void __device_add_disk(struct device *parent, struct gendisk *disk,
 {
 	int ret;
 
+	WARN_ON_ONCE(queue_is_mq(disk->queue) && disk->fops->poll_bio);
+
 	/*
 	 * The disk queue should now be all set with enough information about
 	 * the device for the elevator code to pick an adequate default
include/linux/blkdev.h +1 -0
index fc0ba0b80776..fc63155d2ac4 100644
@@ -1858,6 +1858,7 @@ static inline void blk_ksm_unregister(struct request_queue *q) { }
 
 struct block_device_operations {
 	void (*submit_bio)(struct bio *bio);
+	int (*poll_bio)(struct bio *bio, unsigned int flags);
 	int (*open) (struct block_device *, fmode_t);
 	void (*release) (struct gendisk *, fmode_t);
 	int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int);
Signature · 1 line
2.31.1
author
#3 lore raw

PATCH v3 3/3 dm: support bio polling

In reply to Ming Lei

Support bio(REQ_POLLED) polling in the following approach:

1) only support io polling on normal READ/WRITE, and other abnormal IOs still fallback on IRQ mode, so the target io is exactly inside the dm io.

2) hold one refcnt on io->io_count after submitting this dm bio with REQ_POLLED

3) support dm native bio splitting, any dm io instance associated with current bio will be added into one list which head is bio->bi_end_io which will be recovered before ending this bio

4) implement .poll_bio() callback, call bio_poll() on the single target bio inside the dm io which is retrieved via bio->bi_bio_drv_data; call dec_pending() after the target io is done in .poll_bio()

4) enable QUEUE_FLAG_POLL if all underlying queues enable QUEUE_FLAG_POLL, which is based on Jeffle's previous patch.

---
 drivers/md/dm-table.c |  24 ++++++++
 drivers/md/dm.c       | 131 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 152 insertions(+), 3 deletions(-)
drivers/md/dm-table.c +24 -0
index ee47a332b462..b14b379442d2 100644
@@ -1491,6 +1491,12 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector)
 	return &t->targets[(KEYS_PER_NODE * n) + k];
 }
 
+static int device_not_poll_capable(struct dm_target *ti, struct dm_dev *dev,
+				   sector_t start, sector_t len, void *data)
+{
+	return !blk_queue_poll(bdev_get_queue(dev->bdev));
+}
+
 /*
  * type->iterate_devices() should be called when the sanity check needs to
  * iterate and check all underlying data devices. iterate_devices() will
@@ -1541,6 +1547,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev,
 	return 0;
 }
 
+static int dm_table_supports_poll(struct dm_table *t)
+{
+	return !dm_table_any_dev_attr(t, device_not_poll_capable, NULL);
+}
+
 /*
  * Check whether a table has no data devices attached using each
  * target's iterate_devices method.
@@ -2078,6 +2089,19 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
 
 	dm_update_keyslot_manager(q, t);
 	blk_queue_update_readahead(q);
+
+	/*
+	 * Check for request-based device is remained to
+	 * dm_mq_init_request_queue()->blk_mq_init_allocated_queue().
+	 * For bio-based device, only set QUEUE_FLAG_POLL when all underlying
+	 * devices supporting polling.
+	 */
+	if (__table_type_bio_based(t->type)) {
+		if (dm_table_supports_poll(t))
+			blk_queue_flag_set(QUEUE_FLAG_POLL, q);
+		else
+			blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
+	}
 }
 
 unsigned int dm_table_get_num_targets(struct dm_table *t)
drivers/md/dm.c +128 -3

204 lines not on this page · raw message

Signature · 1 line
2.31.1
#4 lore raw
In reply to Ming Lei
Quoted 30 lines · Ming Lei

Hello Guys,

Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.

Patch 1 adds helper of blk_queue_poll().

Patch 2 adds .bio_poll() callback to block_device_operations, so bio driver can implement its own logic for io polling.

Patch 3 implements bio polling for device mapper.

V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFC

V2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add comment

Ming Lei (3):
block: add helper of blk_queue_poll
block: add ->poll_bio to block_device_operations
dm: support bio polling

Quoted 12 lines
 block/blk-core.c         |  18 +++---
 block/blk-sysfs.c        |   4 +-
 block/genhd.c            |   2 +
 drivers/md/dm-table.c    |  24 +++++++
 drivers/md/dm.c          | 131 ++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/core.c |   2 +-
 include/linux/blkdev.h   |   2 +
 7 files changed, 170 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1

Hey Ming,

I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!

Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?

thanks,
Mike

author
#5 lore raw
Quoted 48 lines · Mike Snitzer
Quoted 30 lines · Ming Lei

Hello Guys,

Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.

Patch 1 adds helper of blk_queue_poll().

Patch 2 adds .bio_poll() callback to block_device_operations, so bio driver can implement its own logic for io polling.

Patch 3 implements bio polling for device mapper.

V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFC

V2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add comment

Ming Lei (3):
block: add helper of blk_queue_poll
block: add ->poll_bio to block_device_operations
dm: support bio polling

Quoted 12 lines
 block/blk-core.c         |  18 +++---
 block/blk-sysfs.c        |   4 +-
 block/genhd.c            |   2 +
 drivers/md/dm-table.c    |  24 +++++++
 drivers/md/dm.c          | 131 ++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/core.c |   2 +-
 include/linux/blkdev.h   |   2 +
 7 files changed, 170 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1

Hey Ming,

Quoted 7 lines

I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!

Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?

Hi Mike,

Actually I have one local v5.17 rebase:

https://github.com/ming1/linux/tree/my_v5.17-dm-io-poll

Also one for-5.18/block rebase which is done just now:

https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll

In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.

Thanks,
Ming

#6 lore raw
Quoted 18 lines · Ming Lei
Quoted 10 lines · Mike Snitzer

Hey Ming,

I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!

Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?

Hi Mike,

Actually I have one local v5.17 rebase:

https://github.com/ming1/linux/tree/my_v5.17-dm-io-poll

Quoted 7 lines

Also one for-5.18/block rebase which is done just now:

https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll

In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.

OK, I've rebased it on dm-5.18.

Can you please share the exact test(s) you were running? I assume you were running directly against a request-based device and then comparing polling perf through dm-linear to the same underlying request-based device?

Thanks,
Mike

author
#7 lore raw
Quoted 29 lines · Mike Snitzer
Quoted 18 lines · Ming Lei
Quoted 10 lines · Mike Snitzer

Hey Ming,

I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!

Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?

Hi Mike,

Actually I have one local v5.17 rebase:

https://github.com/ming1/linux/tree/my_v5.17-dm-io-poll

Quoted 7 lines

Also one for-5.18/block rebase which is done just now:

https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll

In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.

Quoted 6 lines

OK, I've rebased it on dm-5.18.

Can you please share the exact test(s) you were running? I assume you were running directly against a request-based device and then comparing polling perf through dm-linear to the same underlying request-based device?

I run io_uring over dm-linear and dm-stripe, over two nvme disks with 2 poll_queues.

IOPS improvement can be observed, but not big.

Thanks,
Ming