block/dm: support bio polling
Subsystems (3)
v3 cover block/dm: support bio polling
Hello Guys,
Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.
Patch 1 adds helper of blk_queue_poll().
Patch 2 adds .bio_poll() callback to block_device_operations, so bio driver can implement its own logic for io polling.
Patch 3 implements bio polling for device mapper.
V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFC
V2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add comment
Ming Lei (3):
block: add helper of blk_queue_poll
block: add ->poll_bio to block_device_operations
dm: support bio polling
block/blk-core.c | 18 +++--- block/blk-sysfs.c | 4 +- block/genhd.c | 2 + drivers/md/dm-table.c | 24 +++++++ drivers/md/dm.c | 131 ++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 2 + 7 files changed, 170 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1
PATCH v3 1/3 block: add helper of blk_queue_poll
There has been 3 users, and will be more, so add one such helper.
--- block/blk-core.c | 5 ++--- block/blk-sysfs.c | 4 ++-- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 1 + 4 files changed, 6 insertions(+), 6 deletions(-)
block/blk-core.c +2 -3
@@ -835,7 +835,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } -if (!test_bit (QUEUE_FLAG_POLL , &q->queue_flags)) +if (!blk_queue_poll (q)) bio->bi_opf &= ~REQ_POLLED ;switch (bio_op (bio)) {
@@ -1117,8 +1117,7 @@ int bio_poll(struct bio *bio, unsigned int flags)blk_qc_t cookie =READ_ONCE (bio->bi_cookie);int ret; -if (cookie ==BLK_QC_T_NONE || - !test_bit (QUEUE_FLAG_POLL , &q->queue_flags)) +if (cookie ==BLK_QC_T_NONE || !blk_queue_poll (q))return 0 ;if (current->plug)
block/blk-sysfs.c +2 -2
@@ -422,13 +422,13 @@ static ssize_t queue_poll_delay_store(struct request_queue *q, const char *page,static ssize_t queue_poll_show (struct request_queue *q,char *page) { -return queue_var_show (test_bit (QUEUE_FLAG_POLL , &q->queue_flags), page); +return queue_var_show (blk_queue_poll (q), page); }static ssize_t queue_poll_store (struct request_queue *q,const char *page,size_t count) { -if (!test_bit (QUEUE_FLAG_POLL , &q->queue_flags)) +if (!blk_queue_poll (q))return -EINVAL ;pr_info_ratelimited ("writes to the poll attribute are ignored.\n" );pr_info_ratelimited ("please use driver specific parameters instead.\n" );
drivers/nvme/host/core.c +1 -1
@@ -1025,7 +1025,7 @@ static void nvme_execute_rq_polled(struct request_queue *q, {DECLARE_COMPLETION_ONSTACK (wait); -WARN_ON_ONCE (!test_bit (QUEUE_FLAG_POLL , &q->queue_flags)); +WARN_ON_ONCE (!blk_queue_poll (q)); rq->cmd_flags |=REQ_POLLED ; rq->end_io_data = &wait;
include/linux/blkdev.h +1 -0
@@ -677,6 +677,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);#define blk_queue_fua (q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags)#define blk_queue_registered (q) test_bit(QUEUE_FLAG_REGISTERED, &(q)->queue_flags)#define blk_queue_nowait (q) test_bit(QUEUE_FLAG_NOWAIT, &(q)->queue_flags) +#define blk_queue_poll (q) test_bit(QUEUE_FLAG_POLL, &(q)->queue_flags)extern void blk_set_pm_only (struct request_queue *q);extern void blk_clear_pm_only (struct request_queue *q);
Signature · 1 line
2.31.1
PATCH v3 2/3 block: add ->poll_bio to block_device_operations
In reply to Ming LeiPrepare for supporting IO polling for bio based driver.
Add ->poll_bio callback so that bio driver can provide their own logic for polling bio.
--- block/blk-core.c | 13 +++++++++---- block/genhd.c | 2 ++ include/linux/blkdev.h | 1 + 3 files changed, 12 insertions(+), 4 deletions(-)
block/blk-core.c +9 -4
@@ -1113,7 +1113,8 @@ EXPORT_SYMBOL(submit_bio);*/ int bio_poll (struct bio *bio,unsigned int flags) { -struct request_queue *q = bio->bi_bdev->bd_disk->queue; +struct gendisk *disk = bio->bi_bdev->bd_disk; +struct request_queue *q = disk->queue;blk_qc_t cookie =READ_ONCE (bio->bi_cookie);int ret;
@@ -1125,10 +1126,14 @@ int bio_poll(struct bio *bio, unsigned int flags)if (blk_queue_enter (q,BLK_MQ_REQ_NOWAIT ))return 0 ; -if (WARN_ON_ONCE (!queue_is_mq (q))) - ret =0 ;/* not yet implemented, should not happen */ -else + +if (queue_is_mq (q)) ret =blk_mq_poll (q, cookie, flags); +else if (disk->fops->poll_bio) + ret = disk->fops->poll_bio (bio, flags); +else + ret = !WARN_ON_ONCE (1 ); +blk_queue_exit (q);return ret; }
block/genhd.c +2 -0
@@ -471,6 +471,8 @@ static void __device_add_disk(struct device *parent, struct gendisk *disk, {int ret; +WARN_ON_ONCE (queue_is_mq (disk->queue) && disk->fops->poll_bio); +/* * The disk queue should now be all set with enough information about * the device for the elevator code to pick an adequate default
include/linux/blkdev.h +1 -0
@@ -1858,6 +1858,7 @@ static inline void blk_ksm_unregister(struct request_queue *q) { }struct block_device_operations {void (*submit_bio)(struct bio *bio); +int (*poll_bio)(struct bio *bio,unsigned int flags);int (*open) (struct block_device *,fmode_t );void (*release) (struct gendisk *,fmode_t );int (*rw_page)(struct block_device *,sector_t ,struct page *,unsigned int );
Signature · 1 line
2.31.1
PATCH v3 3/3 dm: support bio polling
In reply to Ming LeiSupport bio(REQ_POLLED) polling in the following approach:
1) only support io polling on normal READ/WRITE, and other abnormal IOs still fallback on IRQ mode, so the target io is exactly inside the dm io.
2) hold one refcnt on io->io_count after submitting this dm bio with REQ_POLLED
3) support dm native bio splitting, any dm io instance associated with current bio will be added into one list which head is bio->bi_end_io which will be recovered before ending this bio
4) implement .poll_bio() callback, call bio_poll() on the single target bio inside the dm io which is retrieved via bio->bi_bio_drv_data; call dec_pending() after the target io is done in .poll_bio()
4) enable QUEUE_FLAG_POLL if all underlying queues enable QUEUE_FLAG_POLL, which is based on Jeffle's previous patch.
--- drivers/md/dm-table.c | 24 ++++++++ drivers/md/dm.c | 131 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 152 insertions(+), 3 deletions(-)
drivers/md/dm-table.c +24 -0
@@ -1491,6 +1491,12 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector)return &t->targets[(KEYS_PER_NODE * n) + k]; } +static int device_not_poll_capable (struct dm_target *ti,struct dm_dev *dev, +sector_t start,sector_t len,void *data) +{ +return !blk_queue_poll (bdev_get_queue (dev->bdev)); +} +/* * type->iterate_devices() should be called when the sanity check needs to * iterate and check all underlying data devices. iterate_devices() will
@@ -1541,6 +1547,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev,return 0 ; } +static int dm_table_supports_poll (struct dm_table *t) +{ +return !dm_table_any_dev_attr (t, device_not_poll_capable,NULL ); +} +/* * Check whether a table has no data devices attached using each * target's iterate_devices method.
@@ -2078,6 +2089,19 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,dm_update_keyslot_manager (q, t);blk_queue_update_readahead (q); + +/* +* Check for request-based device is remained to +* dm_mq_init_request_queue()->blk_mq_init_allocated_queue(). +* For bio-based device, only set QUEUE_FLAG_POLL when all underlying +* devices supporting polling. +*/ +if (__table_type_bio_based (t->type)) { +if (dm_table_supports_poll (t)) +blk_queue_flag_set (QUEUE_FLAG_POLL , q); +else +blk_queue_flag_clear (QUEUE_FLAG_POLL , q); + } }unsigned int dm_table_get_num_targets (struct dm_table *t)
drivers/md/dm.c +128 -3
204 lines not on this page · raw message
Signature · 1 line
2.31.1
Quoted 30 lines · Ming Lei
Hello Guys,
Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.
Patch 1 adds helper of
blk_queue_poll().Patch 2 adds .bio_poll() callback to
block_device_operations, so bio driver can implement its own logic for io polling.Patch 3 implements bio polling for device mapper.
V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFCV2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add commentMing Lei (3):
block: add helper ofblk_queue_poll
block: add ->poll_bio toblock_device_operations
dm: support bio polling
Quoted 12 lines
block/blk-core.c | 18 +++--- block/blk-sysfs.c | 4 +- block/genhd.c | 2 + drivers/md/dm-table.c | 24 +++++++ drivers/md/dm.c | 131 ++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 2 + 7 files changed, 170 insertions(+), 13 deletions(-)Signature · 1 line
2.31.1
Hey Ming,
I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!
Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?
thanks,
Mike
Quoted 48 lines · Mike Snitzer
Quoted 30 lines · Ming Lei
Hello Guys,
Based on Christoph's bio based polling model[1], implement DM bio polling with one very simple approach.
Patch 1 adds helper of
blk_queue_poll().Patch 2 adds .bio_poll() callback to
block_device_operations, so bio driver can implement its own logic for io polling.Patch 3 implements bio polling for device mapper.
V3:
- patch style change as suggested by Christoph(2/3)
- fix kernel panic issue caused by nested dm polling, which is found & figured out by Jeffle Xu (3/3)
- re-organize setup polling code (3/3)
- remove RFCV2:
- drop patch to add new fields into bio
- support io polling for dm native bio splitting
- add commentMing Lei (3):
block: add helper ofblk_queue_poll
block: add ->poll_bio toblock_device_operations
dm: support bio pollingQuoted 12 lines
block/blk-core.c | 18 +++--- block/blk-sysfs.c | 4 +- block/genhd.c | 2 + drivers/md/dm-table.c | 24 +++++++ drivers/md/dm.c | 131 ++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 2 + 7 files changed, 170 insertions(+), 13 deletions(-)Signature · 1 line
2.31.1Hey Ming,
Quoted 7 lines
I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!
Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?
Hi Mike,
Actually I have one local v5.17 rebase:
https://github.com/ming1/linux/tree/my_v5.17-dm-io-poll
Also one for-5.18/block rebase which is done just now:
https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll
In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.
Thanks,
Ming
Quoted 18 lines · Ming Lei
Quoted 10 lines · Mike Snitzer
Hey Ming,
I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!
Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?
Hi Mike,
Actually I have one local v5.17 rebase:
Quoted 7 lines
Also one for-5.18/block rebase which is done just now:
https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll
In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.
OK, I've rebased it on dm-5.18.
Can you please share the exact test(s) you were running? I assume you were running directly against a request-based device and then comparing polling perf through dm-linear to the same underlying request-based device?
Thanks,
Mike
Quoted 29 lines · Mike Snitzer
Quoted 18 lines · Ming Lei
Quoted 10 lines · Mike Snitzer
Hey Ming,
I'd like us to follow-through with adding bio-based polling support. Kind of strange none of us that were sent this V3 ever responded, sorry about that!
Do you have interest in rebasing this patchset (against linux-dm.git's "dm-5.18" branch since there has been quite some churn)? Or are you OK with me doing the rebase?
Hi Mike,
Actually I have one local v5.17 rebase:
Quoted 7 lines
Also one for-5.18/block rebase which is done just now:
https://github.com/ming1/linux/tree/my_v5.18-dm-bio-poll
In my previous test on v5.17 rebase, the IOPS improvement is a bit small, so I didn't post it out. Recently not get time to investigate the performance further, so please feel free to work on it.
Quoted 6 lines
OK, I've rebased it on dm-5.18.
Can you please share the exact test(s) you were running? I assume you were running directly against a request-based device and then comparing polling perf through dm-linear to the same underlying request-based device?
I run io_uring over dm-linear and dm-stripe, over two nvme disks with 2 poll_queues.
IOPS improvement can be observed, but not big.
Thanks,
Ming