block/dm: support bio polling
Subsystems (2)
v5 cover block/dm: support bio polling
Hi,
I've rebased Ming's latest [1] ontop of dm-5.18 [2] (which is based on for-5.18/block). End result available in dm-5.18-biopoll branch [3]
These changes add bio polling support to DM. Tested with linear and striped DM targets.
IOPS improvement was ~5% on my baremetal system with a single Intel Optane NVMe device (555K hipri=1 vs 525K hipri=0).
Ming has seen better improvement while testing within a VM:
dm-linear: hipri=1 vs hipri=0 15~20% iops improvement
dm-stripe: hipri=1 vs hipri=0 ~30% iops improvement
I'd like to merge these changes via the DM tree when the 5.18 merge window opens. The first block patch that adds ->poll_bio to block_device_operations will need review so that I can take it through the DM tree. Reason for going through the DM tree is there have been some fairly extensive changes queued in dm-5.18 that build on for-5.18/block. So I think it easiest to just add the block depenency via DM tree since DM is first consumer of ->poll_bio
FYI, Ming does have another DM patch [4] that looks to avoid using hlist but I only just saw it. bio_split() _is_ involved (see dm_split_and_process_bio) so I'm not exactly sure where he is going with that change. But that is DM-implementation detail that we'll sort out. Big thing is we need approval for the first block patch to go to Linus via the DM tree ;)
Thanks,
Mike
[1] https://github.com/ming1/linux/commits/my_v5.18-dm-bio-poll
[2] https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.18
[3] https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.18-biopoll
[4] https://github.com/ming1/linux/commit/c107c30e15041ac1ce672f56809961406e2a3e52
v5: remove WARN_ONs in ->poll_bio interface patch. Fixed comment typo
along the way (found while seeing how other block_device_operations are referenced in block's code comments).
Ming Lei (2):
block: add ->poll_bio to block_device_operations
dm: support bio polling
block/blk-core.c | 14 +++-- block/genhd.c | 4 ++ drivers/md/dm-core.h | 2 + drivers/md/dm-table.c | 27 +++++++++ drivers/md/dm.c | 150 ++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/blkdev.h | 2 + 6 files changed, 191 insertions(+), 8 deletions(-)
Signature · 1 line
2.15.0
PATCH v5 1/2 block: add ->poll_bio to block_device_operations
From: Ming Lei <[email protected]>
Prepare for supporting IO polling for bio-based driver.
Add ->poll_bio callback so that bio-based driver can provide their own logic for polling bio.
Also fix ->submit_bio_bio typo in comment block above __submit_bio_noacct.
--- block/blk-core.c | 14 +++++++++----- block/genhd.c | 4 ++++ include/linux/blkdev.h | 2 ++ 3 files changed, 15 insertions(+), 5 deletions(-)
block/blk-core.c +9 -5
@@ -708,7 +708,7 @@ static void __submit_bio(struct bio *bio)* * bio_list_on_stack[0] contains bios submitted by the current ->submit_bio. * bio_list_on_stack[1] contains bios that were submitted before the current -* ->submit_bio_bio, but that haven't been processed yet. +* ->submit_bio, but that haven't been processed yet. */ static void __submit_bio_noacct (struct bio *bio) {
@@ -975,7 +975,7 @@ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags) {struct request_queue *q =bdev_get_queue (bio->bi_bdev);blk_qc_t cookie =READ_ONCE (bio->bi_cookie); -int ret; +int ret =0 ;if (cookie ==BLK_QC_T_NONE || !test_bit (QUEUE_FLAG_POLL , &q->queue_flags))
@@ -985,10 +985,14 @@ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags)if (blk_queue_enter (q,BLK_MQ_REQ_NOWAIT ))return 0 ; -if (WARN_ON_ONCE (!queue_is_mq (q))) - ret =0 ;/* not yet implemented, should not happen */ -else +if (queue_is_mq (q)) { ret =blk_mq_poll (q, cookie, iob, flags); + }else { +struct gendisk *disk = q->disk; + +if (disk && disk->fops->poll_bio) + ret = disk->fops->poll_bio (bio, iob, flags); + }blk_queue_exit (q);return ret; }
block/genhd.c +4 -0
@@ -410,6 +410,10 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk,struct device *ddev =disk_to_dev (disk);int ret; +/* Only makes sense for bio-based to set ->poll_bio */ +if (queue_is_mq (disk->queue) && disk->fops->poll_bio) +return -EINVAL ; +/* * The disk queue should now be all set with enough information about * the device for the elevator code to pick an adequate default
include/linux/blkdev.h +2 -0
@@ -1455,6 +1455,8 @@ enum blk_unique_id {struct block_device_operations {void (*submit_bio)(struct bio *bio); +int (*poll_bio)(struct bio *bio,struct io_comp_batch *iob, +unsigned int flags);int (*open) (struct block_device *,fmode_t );void (*release) (struct gendisk *,fmode_t );int (*rw_page)(struct block_device *,sector_t ,struct page *,unsigned int );
Signature · 1 line
2.15.0
Re: [PATCH v5 1/2] block: add ->poll_bio to block_device_operations
Looks good,
PATCH v5 2/2 dm: support bio polling
In reply to Mike SnitzerFrom: Ming Lei <[email protected]>
Support bio(REQ_POLLED) polling in the following approach:
1) only support io polling on normal READ/WRITE, and other abnormal IOs still fallback to IRQ mode, so the target io is exactly inside the dm io.
2) hold one refcnt on io->io_count after submitting this dm bio with REQ_POLLED
3) support dm native bio splitting, any dm io instance associated with current bio will be added into one list which head is bio->bi_end_io which will be recovered before ending this bio
4) implement .poll_bio() callback, call bio_poll() on the single target bio inside the dm io which is retrieved via bio->bi_bio_drv_data; call dm_io_dec_pending() after the target io is done in .poll_bio()
5) enable QUEUE_FLAG_POLL if all underlying queues enable QUEUE_FLAG_POLL, which is based on Jeffle's previous patch.
--- drivers/md/dm-core.h | 2 + drivers/md/dm-table.c | 27 +++++++++ drivers/md/dm.c | 150 +++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 176 insertions(+), 3 deletions(-)
drivers/md/dm-core.h +2 -0
@@ -235,6 +235,8 @@ struct dm_io { bool start_io_acct:1;int was_accounted;unsigned long start_time; +void *saved_bio_end_io; +struct hlist_node node;spinlock_t endio_lock;struct dm_stats_aux stats_aux;/* last member of dm_target_io is 'struct bio' */
drivers/md/dm-table.c +27 -0
@@ -1481,6 +1481,14 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector)return &t->targets[(KEYS_PER_NODE * n) + k]; } +static int device_not_poll_capable (struct dm_target *ti,struct dm_dev *dev, +sector_t start,sector_t len,void *data) +{ +struct request_queue *q =bdev_get_queue (dev->bdev); + +return !test_bit (QUEUE_FLAG_POLL , &q->queue_flags); +} +/* * type->iterate_devices() should be called when the sanity check needs to * iterate and check all underlying data devices. iterate_devices() will
@@ -1531,6 +1539,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev,return 0 ; } +static int dm_table_supports_poll (struct dm_table *t) +{ +return !dm_table_any_dev_attr (t, device_not_poll_capable,NULL ); +} +/* * Check whether a table has no data devices attached using each * target's iterate_devices method.
@@ -2067,6 +2080,20 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,dm_update_crypto_profile (q, t);disk_update_readahead (t->md->disk); +/* +* Check for request-based device is left to +* dm_mq_init_request_queue()->blk_mq_init_allocated_queue(). +* +* For bio-based device, only set QUEUE_FLAG_POLL when all +* underlying devices supporting polling. +*/ +if (__table_type_bio_based (t->type)) { +if (dm_table_supports_poll (t)) +blk_queue_flag_set (QUEUE_FLAG_POLL , q); +else +blk_queue_flag_clear (QUEUE_FLAG_POLL , q); + } +return 0 ; }
drivers/md/dm.c +147 -3
211 lines not on this page · raw message
Signature · 1 line
2.15.0
Re: [PATCH v5 2/2] dm: support bio polling
drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471
Quoted 11 lines
drivers/md/dm.c +11
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); + +return (struct hlist_head *)&bio->bi_end_io; +}
So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the end_io handler.
Jens Axboe
maintainerRe: [PATCH v5 2/2] dm: support bio polling
Quoted 9 lines · Christoph Hellwig
Quoted 9 lines
drivers/md/dm.c +9
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); +
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return (struct hlist_head *)&bio->bi_end_io; +}So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the
end_iohandler.
If it's possible, then that would be preferable. But I don't think that's going to be easy to do...
Signature · 1 line
Jens Axboe
Re: [PATCH v5 2/2] dm: support bio polling
Quoted 19 lines · Jens Axboe
Quoted 9 lines · Christoph Hellwig
Quoted 9 lines
drivers/md/dm.c +9
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); +Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return (struct hlist_head *)&bio->bi_end_io; +}So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the
end_iohandler.
Quoted 2 lines
If it's possible, then that would be preferable. But I don't think that's going to be easy to do...
I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...
Thanks,
Ming
Jens Axboe
maintainerRe: [PATCH v5 2/2] dm: support bio polling
Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig
drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471
Quoted 11 lines
drivers/md/dm.c +11
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); + +return (struct hlist_head *)&bio->bi_end_io; +}So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the
end_iohandler.
Quoted 5 lines
Quoted 2 lines
If it's possible, then that would be preferable. But I don't think that's going to be easy to do...
I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...
There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.
Maybe some commenting would help? Is bi_next available at this point?
Signature · 1 line
Jens Axboe
Re: [PATCH v5 2/2] dm: support bio polling
Quoted 27 lines · Jens Axboe
Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig
drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471
Quoted 11 lines
drivers/md/dm.c +11
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); + +return (struct hlist_head *)&bio->bi_end_io; +}So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the
end_iohandler.Quoted 5 lines
Quoted 2 lines
If it's possible, then that would be preferable. But I don't think that's going to be easy to do...
I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...
Quoted 6 lines
There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.
Maybe some commenting would help?
OK, ->bi_end_io is safe because it is only called until the bio is ended, so we can retrieve the list head and recover ->bi_end_io before polling.
Quoted 1 line
Is
bi_nextavailable at this point?
The same bio can be re-submitted to block layer because of splitting, and will be linked to current->bio_list[].
BTW, bio splitting can be very often for some dm target, that is why we don't ignore bio splitting for dm polling.
Thanks,
Ming
Re: [PATCH v5 2/2] dm: support bio polling
Quoted 33 lines · Ming Lei
Quoted 32 lines · Jens Axboe
Quoted 20 lines · Ming Lei
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Christoph Hellwig
drivers/md/dm.c static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, · line 1471
Quoted 11 lines
drivers/md/dm.c +11
-0+/* +* Reuse ->bi_end_io as hlist head for storing all dm_io instances +* associated with this bio, and this bio's bi_end_io has to be +* stored in one of 'dm_io' instance first. +*/ +static inline struct hlist_head *dm_get_bio_hlist_head (struct bio *bio) +{ +WARN_ON_ONCE (!(bio->bi_opf &REQ_DM_POLL_LIST )); + +return (struct hlist_head *)&bio->bi_end_io; +}So this reuse is what I really hated. I still think we should be able to find space in the bio by creatively shifting fields around to just add the hlist there directly, which would remove the need for this override and more importantly the quite cumbersome saving and restoring of the
end_iohandler.Quoted 5 lines
Quoted 2 lines
If it's possible, then that would be preferable. But I don't think that's going to be easy to do...
I agree, now basically there isn't gap inside bio, so either adding one new field or reusing one existed field...
There'd no amount of re-arranging that'll free up 8 bytes, that's just not happening. I'm not a huge fan of growing struct bio for that, and the oddity here is mostly (to me) that ->bi_end_io is the one overlayed. That would usually belong to the owner of the bio.
Quoted 5 lines
Quoted 1 line
Maybe some commenting would help?
OK, ->bi_end_io is safe because it is only called until the bio is ended, so we can retrieve the list head and recover ->bi_end_io before polling.
->bi_private can be reused too, is that better?
Yeah, both belong to owner(higher level storage), then block layer can't touch them inside submit_bio_noacct(), that is also why this trick is safe.
Thanks,
Ming