dm: fix then improve bio splitting
Subsystems (2)
v3 cover dm: fix then improve bio splitting
Hi,
Patches 1 and 2 are queued for me to send to Linus later this week.
Patches 3 and 4 are block core and should get picked up for 5.10. Jens, please pick them up. I revised the header for patch 4 to give better context for use-case where non power-of-2 chunk_sectors occurs. Patch 4 enables DM to switch to using blk_max_size_offset() in Patch 6.
Patches 5 and 6 just show how DM will be enhanced for 5.10 once patches 3 and 4 land in the block tree.
Mike Snitzer (6):
dm: fix bio splitting and its bio completion order for regular IO
dm: fix comment in dm_process_bio()
block: use lcm_not_zero() when stacking chunk_sectors
block: allow 'chunk_sectors' to be non-power-of-2
dm table: stack 'chunk_sectors' limit to account for target-specific splitting
dm: change max_io_len() to use blk_max_size_offset()
block/blk-settings.c | 22 ++++++++++++---------- drivers/md/dm-table.c | 5 +++++ drivers/md/dm.c | 47 +++++++++++++---------------------------------- include/linux/blkdev.h | 12 +++++++++--- 4 files changed, 39 insertions(+), 47 deletions(-)
Signature · 1 line
2.15.0
PATCH v3 1/6 dm: fix bio splitting and its bio completion order for regular IO
dm_queue_split() is removed because __split_and_process_bio() _must_ handle splitting bios to ensure proper bio submission and completion ordering as a bio is split.
Otherwise, multiple recursive calls to ->submit_bio will cause multiple split bios to be allocated from the same ->bio_split mempool at the same time. This would result in deadlock in low memory conditions because no progress could be made (only one bio is available in ->bio_split mempool).
This fix has been verified to still fix the loss of performance, due to excess splitting, that commit 120c9257f5f1 provided.
--- drivers/md/dm.c | 23 ++--------------------- 1 file changed, 2 insertions(+), 21 deletions(-)
drivers/md/dm.c +2 -21
@@ -1724,23 +1724,6 @@ static blk_qc_t __process_bio(struct mapped_device *md, struct dm_table *map,return ret; } -static void dm_queue_split (struct mapped_device *md,struct dm_target *ti,struct bio **bio) -{ -unsigned len, sector_count; - - sector_count =bio_sectors (*bio); - len =min_t (sector_t,max_io_len ((*bio)->bi_iter.bi_sector, ti), sector_count); - -if (sector_count > len) { -struct bio *split =bio_split (*bio, len,GFP_NOIO , &md->queue->bio_split); - -bio_chain (split, *bio); -trace_block_split (md->queue, split, (*bio)->bi_iter.bi_sector); -submit_bio_noacct (*bio); - *bio = split; - } -} -static blk_qc_t dm_process_bio (struct mapped_device *md,struct dm_table *map,struct bio *bio) {
@@ -1768,14 +1751,12 @@ static blk_qc_t dm_process_bio(struct mapped_device *md,if (current->bio_list) {if (is_abnormal_io (bio))blk_queue_split (&bio); -else -dm_queue_split (md, ti, &bio); +/* regular IO is split by __split_and_process_bio */ }if (dm_get_md_type (md) ==DM_TYPE_NVME_BIO_BASED )return __process_bio (md, map, bio, ti); -else -return __split_and_process_bio (md, map, bio); +return __split_and_process_bio (md, map, bio); }static blk_qc_t dm_submit_bio (struct bio *bio)
Signature · 1 line
2.15.0
PATCH v3 2/6 dm: fix comment in dm_process_bio()
In reply to Mike SnitzerRefer to the correct function (->submit_bio instead of ->queue_bio). Also, add details about why using blk_queue_split() isn't needed for dm_wq_work()'s call to dm_process_bio().
--- drivers/md/dm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/md/dm.c +3 -1
@@ -1744,9 +1744,11 @@ static blk_qc_t dm_process_bio(struct mapped_device *md, }/* -* If in ->queue_bio we need to use blk_queue_split(), otherwise +* If in ->submit_bio we need to use blk_queue_split(), otherwise * queue_limits for abnormal requests (e.g. discard, writesame, etc) * won't be imposed. +* If called from dm_wq_work() for deferred bio processing, bio +* was already handled by following code with previous ->submit_bio. */ if (current->bio_list) {if (is_abnormal_io (bio))
Signature · 1 line
2.15.0
PATCH v3 3/6 block: use lcm_not_zero() when stacking chunk_sectors
In reply to Mike SnitzerLike 'io_opt', blk_stack_limits() should stack 'chunk_sectors' using lcm_not_zero() rather than min_not_zero() -- otherwise the final 'chunk_sectors' could result in sub-optimal alignment of IO to component devices in the IO stack.
Also, if 'chunk_sectors' isn't a multiple of 'physical_block_size' then it is a bug in the driver and the device should be flagged as 'misaligned'.
--- block/blk-settings.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
block/blk-settings.c +8 -4
@@ -534,6 +534,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->io_min =max (t->io_min, b->io_min); t->io_opt =lcm_not_zero (t->io_opt, b->io_opt); + t->chunk_sectors =lcm_not_zero (t->chunk_sectors, b->chunk_sectors);/* Physical block size a multiple of the logical block size? */ if (t->physical_block_size & (t->logical_block_size -1 )) {
@@ -556,6 +557,13 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, ret =-1 ; } +/* chunk_sectors a multiple of the physical block size? */ +if ((t->chunk_sectors <<9 ) & (t->physical_block_size -1 )) { + t->chunk_sectors =0 ; + t->misaligned =1 ; + ret =-1 ; + } + t->raid_partial_stripes_expensive =max (t->raid_partial_stripes_expensive, b->raid_partial_stripes_expensive);
@@ -594,10 +602,6 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->discard_granularity; } -if (b->chunk_sectors) - t->chunk_sectors =min_not_zero (t->chunk_sectors, - b->chunk_sectors); - t->zoned =max (t->zoned, b->zoned);return ret; }
Signature · 1 line
2.15.0
Re: [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors
Mike,
Quoted 8 lines
Like '
io_opt',blk_stack_limits()should stack 'chunk_sectors' usinglcm_not_zero()rather thanmin_not_zero()-- otherwise the final 'chunk_sectors' could result in sub-optimal alignment of IO to component devices in the IO stack.Also, if '
chunk_sectors' isn't a multiple of 'physical_block_size' then it is a bug in the driver and the device should be flagged as 'misaligned'.
Looks good.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
PATCH v3 4/6 block: allow 'chunk_sectors' to be non-power-of-2
In reply to Mike SnitzerIt is possible, albeit more unlikely, for a block device to have a non power-of-2 for chunk_sectors (e.g. 10+2 RAID6 with 128K chunk_sectors, which results in a full-stripe size of 1280K. This causes the RAID6's io_opt to be advertised as 1280K, and a stacked device _could_ then be made to use a blocksize, aka chunk_sectors, that matches non power-of-2 io_opt of underlying RAID6 -- resulting in stacked device's chunk_sectors being a non power-of-2).
Update blk_queue_chunk_sectors() and blk_max_size_offset() to accommodate drivers that need a non power-of-2 chunk_sectors.
--- block/blk-settings.c | 10 ++++------ include/linux/blkdev.h | 12 +++++++++--- 2 files changed, 13 insertions(+), 9 deletions(-)
block/blk-settings.c +4 -6
@@ -172,15 +172,13 @@ EXPORT_SYMBOL(blk_queue_max_hw_sectors);* * Description: * If a driver doesn't want IOs to cross a given chunk size, it can set -* this limit and prevent merging across chunks. Note that the chunk size -* must currently be a power-of-2 in sectors. Also note that the block -* layer must accept a page worth of data at any offset. So if the -* crossing of chunks is a hard limitation in the driver, it must still be -* prepared to split single page bios. +* this limit and prevent merging across chunks. Note that the block layer +* must accept a page worth of data at any offset. So if the crossing of +* chunks is a hard limitation in the driver, it must still be prepared +* to split single page bios. **/ void blk_queue_chunk_sectors (struct request_queue *q,unsigned int chunk_sectors) { -BUG_ON (!is_power_of_2 (chunk_sectors)); q->limits.chunk_sectors = chunk_sectors; }EXPORT_SYMBOL (blk_queue_chunk_sectors);
include/linux/blkdev.h +9 -3
@@ -1059,11 +1059,17 @@ static inline unsigned int blk_queue_get_max_sectors(struct request_queue *q,static inline unsigned int blk_max_size_offset (struct request_queue *q,sector_t offset) { -if (!q->limits.chunk_sectors) +unsigned int chunk_sectors = q->limits.chunk_sectors; + +if (!chunk_sectors)return q->limits.max_sectors; -return min (q->limits.max_sectors, (unsigned int )(q->limits.chunk_sectors - - (offset & (q->limits.chunk_sectors -1 )))); +if (likely (is_power_of_2 (chunk_sectors))) + chunk_sectors -= offset & (chunk_sectors -1 ); +else + chunk_sectors -=sector_div (offset, chunk_sectors); + +return min (q->limits.max_sectors, chunk_sectors); } static inline unsigned int blk_rq_get_max_sectors(struct request *rq,
Signature · 1 line
2.15.0
Re: [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2
Mike,
Quoted 10 lines
It is possible, albeit more unlikely, for a block device to have a non power-of-2 for
chunk_sectors(e.g. 10+2 RAID6 with 128Kchunk_sectors, which results in a full-stripe size of 1280K. This causes the RAID6'sio_optto be advertised as 1280K, and a stacked device _could_ then be made to use a blocksize, akachunk_sectors, that matches non power-of-2io_optof underlying RAID6 -- resulting in stacked device'schunk_sectorsbeing a non power-of-2).Update
blk_queue_chunk_sectors()andblk_max_size_offset()to accommodate drivers that need a non power-of-2chunk_sectors.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
PATCH v3 5/6 dm table: stack 'chunk_sectors' limit to account for target-specific splitting
In reply to Mike SnitzerIf target set ti->max_io_len it must be used when stacking DM device's queue_limits to establish a 'chunk_sectors' that is compatible with the IO stack.
By using lcm_not_zero() care is taken to avoid blindly overriding the chunk_sectors limit stacked up by blk_stack_limits().
--- drivers/md/dm-table.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/md/dm-table.c +5 -0
@@ -18,6 +18,7 @@#include <linux/mutex.h> #include <linux/delay.h> #include <linux/atomic.h> +#include <linux/lcm.h> #include <linux/blk-mq.h> #include <linux/mount.h> #include <linux/dax.h>
@@ -1506,6 +1507,10 @@ int dm_calculate_queue_limits(struct dm_table *table, zone_sectors = ti_limits.chunk_sectors; } +/* Stack chunk_sectors if target-specific splitting is required */ +if (ti->max_io_len) + ti_limits.chunk_sectors =lcm_not_zero (ti->max_io_len, + ti_limits.chunk_sectors);/* Set I/O hints portion of queue limits */ if (ti->type->io_hints) ti->type->io_hints (ti, &ti_limits);
Signature · 1 line
2.15.0
PATCH v3 6/6 dm: change max_io_len() to use blk_max_size_offset()
In reply to Mike SnitzerUsing blk_max_size_offset() enables DM core's splitting to impose ti->max_io_len (via q->limits.chunk_sectors) and also fallback to respecting q->limits.max_sectors if chunk_sectors isn't set.
--- drivers/md/dm.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
drivers/md/dm.c +8 -12
@@ -1051,22 +1051,18 @@ static sector_t max_io_len_target_boundary(sector_t sector, struct dm_target *tistatic sector_t max_io_len (sector_t sector,struct dm_target *ti) {sector_t len =max_io_len_target_boundary (sector, ti); -sector_t offset, max_len; +sector_t max_len;/* * Does the target need to split even further? +* - q->limits.chunk_sectors reflects ti->max_io_len so +* blk_max_size_offset() provides required splitting. +* - blk_max_size_offset() also respects q->limits.max_sectors */ -if (ti->max_io_len) { - offset =dm_target_offset (ti, sector); -if (unlikely (ti->max_io_len & (ti->max_io_len -1 ))) - max_len =sector_div (offset, ti->max_io_len); -else - max_len = offset & (ti->max_io_len -1 ); - max_len = ti->max_io_len - max_len; - -if (len > max_len) - len = max_len; - } + max_len =blk_max_size_offset (dm_table_get_md (ti->table)->queue, +dm_target_offset (ti, sector)); +if (len > max_len) + len = max_len;return len; }
Signature · 1 line
2.15.0
Jens Axboe
maintainerQuoted 4 lines · Mike Snitzer
Hi,
Patches 1 and 2 are queued for me to send to Linus later this week.
Quoted 8 lines
Patches 3 and 4 are block core and should get picked up for 5.10. Jens, please pick them up. I revised the header for patch 4 to give better context for use-case where non power-of-2
chunk_sectorsoccurs. Patch 4 enables DM to switch to usingblk_max_size_offset()in Patch 6.Patches 5 and 6 just show how DM will be enhanced for 5.10 once patches 3 and 4 land in the block tree.
Applied 3-4 for 5.10, thanks.
Signature · 1 line
Jens Axboe