dm: fix various issues with bio splitting code
Subsystems (2)
cover dm: fix various issues with bio splitting code
Hi,
These mostly stable@ patches will be sent to Linus for 5.0-rc4 inclusion (would've sent today for 5.0-rc3 but I don't yet have linux-next coverage).
Just wanted to give a heads up because some of the problems fixed could bite other users of the bio_split()+generic_make_request() recursive pattern. We should probably factor out some helpers that all users could share (e.g. so they don't forget to set BIO_QUEUE_ENTERED or call trace_block_split(), etc).
Thanks,
Mike
Mike Snitzer (4):
dm: fix clone_bio() to trigger blk_recount_segments()
dm: fix redundant IO accounting for bios that need splitting
dm: fix missing bio_split() pattern code in __split_and_process_bio()
dm: fix dm_wq_work() to only use __split_and_process_bio() if appropriate
drivers/md/dm.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)
Signature · 1 line
2.15.0
PATCH 1/4 dm: fix clone_bio() to trigger blk_recount_segments()
DM's clone_bio() now benefits from using bio_trim() by fixing the fact that clone_bio() wasn't clearing BIO_SEG_VALID like bio_trim() does; which triggers blk_recount_segments() via bio_phys_segments().
--- drivers/md/dm.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
drivers/md/dm.c +2 -6
@@ -1320,7 +1320,7 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,__bio_clone_fast (clone, bio); -if (unlikely (bio_integrity (bio) !=NULL )) { +if (bio_integrity (bio)) {int r; if (unlikely(!dm_target_has_integrity(tio->ti->type) &&
@@ -1336,11 +1336,7 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,return r; } -bio_advance (clone,to_bytes (sector - clone->bi_iter.bi_sector)); - clone->bi_iter.bi_size =to_bytes (len); - -if (unlikely (bio_integrity (bio) !=NULL )) -bio_integrity_trim (clone); +bio_trim (clone, sector - clone->bi_iter.bi_sector, len);return 0 ; }
Signature · 1 line
2.15.0
Re: [PATCH 1/4] dm: fix clone_bio() to trigger blk_recount_segments()
Quoted 38 lines · Mike Snitzer
DM's
clone_bio()now benefits from usingbio_trim()by fixing the fact thatclone_bio()wasn't clearingBIO_SEG_VALIDlikebio_trim()does; which triggersblk_recount_segments()viabio_phys_segments().Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)drivers/md/dm.c +2
-6@@ -1320,7 +1320,7 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,__bio_clone_fast (clone, bio); -if (unlikely (bio_integrity (bio) !=NULL )) { +if (bio_integrity (bio)) {int r; if (unlikely(!dm_target_has_integrity(tio->ti->type) &&@@ -1336,11 +1336,7 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,return r; } -bio_advance (clone,to_bytes (sector - clone->bi_iter.bi_sector)); - clone->bi_iter.bi_size =to_bytes (len); - -if (unlikely (bio_integrity (bio) !=NULL )) -bio_integrity_trim (clone); +bio_trim (clone, sector - clone->bi_iter.bi_sector, len);return 0 ; }Signature · 1 line
2.15.0
Thanks,
Ming
PATCH 2/4 dm: fix redundant IO accounting for bios that need splitting
In reply to Mike SnitzerThe risk of redundant IO accounting was not taken into consideration when commit 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk") introduced IO splitting in terms of recursion via generic_make_request().
Fix this by subtracting the split bio's payload from the IO stats that were already accounted for by start_io_acct() upon dm_make_request() entry. This repeat oscillation of the IO accounting, up then down, isn't ideal but refactoring DM core's IO splitting to pre-split bios _before_ they are accounted turned out to be an excessive amount of change that will need a full development cycle to refine and verify.
Before this fix:
/dev/mapper/stripe_dev is a 4-way stripe using a 32k chunksize, so bios are split on 32k boundaries.
# fio --name=16M --filename=/dev/mapper/stripe_dev --rw=write --bs=64k --size=16M \
--iodepth=1 --ioengine=libaio --direct=1 --refill_bufferswith debugging added:
[103898.310264] device-mapper: core: start_io_acct: dm-2 WRITE bio->bi_iter.bi_sector=0 len=128 [103898.318704] device-mapper: core: __split_and_process_bio: recursing for following split bio: [103898.329136] device-mapper: core: start_io_acct: dm-2 WRITE bio->bi_iter.bi_sector=64 len=64 ...
16M written yet 136M (278528 * 512b) accounted:
# cat /sys/block/dm-2/stat | awk '{ print $7 }'
278528After this fix:
16M written and 16M (32768 * 512b) accounted:
# cat /sys/block/dm-2/stat | awk '{ print $7 }'
32768--- drivers/md/dm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/md/dm.c +16 -0
@@ -1584,6 +1584,9 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md, ci->sector = bio->bi_iter.bi_sector; } +#define __dm_part_stat_sub (part, field, subnd) \ + (part_stat_get(part, field) -= (subnd)) +/* * Entry point to split a bio into clones and submit them to the targets. */
@@ -1638,6 +1641,19 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md,struct bio *b =bio_split (bio,bio_sectors (bio) - ci.sector_count,GFP_NOIO , &md->queue->bio_split); ci.io->orig_bio = b; + +/* +* Adjust IO stats for each split, otherwise upon queue +* reentry there will be redundant IO accounting. +* NOTE: this is a stop-gap fix, a proper fix involves +* significant refactoring of DM core's bio splitting +* (by eliminating DM's splitting and just using bio_split) +*/ +part_stat_lock (); +__dm_part_stat_sub (&dm_disk (md)->part0, + sectors[op_stat_group (bio_op (bio))], ci.sector_count); +part_stat_unlock (); +bio_chain (b, bio); ret =generic_make_request (bio);break ;
Signature · 1 line
2.15.0
Re: [PATCH 2/4] dm: fix redundant IO accounting for bios that need splitting
Quoted 68 lines · Mike Snitzer
The risk of redundant IO accounting was not taken into consideration when commit
18a25da84354("dm: ensure bio submission follows a depth-first tree walk") introduced IO splitting in terms of recursion viageneric_make_request().Fix this by subtracting the split bio's payload from the IO stats that were already accounted for by
start_io_acct()upondm_make_request()entry. This repeat oscillation of the IO accounting, up then down, isn't ideal but refactoring DM core's IO splitting to pre-split bios _before_ they are accounted turned out to be an excessive amount of change that will need a full development cycle to refine and verify.Before this fix:
/dev/mapper/stripe_devis a 4-way stripe using a 32k chunksize, so bios are split on 32k boundaries.# fio --name=16M --filename=/dev/mapper/stripe_dev --rw=write --bs=64k --size=16M \ --iodepth=1 --ioengine=libaio --direct=1 --refill_bufferswith debugging added:
[103898.310264] device-mapper: core: start_io_acct: dm-2 WRITE bio->bi_iter.bi_sector=0 len=128 [103898.318704] device-mapper: core: __split_and_process_bio: recursing for following split bio: [103898.329136] device-mapper: core: start_io_acct: dm-2 WRITE bio->bi_iter.bi_sector=64 len=64 ...16M written yet 136M (278528 * 512b) accounted:
# cat /sys/block/dm-2/stat | awk '{ print $7 }' 278528After this fix:
16M written and 16M (32768 * 512b) accounted:
# cat /sys/block/dm-2/stat | awk '{ print $7 }' 32768Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Reported-by Bryan Gurney <[email protected]>Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/md/dm.c +9
-0@@ -1584,6 +1584,9 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md, ci->sector = bio->bi_iter.bi_sector; } +#define __dm_part_stat_sub (part, field, subnd) \ + (part_stat_get(part, field) -= (subnd)) +/* * Entry point to split a bio into clones and submit them to the targets. */ @@ -1638,6 +1641,19 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md,struct bio *b =bio_split (bio,bio_sectors (bio) - ci.sector_count,GFP_NOIO , &md->queue->bio_split); ci.io->orig_bio = b; + +/* +* Adjust IO stats for each split, otherwise upon queue +* reentry there will be redundant IO accounting. +* NOTE: this is a stop-gap fix, a proper fix involves +* significant refactoring of DM core's bio splitting
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1659
Quoted 10 lines
drivers/md/dm.c +7
-0+* (by eliminating DM's splitting and just using bio_split) +*/ +part_stat_lock (); +__dm_part_stat_sub (&dm_disk (md)->part0, + sectors[op_stat_group (bio_op (bio))], ci.sector_count); +part_stat_unlock (); +bio_chain (b, bio); ret =generic_make_request (bio);break ;
This ways is a bit ugly, but looks it works and it is simple, especially DM target may accept partial bio, so:
Thanks,
Ming
Re: [PATCH 2/4] dm: fix redundant IO accounting for bios that need splitting
In reply to Mike SnitzerHi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit: 18a25da84354 dm: ensure bio submission follows a depth-first tree walk.
The bot has tested the following trees: v4.20.3, v4.19.16.
v4.20.3: Build failed! Errors:
drivers/md/dm.c:1582:3: error: implicit declaration of function ‘part_stat_get’; did you mean ‘part_stat_dec’? [-Werror=implicit-function-declaration] drivers/md/dm.c:1639:10: error: ‘sectors’ undeclared (first use in this function); did you mean ‘Sector’?
v4.19.16: Build failed! Errors:
drivers/md/dm.c:1581:3: error: implicit declaration of function ‘part_stat_get’; did you mean ‘part_stat_dec’? [-Werror=implicit-function-declaration] drivers/md/dm.c:1641:10: error: ‘sectors’ undeclared (first use in this function); did you mean ‘Sector’?
How should we proceed with this patch?
--
Thanks,
Sasha
PATCH 3/4 dm: fix missing bio_split() pattern code in __split_and_process_bio()
In reply to Mike SnitzerUse the same BIO_QUEUE_ENTERED pattern that was established by commit cd4a4ae4683dc ("block: don't use blocking queue entered for recursive bio submits") by setting BIO_QUEUE_ENTERED after bio_split() and before recursing via generic_make_request().
Also add trace_block_split() because it provides useful context about bio splits in blktrace.
Depends-on: cd4a4ae4683dc ("block: don't use blocking queue entered for recursive bio submits")
--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/md/dm.c +2 -0
@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }
Signature · 1 line
2.15.0
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 21 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 9 lines
drivers/md/dm.c +2
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }
In theory, BIO_QUEUE_ENTERED is only required when __split_and_process_bio() is called from generic_make_request(). However, it may be called from dm_wq_work(), this way might cause trouble on operation to q->q_usage_counter.
Thanks,
Ming
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 25 lines · Ming Lei
Quoted 24 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock ();
Quoted 10 lines
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 6 lines
drivers/md/dm.c +2
-0+bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }In theory,
BIO_QUEUE_ENTEREDis only required when__split_and_process_bio()is called fromgeneric_make_request(). However, it may be called fromdm_wq_work(), this way might cause trouble on operation toq->q_usage_counter.
Good point, I've tweaked this patch to clear BIO_QUEUE_ENTERED in dm_make_request().
And to Neil's point: yes, these changes really do need to made common since it appears all bio_split() callers do go on to call generic_make_request().
Anyway, here is the updated patch that is now staged in linux-next:
From: Mike Snitzer <[email protected]>
Date: Fri, 18 Jan 2019 01:21:11 -0500
Subject: [PATCH v2] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Use the same BIO_QUEUE_ENTERED pattern that was established by commit cd4a4ae4683dc ("block: don't use blocking queue entered for recursive bio submits") by setting BIO_QUEUE_ENTERED after bio_split() and before recursing via generic_make_request().
Also add trace_block_split() because it provides useful context about bio splits in blktrace.
Depends-on: cd4a4ae4683dc ("block: don't use blocking queue entered for recursive bio submits")
--- drivers/md/dm.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/md/dm.c +9 -0
@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }
@@ -1734,6 +1736,13 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) map =dm_get_live_table (md, &srcu_idx); +/* +* Clear the bio-reentered-generic_make_request() flag, +* will be set again as needed if bio needs to be split. +*/ +if (bio_flagged (bio,BIO_QUEUE_ENTERED )) +bio_clear_flag (bio,BIO_QUEUE_ENTERED ); +/* if we're suspended, we have to queue this io for later */ if (unlikely (test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags))) {dm_put_live_table (md, srcu_idx);
Signature · 1 line
2.15.0
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 99 lines · Mike Snitzer
Quoted 25 lines · Ming Lei
Quoted 24 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock ();Quoted 10 lines
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 6 lines
drivers/md/dm.c +2
-0+bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }In theory,
BIO_QUEUE_ENTEREDis only required when__split_and_process_bio()is called fromgeneric_make_request(). However, it may be called fromdm_wq_work(), this way might cause trouble on operation toq->q_usage_counter.Good point, I've tweaked this patch to clear
BIO_QUEUE_ENTEREDindm_make_request().And to Neil's point: yes, these changes really do need to made common since it appears all
bio_split()callers do go on to callgeneric_make_request().Anyway, here is the updated patch that is now staged in linux-next:
From: Mike Snitzer <[email protected]>
Date: Fri, 18 Jan 2019 01:21:11 -0500
Subject: [PATCH v2] dm: fix missingbio_split()pattern code in__split_and_process_bio()Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 9 +++++++++ 1 file changed, 9 insertions(+)drivers/md/dm.c +9
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }@@ -1734,6 +1736,13 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) map =dm_get_live_table (md, &srcu_idx); +/* +* Clear the bio-reentered-generic_make_request() flag, +* will be set again as needed if bio needs to be split. +*/ +if (bio_flagged (bio,BIO_QUEUE_ENTERED )) +bio_clear_flag (bio,BIO_QUEUE_ENTERED ); +/* if we're suspended, we have to queue this io for later */ if (unlikely (test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags))) {dm_put_live_table (md, srcu_idx);Signature · 1 line
2.15.0
Hi Mike,
I'd suggest to fix this kind issue in the following way, then we can avoid to touch this flag from drivers:
block/blk-core.c +4 -0
@@ -1024,6 +1024,8 @@ blk_qc_t generic_make_request(struct bio *bio)else bio_io_error (bio);return ret; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); }if (!generic_make_request_checks (bio))
@@ -1074,6 +1076,8 @@ blk_qc_t generic_make_request(struct bio *bio)if (blk_queue_enter (q, flags) <0 ) { enter_succeeded = false; q =NULL ; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); } }
block/blk-merge.c +0 -10
@@ -339,16 +339,6 @@ void blk_queue_split(struct request_queue *q, struct bio **bio)/* there isn't chance to merge the splitted bio */ split->bi_opf |=REQ_NOMERGE ; -/* -* Since we're recursing into make_request here, ensure -* that we mark this bio as already having entered the queue. -* If not, and the queue is going away, we can get stuck -* forever on waiting for the queue reference to drop. But -* that will never happen, as we're already holding a -* reference to it. -*/ -bio_set_flag (*bio,BIO_QUEUE_ENTERED ); -bio_chain (split, *bio);trace_block_split (q, split, (*bio)->bi_iter.bi_sector);generic_make_request (*bio);
Thanks,
Ming
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 140 lines · Ming Lei
Quoted 99 lines · Mike Snitzer
Quoted 25 lines · Ming Lei
Quoted 24 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock ();Quoted 10 lines
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 6 lines
drivers/md/dm.c +2
-0+bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }In theory,
BIO_QUEUE_ENTEREDis only required when__split_and_process_bio()is called fromgeneric_make_request(). However, it may be called fromdm_wq_work(), this way might cause trouble on operation toq->q_usage_counter.Good point, I've tweaked this patch to clear
BIO_QUEUE_ENTEREDindm_make_request().And to Neil's point: yes, these changes really do need to made common since it appears all
bio_split()callers do go on to callgeneric_make_request().Anyway, here is the updated patch that is now staged in linux-next:
From: Mike Snitzer <[email protected]>
Date: Fri, 18 Jan 2019 01:21:11 -0500
Subject: [PATCH v2] dm: fix missingbio_split()pattern code in__split_and_process_bio()Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 9 +++++++++ 1 file changed, 9 insertions(+)drivers/md/dm.c +9
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }@@ -1734,6 +1736,13 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) map =dm_get_live_table (md, &srcu_idx); +/* +* Clear the bio-reentered-generic_make_request() flag, +* will be set again as needed if bio needs to be split. +*/ +if (bio_flagged (bio,BIO_QUEUE_ENTERED )) +bio_clear_flag (bio,BIO_QUEUE_ENTERED ); +/* if we're suspended, we have to queue this io for later */ if (unlikely (test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags))) {dm_put_live_table (md, srcu_idx);Signature · 1 line
2.15.0Hi Mike,
I'd suggest to fix this kind issue in the following way, then we can avoid to touch this flag from drivers:
block/blk-core.c +4
-0@@ -1024,6 +1024,8 @@ blk_qc_t generic_make_request(struct bio *bio)else bio_io_error (bio);return ret; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); }if (!generic_make_request_checks (bio))@@ -1074,6 +1076,8 @@ blk_qc_t generic_make_request(struct bio *bio)if (blk_queue_enter (q, flags) <0 ) { enter_succeeded = false; q =NULL ; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); } }block/blk-merge.c +0
-3@@ -339,16 +339,6 @@ void blk_queue_split(struct request_queue *q, struct bio **bio)/* there isn't chance to merge the splitted bio */ split->bi_opf |=REQ_NOMERGE ; -/* -* Since we're recursing into make_request here, ensure -* that we mark this bio as already having entered the queue.
block/blk-merge.c void blk_queue_split(struct request_queue *q, struct bio **bio) · line 344
Quoted 10 lines
block/blk-merge.c +0
-7-* If not, and the queue is going away, we can get stuck -* forever on waiting for the queue reference to drop. But -* that will never happen, as we're already holding a -* reference to it. -*/ -bio_set_flag (*bio,BIO_QUEUE_ENTERED ); -bio_chain (split, *bio);trace_block_split (q, split, (*bio)->bi_iter.bi_sector);generic_make_request (*bio);
Not opposed to this.
Thanks,
Mike
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 139 lines · Mike Snitzer
Quoted 136 lines · Ming Lei
Quoted 99 lines · Mike Snitzer
Quoted 25 lines · Ming Lei
Quoted 24 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock ();Quoted 10 lines
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 6 lines
drivers/md/dm.c +2
-0+bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }In theory,
BIO_QUEUE_ENTEREDis only required when__split_and_process_bio()is called fromgeneric_make_request(). However, it may be called fromdm_wq_work(), this way might cause trouble on operation toq->q_usage_counter.Good point, I've tweaked this patch to clear
BIO_QUEUE_ENTEREDindm_make_request().And to Neil's point: yes, these changes really do need to made common since it appears all
bio_split()callers do go on to callgeneric_make_request().Anyway, here is the updated patch that is now staged in linux-next:
From: Mike Snitzer <[email protected]>
Date: Fri, 18 Jan 2019 01:21:11 -0500
Subject: [PATCH v2] dm: fix missingbio_split()pattern code in__split_and_process_bio()Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 9 +++++++++ 1 file changed, 9 insertions(+)drivers/md/dm.c +9
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }@@ -1734,6 +1736,13 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) map =dm_get_live_table (md, &srcu_idx); +/* +* Clear the bio-reentered-generic_make_request() flag, +* will be set again as needed if bio needs to be split. +*/ +if (bio_flagged (bio,BIO_QUEUE_ENTERED )) +bio_clear_flag (bio,BIO_QUEUE_ENTERED ); +/* if we're suspended, we have to queue this io for later */ if (unlikely (test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags))) {dm_put_live_table (md, srcu_idx);Signature · 1 line
2.15.0Hi Mike,
I'd suggest to fix this kind issue in the following way, then we can avoid to touch this flag from drivers:
block/blk-core.c +4
-0@@ -1024,6 +1024,8 @@ blk_qc_t generic_make_request(struct bio *bio)else bio_io_error (bio);return ret; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); }if (!generic_make_request_checks (bio))@@ -1074,6 +1076,8 @@ blk_qc_t generic_make_request(struct bio *bio)if (blk_queue_enter (q, flags) <0 ) { enter_succeeded = false; q =NULL ; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); } }block/blk-merge.c +0
-0@@ -339,16 +339,6 @@ void blk_queue_split(struct request_queue *q, struct bio **bio)/* there isn't chance to merge the splitted bio */ split->bi_opf |=REQ_NOMERGE ;
Quoted 16 lines
Quoted 4 lines
block/blk-merge.c +0
-3-/* -* Since we're recursing into make_request here, ensure -* that we mark this bio as already having entered the queue. block/blk-merge.c void blk_queue_split(struct request_queue *q, struct bio **bio) · line 344
Quoted 10 lines
block/blk-merge.c +0
-7-* If not, and the queue is going away, we can get stuck -* forever on waiting for the queue reference to drop. But -* that will never happen, as we're already holding a -* reference to it. -*/ -bio_set_flag (*bio,BIO_QUEUE_ENTERED ); -bio_chain (split, *bio);trace_block_split (q, split, (*bio)->bi_iter.bi_sector);generic_make_request (*bio);Not opposed to this.
But thinking further: when you have a stack of cascading q->make_request_fn it could easily be that work done the next layer down end up causing the bio to recurse to generic_make_request() but not directly (e.g. dm_wq_work)... yet BIO_QUEUE_ENTERED will still be set when it really isn't appropriate.
Getting too cute with setting bio flags but not clearing them on different device boundaries could render the flags useless (or worse:
incorrect).
I'm not out for enaging in a focused audit/churn in this area that becomes a slippery slope during the rest of 5.0-rcX.
That is why I was going for a local DM change for 5.0 and, in parallel, work on the more generic fixes for 5.1.
So I'm back to preferring that...
But if you, Jens or others feel strongly about it I'm open to discuss it further.
Think we need to set REQ_NOMERGE in the split too (like blk_queue_split() is doing). Again, a comprehensive cleanup and consolidation of bio_split+generic_make_request pattern is needed. MD has a lot of it, DM has it, and then there is blk_queue_split(). Basically blk_queue_split()'s bio_split+bio_chain+generic_make_request and all the flags that get set inbetween should be factored out for all to use.
Mike
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
Quoted 159 lines · Mike Snitzer
Quoted 139 lines · Mike Snitzer
Quoted 136 lines · Ming Lei
Quoted 99 lines · Mike Snitzer
Quoted 25 lines · Ming Lei
Quoted 24 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock ();Quoted 10 lines
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 6 lines
drivers/md/dm.c +2
-0+bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }In theory,
BIO_QUEUE_ENTEREDis only required when__split_and_process_bio()is called fromgeneric_make_request(). However, it may be called fromdm_wq_work(), this way might cause trouble on operation toq->q_usage_counter.Good point, I've tweaked this patch to clear
BIO_QUEUE_ENTEREDindm_make_request().And to Neil's point: yes, these changes really do need to made common since it appears all
bio_split()callers do go on to callgeneric_make_request().Anyway, here is the updated patch that is now staged in linux-next:
From: Mike Snitzer <[email protected]>
Date: Fri, 18 Jan 2019 01:21:11 -0500
Subject: [PATCH v2] dm: fix missingbio_split()pattern code in__split_and_process_bio()Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 9 +++++++++ 1 file changed, 9 insertions(+)drivers/md/dm.c +9
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }@@ -1734,6 +1736,13 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) map =dm_get_live_table (md, &srcu_idx); +/* +* Clear the bio-reentered-generic_make_request() flag, +* will be set again as needed if bio needs to be split. +*/ +if (bio_flagged (bio,BIO_QUEUE_ENTERED )) +bio_clear_flag (bio,BIO_QUEUE_ENTERED ); +/* if we're suspended, we have to queue this io for later */ if (unlikely (test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags))) {dm_put_live_table (md, srcu_idx);Signature · 1 line
2.15.0Hi Mike,
I'd suggest to fix this kind issue in the following way, then we can avoid to touch this flag from drivers:
block/blk-core.c +4
-0@@ -1024,6 +1024,8 @@ blk_qc_t generic_make_request(struct bio *bio)else bio_io_error (bio);return ret; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); }if (!generic_make_request_checks (bio))@@ -1074,6 +1076,8 @@ blk_qc_t generic_make_request(struct bio *bio)if (blk_queue_enter (q, flags) <0 ) { enter_succeeded = false; q =NULL ; + }else { +bio_set_flag (bio,BIO_QUEUE_ENTERED ); } }block/blk-merge.c +0
-0@@ -339,16 +339,6 @@ void blk_queue_split(struct request_queue *q, struct bio **bio)/* there isn't chance to merge the splitted bio */ split->bi_opf |=REQ_NOMERGE ;Quoted 16 lines
Quoted 4 lines
block/blk-merge.c +0
-3-/* -* Since we're recursing into make_request here, ensure -* that we mark this bio as already having entered the queue. block/blk-merge.c void blk_queue_split(struct request_queue *q, struct bio **bio) · line 344
Quoted 10 lines
block/blk-merge.c +0
-7-* If not, and the queue is going away, we can get stuck -* forever on waiting for the queue reference to drop. But -* that will never happen, as we're already holding a -* reference to it. -*/ -bio_set_flag (*bio,BIO_QUEUE_ENTERED ); -bio_chain (split, *bio);trace_block_split (q, split, (*bio)->bi_iter.bi_sector);generic_make_request (*bio);Not opposed to this.
Quoted 5 lines
But thinking further: when you have a stack of cascading
q->make_request_fnit could easily be that work done the next layer down end up causing the bio to recurse togeneric_make_request()but not directly (e.g.dm_wq_work)... yetBIO_QUEUE_ENTEREDwill still be set when it really isn't appropriate.
That is true, in theory, we need a per-queue stack variable to record if queue usage counter is held. But it is quite hard to do that in kernel because we don't have stack variable allocator, otherwise this issue can be solved clean & simple.
Quoted 4 lines
Getting too cute with setting bio flags but not clearing them on different device boundaries could render the flags useless (or worse:
incorrect).
How about clearing the flag just following q->make_request_fn() in generic_make_request()?
Quoted 11 lines
I'm not out for enaging in a focused audit/churn in this area that becomes a slippery slope during the rest of 5.0-rcX.
That is why I was going for a local DM change for 5.0 and, in parallel, work on the more generic fixes for 5.1.
So I'm back to preferring that...
But if you, Jens or others feel strongly about it I'm open to discuss it further.
One concern is that if this flag starts to be used by drivers, sooner or later it may be difficult to maintain.
Quoted 8 lines
Think we need to set
REQ_NOMERGEin the split too (likeblk_queue_split()is doing). Again, a comprehensive cleanup and consolidation of bio_split+generic_make_request pattern is needed. MD has a lot of it, DM has it, and then there isblk_queue_split(). Basically blk_queue_split()'s bio_split+bio_chain+generic_make_request and all the flags that get set inbetween should be factored out for all to use.
Sounds a good topic and I am interested in, maybe you can submit a lsfmm proposal, :-)
Thanks,
Ming
Re: [dm-devel] [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
In reply to Mike SnitzerQuoted 21 lines · Mike Snitzer
Use the same
BIO_QUEUE_ENTEREDpattern that was established by commitcd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits") by settingBIO_QUEUE_ENTEREDafterbio_split()and before recursing viageneric_make_request().Also add
trace_block_split()because it provides useful context about bio splits in blktrace.Depends-on:
cd4a4ae4683dc("block: don't use blocking queue entered for recursive bio submits")Fixes 18a25da84354 ("dm: ensure bio submission follows a depth-first tree walk")Cc [email protected] # 4.16+Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 2 ++ 1 file changed, 2 insertions(+)drivers/md/dm.c +0
-0
drivers/md/dm.c static blk_qc_t __split_and_process_bio(struct mapped_device *md, · line 1662
Quoted 9 lines
drivers/md/dm.c +2
-0@@ -1654,7 +1654,9 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, sectors[op_stat_group(bio_op(bio))], ci.sector_count);part_stat_unlock (); +bio_set_flag (bio,BIO_QUEUE_ENTERED );bio_chain (b, bio); +trace_block_split (md->queue, b, bio->bi_iter.bi_sector); ret =generic_make_request (bio);break ; }
Thanks Mike...
If I understand this correctly, then we need to make the same change for all other callers of bio_split(), except blk_queue_split(). Maybe we should just set the flag and do the trace in bio_split(). Do you see any harm with doing it that way (in the next merge-window, I don't suggest you change this patch).
Thanks,
NeilBrown
Quoted 7 lines
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/dm-devel
Re: [PATCH 3/4] dm: fix missing bio_split() pattern code in __split_and_process_bio()
In reply to Mike SnitzerHi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit: 18a25da84354 dm: ensure bio submission follows a depth-first tree walk.
The bot has tested the following trees: v4.20.3, v4.19.16.
v4.20.3: Failed to apply! Possible dependencies:
7ed15898163a ("dm: fix redundant IO accounting for bios that need splitting")
v4.19.16: Failed to apply! Possible dependencies:
7ed15898163a ("dm: fix redundant IO accounting for bios that need splitting")
How should we proceed with this patch?
--
Thanks,
Sasha
PATCH 4/4 dm: fix dm_wq_work() to only use __split_and_process_bio() if appropriate
In reply to Mike SnitzerOtherwise targets that don't support/expect IO splitting could resubmit bios using code paths with unnecessary IO splitting complexity.
Depends-on: 24113d487843 ("dm: avoid indirect call in __dm_make_request")
--- drivers/md/dm.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
drivers/md/dm.c +12 -6
@@ -1727,6 +1727,15 @@ static blk_qc_t __process_bio(struct mapped_device *md,return ret; } +static blk_qc_t dm_process_bio (struct mapped_device *md, +struct dm_table *map,struct bio *bio) +{ +if (dm_get_md_type (md) ==DM_TYPE_NVME_BIO_BASED ) +return __process_bio (md, map, bio); +else +return __split_and_process_bio (md, map, bio); +} +static blk_qc_t dm_make_request (struct request_queue *q,struct bio *bio) {struct mapped_device *md = q->queuedata;
@@ -1747,10 +1756,7 @@ static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio)return ret; } -if (dm_get_md_type (md) ==DM_TYPE_NVME_BIO_BASED ) - ret =__process_bio (md, map, bio); -else - ret =__split_and_process_bio (md, map, bio); + ret =dm_process_bio (md, map, bio);dm_put_live_table (md, srcu_idx);return ret;
@@ -2429,9 +2435,9 @@ static void dm_wq_work(struct work_struct *work)break ;if (dm_request_based (md)) -generic_make_request (c); + (void )generic_make_request (c);else -__split_and_process_bio (md, map, c); + (void )dm_process_bio (md, map, c); }dm_put_live_table (md, srcu_idx);
Signature · 1 line
2.15.0