dm: changes staged for 5.19
Subsystems (3)
Mike Snitzer
maintainercover dm: changes staged for 5.19
Hi,
This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19
It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)
I can still make changes or add Reviewed-by:s, etc. So please feel free to review.
Jens, I'd appreciate it if you could pickup the first patch:
"block: change exported IO accounting interface from gendisk to bdev" (still not in love with that subject but...)
Thanks,
Mike
Mike Snitzer (13):
dm: conditionally enable BIOSET_PERCPU_CACHE for dm_io bioset
dm: factor out dm_io_set_error and __dm_io_dec_pending
dm: simplify dm_io access in dm_split_and_process_bio
dm: simplify dm_start_io_acct
dm: mark various branches unlikely
dm: add local variables to clone_endio and __map_bio
dm: move hot dm_io members to same cacheline as dm_target_io
dm: introduce dm_{get,put}_live_table_bio called from dm_submit_bio
dm: conditionally enable branching for less used features
dm: simplify basic targets
dm: use bio_sectors in dm_aceept_partial_bio
dm: simplify bio-based IO accounting further
dm: improve abnormal bio processing
Ming Lei (8):
block: change exported IO accounting interface from gendisk to bdev
dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct
dm: pass dm_io instance to dm_io_acct directly
dm: switch to bdev based IO accounting interfaces
dm: improve bio splitting and associated IO accounting
dm: don't grab target io reference in dm_zone_map_bio
dm: improve dm_io reference counting
dm: put all polled dm_io instances into a single list
block/blk-core.c | 52 ++--- drivers/block/zram/zram_drv.c | 5 +- drivers/md/dm-core.h | 38 ++-- drivers/md/dm-delay.c | 3 +- drivers/md/dm-flakey.c | 4 +- drivers/md/dm-linear.c | 11 +- drivers/md/dm-stats.c | 3 + drivers/md/dm-table.c | 16 +- drivers/md/dm-zone.c | 10 - drivers/md/dm.c | 515 ++++++++++++++++++++++++------------------ drivers/md/dm.h | 4 +- include/linux/blkdev.h | 7 +- 12 files changed, 367 insertions(+), 301 deletions(-)
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 1/21 block: change exported IO accounting interface from gendisk to bdev
From: Ming Lei <[email protected]>
Export IO accounting interfaces in terms of block_device now that gendisk has become more internal to block core.
Rename __part_{start,end}_io_acct's first argument from part to bdev. Rename __part_{start,end}_io_acct to bdev_{start,end}_io_acct and export them. Remove disk_{start,end}_io_acct and update caller (zram) to use bdev_{start,end}_io_acct.
DM can now be updated to use bdev_{start,end}_io_acct.
--- block/blk-core.c | 52 +++++++++++++++++-------------------------- drivers/block/zram/zram_drv.c | 5 +++-- include/linux/blkdev.h | 7 +++--- 3 files changed, 27 insertions(+), 37 deletions(-)
block/blk-core.c +20 -32
@@ -1022,21 +1022,22 @@ void update_io_ticks(struct block_device *part, unsigned long now, bool end) } } -static unsigned long __part_start_io_acct (struct block_device *part, -unsigned int sectors,unsigned int op, -unsigned long start_time) +unsigned long bdev_start_io_acct (struct block_device *bdev, +unsigned int sectors,unsigned int op, +unsigned long start_time) {const int sgrp =op_stat_group (op);part_stat_lock (); -update_io_ticks (part, start_time, false); -part_stat_inc (part, ios[sgrp]); -part_stat_add (part, sectors[sgrp], sectors); -part_stat_local_inc (part, in_flight[op_is_write (op)]); +update_io_ticks (bdev, start_time, false); +part_stat_inc (bdev, ios[sgrp]); +part_stat_add (bdev, sectors[sgrp], sectors); +part_stat_local_inc (bdev, in_flight[op_is_write (op)]);part_stat_unlock ();return start_time; } +EXPORT_SYMBOL (bdev_start_io_acct);/** * bio_start_io_acct_time - start I/O accounting for bio based drivers
@@ -1045,8 +1046,8 @@ static unsigned long __part_start_io_acct(struct block_device *part,*/ void bio_start_io_acct_time (struct bio *bio,unsigned long start_time) { -__part_start_io_acct (bio->bi_bdev,bio_sectors (bio), -bio_op (bio), start_time); +bdev_start_io_acct (bio->bi_bdev,bio_sectors (bio), +bio_op (bio), start_time); }EXPORT_SYMBOL_GPL (bio_start_io_acct_time);
@@ -1058,46 +1059,33 @@ EXPORT_SYMBOL_GPL(bio_start_io_acct_time);*/ unsigned long bio_start_io_acct (struct bio *bio) { -return __part_start_io_acct (bio->bi_bdev,bio_sectors (bio), -bio_op (bio), jiffies); +return bdev_start_io_acct (bio->bi_bdev,bio_sectors (bio), +bio_op (bio), jiffies); }EXPORT_SYMBOL_GPL (bio_start_io_acct); -unsigned long disk_start_io_acct (struct gendisk *disk,unsigned int sectors, -unsigned int op) -{ -return __part_start_io_acct (disk->part0, sectors, op, jiffies); -} -EXPORT_SYMBOL (disk_start_io_acct); - -static void __part_end_io_acct (struct block_device *part,unsigned int op, -unsigned long start_time) +void bdev_end_io_acct (struct block_device *bdev,unsigned int op, +unsigned long start_time) {const int sgrp =op_stat_group (op);unsigned long now =READ_ONCE (jiffies);unsigned long duration = now - start_time;part_stat_lock (); -update_io_ticks (part, now, true); -part_stat_add (part, nsecs[sgrp],jiffies_to_nsecs (duration)); -part_stat_local_dec (part, in_flight[op_is_write (op)]); +update_io_ticks (bdev, now, true); +part_stat_add (bdev, nsecs[sgrp],jiffies_to_nsecs (duration)); +part_stat_local_dec (bdev, in_flight[op_is_write (op)]);part_stat_unlock (); } +EXPORT_SYMBOL (bdev_end_io_acct);void bio_end_io_acct_remapped (struct bio *bio,unsigned long start_time, -struct block_device *orig_bdev) +struct block_device *orig_bdev) { -__part_end_io_acct (orig_bdev,bio_op (bio), start_time); +bdev_end_io_acct (orig_bdev,bio_op (bio), start_time); }EXPORT_SYMBOL_GPL (bio_end_io_acct_remapped); -void disk_end_io_acct (struct gendisk *disk,unsigned int op, -unsigned long start_time) -{ -__part_end_io_acct (disk->part0, op, start_time); -} -EXPORT_SYMBOL (disk_end_io_acct); -/** * blk_lld_busy - Check if underlying low-level drivers of a device are busy * @q : the queue of the device being checked
drivers/block/zram/zram_drv.c +3 -2
@@ -1675,9 +1675,10 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector, bv.bv_len =PAGE_SIZE ; bv.bv_offset =0 ; - start_time =disk_start_io_acct (bdev->bd_disk,SECTORS_PER_PAGE , op); + start_time =bdev_start_io_acct (bdev->bd_disk->part0, +SECTORS_PER_PAGE , op, jiffies); ret =zram_bvec_rw (zram, &bv, index, offset, op,NULL ); -disk_end_io_acct (bdev->bd_disk, op, start_time); +bdev_end_io_acct (bdev->bd_disk->part0, op, start_time); out:/* * If I/O fails, just return error(ie, non-zero) without
include/linux/blkdev.h +4 -3
@@ -1491,9 +1491,10 @@ static inline void blk_wake_io_task(struct task_struct *waiter)wake_up_process (waiter); } -unsigned long disk_start_io_acct (struct gendisk *disk,unsigned int sectors, -unsigned int op); -void disk_end_io_acct (struct gendisk *disk,unsigned int op, +unsigned long bdev_start_io_acct (struct block_device *bdev, +unsigned int sectors,unsigned int op, +unsigned long start_time); +void bdev_end_io_acct (struct block_device *bdev,unsigned int op,unsigned long start_time);void bio_start_io_acct_time (struct bio *bio,unsigned long start_time);
Signature · 1 line
2.15.0
Mike Snitzer
maintainerPATCH 2/21 dm: conditionally enable BIOSET_PERCPU_CACHE for dm_io bioset
In reply to Mike SnitzerA bioset's per-cpu alloc cache may have broader utility in the future but for now constrain it to being tightly coupled to QUEUE_FLAG_POLL.
Also change dm_io_complete() to use bio_clear_polled() so that it properly clears all associated bio state on requeue.
This commit improves DM's hipri bio polling (REQ_POLLED) perf by 7 - 20% depending on the system.
--- drivers/md/dm-table.c | 11 ++++++++--- drivers/md/dm.c | 8 ++++---- drivers/md/dm.h | 4 ++-- 3 files changed, 14 insertions(+), 9 deletions(-)
drivers/md/dm-table.c +8 -3
@@ -1002,6 +1002,8 @@ bool dm_table_request_based(struct dm_table *t)return __table_type_request_based (dm_table_get_type (t)); } +static int dm_table_supports_poll (struct dm_table *t); +static int dm_table_alloc_md_mempools (struct dm_table *t,struct mapped_device *md) {enum dm_queue_mode type =dm_table_get_type (t);
@@ -1009,21 +1011,24 @@ static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *unsigned min_pool_size =0 ;struct dm_target *ti;unsigned i; +bool poll_supported = false;if (unlikely (type ==DM_TYPE_NONE )) {DMWARN ("no table type is set, can't allocate mempools" );return -EINVAL ; } -if (__table_type_bio_based (type)) +if (__table_type_bio_based (type)) {for (i =0 ; i < t->num_targets; i++) { ti = t->targets + i; per_io_data_size =max (per_io_data_size, ti->per_io_data_size); min_pool_size =max (min_pool_size, ti->num_flush_bios); } + poll_supported = !!dm_table_supports_poll (t); + } - t->mempools =dm_alloc_md_mempools (md, type, t->integrity_supported, - per_io_data_size, min_pool_size); + t->mempools =dm_alloc_md_mempools (md, type, per_io_data_size, min_pool_size, + t->integrity_supported, poll_supported);if (!t->mempools)return -ENOMEM ;
drivers/md/dm.c +4 -4
@@ -899,7 +899,7 @@ static void dm_io_complete(struct dm_io *io)* may only reflect a subset of the pre-split original) * so clear REQ_POLLED in case of requeue. */ - bio->bi_opf &= ~REQ_POLLED ; +bio_clear_polled (bio);if (io_error ==BLK_STS_AGAIN ) {/* io_uring doesn't handle BLK_STS_AGAIN (yet) */ queue_io (md, bio);
@@ -2902,8 +2902,8 @@ int dm_noflush_suspending(struct dm_target *ti)EXPORT_SYMBOL_GPL (dm_noflush_suspending);struct dm_md_mempools *dm_alloc_md_mempools (struct mapped_device *md,enum dm_queue_mode type, -unsigned integrity,unsigned per_io_data_size, -unsigned min_pool_size) +unsigned per_io_data_size,unsigned min_pool_size, +bool integrity,bool poll) {struct dm_md_mempools *pools =kzalloc_node (sizeof (*pools),GFP_KERNEL , md->numa_node_id);unsigned int pool_size =0 ;
@@ -2919,7 +2919,7 @@ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_qu pool_size =max (dm_get_reserved_bio_based_ios (), min_pool_size); front_pad =roundup (per_io_data_size, __alignof__(struct dm_target_io )) +DM_TARGET_IO_BIO_OFFSET ; io_front_pad =roundup (per_io_data_size, __alignof__(struct dm_io )) +DM_IO_BIO_OFFSET ; - ret =bioset_init (&pools->io_bs, pool_size, io_front_pad,0 ); + ret =bioset_init (&pools->io_bs, pool_size, io_front_pad, poll ?BIOSET_PERCPU_CACHE :0 );if (ret) goto out;if (integrity &&bioset_integrity_create (&pools->io_bs, pool_size))
drivers/md/dm.h +2 -2
@@ -221,8 +221,8 @@ void dm_kcopyd_exit(void);* Mempool operations */ struct dm_md_mempools *dm_alloc_md_mempools (struct mapped_device *md,enum dm_queue_mode type, -unsigned integrity,unsigned per_bio_data_size, -unsigned min_pool_size); +unsigned per_io_data_size,unsigned min_pool_size, +bool integrity,bool poll);void dm_free_md_mempools (struct dm_md_mempools *pools);/*
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 3/21 dm: factor out dm_io_set_error and __dm_io_dec_pending
In reply to Mike SnitzerAlso eliminate need to use errno_to_blk_status().
--- drivers/md/dm.c | 64 ++++++++++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 28 deletions(-)
drivers/md/dm.c +36 -28
@@ -578,7 +578,7 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) io = container_of(tio, struct dm_io, tio); io->magic =DM_IO_MAGIC ; - io->status =0 ; + io->status =BLK_STS_OK ;atomic_set (&io->io_count,1 );this_cpu_inc (*md->pending_io); io->orig_bio =NULL ;
@@ -933,20 +933,31 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)* Decrements the number of outstanding ios that a bio has been * cloned into, completing the original io if necc. */ -void dm_io_dec_pending (struct dm_io *io,blk_status_t error) +static inline void __dm_io_dec_pending (struct dm_io *io) +{ +if (atomic_dec_and_test (&io->io_count)) +dm_io_complete (io); +} + +static void dm_io_set_error (struct dm_io *io,blk_status_t error) { +unsigned long flags; +/* Push-back supersedes any I/O errors */ -if (unlikely (error)) { -unsigned long flags; -spin_lock_irqsave (&io->lock, flags); -if (!(io->status ==BLK_STS_DM_REQUEUE && -__noflush_suspending (io->md))) - io->status = error; -spin_unlock_irqrestore (&io->lock, flags); +spin_lock_irqsave (&io->lock, flags); +if (!(io->status ==BLK_STS_DM_REQUEUE && +__noflush_suspending (io->md))) { + io->status = error; } +spin_unlock_irqrestore (&io->lock, flags); +} -if (atomic_dec_and_test (&io->io_count)) -dm_io_complete (io); +void dm_io_dec_pending (struct dm_io *io,blk_status_t error) +{ +if (unlikely (error)) +dm_io_set_error (io, error); + +__dm_io_dec_pending (io); }void disable_discard (struct mapped_device *md)
@@ -1429,7 +1440,7 @@ static bool is_abnormal_io(struct bio *bio) }static bool __process_abnormal_io (struct clone_info *ci,struct dm_target *ti, -int *result) +blk_status_t *status) {unsigned num_bios =0 ;
@@ -1453,11 +1464,11 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,* reconfiguration might also have changed that since the * check was performed. */ -if (!num_bios) - *result = -EOPNOTSUPP ; +if (unlikely (!num_bios)) + *status =BLK_STS_NOTSUPP ;else {__send_changing_extent_only (ci, ti, num_bios); - *result =0 ; + *status =BLK_STS_OK ; }return true; }
@@ -1506,19 +1517,16 @@ static void dm_queue_poll_io(struct bio *bio, struct dm_io *io)/* * Select the correct strategy for processing a non-flush bio. */ -static int __split_and_process_bio (struct clone_info *ci) +static blk_status_t __split_and_process_bio (struct clone_info *ci) {struct bio *clone;struct dm_target *ti;unsigned len; -int r; +blk_status_t error =BLK_STS_IOERR ; ti =dm_table_find_target (ci->map, ci->sector); -if (!ti) -return -EIO ; - -if (__process_abnormal_io (ci, ti, &r)) -return r; +if (unlikely (!ti ||__process_abnormal_io (ci, ti, &error))) +return error;/* * Only support bio polling for normal IO, and the target io is
@@ -1533,7 +1541,7 @@ static int __split_and_process_bio(struct clone_info *ci) ci->sector += len; ci->sector_count -= len; -return 0 ; +return BLK_STS_OK ; } static void init_clone_info(struct clone_info *ci, struct mapped_device *md,
@@ -1559,7 +1567,7 @@ static void dm_split_and_process_bio(struct mapped_device *md, {struct clone_info ci;struct bio *orig_bio =NULL ; -int error =0 ; +blk_status_t error =BLK_STS_OK ;init_clone_info (&ci, md, map, bio);
@@ -1601,7 +1609,7 @@ static void dm_split_and_process_bio(struct mapped_device *md,* bio->bi_private, so that dm_poll_bio can poll them all. */ if (error || !ci.submit_as_polled) -dm_io_dec_pending (ci.io,errno_to_blk_status (error)); +dm_io_dec_pending (ci.io, error);else dm_queue_poll_io (bio, ci.io); }
@@ -1682,10 +1690,10 @@ static int dm_poll_bio(struct bio *bio, struct io_comp_batch *iob,if (dm_poll_dm_io (io, iob, flags)) {hlist_del_init (&io->node);/* -* clone_endio() has already occurred, so passing -* error as 0 here doesn't override io->status +* clone_endio() has already occurred, so no +* error handling is needed here. */ -dm_io_dec_pending (io,0 ); +__dm_io_dec_pending (io); } }
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 4/21 dm: simplify dm_io access in dm_split_and_process_bio
In reply to Mike SnitzerUse local variable instead of redudant access using ci.io
--- drivers/md/dm.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
drivers/md/dm.c +8 -6
@@ -1566,10 +1566,12 @@ static void dm_split_and_process_bio(struct mapped_device *md, struct dm_table *map, struct bio *bio) { struct clone_info ci; +struct dm_io *io;struct bio *orig_bio =NULL ;blk_status_t error =BLK_STS_OK ;init_clone_info (&ci, md, map, bio); + io = ci.io;if (bio->bi_opf &REQ_PREFLUSH ) {__send_empty_flush (&ci);
@@ -1578,14 +1580,14 @@ static void dm_split_and_process_bio(struct mapped_device *md, } error =__split_and_process_bio (&ci); - ci.io->map_task =NULL ; + io->map_task =NULL ;if (error || !ci.sector_count) goto out;/* * Remainder must be passed to submit_bio_noacct() so it gets handled * *after* bios already submitted have been completely processed. -* We take a clone of the original to store in ci.io->orig_bio to be +* We take a clone of the original to store in io->orig_bio to be * used by dm_end_io_acct() and for dm_io_complete() to use for * completion handling. */
@@ -1597,9 +1599,9 @@ static void dm_split_and_process_bio(struct mapped_device *md, out:if (!orig_bio) orig_bio = bio; -smp_store_release (&ci.io->orig_bio, orig_bio); -if (dm_io_flagged (ci.io,DM_IO_START_ACCT )) -dm_start_io_acct (ci.io,NULL ); +smp_store_release (&io->orig_bio, orig_bio); +if (dm_io_flagged (io,DM_IO_START_ACCT )) +dm_start_io_acct (io,NULL );/* * Drop the extra reference count for non-POLLED bio, and hold one
@@ -1611,7 +1613,7 @@ static void dm_split_and_process_bio(struct mapped_device *md,if (error || !ci.submit_as_polled)dm_io_dec_pending (ci.io, error);else -dm_queue_poll_io (bio, ci.io); +dm_queue_poll_io (bio, io); }static void dm_submit_bio (struct bio *bio)
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 5/21 dm: simplify dm_start_io_acct
In reply to Mike SnitzerPull common DM_IO_ACCOUNTED check out to beginning of dm_start_io_acct. Also, use dm_tio_is_normal (and move it to dm-core.h).
--- drivers/md/dm-core.h | 6 ++++++ drivers/md/dm.c | 18 +++++------------- 2 files changed, 11 insertions(+), 13 deletions(-)
drivers/md/dm-core.h +6 -0
@@ -237,6 +237,12 @@ static inline void dm_tio_set_flag(struct dm_target_io *tio, unsigned int bit) tio->flags |= (1U << bit); } +static inline bool dm_tio_is_normal (struct dm_target_io *tio) +{ +return (dm_tio_flagged (tio,DM_TIO_INSIDE_DM_IO ) && + !dm_tio_flagged (tio,DM_TIO_IS_DUPLICATE_BIO )); +} +/* * One of these is allocated per original bio. * It contains the first clone used for that original.
drivers/md/dm.c +5 -13
@@ -538,17 +538,15 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone)/* * Ensure IO accounting is only ever started once. -* Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO. */ -if (!clone || -likely (!dm_tio_flagged (clone_to_tio (clone),DM_TIO_IS_DUPLICATE_BIO ))) { -if (WARN_ON_ONCE (dm_io_flagged (io,DM_IO_ACCOUNTED ))) -return ; +if (dm_io_flagged (io,DM_IO_ACCOUNTED )) +return ; + +/* Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO. */ +if (!clone ||likely (dm_tio_is_normal (clone_to_tio (clone)))) {dm_io_set_flag (io,DM_IO_ACCOUNTED ); }else {unsigned long flags; -if (dm_io_flagged (io,DM_IO_ACCOUNTED )) -return ;/* Can afford locking given DM_TIO_IS_DUPLICATE_BIO */ spin_lock_irqsave (&io->lock, flags);dm_io_set_flag (io,DM_IO_ACCOUNTED );
@@ -923,12 +921,6 @@ static void dm_io_complete(struct dm_io *io) } } -static inline bool dm_tio_is_normal (struct dm_target_io *tio) -{ -return (dm_tio_flagged (tio,DM_TIO_INSIDE_DM_IO ) && - !dm_tio_flagged (tio,DM_TIO_IS_DUPLICATE_BIO )); -} -/* * Decrements the number of outstanding ios that a bio has been * cloned into, completing the original io if necc.
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 6/21 dm: mark various branches unlikely
In reply to Mike Snitzer--- drivers/md/dm.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/md/dm.c +3 -3
@@ -992,7 +992,7 @@ static void clone_endio(struct bio *bio)disable_write_zeroes (md); } -if (blk_queue_is_zoned (q)) +if (unlikely (blk_queue_is_zoned (q)))dm_zone_endio (io, bio);if (endio) {
@@ -1289,7 +1289,7 @@ static void __map_bio(struct bio *clone)* on zoned target. In this case, dm_zone_map_bio() calls the target * map operation. */ -if (dm_emulate_zone_append (io->md)) +if (unlikely (dm_emulate_zone_append (io->md))) r =dm_zone_map_bio (tio);else r = ti->type->map (ti, clone);
@@ -1632,7 +1632,7 @@ static void dm_submit_bio(struct bio *bio)* Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc) * otherwise associated queue_limits won't be imposed. */ -if (is_abnormal_io (bio)) +if (unlikely (is_abnormal_io (bio)))blk_queue_split (&bio);dm_split_and_process_bio (md, map, bio);
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 7/21 dm: add local variables to clone_endio and __map_bio
In reply to Mike SnitzerAvoid redundant dereferences in both functions.
--- drivers/md/dm.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)
drivers/md/dm.c +12 -13
@@ -977,11 +977,12 @@ static bool swap_bios_limit(struct dm_target *ti, struct bio *bio)static void clone_endio (struct bio *bio) {blk_status_t error = bio->bi_status; +struct request_queue *q = bio->bi_bdev->bd_disk->queue;struct dm_target_io *tio =clone_to_tio (bio); +struct dm_target *ti = tio->ti; +dm_endio_fn endio = ti->type->end_io;struct dm_io *io = tio->io; -struct mapped_device *md = tio->io->md; -dm_endio_fn endio = tio->ti->type->end_io; -struct request_queue *q = bio->bi_bdev->bd_disk->queue; +struct mapped_device *md = io->md;if (unlikely (error ==BLK_STS_TARGET )) { if (bio_op(bio) == REQ_OP_DISCARD &&
@@ -996,7 +997,7 @@ static void clone_endio(struct bio *bio)dm_zone_endio (io, bio);if (endio) { -int r =endio (tio->ti, bio, &error); +int r =endio (ti, bio, &error);switch (r) {case DM_ENDIO_REQUEUE :/*
@@ -1020,10 +1021,8 @@ static void clone_endio(struct bio *bio) } } -if (unlikely (swap_bios_limit (tio->ti, bio))) { -struct mapped_device *md = io->md; +if (unlikely (swap_bios_limit (ti, bio)))up (&md->swap_bios_semaphore); - }free_tio (bio);dm_io_dec_pending (io, error);
@@ -1264,9 +1263,10 @@ static noinline void __set_swap_bios_limit(struct mapped_device *md, int latch)static void __map_bio (struct bio *clone) {struct dm_target_io *tio =clone_to_tio (clone); -int r; -struct dm_io *io = tio->io;struct dm_target *ti = tio->ti; +struct dm_io *io = tio->io; +struct mapped_device *md = io->md; +int r; clone->bi_end_io = clone_endio;
@@ -1277,7 +1277,6 @@ static void __map_bio(struct bio *clone) tio->old_sector = clone->bi_iter.bi_sector;if (unlikely (swap_bios_limit (ti, clone))) { -struct mapped_device *md = io->md;int latch =get_swap_bios ();if (unlikely (latch != md->swap_bios))__set_swap_bios_limit (md, latch);
@@ -1289,7 +1288,7 @@ static void __map_bio(struct bio *clone)* on zoned target. In this case, dm_zone_map_bio() calls the target * map operation. */ -if (unlikely (dm_emulate_zone_append (io->md))) +if (unlikely (dm_emulate_zone_append (md))) r =dm_zone_map_bio (tio);else r = ti->type->map (ti, clone);
@@ -1305,14 +1304,14 @@ static void __map_bio(struct bio *clone)* the bio has been remapped so dispatch it, but defer * dm_start_io_acct() until after possible bio_split(). */ -__dm_submit_bio_remap (clone,disk_devt (io->md->disk), +__dm_submit_bio_remap (clone,disk_devt (md->disk), tio->old_sector);dm_io_set_flag (io,DM_IO_START_ACCT );break ;case DM_MAPIO_KILL :case DM_MAPIO_REQUEUE :if (unlikely (swap_bios_limit (ti, clone))) -up (&io->md->swap_bios_semaphore); +up (&md->swap_bios_semaphore);free_tio (clone);if (r ==DM_MAPIO_KILL )dm_io_dec_pending (io,BLK_STS_IOERR );
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 8/21 dm: move hot dm_io members to same cacheline as dm_target_io
In reply to Mike SnitzerJust saves some cacheline bouncing for members accessed during cloned bio submission and completion.
--- drivers/md/dm-core.h | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/md/dm-core.h +7 -5
@@ -250,17 +250,19 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)#define DM_IO_MAGIC 19577struct dm_io {unsigned short magic; -blk_short_t flags; -atomic_t io_count; -struct mapped_device *md; -struct bio *orig_bio; -blk_status_t status; +spinlock_t lock;unsigned long start_time;void *data;struct hlist_node node;struct task_struct *map_task;struct dm_stats_aux stats_aux; + +blk_short_t flags; +blk_status_t status; +atomic_t io_count; +struct mapped_device *md; +struct bio *orig_bio;/* last member of dm_target_io is 'struct bio' */ struct dm_target_io tio; };
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 9/21 dm: introduce dm_{get, put}_live_table_bio called from dm_submit_bio
In reply to Mike SnitzerIf a bio is marked REQ_NOWAIT optimize dm_submit_bio()'s dm_table RCU usage to dm_{get,put}_live_table_fast.
DM core offers protection against blocking (via suspend) if REQ_NOWAIT.
--- drivers/md/dm.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
drivers/md/dm.c +24 -4
@@ -661,14 +661,16 @@ static void queue_io(struct mapped_device *md, struct bio *bio)* function to access the md->map field, and make sure they call * dm_put_live_table() when finished. */ -struct dm_table *dm_get_live_table(struct mapped_device *md, int *srcu_idx) __acquires(md->io_barrier) +struct dm_table *dm_get_live_table (struct mapped_device *md, + int *srcu_idx) __acquires(md->io_barrier) { *srcu_idx =srcu_read_lock (&md->io_barrier);return srcu_dereference (md->map, &md->io_barrier); } -void dm_put_live_table(struct mapped_device *md, int srcu_idx) __releases(md->io_barrier) +void dm_put_live_table (struct mapped_device *md, + int srcu_idx) __releases(md->io_barrier) {srcu_read_unlock (&md->io_barrier, srcu_idx); }
@@ -694,6 +696,24 @@ static void dm_put_live_table_fast(struct mapped_device *md) __releases(RCU)rcu_read_unlock (); } +static inline struct dm_table *dm_get_live_table_bio (struct mapped_device *md, +int *srcu_idx,struct bio *bio) +{ +if (bio->bi_opf &REQ_NOWAIT ) +return dm_get_live_table_fast (md); +else +return dm_get_live_table (md, srcu_idx); +} + +static inline void dm_put_live_table_bio (struct mapped_device *md,int srcu_idx, +struct bio *bio) +{ +if (bio->bi_opf &REQ_NOWAIT ) +dm_put_live_table_fast (md); +else +dm_put_live_table (md, srcu_idx); +} +static char *_dm_claim_ptr ="I belong to device-mapper" ;/*
@@ -1613,7 +1633,7 @@ static void dm_submit_bio(struct bio *bio)int srcu_idx;struct dm_table *map; - map =dm_get_live_table (md, &srcu_idx); + map =dm_get_live_table_bio (md, &srcu_idx, bio);/* If suspended, or map not yet available, queue this IO for later */ if (unlikely(test_bit(DMF_BLOCK_IO_FOR_SUSPEND, &md->flags)) ||
@@ -1636,7 +1656,7 @@ static void dm_submit_bio(struct bio *bio)dm_split_and_process_bio (md, map, bio); out: -dm_put_live_table (md, srcu_idx); +dm_put_live_table_bio (md, srcu_idx, bio); } static bool dm_poll_dm_io(struct dm_io *io, struct io_comp_batch *iob,
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 10/21 dm: conditionally enable branching for less used features
In reply to Mike SnitzerUse jump_labels to further reduce cost of unlikely branches for zoned block devices, dm-stats and swap_bios throttling.
--- drivers/md/dm-core.h | 5 ++++ drivers/md/dm-stats.c | 3 +++ drivers/md/dm-table.c | 5 ++++ drivers/md/dm.c | 63 ++++++++++++++++++++++++++++++++------------------- 4 files changed, 53 insertions(+), 23 deletions(-)
drivers/md/dm-core.h +5 -0
@@ -13,6 +13,7 @@#include <linux/ktime.h> #include <linux/blk-mq.h> #include <linux/blk-crypto-profile.h> +#include <linux/jump_label.h> #include <trace/events/block.h>
@@ -154,6 +155,10 @@ static inline struct dm_stats *dm_get_stats(struct mapped_device *md)return &md->stats; } +DECLARE_STATIC_KEY_FALSE (stats_enabled); +DECLARE_STATIC_KEY_FALSE (swap_bios_enabled); +DECLARE_STATIC_KEY_FALSE (zoned_enabled); +static inline bool dm_emulate_zone_append (struct mapped_device *md) {if (blk_queue_is_zoned (md->queue))
drivers/md/dm-stats.c +3 -0
@@ -396,6 +396,9 @@ static int dm_stats_create(struct dm_stats *stats, sector_t start, sector_t end,dm_stats_recalc_precise_timestamps (stats); +if (!static_key_enabled (&stats_enabled.key)) +static_branch_enable (&stats_enabled); +mutex_unlock (&stats->mutex);resume_callback (md);
drivers/md/dm-table.c +5 -0
@@ -719,6 +719,9 @@ int dm_table_add_target(struct dm_table *t, const char *type,DMWARN ("%s: %s: ignoring discards_supported because num_discard_bios is zero." ,dm_device_name (t->md), type); +if (tgt->limit_swap_bios && !static_key_enabled (&swap_bios_enabled.key)) +static_branch_enable (&swap_bios_enabled); +return 0 ; bad:
@@ -2051,6 +2054,8 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, r =dm_set_zones_restrictions (t, q);if (r)return r; +if (!static_key_enabled (&zoned_enabled.key)) +static_branch_enable (&zoned_enabled); }dm_update_crypto_profile (q, t);
drivers/md/dm.c +40 -23
123 lines not on this page · raw message
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 11/21 dm: simplify basic targets
In reply to Mike SnitzerRemove needless factoring and remap bi_sector regardless of bio_sectors() being non-zero.
--- drivers/md/dm-delay.c | 3 +-- drivers/md/dm-flakey.c | 4 +--- drivers/md/dm-linear.c | 11 ++--------- 3 files changed, 4 insertions(+), 14 deletions(-)
drivers/md/dm-delay.c +1 -2
@@ -296,8 +296,7 @@ static int delay_map(struct dm_target *ti, struct bio *bio) } delayed->class = c;bio_set_dev (bio, c->dev->bdev); -if (bio_sectors (bio)) - bio->bi_iter.bi_sector = c->start +dm_target_offset (ti, bio->bi_iter.bi_sector); + bio->bi_iter.bi_sector = c->start +dm_target_offset (ti, bio->bi_iter.bi_sector);return delay_bio (dc, c, bio); }
drivers/md/dm-flakey.c +1 -3
@@ -280,9 +280,7 @@ static void flakey_map_bio(struct dm_target *ti, struct bio *bio)struct flakey_c *fc = ti->private;bio_set_dev (bio, fc->dev->bdev); -if (bio_sectors (bio) ||op_is_zone_mgmt (bio_op (bio))) - bio->bi_iter.bi_sector = -flakey_map_sector (ti, bio->bi_iter.bi_sector); + bio->bi_iter.bi_sector =flakey_map_sector (ti, bio->bi_iter.bi_sector); }static void corrupt_bio_data (struct bio *bio,struct flakey_c *fc)
drivers/md/dm-linear.c +2 -9
@@ -84,19 +84,12 @@ static sector_t linear_map_sector(struct dm_target *ti, sector_t bi_sector)return lc->start +dm_target_offset (ti, bi_sector); } -static void linear_map_bio (struct dm_target *ti,struct bio *bio) +static int linear_map (struct dm_target *ti,struct bio *bio) {struct linear_c *lc = ti->private;bio_set_dev (bio, lc->dev->bdev); -if (bio_sectors (bio) ||op_is_zone_mgmt (bio_op (bio))) - bio->bi_iter.bi_sector = -linear_map_sector (ti, bio->bi_iter.bi_sector); -} - -static int linear_map (struct dm_target *ti,struct bio *bio) -{ -linear_map_bio (ti, bio); + bio->bi_iter.bi_sector =linear_map_sector (ti, bio->bi_iter.bi_sector);return DM_MAPIO_REMAPPED ; }
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 12/21 dm: use bio_sectors in dm_aceept_partial_bio
In reply to Mike SnitzerRename 'bi_size' to 'bio_sectors' given bi_size is being stored in sectors. Also, use bio_sectors() rather than open-coding it.
--- drivers/md/dm.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/md/dm.c +5 -5
@@ -1195,7 +1195,7 @@ static int dm_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff,* +--------------------+---------------+-------+ * * <-------------- *tio->len_ptr ---------------> -* <------- bi_size -------> +* <----- bio_sectors -----> * <-- n_sectors --> * * Region 1 was already iterated over with bio_advance or similar function.
@@ -1212,15 +1212,15 @@ static int dm_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff,void dm_accept_partial_bio (struct bio *bio,unsigned n_sectors) {struct dm_target_io *tio =clone_to_tio (bio); -unsigned bi_size = bio->bi_iter.bi_size >>SECTOR_SHIFT ; +unsigned bio_sectors =bio_sectors (bio);BUG_ON (dm_tio_flagged (tio,DM_TIO_IS_DUPLICATE_BIO ));BUG_ON (op_is_zone_mgmt (bio_op (bio)));BUG_ON (bio_op (bio) ==REQ_OP_ZONE_APPEND ); -BUG_ON (bi_size > *tio->len_ptr); -BUG_ON (n_sectors > bi_size); +BUG_ON (bio_sectors > *tio->len_ptr); +BUG_ON (n_sectors > bio_sectors); - *tio->len_ptr -= bi_size - n_sectors; + *tio->len_ptr -= bio_sectors - n_sectors; bio->bi_iter.bi_size = n_sectors <<SECTOR_SHIFT ; }EXPORT_SYMBOL_GPL (dm_accept_partial_bio);
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 13/21 dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 14/21 dm: pass dm_io instance to dm_io_acct directly
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 15/21 dm: switch to bdev based IO accounting interfaces
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 16/21 dm: improve bio splitting and associated IO accounting
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 17/21 dm: don't grab target io reference in dm_zone_map_bio
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 18/21 dm: improve dm_io reference counting
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 19/21 dm: put all polled dm_io instances into a single list
In reply to Mike Snitzer--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 20/21 dm: simplify bio-based IO accounting further
In reply to Mike SnitzerNow that io splitting is recorded prior to, or during, ->map IO accounting can happen immediately rather than defer until after bio splitting in dm_split_and_process_bio().
Remove the DM_IO_START_ACCT flag and also remove dm_io's map_task member because there is no longer any need to wait for splitting to occur before accounting.
Also move dm_io struct's 'flags' member to consolidate struct holes.
--- drivers/md/dm-core.h | 6 +----- drivers/md/dm.c | 34 +++++----------------------------- 2 files changed, 6 insertions(+), 34 deletions(-)
drivers/md/dm-core.h +1 -5
@@ -255,15 +255,12 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)#define DM_IO_MAGIC 19577struct dm_io {unsigned short magic; - +blk_short_t flags;spinlock_t lock;unsigned long start_time;void *data;struct dm_io *next; -struct task_struct *map_task;struct dm_stats_aux stats_aux; - -blk_short_t flags;blk_status_t status;atomic_t io_count;struct mapped_device *md;
@@ -281,7 +278,6 @@ struct dm_io {* dm_io flags */ enum { -DM_IO_START_ACCT ,DM_IO_ACCOUNTED ,DM_IO_WAS_SPLIT };
drivers/md/dm.c +5 -29
@@ -596,7 +596,6 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)this_cpu_inc (*md->pending_io); io->orig_bio = bio; io->md = md; - io->map_task = current;spin_lock_init (&io->lock); io->start_time = jiffies; io->flags =0 ;
@@ -1242,13 +1241,6 @@ void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors) }EXPORT_SYMBOL_GPL (dm_accept_partial_bio); -static inline void __dm_submit_bio_remap (struct bio *clone, -dev_t dev,sector_t old_sector) -{ -trace_block_bio_remap (clone, dev, old_sector); -submit_bio_noacct (clone); -} -/* * @clone: clone bio that DM core passed to target's .map function * @tgt_clone: clone of @clone bio that target needs submitted
@@ -1263,8 +1255,6 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)struct dm_target_io *tio =clone_to_tio (clone);struct dm_io *io = tio->io; -WARN_ON_ONCE (!tio->ti->accounts_remapped_io); -/* establish bio that will get submitted */ if (!tgt_clone) tgt_clone = clone;
@@ -1273,15 +1263,11 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)* Account io->origin_bio to DM dev on behalf of target * that took ownership of IO with DM_MAPIO_SUBMITTED. */ -if (io->map_task == current) { -/* Still in target's map function */ -dm_io_set_flag (io,DM_IO_START_ACCT ); - }else { -dm_start_io_acct (io, clone); - } +dm_start_io_acct (io, clone); -__dm_submit_bio_remap (tgt_clone,disk_devt (io->md->disk), +trace_block_bio_remap (tgt_clone,disk_devt (io->md->disk), tio->old_sector); +submit_bio_noacct (tgt_clone); }EXPORT_SYMBOL_GPL (dm_submit_bio_remap);
@@ -1341,16 +1327,10 @@ static void __map_bio(struct bio *clone)case DM_MAPIO_SUBMITTED :/* target has assumed ownership of this io */ if (!ti->accounts_remapped_io) -dm_io_set_flag (io,DM_IO_START_ACCT ); +dm_start_io_acct (io, clone);break ;case DM_MAPIO_REMAPPED : -/* -* the bio has been remapped so dispatch it, but defer -* dm_start_io_acct() until after possible bio_split(). -*/ -__dm_submit_bio_remap (clone,disk_devt (md->disk), - tio->old_sector); -dm_io_set_flag (io,DM_IO_START_ACCT ); +dm_submit_bio_remap (clone,NULL );break ;case DM_MAPIO_KILL :case DM_MAPIO_REQUEUE :
@@ -1668,7 +1648,6 @@ static void dm_split_and_process_bio(struct mapped_device *md, } error =__split_and_process_bio (&ci); - io->map_task =NULL ;if (error || !ci.sector_count) goto out;/*
@@ -1680,9 +1659,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,bio_inc_remaining (bio);submit_bio_noacct (bio); out: -if (dm_io_flagged (io,DM_IO_START_ACCT )) -dm_start_io_acct (io,NULL ); -/* * Drop the extra reference count for non-POLLED bio, and hold one * reference for POLLED bio, which will be released in dm_poll_bio
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerPATCH 21/21 dm: improve abnormal bio processing
In reply to Mike SnitzerRead/write/flush are the most common operations, optimize switch in is_abnormal_io() for those cases. Follows same pattern established in block perf-wip commit ("block: optimise blk_may_split for normal rw")
Also, push is_abnormal_io() check and blk_queue_split() down from dm_submit_bio() to dm_split_and_process_bio() and set new 'is_abnormal_io' flag in clone_info. Optimize __split_and_process_bio and __process_abnormal_io by leveraging ci.is_abnormal_io flag.
--- drivers/md/dm.c | 60 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 29 deletions(-)
drivers/md/dm.c +31 -29
@@ -84,7 +84,8 @@ struct clone_info {struct dm_io *io;sector_t sector;unsigned sector_count; -bool submit_as_polled; + bool is_abnormal_io:1; + bool submit_as_polled:1; };#define DM_TARGET_IO_BIO_OFFSET (offsetof(struct dm_target_io, clone))
@@ -1492,21 +1493,24 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_targetstatic bool is_abnormal_io (struct bio *bio) { -bool r = false; +unsigned int op =bio_op (bio); -switch (bio_op (bio)) { -case REQ_OP_DISCARD : -case REQ_OP_SECURE_ERASE : -case REQ_OP_WRITE_ZEROES : - r = true; -break ; +if (op !=REQ_OP_READ && op !=REQ_OP_WRITE && op !=REQ_OP_FLUSH ) { +switch (op) { +case REQ_OP_DISCARD : +case REQ_OP_SECURE_ERASE : +case REQ_OP_WRITE_ZEROES : +return true; +default : +break ; + } } -return r; +return false; } -static bool __process_abnormal_io (struct clone_info *ci,struct dm_target *ti, -blk_status_t *status) +static blk_status_t __process_abnormal_io (struct clone_info *ci, +struct dm_target *ti) {unsigned num_bios =0 ;
@@ -1520,8 +1524,6 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,case REQ_OP_WRITE_ZEROES : num_bios = ti->num_write_zeroes_bios;break ; -default : -return false; }/*
@@ -1531,12 +1533,10 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,* check was performed. */ if (unlikely (!num_bios)) - *status =BLK_STS_NOTSUPP ; -else { -__send_changing_extent_only (ci, ti, num_bios); - *status =BLK_STS_OK ; - } -return true; +return BLK_STS_NOTSUPP ; + +__send_changing_extent_only (ci, ti, num_bios); +return BLK_STS_OK ; }/*
@@ -1589,11 +1589,12 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci)struct bio *clone;struct dm_target *ti;unsigned len; -blk_status_t error =BLK_STS_IOERR ; ti =dm_table_find_target (ci->map, ci->sector); -if (unlikely (!ti ||__process_abnormal_io (ci, ti, &error))) -return error; +if (unlikely (!ti)) +return BLK_STS_IOERR ; +else if (unlikely (ci->is_abnormal_io)) +return __process_abnormal_io (ci, ti);/* * Only support bio polling for normal IO, and the target io is
@@ -1618,6 +1619,7 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md, ci->map = map; ci->io =alloc_io (md, bio); ci->bio = bio; + ci->is_abnormal_io = false; ci->submit_as_polled = false; ci->sector = bio->bi_iter.bi_sector; ci->sector_count =bio_sectors (bio);
@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out; + }else if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);
@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio) goto out; } -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc) -* otherwise associated queue_limits won't be imposed. -*/ -if (unlikely (is_abnormal_io (bio))) -blk_queue_split (&bio); -dm_split_and_process_bio (md, map, bio); out:dm_put_live_table_bio (md, srcu_idx, bio);
Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing
Quoted 21 lines · Mike Snitzer
Read/write/flush are the most common operations, optimize switch in
is_abnormal_io()for those cases. Follows same pattern established in block perf-wip commit ("block: optimiseblk_may_splitfor normal rw")Also, push
is_abnormal_io()check andblk_queue_split()down fromdm_submit_bio()todm_split_and_process_bio()and set new 'is_abnormal_io' flag inclone_info. Optimize__split_and_process_bioand__process_abnormal_ioby leveraging ci.is_abnormal_io flag.Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 60 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 29 deletions(-)drivers/md/dm.c +0
-0@@ -84,7 +84,8 @@ struct clone_info {struct dm_io *io;sector_t sector;
(snip)
Quoted 6 lines
drivers/md/dm.c +0
-0ci->sector = bio->bi_iter.bi_sector; ci->sector_count =bio_sectors (bio);@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out;
drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659
Quoted 10 lines
drivers/md/dm.c +7
-0+ }else if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);
Hi Mike,
The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the blk_queue_split(&bio) for abnormal I/O is called after init_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning of dm_split_and_process_bio(), so that blk_queue_split() is called before init_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.
Quoted 22 lines
drivers/md/dm.c +0
-7@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio) goto out; } -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc) -* otherwise associated queue_limits won't be imposed. -*/ -if (unlikely (is_abnormal_io (bio))) -blk_queue_split (&bio); -dm_split_and_process_bio (md, map, bio); out:dm_put_live_table_bio (md, srcu_idx, bio);Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
[1]
WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150 Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs] CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2 Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015 RIP: 0010:bio_trim+0x10a/0x150 Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89 RSP: 0018:ffff8881977d7928 EFLAGS: 00010206 RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000 RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00 RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057 R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00 R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054 FS: 00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0 Call Trace: <TASK> dm_submit_bio+0x5fa/0x13b0 ? dm_dax_direct_access+0x1c0/0x1c0 ? lock_release+0x3a7/0x6c0 ? lock_downgrade+0x6a0/0x6a0 __submit_bio+0x1c2/0x2c0 ? __bio_queue_enter+0x570/0x570 submit_bio_noacct_nocheck+0x2f7/0x810 ? should_fail_request+0x70/0x70 ? rcu_read_lock_sched_held+0x3f/0x70 ? __lock_acquire+0x1591/0x5030 submit_bio+0x10a/0x270 ? submit_bio_noacct+0x15c0/0x15c0 submit_bio_wait+0xf2/0x1d0 ? submit_bio_wait_endio+0x40/0x40 ? lock_release+0x6c0/0x6c0 blkdev_issue_discard+0xb5/0x110 ? __blkdev_issue_discard+0x590/0x590 ? truncate_bdev_range+0x15d/0x240 blkdev_common_ioctl+0x853/0x1670 ? blkdev_bszset+0x160/0x160 ? __ia32_sys_readlinkat+0x87/0xf0 ? __ia32_compat_sys_newlstat+0x70/0x70 ? count_memcg_events.constprop.0+0x44/0x50 blkdev_ioctl+0x23b/0x690 ? blkdev_common_ioctl+0x1670/0x1670 ? security_file_ioctl+0x50/0x90 __x64_sys_ioctl+0x127/0x190 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7ff3a970731b Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48 RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004 RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997 R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000 </TASK> irq event stamp: 10213 hardirqs last enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70 hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70 softirqs last enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 ---[ end trace 0000000000000000 ]---
[2]
drivers/md/dm.c +9 -7
@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,struct dm_io *io;blk_status_t error =BLK_STS_OK ; +if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; + } +init_clone_info (&ci, md, map, bio); io = ci.io;
@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out; - }else if (unlikely (is_abnormal_io (bio))) { -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, etc) -* otherwise associated queue_limits won't be imposed. -*/ -blk_queue_split (&bio); - ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);
Signature · 7 lines
Best Regards, Shin'ichiro Kawasaki -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing
Quoted 181 lines · Shinichiro Kawasaki
Quoted 21 lines · Mike Snitzer
Read/write/flush are the most common operations, optimize switch in
is_abnormal_io()for those cases. Follows same pattern established in block perf-wip commit ("block: optimiseblk_may_splitfor normal rw")Also, push
is_abnormal_io()check andblk_queue_split()down fromdm_submit_bio()todm_split_and_process_bio()and set new 'is_abnormal_io' flag inclone_info. Optimize__split_and_process_bioand__process_abnormal_ioby leveraging ci.is_abnormal_io flag.Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 60 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 29 deletions(-)drivers/md/dm.c +0
-0@@ -84,7 +84,8 @@ struct clone_info {struct dm_io *io;sector_t sector;(snip)
Quoted 6 lines
drivers/md/dm.c +0
-0ci->sector = bio->bi_iter.bi_sector; ci->sector_count =bio_sectors (bio);@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out;drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659
Quoted 10 lines
drivers/md/dm.c +7
-0+ }else if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);Hi Mike,
The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the
blk_queue_split(&bio)for abnormal I/O is called afterinit_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning ofdm_split_and_process_bio(), so thatblk_queue_split()is called beforeinit_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.Quoted 22 lines
drivers/md/dm.c +0
-7@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio) goto out; } -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc) -* otherwise associated queue_limits won't be imposed. -*/ -if (unlikely (is_abnormal_io (bio))) -blk_queue_split (&bio); -dm_split_and_process_bio (md, map, bio); out:dm_put_live_table_bio (md, srcu_idx, bio);Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel[1]
WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150 Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs] CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2 Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015 RIP: 0010:bio_trim+0x10a/0x150 Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89 RSP: 0018:ffff8881977d7928 EFLAGS: 00010206 RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000 RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00 RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057 R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00 R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054 FS: 00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0 Call Trace: <TASK> dm_submit_bio+0x5fa/0x13b0 ? dm_dax_direct_access+0x1c0/0x1c0 ? lock_release+0x3a7/0x6c0 ? lock_downgrade+0x6a0/0x6a0 __submit_bio+0x1c2/0x2c0 ? __bio_queue_enter+0x570/0x570 submit_bio_noacct_nocheck+0x2f7/0x810 ? should_fail_request+0x70/0x70 ? rcu_read_lock_sched_held+0x3f/0x70 ? __lock_acquire+0x1591/0x5030 submit_bio+0x10a/0x270 ? submit_bio_noacct+0x15c0/0x15c0 submit_bio_wait+0xf2/0x1d0 ? submit_bio_wait_endio+0x40/0x40 ? lock_release+0x6c0/0x6c0 blkdev_issue_discard+0xb5/0x110 ? __blkdev_issue_discard+0x590/0x590 ? truncate_bdev_range+0x15d/0x240 blkdev_common_ioctl+0x853/0x1670 ? blkdev_bszset+0x160/0x160 ? __ia32_sys_readlinkat+0x87/0xf0 ? __ia32_compat_sys_newlstat+0x70/0x70 ? count_memcg_events.constprop.0+0x44/0x50 blkdev_ioctl+0x23b/0x690 ? blkdev_common_ioctl+0x1670/0x1670 ? security_file_ioctl+0x50/0x90 __x64_sys_ioctl+0x127/0x190 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7ff3a970731b Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48 RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004 RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997 R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000 </TASK> irq event stamp: 10213 hardirqs last enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70 hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70 softirqs last enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 ---[ end trace 0000000000000000 ]---[2]
drivers/md/dm.c +9
-7@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,struct dm_io *io;blk_status_t error =BLK_STS_OK ; +if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; + } +init_clone_info (&ci, md, map, bio); io = ci.io;@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out; - }else if (unlikely (is_abnormal_io (bio))) { -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, etc) -* otherwise associated queue_limits won't be imposed. -*/ -blk_queue_split (&bio); - ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);Signature · 3 lines
-- Best Regards, Shin'ichiro Kawasaki
Thanks for your report, I'll sort it out but your patch doesn't seem right.
init_clone_info() will reset ci.is_abnormal_io so the bio won't be processed properly.
Mike
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing
Quoted 187 lines · Mike Snitzer
Quoted 181 lines · Shinichiro Kawasaki
Quoted 21 lines · Mike Snitzer
Read/write/flush are the most common operations, optimize switch in
is_abnormal_io()for those cases. Follows same pattern established in block perf-wip commit ("block: optimiseblk_may_splitfor normal rw")Also, push
is_abnormal_io()check andblk_queue_split()down fromdm_submit_bio()todm_split_and_process_bio()and set new 'is_abnormal_io' flag inclone_info. Optimize__split_and_process_bioand__process_abnormal_ioby leveraging ci.is_abnormal_io flag.Signed-off-by Mike Snitzer <[email protected]>--- drivers/md/dm.c | 60 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 29 deletions(-)drivers/md/dm.c +0
-0@@ -84,7 +84,8 @@ struct clone_info {struct dm_io *io;sector_t sector;(snip)
Quoted 6 lines
drivers/md/dm.c +0
-0ci->sector = bio->bi_iter.bi_sector; ci->sector_count =bio_sectors (bio);@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out;drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659
Quoted 10 lines
drivers/md/dm.c +7
-0+ }else if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);Hi Mike,
The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the
blk_queue_split(&bio)for abnormal I/O is called afterinit_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning ofdm_split_and_process_bio(), so thatblk_queue_split()is called beforeinit_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.Quoted 22 lines
drivers/md/dm.c +0
-7@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio) goto out; } -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc) -* otherwise associated queue_limits won't be imposed. -*/ -if (unlikely (is_abnormal_io (bio))) -blk_queue_split (&bio); -dm_split_and_process_bio (md, map, bio); out:dm_put_live_table_bio (md, srcu_idx, bio);Signature · 6 lines
2.15.0 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel[1]
WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150 Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs] CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2 Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015 RIP: 0010:bio_trim+0x10a/0x150 Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89 RSP: 0018:ffff8881977d7928 EFLAGS: 00010206 RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000 RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00 RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057 R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00 R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054 FS: 00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0 Call Trace: <TASK> dm_submit_bio+0x5fa/0x13b0 ? dm_dax_direct_access+0x1c0/0x1c0 ? lock_release+0x3a7/0x6c0 ? lock_downgrade+0x6a0/0x6a0 __submit_bio+0x1c2/0x2c0 ? __bio_queue_enter+0x570/0x570 submit_bio_noacct_nocheck+0x2f7/0x810 ? should_fail_request+0x70/0x70 ? rcu_read_lock_sched_held+0x3f/0x70 ? __lock_acquire+0x1591/0x5030 submit_bio+0x10a/0x270 ? submit_bio_noacct+0x15c0/0x15c0 submit_bio_wait+0xf2/0x1d0 ? submit_bio_wait_endio+0x40/0x40 ? lock_release+0x6c0/0x6c0 blkdev_issue_discard+0xb5/0x110 ? __blkdev_issue_discard+0x590/0x590 ? truncate_bdev_range+0x15d/0x240 blkdev_common_ioctl+0x853/0x1670 ? blkdev_bszset+0x160/0x160 ? __ia32_sys_readlinkat+0x87/0xf0 ? __ia32_compat_sys_newlstat+0x70/0x70 ? count_memcg_events.constprop.0+0x44/0x50 blkdev_ioctl+0x23b/0x690 ? blkdev_common_ioctl+0x1670/0x1670 ? security_file_ioctl+0x50/0x90 __x64_sys_ioctl+0x127/0x190 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7ff3a970731b Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48 RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004 RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997 R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000 </TASK> irq event stamp: 10213 hardirqs last enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70 hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70 softirqs last enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260 ---[ end trace 0000000000000000 ]---[2]
drivers/md/dm.c +9
-7@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,struct dm_io *io;blk_status_t error =BLK_STS_OK ; +if (unlikely (is_abnormal_io (bio))) { +/* +* Use blk_queue_split() for abnormal IO (e.g. discard, etc) +* otherwise associated queue_limits won't be imposed. +*/ +blk_queue_split (&bio); + ci.is_abnormal_io = true; + } +init_clone_info (&ci, md, map, bio); io = ci.io;@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,__send_empty_flush (&ci);/* dm_io_complete submits any data associated with flush */ goto out; - }else if (unlikely (is_abnormal_io (bio))) { -/* -* Use blk_queue_split() for abnormal IO (e.g. discard, etc) -* otherwise associated queue_limits won't be imposed. -*/ -blk_queue_split (&bio); - ci.is_abnormal_io = true; } error =__split_and_process_bio (&ci);Signature · 3 lines
-- Best Regards, Shin'ichiro KawasakiThanks for your report, I'll sort it out but your patch doesn't seem right.
Quoted 2 lines
init_clone_info()will reset ci.is_abnormal_io so the bio won't be processed properly.
FYI, I will be pushing a rebased commit with the following patch folded in:
drivers/md/dm.c +1 -1
@@ -1613,12 +1613,12 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) }static void init_clone_info (struct clone_info *ci,struct mapped_device *md, -struct dm_table *map,struct bio *bio) +struct dm_table *map,struct bio *bio,bool is_abnormal) {
ci->map = map; ci->io = alloc_io(md, bio); ci->bio = bio; - ci->is_abnormal_io = false; + ci->is_abnormal_io = is_abnormal; ci->submit_as_polled = false; ci->sector = bio->bi_iter.bi_sector; ci->sector_count = bio_sectors(bio); @@ -1638,21 +1638,24 @@ static void dm_split_and_process_bio(struct mapped_device *md, struct clone_info ci; struct dm_io *io; blk_status_t error = BLK_STS_OK;
+ bool is_abnormal;
- init_clone_info(&ci, md, map, bio);
+ is_abnormal = is_abnormal_io(bio);
+ if (unlikely(is_abnormal)) {
+ /*
+ * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+ * otherwise associated queue_limits won't be imposed.
+ */
+ blk_queue_split(&bio);
+ }
+
+ init_clone_info(&ci, md, map, bio, is_abnormal);
io = ci.io;
if (bio->bi_opf & REQ_PREFLUSH) {
__send_empty_flush(&ci);
/* dm_io_complete submits any data associated with flush */
goto out;- } else if (unlikely(is_abnormal_io(bio))) {
- /*
- * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
- * otherwise associated queue_limits won't be imposed.
- */
- blk_queue_split(&bio);
- ci.is_abnormal_io = true; } error = __split_and_process_bio(&ci);
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing
Quoted 2 lines · Mike Snitzer
On Fri, Apr 22 2022 at 9:13P -0400,
Mike Snitzer <[email protected]> wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Thanks for your report, I'll sort it out but your patch doesn't seem right.
init_clone_info()will reset ci.is_abnormal_io so the bio won't be processed properly.
Right, I missed that point.
drivers/md/dm.c
Quoted 6 lines
FYI, I will be pushing a rebased commit with the following patch folded in:
drivers/md/dm.c +0
-0
[...]
I tested the dm-5.19 branch at git hash e7ea9556985f including the fix above, and observed it passed all of my test sets. Thanks!
Signature · 7 lines
Best Regards, Shin'ichiro Kawasaki -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Quoted 7 lines · Mike Snitzer
Hi,
This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)
Mike, thanks for posting. We will run this on zoned stuff to check. Note that patches 13 to 20 are empty...
Signature · 7 lines
Damien Le Moal Western Digital Research -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Mike Snitzer
maintainerQuoted 10 lines · Damien Le Moal
Quoted 7 lines · Mike Snitzer
Hi,
This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)
Mike, thanks for posting. We will run this on zoned stuff to check.
OK, I appreciate it..
Quoted 1 line
Note that patches 13 to 20 are empty...
Not sure what's going on there... basically any patch that wasn't from me (so 1, 13-19) isn't showing up in patchwork or the dm-devel archive.
This is my first submission of patches using my [email protected] alias with redhat's smtp server that is really anal about not sending email that isn't from me. Yet it sent the email but seized on the following to have the body be empty:
"From: Ming Lei <[email protected]>"
Best part is my email account shows all the emails sent like all was perfect. Sorting this out will have to wait until Tuesday.
Sorry, but please just pull the dm-5.19 branch via git ;)
Thanks,
Mike
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Jens Axboe
maintainerQuoted 14 lines · Mike Snitzer
Quoted 10 lines · Damien Le Moal
Quoted 7 lines · Mike Snitzer
Hi,
This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)
Mike, thanks for posting. We will run this on zoned stuff to check.
Quoted 7 lines
OK, I appreciate it..
Quoted 1 line
Note that patches 13 to 20 are empty...
Not sure what's going on there... basically any patch that wasn't from me (so 1, 13-19) isn't showing up in patchwork or the dm-devel archive.
The patches I received look normal, fwiw.
Signature · 6 lines
Jens Axboe -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] (subset) [dm-5.19 PATCH 00/21] dm: changes staged for 5.19
In reply to Mike SnitzerQuoted 10 lines · Mike Snitzer
This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)
I can still make changes or add Reviewed-by:s, etc. So please feel free to review.
[...]
Applied, thanks!
[01/21] block: change exported IO accounting interface from gendisk to bdev
commit: 5f0614a55ecebdf55f1a17db0b5f6b787ed009f1
Best regards,
Signature · 7 lines
Jens Axboe -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel