Skip to contentSkip to conversation

dm: changes staged for 5.19

PATCH Applied Mike Snitzer 5 participants 30 messages Apr 18, 2022 – Apr 27, 2022

Subsystems (3)
lore
Share & tools
lore

Mike Snitzer

author maintainer
#0 lore raw

cover dm: changes staged for 5.19

Hi,

This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)

I can still make changes or add Reviewed-by:s, etc. So please feel free to review.

Jens, I'd appreciate it if you could pickup the first patch:
"block: change exported IO accounting interface from gendisk to bdev" (still not in love with that subject but...)

Thanks,
Mike

Mike Snitzer (13):
dm: conditionally enable BIOSET_PERCPU_CACHE for dm_io bioset
dm: factor out dm_io_set_error and __dm_io_dec_pending
dm: simplify dm_io access in dm_split_and_process_bio
dm: simplify dm_start_io_acct
dm: mark various branches unlikely
dm: add local variables to clone_endio and __map_bio
dm: move hot dm_io members to same cacheline as dm_target_io
dm: introduce dm_{get,put}_live_table_bio called from dm_submit_bio
dm: conditionally enable branching for less used features
dm: simplify basic targets
dm: use bio_sectors in dm_aceept_partial_bio
dm: simplify bio-based IO accounting further
dm: improve abnormal bio processing

Ming Lei (8):
block: change exported IO accounting interface from gendisk to bdev
dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct
dm: pass dm_io instance to dm_io_acct directly
dm: switch to bdev based IO accounting interfaces
dm: improve bio splitting and associated IO accounting
dm: don't grab target io reference in dm_zone_map_bio
dm: improve dm_io reference counting
dm: put all polled dm_io instances into a single list

 block/blk-core.c              |  52 ++---
 drivers/block/zram/zram_drv.c |   5 +-
 drivers/md/dm-core.h          |  38 ++--
 drivers/md/dm-delay.c         |   3 +-
 drivers/md/dm-flakey.c        |   4 +-
 drivers/md/dm-linear.c        |  11 +-
 drivers/md/dm-stats.c         |   3 +
 drivers/md/dm-table.c         |  16 +-
 drivers/md/dm-zone.c          |  10 -
 drivers/md/dm.c               | 515 ++++++++++++++++++++++++------------------
 drivers/md/dm.h               |   4 +-
 include/linux/blkdev.h        |   7 +-
 12 files changed, 367 insertions(+), 301 deletions(-)
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#1 lore raw

PATCH 1/21 block: change exported IO accounting interface from gendisk to bdev

From: Ming Lei <[email protected]>

Export IO accounting interfaces in terms of block_device now that gendisk has become more internal to block core.

Rename __part_{start,end}_io_acct's first argument from part to bdev. Rename __part_{start,end}_io_acct to bdev_{start,end}_io_acct and export them. Remove disk_{start,end}_io_acct and update caller (zram) to use bdev_{start,end}_io_acct.

DM can now be updated to use bdev_{start,end}_io_acct.

---
 block/blk-core.c              | 52 +++++++++++++++++--------------------------
 drivers/block/zram/zram_drv.c |  5 +++--
 include/linux/blkdev.h        |  7 +++---
 3 files changed, 27 insertions(+), 37 deletions(-)
block/blk-core.c +20 -32
index 937bb6b86331..99deb4603be3 100644
@@ -1022,21 +1022,22 @@ void update_io_ticks(struct block_device *part, unsigned long now, bool end)
 	}
 }
 
-static unsigned long __part_start_io_acct(struct block_device *part,
-					  unsigned int sectors, unsigned int op,
-					  unsigned long start_time)
+unsigned long bdev_start_io_acct(struct block_device *bdev,
+				 unsigned int sectors, unsigned int op,
+				 unsigned long start_time)
 {
 	const int sgrp = op_stat_group(op);
 
 	part_stat_lock();
-	update_io_ticks(part, start_time, false);
-	part_stat_inc(part, ios[sgrp]);
-	part_stat_add(part, sectors[sgrp], sectors);
-	part_stat_local_inc(part, in_flight[op_is_write(op)]);
+	update_io_ticks(bdev, start_time, false);
+	part_stat_inc(bdev, ios[sgrp]);
+	part_stat_add(bdev, sectors[sgrp], sectors);
+	part_stat_local_inc(bdev, in_flight[op_is_write(op)]);
 	part_stat_unlock();
 
 	return start_time;
 }
+EXPORT_SYMBOL(bdev_start_io_acct);
 
 /**
  * bio_start_io_acct_time - start I/O accounting for bio based drivers
@@ -1045,8 +1046,8 @@ static unsigned long __part_start_io_acct(struct block_device *part,
  */
 void bio_start_io_acct_time(struct bio *bio, unsigned long start_time)
 {
-	__part_start_io_acct(bio->bi_bdev, bio_sectors(bio),
-			     bio_op(bio), start_time);
+	bdev_start_io_acct(bio->bi_bdev, bio_sectors(bio),
+			   bio_op(bio), start_time);
 }
 EXPORT_SYMBOL_GPL(bio_start_io_acct_time);
 
@@ -1058,46 +1059,33 @@ EXPORT_SYMBOL_GPL(bio_start_io_acct_time);
  */
 unsigned long bio_start_io_acct(struct bio *bio)
 {
-	return __part_start_io_acct(bio->bi_bdev, bio_sectors(bio),
-				    bio_op(bio), jiffies);
+	return bdev_start_io_acct(bio->bi_bdev, bio_sectors(bio),
+				  bio_op(bio), jiffies);
 }
 EXPORT_SYMBOL_GPL(bio_start_io_acct);
 
-unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors,
-				 unsigned int op)
-{
-	return __part_start_io_acct(disk->part0, sectors, op, jiffies);
-}
-EXPORT_SYMBOL(disk_start_io_acct);
-
-static void __part_end_io_acct(struct block_device *part, unsigned int op,
-			       unsigned long start_time)
+void bdev_end_io_acct(struct block_device *bdev, unsigned int op,
+		      unsigned long start_time)
 {
 	const int sgrp = op_stat_group(op);
 	unsigned long now = READ_ONCE(jiffies);
 	unsigned long duration = now - start_time;
 
 	part_stat_lock();
-	update_io_ticks(part, now, true);
-	part_stat_add(part, nsecs[sgrp], jiffies_to_nsecs(duration));
-	part_stat_local_dec(part, in_flight[op_is_write(op)]);
+	update_io_ticks(bdev, now, true);
+	part_stat_add(bdev, nsecs[sgrp], jiffies_to_nsecs(duration));
+	part_stat_local_dec(bdev, in_flight[op_is_write(op)]);
 	part_stat_unlock();
 }
+EXPORT_SYMBOL(bdev_end_io_acct);
 
 void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time,
-		struct block_device *orig_bdev)
+			      struct block_device *orig_bdev)
 {
-	__part_end_io_acct(orig_bdev, bio_op(bio), start_time);
+	bdev_end_io_acct(orig_bdev, bio_op(bio), start_time);
 }
 EXPORT_SYMBOL_GPL(bio_end_io_acct_remapped);
 
-void disk_end_io_acct(struct gendisk *disk, unsigned int op,
-		      unsigned long start_time)
-{
-	__part_end_io_acct(disk->part0, op, start_time);
-}
-EXPORT_SYMBOL(disk_end_io_acct);
-
 /**
  * blk_lld_busy - Check if underlying low-level drivers of a device are busy
  * @q : the queue of the device being checked
drivers/block/zram/zram_drv.c +3 -2
index e9474b02012d..adb5209a556a 100644
@@ -1675,9 +1675,10 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector,
 	bv.bv_len = PAGE_SIZE;
 	bv.bv_offset = 0;
 
-	start_time = disk_start_io_acct(bdev->bd_disk, SECTORS_PER_PAGE, op);
+	start_time = bdev_start_io_acct(bdev->bd_disk->part0,
+			SECTORS_PER_PAGE, op, jiffies);
 	ret = zram_bvec_rw(zram, &bv, index, offset, op, NULL);
-	disk_end_io_acct(bdev->bd_disk, op, start_time);
+	bdev_end_io_acct(bdev->bd_disk->part0, op, start_time);
 out:
 	/*
 	 * If I/O fails, just return error(ie, non-zero) without
include/linux/blkdev.h +4 -3
index 60d016138997..f680ba6f0ab2 100644
@@ -1491,9 +1491,10 @@ static inline void blk_wake_io_task(struct task_struct *waiter)
 		wake_up_process(waiter);
 }
 
-unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors,
-		unsigned int op);
-void disk_end_io_acct(struct gendisk *disk, unsigned int op,
+unsigned long bdev_start_io_acct(struct block_device *bdev,
+				 unsigned int sectors, unsigned int op,
+				 unsigned long start_time);
+void bdev_end_io_acct(struct block_device *bdev, unsigned int op,
 		unsigned long start_time);
 
 void bio_start_io_acct_time(struct bio *bio, unsigned long start_time);
Signature · 1 line
2.15.0
author maintainer
#2 lore raw

PATCH 2/21 dm: conditionally enable BIOSET_PERCPU_CACHE for dm_io bioset

In reply to Mike Snitzer

A bioset's per-cpu alloc cache may have broader utility in the future but for now constrain it to being tightly coupled to QUEUE_FLAG_POLL.

Also change dm_io_complete() to use bio_clear_polled() so that it properly clears all associated bio state on requeue.

This commit improves DM's hipri bio polling (REQ_POLLED) perf by 7 - 20% depending on the system.

---
 drivers/md/dm-table.c | 11 ++++++++---
 drivers/md/dm.c       |  8 ++++----
 drivers/md/dm.h       |  4 ++--
 3 files changed, 14 insertions(+), 9 deletions(-)
drivers/md/dm-table.c +8 -3
index 03541cfc2317..ffd97cb6d1c1 100644
@@ -1002,6 +1002,8 @@ bool dm_table_request_based(struct dm_table *t)
 	return __table_type_request_based(dm_table_get_type(t));
 }
 
+static int dm_table_supports_poll(struct dm_table *t);
+
 static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *md)
 {
 	enum dm_queue_mode type = dm_table_get_type(t);
@@ -1009,21 +1011,24 @@ static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *
 	unsigned min_pool_size = 0;
 	struct dm_target *ti;
 	unsigned i;
+	bool poll_supported = false;
 
 	if (unlikely(type == DM_TYPE_NONE)) {
 		DMWARN("no table type is set, can't allocate mempools");
 		return -EINVAL;
 	}
 
-	if (__table_type_bio_based(type))
+	if (__table_type_bio_based(type)) {
 		for (i = 0; i < t->num_targets; i++) {
 			ti = t->targets + i;
 			per_io_data_size = max(per_io_data_size, ti->per_io_data_size);
 			min_pool_size = max(min_pool_size, ti->num_flush_bios);
 		}
+		poll_supported = !!dm_table_supports_poll(t);
+	}
 
-	t->mempools = dm_alloc_md_mempools(md, type, t->integrity_supported,
-					   per_io_data_size, min_pool_size);
+	t->mempools = dm_alloc_md_mempools(md, type, per_io_data_size, min_pool_size,
+					   t->integrity_supported, poll_supported);
 	if (!t->mempools)
 		return -ENOMEM;
 
drivers/md/dm.c +4 -4
index 82957bd460e8..61a52f585966 100644
@@ -899,7 +899,7 @@ static void dm_io_complete(struct dm_io *io)
 			 * may only reflect a subset of the pre-split original)
 			 * so clear REQ_POLLED in case of requeue.
 			 */
-			bio->bi_opf &= ~REQ_POLLED;
+			bio_clear_polled(bio);
 			if (io_error == BLK_STS_AGAIN) {
 				/* io_uring doesn't handle BLK_STS_AGAIN (yet) */
 				queue_io(md, bio);
@@ -2902,8 +2902,8 @@ int dm_noflush_suspending(struct dm_target *ti)
 EXPORT_SYMBOL_GPL(dm_noflush_suspending);
 
 struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_queue_mode type,
-					    unsigned integrity, unsigned per_io_data_size,
-					    unsigned min_pool_size)
+					    unsigned per_io_data_size, unsigned min_pool_size,
+					    bool integrity, bool poll)
 {
 	struct dm_md_mempools *pools = kzalloc_node(sizeof(*pools), GFP_KERNEL, md->numa_node_id);
 	unsigned int pool_size = 0;
@@ -2919,7 +2919,7 @@ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_qu
 		pool_size = max(dm_get_reserved_bio_based_ios(), min_pool_size);
 		front_pad = roundup(per_io_data_size, __alignof__(struct dm_target_io)) + DM_TARGET_IO_BIO_OFFSET;
 		io_front_pad = roundup(per_io_data_size,  __alignof__(struct dm_io)) + DM_IO_BIO_OFFSET;
-		ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, 0);
+		ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, poll ? BIOSET_PERCPU_CACHE : 0);
 		if (ret)
 			goto out;
 		if (integrity && bioset_integrity_create(&pools->io_bs, pool_size))
drivers/md/dm.h +2 -2
index 9013dc1a7b00..3f89664fea01 100644
@@ -221,8 +221,8 @@ void dm_kcopyd_exit(void);
  * Mempool operations
  */
 struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_queue_mode type,
-					    unsigned integrity, unsigned per_bio_data_size,
-					    unsigned min_pool_size);
+					    unsigned per_io_data_size, unsigned min_pool_size,
+					    bool integrity, bool poll);
 void dm_free_md_mempools(struct dm_md_mempools *pools);
 
 /*
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#3 lore raw

PATCH 3/21 dm: factor out dm_io_set_error and __dm_io_dec_pending

In reply to Mike Snitzer

Also eliminate need to use errno_to_blk_status().

---
 drivers/md/dm.c | 64 ++++++++++++++++++++++++++++++++-------------------------
 1 file changed, 36 insertions(+), 28 deletions(-)
drivers/md/dm.c +36 -28
index 61a52f585966..88ff5c3311aa 100644
@@ -578,7 +578,7 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
 
 	io = container_of(tio, struct dm_io, tio);
 	io->magic = DM_IO_MAGIC;
-	io->status = 0;
+	io->status = BLK_STS_OK;
 	atomic_set(&io->io_count, 1);
 	this_cpu_inc(*md->pending_io);
 	io->orig_bio = NULL;
@@ -933,20 +933,31 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)
  * Decrements the number of outstanding ios that a bio has been
  * cloned into, completing the original io if necc.
  */
-void dm_io_dec_pending(struct dm_io *io, blk_status_t error)
+static inline void __dm_io_dec_pending(struct dm_io *io)
+{
+	if (atomic_dec_and_test(&io->io_count))
+		dm_io_complete(io);
+}
+
+static void dm_io_set_error(struct dm_io *io, blk_status_t error)
 {
+	unsigned long flags;
+
 	/* Push-back supersedes any I/O errors */
-	if (unlikely(error)) {
-		unsigned long flags;
-		spin_lock_irqsave(&io->lock, flags);
-		if (!(io->status == BLK_STS_DM_REQUEUE &&
-		      __noflush_suspending(io->md)))
-			io->status = error;
-		spin_unlock_irqrestore(&io->lock, flags);
+	spin_lock_irqsave(&io->lock, flags);
+	if (!(io->status == BLK_STS_DM_REQUEUE &&
+	      __noflush_suspending(io->md))) {
+		io->status = error;
 	}
+	spin_unlock_irqrestore(&io->lock, flags);
+}
 
-	if (atomic_dec_and_test(&io->io_count))
-		dm_io_complete(io);
+void dm_io_dec_pending(struct dm_io *io, blk_status_t error)
+{
+	if (unlikely(error))
+		dm_io_set_error(io, error);
+
+	__dm_io_dec_pending(io);
 }
 
 void disable_discard(struct mapped_device *md)
@@ -1429,7 +1440,7 @@ static bool is_abnormal_io(struct bio *bio)
 }
 
 static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,
-				  int *result)
+				  blk_status_t *status)
 {
 	unsigned num_bios = 0;
 
@@ -1453,11 +1464,11 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,
 	 * reconfiguration might also have changed that since the
 	 * check was performed.
 	 */
-	if (!num_bios)
-		*result = -EOPNOTSUPP;
+	if (unlikely(!num_bios))
+		*status = BLK_STS_NOTSUPP;
 	else {
 		__send_changing_extent_only(ci, ti, num_bios);
-		*result = 0;
+		*status = BLK_STS_OK;
 	}
 	return true;
 }
@@ -1506,19 +1517,16 @@ static void dm_queue_poll_io(struct bio *bio, struct dm_io *io)
 /*
  * Select the correct strategy for processing a non-flush bio.
  */
-static int __split_and_process_bio(struct clone_info *ci)
+static blk_status_t __split_and_process_bio(struct clone_info *ci)
 {
 	struct bio *clone;
 	struct dm_target *ti;
 	unsigned len;
-	int r;
+	blk_status_t error = BLK_STS_IOERR;
 
 	ti = dm_table_find_target(ci->map, ci->sector);
-	if (!ti)
-		return -EIO;
-
-	if (__process_abnormal_io(ci, ti, &r))
-		return r;
+	if (unlikely(!ti || __process_abnormal_io(ci, ti, &error)))
+		return error;
 
 	/*
 	 * Only support bio polling for normal IO, and the target io is
@@ -1533,7 +1541,7 @@ static int __split_and_process_bio(struct clone_info *ci)
 	ci->sector += len;
 	ci->sector_count -= len;
 
-	return 0;
+	return BLK_STS_OK;
 }
 
 static void init_clone_info(struct clone_info *ci, struct mapped_device *md,
@@ -1559,7 +1567,7 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 {
 	struct clone_info ci;
 	struct bio *orig_bio = NULL;
-	int error = 0;
+	blk_status_t error = BLK_STS_OK;
 
 	init_clone_info(&ci, md, map, bio);
 
@@ -1601,7 +1609,7 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	 * bio->bi_private, so that dm_poll_bio can poll them all.
 	 */
 	if (error || !ci.submit_as_polled)
-		dm_io_dec_pending(ci.io, errno_to_blk_status(error));
+		dm_io_dec_pending(ci.io, error);
 	else
 		dm_queue_poll_io(bio, ci.io);
 }
@@ -1682,10 +1690,10 @@ static int dm_poll_bio(struct bio *bio, struct io_comp_batch *iob,
 		if (dm_poll_dm_io(io, iob, flags)) {
 			hlist_del_init(&io->node);
 			/*
-			 * clone_endio() has already occurred, so passing
-			 * error as 0 here doesn't override io->status
+			 * clone_endio() has already occurred, so no
+			 * error handling is needed here.
 			 */
-			dm_io_dec_pending(io, 0);
+			__dm_io_dec_pending(io);
 		}
 	}
 
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#4 lore raw

PATCH 4/21 dm: simplify dm_io access in dm_split_and_process_bio

In reply to Mike Snitzer

Use local variable instead of redudant access using ci.io

---
 drivers/md/dm.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
drivers/md/dm.c +8 -6
index 88ff5c3311aa..48f93c55c992 100644
@@ -1566,10 +1566,12 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 				     struct dm_table *map, struct bio *bio)
 {
 	struct clone_info ci;
+	struct dm_io *io;
 	struct bio *orig_bio = NULL;
 	blk_status_t error = BLK_STS_OK;
 
 	init_clone_info(&ci, md, map, bio);
+	io = ci.io;
 
 	if (bio->bi_opf & REQ_PREFLUSH) {
 		__send_empty_flush(&ci);
@@ -1578,14 +1580,14 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	}
 
 	error = __split_and_process_bio(&ci);
-	ci.io->map_task = NULL;
+	io->map_task = NULL;
 	if (error || !ci.sector_count)
 		goto out;
 
 	/*
 	 * Remainder must be passed to submit_bio_noacct() so it gets handled
 	 * *after* bios already submitted have been completely processed.
-	 * We take a clone of the original to store in ci.io->orig_bio to be
+	 * We take a clone of the original to store in io->orig_bio to be
 	 * used by dm_end_io_acct() and for dm_io_complete() to use for
 	 * completion handling.
 	 */
@@ -1597,9 +1599,9 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 out:
 	if (!orig_bio)
 		orig_bio = bio;
-	smp_store_release(&ci.io->orig_bio, orig_bio);
-	if (dm_io_flagged(ci.io, DM_IO_START_ACCT))
-		dm_start_io_acct(ci.io, NULL);
+	smp_store_release(&io->orig_bio, orig_bio);
+	if (dm_io_flagged(io, DM_IO_START_ACCT))
+		dm_start_io_acct(io, NULL);
 
 	/*
 	 * Drop the extra reference count for non-POLLED bio, and hold one
@@ -1611,7 +1613,7 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	if (error || !ci.submit_as_polled)
 		dm_io_dec_pending(ci.io, error);
 	else
-		dm_queue_poll_io(bio, ci.io);
+		dm_queue_poll_io(bio, io);
 }
 
 static void dm_submit_bio(struct bio *bio)
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#5 lore raw

PATCH 5/21 dm: simplify dm_start_io_acct

In reply to Mike Snitzer

Pull common DM_IO_ACCOUNTED check out to beginning of dm_start_io_acct. Also, use dm_tio_is_normal (and move it to dm-core.h).

---
 drivers/md/dm-core.h |  6 ++++++
 drivers/md/dm.c      | 18 +++++-------------
 2 files changed, 11 insertions(+), 13 deletions(-)
drivers/md/dm-core.h +6 -0
index 4277853c7535..db069fa9cee5 100644
@@ -237,6 +237,12 @@ static inline void dm_tio_set_flag(struct dm_target_io *tio, unsigned int bit)
 	tio->flags |= (1U << bit);
 }
 
+static inline bool dm_tio_is_normal(struct dm_target_io *tio)
+{
+	return (dm_tio_flagged(tio, DM_TIO_INSIDE_DM_IO) &&
+		!dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO));
+}
+
 /*
  * One of these is allocated per original bio.
  * It contains the first clone used for that original.
drivers/md/dm.c +5 -13
index 48f93c55c992..0974498c68e7 100644
@@ -538,17 +538,15 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone)
 
 	/*
 	 * Ensure IO accounting is only ever started once.
-	 * Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO.
 	 */
-	if (!clone ||
-	    likely(!dm_tio_flagged(clone_to_tio(clone), DM_TIO_IS_DUPLICATE_BIO))) {
-		if (WARN_ON_ONCE(dm_io_flagged(io, DM_IO_ACCOUNTED)))
-			return;
+	if (dm_io_flagged(io, DM_IO_ACCOUNTED))
+		return;
+
+	/* Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO. */
+	if (!clone || likely(dm_tio_is_normal(clone_to_tio(clone)))) {
 		dm_io_set_flag(io, DM_IO_ACCOUNTED);
 	} else {
 		unsigned long flags;
-		if (dm_io_flagged(io, DM_IO_ACCOUNTED))
-			return;
 		/* Can afford locking given DM_TIO_IS_DUPLICATE_BIO */
 		spin_lock_irqsave(&io->lock, flags);
 		dm_io_set_flag(io, DM_IO_ACCOUNTED);
@@ -923,12 +921,6 @@ static void dm_io_complete(struct dm_io *io)
 	}
 }
 
-static inline bool dm_tio_is_normal(struct dm_target_io *tio)
-{
-	return (dm_tio_flagged(tio, DM_TIO_INSIDE_DM_IO) &&
-		!dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO));
-}
-
 /*
  * Decrements the number of outstanding ios that a bio has been
  * cloned into, completing the original io if necc.
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#6 lore raw

PATCH 6/21 dm: mark various branches unlikely

In reply to Mike Snitzer
---
 drivers/md/dm.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/md/dm.c +3 -3
index 0974498c68e7..630c1880023a 100644
@@ -992,7 +992,7 @@ static void clone_endio(struct bio *bio)
 			disable_write_zeroes(md);
 	}
 
-	if (blk_queue_is_zoned(q))
+	if (unlikely(blk_queue_is_zoned(q)))
 		dm_zone_endio(io, bio);
 
 	if (endio) {
@@ -1289,7 +1289,7 @@ static void __map_bio(struct bio *clone)
 	 * on zoned target. In this case, dm_zone_map_bio() calls the target
 	 * map operation.
 	 */
-	if (dm_emulate_zone_append(io->md))
+	if (unlikely(dm_emulate_zone_append(io->md)))
 		r = dm_zone_map_bio(tio);
 	else
 		r = ti->type->map(ti, clone);
@@ -1632,7 +1632,7 @@ static void dm_submit_bio(struct bio *bio)
 	 * Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc)
 	 * otherwise associated queue_limits won't be imposed.
 	 */
-	if (is_abnormal_io(bio))
+	if (unlikely(is_abnormal_io(bio)))
 		blk_queue_split(&bio);
 
 	dm_split_and_process_bio(md, map, bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#7 lore raw

PATCH 7/21 dm: add local variables to clone_endio and __map_bio

In reply to Mike Snitzer

Avoid redundant dereferences in both functions.

---
 drivers/md/dm.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
drivers/md/dm.c +12 -13
index 630c1880023a..4793225722dd 100644
@@ -977,11 +977,12 @@ static bool swap_bios_limit(struct dm_target *ti, struct bio *bio)
 static void clone_endio(struct bio *bio)
 {
 	blk_status_t error = bio->bi_status;
+	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
 	struct dm_target_io *tio = clone_to_tio(bio);
+	struct dm_target *ti = tio->ti;
+	dm_endio_fn endio = ti->type->end_io;
 	struct dm_io *io = tio->io;
-	struct mapped_device *md = tio->io->md;
-	dm_endio_fn endio = tio->ti->type->end_io;
-	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
+	struct mapped_device *md = io->md;
 
 	if (unlikely(error == BLK_STS_TARGET)) {
 		if (bio_op(bio) == REQ_OP_DISCARD &&
@@ -996,7 +997,7 @@ static void clone_endio(struct bio *bio)
 		dm_zone_endio(io, bio);
 
 	if (endio) {
-		int r = endio(tio->ti, bio, &error);
+		int r = endio(ti, bio, &error);
 		switch (r) {
 		case DM_ENDIO_REQUEUE:
 			/*
@@ -1020,10 +1021,8 @@ static void clone_endio(struct bio *bio)
 		}
 	}
 
-	if (unlikely(swap_bios_limit(tio->ti, bio))) {
-		struct mapped_device *md = io->md;
+	if (unlikely(swap_bios_limit(ti, bio)))
 		up(&md->swap_bios_semaphore);
-	}
 
 	free_tio(bio);
 	dm_io_dec_pending(io, error);
@@ -1264,9 +1263,10 @@ static noinline void __set_swap_bios_limit(struct mapped_device *md, int latch)
 static void __map_bio(struct bio *clone)
 {
 	struct dm_target_io *tio = clone_to_tio(clone);
-	int r;
-	struct dm_io *io = tio->io;
 	struct dm_target *ti = tio->ti;
+	struct dm_io *io = tio->io;
+	struct mapped_device *md = io->md;
+	int r;
 
 	clone->bi_end_io = clone_endio;
 
@@ -1277,7 +1277,6 @@ static void __map_bio(struct bio *clone)
 	tio->old_sector = clone->bi_iter.bi_sector;
 
 	if (unlikely(swap_bios_limit(ti, clone))) {
-		struct mapped_device *md = io->md;
 		int latch = get_swap_bios();
 		if (unlikely(latch != md->swap_bios))
 			__set_swap_bios_limit(md, latch);
@@ -1289,7 +1288,7 @@ static void __map_bio(struct bio *clone)
 	 * on zoned target. In this case, dm_zone_map_bio() calls the target
 	 * map operation.
 	 */
-	if (unlikely(dm_emulate_zone_append(io->md)))
+	if (unlikely(dm_emulate_zone_append(md)))
 		r = dm_zone_map_bio(tio);
 	else
 		r = ti->type->map(ti, clone);
@@ -1305,14 +1304,14 @@ static void __map_bio(struct bio *clone)
 		 * the bio has been remapped so dispatch it, but defer
 		 * dm_start_io_acct() until after possible bio_split().
 		 */
-		__dm_submit_bio_remap(clone, disk_devt(io->md->disk),
+		__dm_submit_bio_remap(clone, disk_devt(md->disk),
 				      tio->old_sector);
 		dm_io_set_flag(io, DM_IO_START_ACCT);
 		break;
 	case DM_MAPIO_KILL:
 	case DM_MAPIO_REQUEUE:
 		if (unlikely(swap_bios_limit(ti, clone)))
-			up(&io->md->swap_bios_semaphore);
+			up(&md->swap_bios_semaphore);
 		free_tio(clone);
 		if (r == DM_MAPIO_KILL)
 			dm_io_dec_pending(io, BLK_STS_IOERR);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#8 lore raw

PATCH 8/21 dm: move hot dm_io members to same cacheline as dm_target_io

In reply to Mike Snitzer

Just saves some cacheline bouncing for members accessed during cloned bio submission and completion.

---
 drivers/md/dm-core.h | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/md/dm-core.h +7 -5
index db069fa9cee5..41d6511dc7cf 100644
@@ -250,17 +250,19 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)
 #define DM_IO_MAGIC 19577
 struct dm_io {
 	unsigned short magic;
-	blk_short_t flags;
-	atomic_t io_count;
-	struct mapped_device *md;
-	struct bio *orig_bio;
-	blk_status_t status;
+
 	spinlock_t lock;
 	unsigned long start_time;
 	void *data;
 	struct hlist_node node;
 	struct task_struct *map_task;
 	struct dm_stats_aux stats_aux;
+
+	blk_short_t flags;
+	blk_status_t status;
+	atomic_t io_count;
+	struct mapped_device *md;
+	struct bio *orig_bio;
 	/* last member of dm_target_io is 'struct bio' */
 	struct dm_target_io tio;
 };
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#9 lore raw

PATCH 9/21 dm: introduce dm_{get, put}_live_table_bio called from dm_submit_bio

In reply to Mike Snitzer

If a bio is marked REQ_NOWAIT optimize dm_submit_bio()'s dm_table RCU usage to dm_{get,put}_live_table_fast.

DM core offers protection against blocking (via suspend) if REQ_NOWAIT.

---
 drivers/md/dm.c | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
drivers/md/dm.c +24 -4
index 4793225722dd..073b41ce7a30 100644
@@ -661,14 +661,16 @@ static void queue_io(struct mapped_device *md, struct bio *bio)
  * function to access the md->map field, and make sure they call
  * dm_put_live_table() when finished.
  */
-struct dm_table *dm_get_live_table(struct mapped_device *md, int *srcu_idx) __acquires(md->io_barrier)
+struct dm_table *dm_get_live_table(struct mapped_device *md,
+				   int *srcu_idx) __acquires(md->io_barrier)
 {
 	*srcu_idx = srcu_read_lock(&md->io_barrier);
 
 	return srcu_dereference(md->map, &md->io_barrier);
 }
 
-void dm_put_live_table(struct mapped_device *md, int srcu_idx) __releases(md->io_barrier)
+void dm_put_live_table(struct mapped_device *md,
+		       int srcu_idx) __releases(md->io_barrier)
 {
 	srcu_read_unlock(&md->io_barrier, srcu_idx);
 }
@@ -694,6 +696,24 @@ static void dm_put_live_table_fast(struct mapped_device *md) __releases(RCU)
 	rcu_read_unlock();
 }
 
+static inline struct dm_table *dm_get_live_table_bio(struct mapped_device *md,
+						     int *srcu_idx, struct bio *bio)
+{
+	if (bio->bi_opf & REQ_NOWAIT)
+		return dm_get_live_table_fast(md);
+	else
+		return dm_get_live_table(md, srcu_idx);
+}
+
+static inline void dm_put_live_table_bio(struct mapped_device *md, int srcu_idx,
+					 struct bio *bio)
+{
+	if (bio->bi_opf & REQ_NOWAIT)
+		dm_put_live_table_fast(md);
+	else
+		dm_put_live_table(md, srcu_idx);
+}
+
 static char *_dm_claim_ptr = "I belong to device-mapper";
 
 /*
@@ -1613,7 +1633,7 @@ static void dm_submit_bio(struct bio *bio)
 	int srcu_idx;
 	struct dm_table *map;
 
-	map = dm_get_live_table(md, &srcu_idx);
+	map = dm_get_live_table_bio(md, &srcu_idx, bio);
 
 	/* If suspended, or map not yet available, queue this IO for later */
 	if (unlikely(test_bit(DMF_BLOCK_IO_FOR_SUSPEND, &md->flags)) ||
@@ -1636,7 +1656,7 @@ static void dm_submit_bio(struct bio *bio)
 
 	dm_split_and_process_bio(md, map, bio);
 out:
-	dm_put_live_table(md, srcu_idx);
+	dm_put_live_table_bio(md, srcu_idx, bio);
 }
 
 static bool dm_poll_dm_io(struct dm_io *io, struct io_comp_batch *iob,
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#10 lore raw

PATCH 10/21 dm: conditionally enable branching for less used features

In reply to Mike Snitzer

Use jump_labels to further reduce cost of unlikely branches for zoned block devices, dm-stats and swap_bios throttling.

---
 drivers/md/dm-core.h  |  5 ++++
 drivers/md/dm-stats.c |  3 +++
 drivers/md/dm-table.c |  5 ++++
 drivers/md/dm.c       | 63 ++++++++++++++++++++++++++++++++-------------------
 4 files changed, 53 insertions(+), 23 deletions(-)
drivers/md/dm-core.h +5 -0
index 41d6511dc7cf..8ba99eaa0872 100644
@@ -13,6 +13,7 @@
 #include <linux/ktime.h>
 #include <linux/blk-mq.h>
 #include <linux/blk-crypto-profile.h>
+#include <linux/jump_label.h>
 
 #include <trace/events/block.h>
 
@@ -154,6 +155,10 @@ static inline struct dm_stats *dm_get_stats(struct mapped_device *md)
 	return &md->stats;
 }
 
+DECLARE_STATIC_KEY_FALSE(stats_enabled);
+DECLARE_STATIC_KEY_FALSE(swap_bios_enabled);
+DECLARE_STATIC_KEY_FALSE(zoned_enabled);
+
 static inline bool dm_emulate_zone_append(struct mapped_device *md)
 {
 	if (blk_queue_is_zoned(md->queue))
drivers/md/dm-stats.c +3 -0
index 0e039a8c0bf2..86e0697330e8 100644
@@ -396,6 +396,9 @@ static int dm_stats_create(struct dm_stats *stats, sector_t start, sector_t end,
 
 	dm_stats_recalc_precise_timestamps(stats);
 
+	if (!static_key_enabled(&stats_enabled.key))
+		static_branch_enable(&stats_enabled);
+
 	mutex_unlock(&stats->mutex);
 
 	resume_callback(md);
drivers/md/dm-table.c +5 -0
index ffd97cb6d1c1..a9d2fc5d9ad9 100644
@@ -719,6 +719,9 @@ int dm_table_add_target(struct dm_table *t, const char *type,
 		DMWARN("%s: %s: ignoring discards_supported because num_discard_bios is zero.",
 		       dm_device_name(t->md), type);
 
+	if (tgt->limit_swap_bios && !static_key_enabled(&swap_bios_enabled.key))
+		static_branch_enable(&swap_bios_enabled);
+
 	return 0;
 
  bad:
@@ -2051,6 +2054,8 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
 		r = dm_set_zones_restrictions(t, q);
 		if (r)
 			return r;
+		if (!static_key_enabled(&zoned_enabled.key))
+			static_branch_enable(&zoned_enabled);
 	}
 
 	dm_update_crypto_profile(q, t);
drivers/md/dm.c +40 -23

123 lines not on this page · raw message

Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#11 lore raw

PATCH 11/21 dm: simplify basic targets

In reply to Mike Snitzer

Remove needless factoring and remap bi_sector regardless of bio_sectors() being non-zero.

---
 drivers/md/dm-delay.c  |  3 +--
 drivers/md/dm-flakey.c |  4 +---
 drivers/md/dm-linear.c | 11 ++---------
 3 files changed, 4 insertions(+), 14 deletions(-)
drivers/md/dm-delay.c +1 -2
index 9a51bf51a859..869afef5654a 100644
@@ -296,8 +296,7 @@ static int delay_map(struct dm_target *ti, struct bio *bio)
 	}
 	delayed->class = c;
 	bio_set_dev(bio, c->dev->bdev);
-	if (bio_sectors(bio))
-		bio->bi_iter.bi_sector = c->start + dm_target_offset(ti, bio->bi_iter.bi_sector);
+	bio->bi_iter.bi_sector = c->start + dm_target_offset(ti, bio->bi_iter.bi_sector);
 
 	return delay_bio(dc, c, bio);
 }
drivers/md/dm-flakey.c +1 -3
index 345229d7e59c..f2305eb758a2 100644
@@ -280,9 +280,7 @@ static void flakey_map_bio(struct dm_target *ti, struct bio *bio)
 	struct flakey_c *fc = ti->private;
 
 	bio_set_dev(bio, fc->dev->bdev);
-	if (bio_sectors(bio) || op_is_zone_mgmt(bio_op(bio)))
-		bio->bi_iter.bi_sector =
-			flakey_map_sector(ti, bio->bi_iter.bi_sector);
+	bio->bi_iter.bi_sector = flakey_map_sector(ti, bio->bi_iter.bi_sector);
 }
 
 static void corrupt_bio_data(struct bio *bio, struct flakey_c *fc)
drivers/md/dm-linear.c +2 -9
index 76b486e4d2be..0a6abbbe3745 100644
@@ -84,19 +84,12 @@ static sector_t linear_map_sector(struct dm_target *ti, sector_t bi_sector)
 	return lc->start + dm_target_offset(ti, bi_sector);
 }
 
-static void linear_map_bio(struct dm_target *ti, struct bio *bio)
+static int linear_map(struct dm_target *ti, struct bio *bio)
 {
 	struct linear_c *lc = ti->private;
 
 	bio_set_dev(bio, lc->dev->bdev);
-	if (bio_sectors(bio) || op_is_zone_mgmt(bio_op(bio)))
-		bio->bi_iter.bi_sector =
-			linear_map_sector(ti, bio->bi_iter.bi_sector);
-}
-
-static int linear_map(struct dm_target *ti, struct bio *bio)
-{
-	linear_map_bio(ti, bio);
+	bio->bi_iter.bi_sector = linear_map_sector(ti, bio->bi_iter.bi_sector);
 
 	return DM_MAPIO_REMAPPED;
 }
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#12 lore raw

PATCH 12/21 dm: use bio_sectors in dm_aceept_partial_bio

In reply to Mike Snitzer

Rename 'bi_size' to 'bio_sectors' given bi_size is being stored in sectors. Also, use bio_sectors() rather than open-coding it.

---
 drivers/md/dm.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/md/dm.c +5 -5
index 3359b55daa99..c8933d7e6a78 100644
@@ -1195,7 +1195,7 @@ static int dm_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff,
  * +--------------------+---------------+-------+
  *
  * <-------------- *tio->len_ptr --------------->
- *                      <------- bi_size ------->
+ *                      <----- bio_sectors ----->
  *                      <-- n_sectors -->
  *
  * Region 1 was already iterated over with bio_advance or similar function.
@@ -1212,15 +1212,15 @@ static int dm_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff,
 void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors)
 {
 	struct dm_target_io *tio = clone_to_tio(bio);
-	unsigned bi_size = bio->bi_iter.bi_size >> SECTOR_SHIFT;
+	unsigned bio_sectors = bio_sectors(bio);
 
 	BUG_ON(dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO));
 	BUG_ON(op_is_zone_mgmt(bio_op(bio)));
 	BUG_ON(bio_op(bio) == REQ_OP_ZONE_APPEND);
-	BUG_ON(bi_size > *tio->len_ptr);
-	BUG_ON(n_sectors > bi_size);
+	BUG_ON(bio_sectors > *tio->len_ptr);
+	BUG_ON(n_sectors > bio_sectors);
 
-	*tio->len_ptr -= bi_size - n_sectors;
+	*tio->len_ptr -= bio_sectors - n_sectors;
 	bio->bi_iter.bi_size = n_sectors << SECTOR_SHIFT;
 }
 EXPORT_SYMBOL_GPL(dm_accept_partial_bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#20 lore raw

PATCH 20/21 dm: simplify bio-based IO accounting further

In reply to Mike Snitzer

Now that io splitting is recorded prior to, or during, ->map IO accounting can happen immediately rather than defer until after bio splitting in dm_split_and_process_bio().

Remove the DM_IO_START_ACCT flag and also remove dm_io's map_task member because there is no longer any need to wait for splitting to occur before accounting.

Also move dm_io struct's 'flags' member to consolidate struct holes.

---
 drivers/md/dm-core.h |  6 +-----
 drivers/md/dm.c      | 34 +++++-----------------------------
 2 files changed, 6 insertions(+), 34 deletions(-)
drivers/md/dm-core.h +1 -5
index f3cfc7affd12..d21648a923ea 100644
@@ -255,15 +255,12 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio)
 #define DM_IO_MAGIC 19577
 struct dm_io {
 	unsigned short magic;
-
+	blk_short_t flags;
 	spinlock_t lock;
 	unsigned long start_time;
 	void *data;
 	struct dm_io *next;
-	struct task_struct *map_task;
 	struct dm_stats_aux stats_aux;
-
-	blk_short_t flags;
 	blk_status_t status;
 	atomic_t io_count;
 	struct mapped_device *md;
@@ -281,7 +278,6 @@ struct dm_io {
  * dm_io flags
  */
 enum {
-	DM_IO_START_ACCT,
 	DM_IO_ACCOUNTED,
 	DM_IO_WAS_SPLIT
 };
drivers/md/dm.c +5 -29
index c5a79712de1d..3b87d151ef88 100644
@@ -596,7 +596,6 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio)
 	this_cpu_inc(*md->pending_io);
 	io->orig_bio = bio;
 	io->md = md;
-	io->map_task = current;
 	spin_lock_init(&io->lock);
 	io->start_time = jiffies;
 	io->flags = 0;
@@ -1242,13 +1241,6 @@ void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors)
 }
 EXPORT_SYMBOL_GPL(dm_accept_partial_bio);
 
-static inline void __dm_submit_bio_remap(struct bio *clone,
-					 dev_t dev, sector_t old_sector)
-{
-	trace_block_bio_remap(clone, dev, old_sector);
-	submit_bio_noacct(clone);
-}
-
 /*
  * @clone: clone bio that DM core passed to target's .map function
  * @tgt_clone: clone of @clone bio that target needs submitted
@@ -1263,8 +1255,6 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)
 	struct dm_target_io *tio = clone_to_tio(clone);
 	struct dm_io *io = tio->io;
 
-	WARN_ON_ONCE(!tio->ti->accounts_remapped_io);
-
 	/* establish bio that will get submitted */
 	if (!tgt_clone)
 		tgt_clone = clone;
@@ -1273,15 +1263,11 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)
 	 * Account io->origin_bio to DM dev on behalf of target
 	 * that took ownership of IO with DM_MAPIO_SUBMITTED.
 	 */
-	if (io->map_task == current) {
-		/* Still in target's map function */
-		dm_io_set_flag(io, DM_IO_START_ACCT);
-	} else {
-		dm_start_io_acct(io, clone);
-	}
+	dm_start_io_acct(io, clone);
 
-	__dm_submit_bio_remap(tgt_clone, disk_devt(io->md->disk),
+	trace_block_bio_remap(tgt_clone, disk_devt(io->md->disk),
 			      tio->old_sector);
+	submit_bio_noacct(tgt_clone);
 }
 EXPORT_SYMBOL_GPL(dm_submit_bio_remap);
 
@@ -1341,16 +1327,10 @@ static void __map_bio(struct bio *clone)
 	case DM_MAPIO_SUBMITTED:
 		/* target has assumed ownership of this io */
 		if (!ti->accounts_remapped_io)
-			dm_io_set_flag(io, DM_IO_START_ACCT);
+			dm_start_io_acct(io, clone);
 		break;
 	case DM_MAPIO_REMAPPED:
-		/*
-		 * the bio has been remapped so dispatch it, but defer
-		 * dm_start_io_acct() until after possible bio_split().
-		 */
-		__dm_submit_bio_remap(clone, disk_devt(md->disk),
-				      tio->old_sector);
-		dm_io_set_flag(io, DM_IO_START_ACCT);
+		dm_submit_bio_remap(clone, NULL);
 		break;
 	case DM_MAPIO_KILL:
 	case DM_MAPIO_REQUEUE:
@@ -1668,7 +1648,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	}
 
 	error = __split_and_process_bio(&ci);
-	io->map_task = NULL;
 	if (error || !ci.sector_count)
 		goto out;
 	/*
@@ -1680,9 +1659,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	bio_inc_remaining(bio);
 	submit_bio_noacct(bio);
 out:
-	if (dm_io_flagged(io, DM_IO_START_ACCT))
-		dm_start_io_acct(io, NULL);
-
 	/*
 	 * Drop the extra reference count for non-POLLED bio, and hold one
 	 * reference for POLLED bio, which will be released in dm_poll_bio
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#21 lore raw

PATCH 21/21 dm: improve abnormal bio processing

In reply to Mike Snitzer

Read/write/flush are the most common operations, optimize switch in is_abnormal_io() for those cases. Follows same pattern established in block perf-wip commit ("block: optimise blk_may_split for normal rw")

Also, push is_abnormal_io() check and blk_queue_split() down from dm_submit_bio() to dm_split_and_process_bio() and set new 'is_abnormal_io' flag in clone_info. Optimize __split_and_process_bio and __process_abnormal_io by leveraging ci.is_abnormal_io flag.

---
 drivers/md/dm.c | 60 +++++++++++++++++++++++++++++----------------------------
 1 file changed, 31 insertions(+), 29 deletions(-)
drivers/md/dm.c +31 -29
index 3b87d151ef88..b9c30dfe0f2a 100644
@@ -84,7 +84,8 @@ struct clone_info {
 	struct dm_io *io;
 	sector_t sector;
 	unsigned sector_count;
-	bool submit_as_polled;
+	bool is_abnormal_io:1;
+	bool submit_as_polled:1;
 };
 
 #define DM_TARGET_IO_BIO_OFFSET (offsetof(struct dm_target_io, clone))
@@ -1492,21 +1493,24 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_target
 
 static bool is_abnormal_io(struct bio *bio)
 {
-	bool r = false;
+	unsigned int op = bio_op(bio);
 
-	switch (bio_op(bio)) {
-	case REQ_OP_DISCARD:
-	case REQ_OP_SECURE_ERASE:
-	case REQ_OP_WRITE_ZEROES:
-		r = true;
-		break;
+	if (op != REQ_OP_READ && op != REQ_OP_WRITE && op != REQ_OP_FLUSH) {
+		switch (op) {
+		case REQ_OP_DISCARD:
+		case REQ_OP_SECURE_ERASE:
+		case REQ_OP_WRITE_ZEROES:
+			return true;
+		default:
+			break;
+		}
 	}
 
-	return r;
+	return false;
 }
 
-static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,
-				  blk_status_t *status)
+static blk_status_t __process_abnormal_io(struct clone_info *ci,
+					  struct dm_target *ti)
 {
 	unsigned num_bios = 0;
 
@@ -1520,8 +1524,6 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,
 	case REQ_OP_WRITE_ZEROES:
 		num_bios = ti->num_write_zeroes_bios;
 		break;
-	default:
-		return false;
 	}
 
 	/*
@@ -1531,12 +1533,10 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti,
 	 * check was performed.
 	 */
 	if (unlikely(!num_bios))
-		*status = BLK_STS_NOTSUPP;
-	else {
-		__send_changing_extent_only(ci, ti, num_bios);
-		*status = BLK_STS_OK;
-	}
-	return true;
+		return BLK_STS_NOTSUPP;
+
+	__send_changing_extent_only(ci, ti, num_bios);
+	return BLK_STS_OK;
 }
 
 /*
@@ -1589,11 +1589,12 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci)
 	struct bio *clone;
 	struct dm_target *ti;
 	unsigned len;
-	blk_status_t error = BLK_STS_IOERR;
 
 	ti = dm_table_find_target(ci->map, ci->sector);
-	if (unlikely(!ti || __process_abnormal_io(ci, ti, &error)))
-		return error;
+	if (unlikely(!ti))
+		return BLK_STS_IOERR;
+	else if (unlikely(ci->is_abnormal_io))
+		return __process_abnormal_io(ci, ti);
 
 	/*
 	 * Only support bio polling for normal IO, and the target io is
@@ -1618,6 +1619,7 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md,
 	ci->map = map;
 	ci->io = alloc_io(md, bio);
 	ci->bio = bio;
+	ci->is_abnormal_io = false;
 	ci->submit_as_polled = false;
 	ci->sector = bio->bi_iter.bi_sector;
 	ci->sector_count = bio_sectors(bio);
@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;
+	} else if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);
@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio)
 		goto out;
 	}
 
-	/*
-	 * Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc)
-	 * otherwise associated queue_limits won't be imposed.
-	 */
-	if (unlikely(is_abnormal_io(bio)))
-		blk_queue_split(&bio);
-
 	dm_split_and_process_bio(md, map, bio);
 out:
 	dm_put_live_table_bio(md, srcu_idx, bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#22 lore raw

Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing

Quoted 21 lines · Mike Snitzer

Read/write/flush are the most common operations, optimize switch in is_abnormal_io() for those cases. Follows same pattern established in block perf-wip commit ("block: optimise blk_may_split for normal rw")

Also, push is_abnormal_io() check and blk_queue_split() down from dm_submit_bio() to dm_split_and_process_bio() and set new 'is_abnormal_io' flag in clone_info. Optimize __split_and_process_bio and __process_abnormal_io by leveraging ci.is_abnormal_io flag.

---
 drivers/md/dm.c | 60 +++++++++++++++++++++++++++++----------------------------
 1 file changed, 31 insertions(+), 29 deletions(-)
drivers/md/dm.c +0 -0
index 3b87d151ef88..b9c30dfe0f2a 100644
@@ -84,7 +84,8 @@ struct clone_info {
 	struct dm_io *io;
 	sector_t sector;

(snip)

Quoted 6 lines
drivers/md/dm.c +0 -0
 	ci->sector = bio->bi_iter.bi_sector;
 	ci->sector_count = bio_sectors(bio);
@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;

drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659

Quoted 10 lines
drivers/md/dm.c +7 -0
+	} else if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);

Hi Mike,

The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the blk_queue_split(&bio) for abnormal I/O is called after init_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning of dm_split_and_process_bio(), so that blk_queue_split() is called before init_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.

Quoted 22 lines
drivers/md/dm.c +0 -7
@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio)
 		goto out;
 	}
 
-	/*
-	 * Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc)
-	 * otherwise associated queue_limits won't be imposed.
-	 */
-	if (unlikely(is_abnormal_io(bio)))
-		blk_queue_split(&bio);
-
 	dm_split_and_process_bio(md, map, bio);
 out:
 	dm_put_live_table_bio(md, srcu_idx, bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

[1]

WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150
Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev
 intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs]
CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:bio_trim+0x10a/0x150
Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89
RSP: 0018:ffff8881977d7928 EFLAGS: 00010206
RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000
RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00
RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057
R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00
R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054
FS:  00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0
Call Trace:
 <TASK>
 dm_submit_bio+0x5fa/0x13b0
 ? dm_dax_direct_access+0x1c0/0x1c0
 ? lock_release+0x3a7/0x6c0
 ? lock_downgrade+0x6a0/0x6a0
 __submit_bio+0x1c2/0x2c0
 ? __bio_queue_enter+0x570/0x570
 submit_bio_noacct_nocheck+0x2f7/0x810
 ? should_fail_request+0x70/0x70
 ? rcu_read_lock_sched_held+0x3f/0x70
 ? __lock_acquire+0x1591/0x5030
 submit_bio+0x10a/0x270
 ? submit_bio_noacct+0x15c0/0x15c0
 submit_bio_wait+0xf2/0x1d0
 ? submit_bio_wait_endio+0x40/0x40
 ? lock_release+0x6c0/0x6c0
 blkdev_issue_discard+0xb5/0x110
 ? __blkdev_issue_discard+0x590/0x590
 ? truncate_bdev_range+0x15d/0x240
 blkdev_common_ioctl+0x853/0x1670
 ? blkdev_bszset+0x160/0x160
 ? __ia32_sys_readlinkat+0x87/0xf0
 ? __ia32_compat_sys_newlstat+0x70/0x70
 ? count_memcg_events.constprop.0+0x44/0x50
 blkdev_ioctl+0x23b/0x690
 ? blkdev_common_ioctl+0x1670/0x1670
 ? security_file_ioctl+0x50/0x90
 __x64_sys_ioctl+0x127/0x190
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7ff3a970731b
Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b
RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004
RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000
 </TASK>
irq event stamp: 10213
hardirqs last  enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70
hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70
softirqs last  enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
---[ end trace 0000000000000000 ]---

[2]

drivers/md/dm.c +9 -7
index 7e3b5bdcf520..50382c98d7b3 100644
@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	struct dm_io *io;
 	blk_status_t error = BLK_STS_OK;
 
+	if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
+	}
+
 	init_clone_info(&ci, md, map, bio);
 	io = ci.io;
 
@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;
-	} else if (unlikely(is_abnormal_io(bio))) {
-		/*
-		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
-		 * otherwise associated queue_limits won't be imposed.
-		 */
-		blk_queue_split(&bio);
-		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);
Signature · 7 lines
Best Regards,
Shin'ichiro Kawasaki

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#23 lore raw

Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing

Quoted 181 lines · Shinichiro Kawasaki
Quoted 21 lines · Mike Snitzer

Read/write/flush are the most common operations, optimize switch in is_abnormal_io() for those cases. Follows same pattern established in block perf-wip commit ("block: optimise blk_may_split for normal rw")

Also, push is_abnormal_io() check and blk_queue_split() down from dm_submit_bio() to dm_split_and_process_bio() and set new 'is_abnormal_io' flag in clone_info. Optimize __split_and_process_bio and __process_abnormal_io by leveraging ci.is_abnormal_io flag.

---
 drivers/md/dm.c | 60 +++++++++++++++++++++++++++++----------------------------
 1 file changed, 31 insertions(+), 29 deletions(-)
drivers/md/dm.c +0 -0
index 3b87d151ef88..b9c30dfe0f2a 100644
@@ -84,7 +84,8 @@ struct clone_info {
 	struct dm_io *io;
 	sector_t sector;

(snip)

Quoted 6 lines
drivers/md/dm.c +0 -0
 	ci->sector = bio->bi_iter.bi_sector;
 	ci->sector_count = bio_sectors(bio);
@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;

drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659

Quoted 10 lines
drivers/md/dm.c +7 -0
+	} else if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);

Hi Mike,

The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the blk_queue_split(&bio) for abnormal I/O is called after init_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning of dm_split_and_process_bio(), so that blk_queue_split() is called before init_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.

Quoted 22 lines
drivers/md/dm.c +0 -7
@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio)
 		goto out;
 	}
 
-	/*
-	 * Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc)
-	 * otherwise associated queue_limits won't be imposed.
-	 */
-	if (unlikely(is_abnormal_io(bio)))
-		blk_queue_split(&bio);
-
 	dm_split_and_process_bio(md, map, bio);
 out:
 	dm_put_live_table_bio(md, srcu_idx, bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

[1]

WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150
Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev
 intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs]
CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:bio_trim+0x10a/0x150
Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89
RSP: 0018:ffff8881977d7928 EFLAGS: 00010206
RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000
RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00
RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057
R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00
R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054
FS:  00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0
Call Trace:
 <TASK>
 dm_submit_bio+0x5fa/0x13b0
 ? dm_dax_direct_access+0x1c0/0x1c0
 ? lock_release+0x3a7/0x6c0
 ? lock_downgrade+0x6a0/0x6a0
 __submit_bio+0x1c2/0x2c0
 ? __bio_queue_enter+0x570/0x570
 submit_bio_noacct_nocheck+0x2f7/0x810
 ? should_fail_request+0x70/0x70
 ? rcu_read_lock_sched_held+0x3f/0x70
 ? __lock_acquire+0x1591/0x5030
 submit_bio+0x10a/0x270
 ? submit_bio_noacct+0x15c0/0x15c0
 submit_bio_wait+0xf2/0x1d0
 ? submit_bio_wait_endio+0x40/0x40
 ? lock_release+0x6c0/0x6c0
 blkdev_issue_discard+0xb5/0x110
 ? __blkdev_issue_discard+0x590/0x590
 ? truncate_bdev_range+0x15d/0x240
 blkdev_common_ioctl+0x853/0x1670
 ? blkdev_bszset+0x160/0x160
 ? __ia32_sys_readlinkat+0x87/0xf0
 ? __ia32_compat_sys_newlstat+0x70/0x70
 ? count_memcg_events.constprop.0+0x44/0x50
 blkdev_ioctl+0x23b/0x690
 ? blkdev_common_ioctl+0x1670/0x1670
 ? security_file_ioctl+0x50/0x90
 __x64_sys_ioctl+0x127/0x190
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7ff3a970731b
Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b
RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004
RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000
 </TASK>
irq event stamp: 10213
hardirqs last  enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70
hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70
softirqs last  enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
---[ end trace 0000000000000000 ]---

[2]

drivers/md/dm.c +9 -7
index 7e3b5bdcf520..50382c98d7b3 100644
@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	struct dm_io *io;
 	blk_status_t error = BLK_STS_OK;
 
+	if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
+	}
+
 	init_clone_info(&ci, md, map, bio);
 	io = ci.io;
 
@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;
-	} else if (unlikely(is_abnormal_io(bio))) {
-		/*
-		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
-		 * otherwise associated queue_limits won't be imposed.
-		 */
-		blk_queue_split(&bio);
-		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);
Signature · 3 lines
-- 
Best Regards,
Shin'ichiro Kawasaki

Thanks for your report, I'll sort it out but your patch doesn't seem right.

init_clone_info() will reset ci.is_abnormal_io so the bio won't be processed properly.

Mike

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#24 lore raw

Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing

Quoted 187 lines · Mike Snitzer
Quoted 181 lines · Shinichiro Kawasaki
Quoted 21 lines · Mike Snitzer

Read/write/flush are the most common operations, optimize switch in is_abnormal_io() for those cases. Follows same pattern established in block perf-wip commit ("block: optimise blk_may_split for normal rw")

Also, push is_abnormal_io() check and blk_queue_split() down from dm_submit_bio() to dm_split_and_process_bio() and set new 'is_abnormal_io' flag in clone_info. Optimize __split_and_process_bio and __process_abnormal_io by leveraging ci.is_abnormal_io flag.

---
 drivers/md/dm.c | 60 +++++++++++++++++++++++++++++----------------------------
 1 file changed, 31 insertions(+), 29 deletions(-)
drivers/md/dm.c +0 -0
index 3b87d151ef88..b9c30dfe0f2a 100644
@@ -84,7 +84,8 @@ struct clone_info {
 	struct dm_io *io;
 	sector_t sector;

(snip)

Quoted 6 lines
drivers/md/dm.c +0 -0
 	ci->sector = bio->bi_iter.bi_sector;
 	ci->sector_count = bio_sectors(bio);
@@ -1645,6 +1647,13 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;

drivers/md/dm.c static void dm_split_and_process_bio(struct mapped_device *md, · line 1659

Quoted 10 lines
drivers/md/dm.c +7 -0
+	} else if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);

Hi Mike,

The hunk above triggered a WARNING [1] which is observed at mkfs.xfs for dm- zoned devices. With the hunk, the blk_queue_split(&bio) for abnormal I/O is called after init_clone_info(). I think it made the cloned bio different from the split bio. I suggest to move the "if (unlikely(is_abnormal_io(bio)))" block at the beginning of dm_split_and_process_bio(), so that blk_queue_split() is called before init_clone_info(). I created a patch for such change [2], and confirmed the WARNING goes away.

Quoted 22 lines
drivers/md/dm.c +0 -7
@@ -1698,13 +1707,6 @@ static void dm_submit_bio(struct bio *bio)
 		goto out;
 	}
 
-	/*
-	 * Use blk_queue_split() for abnormal IO (e.g. discard, writesame, etc)
-	 * otherwise associated queue_limits won't be imposed.
-	 */
-	if (unlikely(is_abnormal_io(bio)))
-		blk_queue_split(&bio);
-
 	dm_split_and_process_bio(md, map, bio);
 out:
 	dm_put_live_table_bio(md, srcu_idx, bio);
Signature · 6 lines
2.15.0

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

[1]

WARNING: CPU: 3 PID: 79434 at block/bio.c:1629 bio_trim+0x10a/0x150
Modules linked in: dm_zoned null_blk f2fs crc32_generic dm_flakey iscsi_target_mod tcm_loop target_core_pscsi target_core_file target_core_iblock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw rfkill iptable_security ip_set target_core_user nfnetlink target_core_mod ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qrtr sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel at24 iTCO_wdt intel_pmc_bxt iTCO_vendor_support btrfs kvm irqbypass rapl raid6_pq intel_cstate zstd_compress joydev
 intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal lpc_ich ses enclosure ie31200_edac video zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper drm_kms_helper drm_ttm_helper ttm igb drm mpt3sas e1000e dca i2c_algo_bit raid_class scsi_transport_sas fuse [last unloaded: zonefs]
CPU: 3 PID: 79434 Comm: mkfs.xfs Kdump: loaded Not tainted 5.18.0-rc3+ #2
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:bio_trim+0x10a/0x150
Code: 8d 7c 24 68 48 89 fa 48 c1 ea 03 80 3c 02 00 75 58 49 83 7c 24 68 00 74 13 48 83 c4 10 4c 89 e7 5d 41 5c 41 5d e9 f6 aa 0f 00 <0f> 0b 48 83 c4 10 5d 41 5c 41 5d c3 4c 89 ef 48 89 54 24 08 48 89
RSP: 0018:ffff8881977d7928 EFLAGS: 00010206
RAX: 0000000008000000 RBX: ffff88852b000000 RCX: 0000000000040000
RDX: 00000000003c0000 RSI: 0000000000040000 RDI: ffff8883215a3d00
RBP: 0000000000400000 R08: 0000000000000001 R09: ffff8881124b9057
R10: ffffed102249720a R11: 0000000000000001 R12: ffff8883215a3d00
R13: ffff8883215a3d28 R14: ffff8881124b9098 R15: ffff8881124b9054
FS:  00007ff3a993dbc0(0000) GS:ffff8886ecf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a1bf241900 CR3: 0000000177074002 CR4: 00000000001706e0
Call Trace:
 <TASK>
 dm_submit_bio+0x5fa/0x13b0
 ? dm_dax_direct_access+0x1c0/0x1c0
 ? lock_release+0x3a7/0x6c0
 ? lock_downgrade+0x6a0/0x6a0
 __submit_bio+0x1c2/0x2c0
 ? __bio_queue_enter+0x570/0x570
 submit_bio_noacct_nocheck+0x2f7/0x810
 ? should_fail_request+0x70/0x70
 ? rcu_read_lock_sched_held+0x3f/0x70
 ? __lock_acquire+0x1591/0x5030
 submit_bio+0x10a/0x270
 ? submit_bio_noacct+0x15c0/0x15c0
 submit_bio_wait+0xf2/0x1d0
 ? submit_bio_wait_endio+0x40/0x40
 ? lock_release+0x6c0/0x6c0
 blkdev_issue_discard+0xb5/0x110
 ? __blkdev_issue_discard+0x590/0x590
 ? truncate_bdev_range+0x15d/0x240
 blkdev_common_ioctl+0x853/0x1670
 ? blkdev_bszset+0x160/0x160
 ? __ia32_sys_readlinkat+0x87/0xf0
 ? __ia32_compat_sys_newlstat+0x70/0x70
 ? count_memcg_events.constprop.0+0x44/0x50
 blkdev_ioctl+0x23b/0x690
 ? blkdev_common_ioctl+0x1670/0x1670
 ? security_file_ioctl+0x50/0x90
 __x64_sys_ioctl+0x127/0x190
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7ff3a970731b
Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d dd 2a 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffd79c8db68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00000000e8000000 RCX: 00007ff3a970731b
RDX: 00007ffd79c8db80 RSI: 0000000000001277 RDI: 0000000000000004
RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffd79c8d997
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd79c8db80
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000080000000
 </TASK>
irq event stamp: 10213
hardirqs last  enabled at (10223): [<ffffffffaa36e02e>] __up_console_sem+0x5e/0x70
hardirqs last disabled at (10232): [<ffffffffaa36e013>] __up_console_sem+0x43/0x70
softirqs last  enabled at (10190): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
softirqs last disabled at (10185): [<ffffffffaa1faf06>] __irq_exit_rcu+0x1c6/0x260
---[ end trace 0000000000000000 ]---

[2]

drivers/md/dm.c +9 -7
index 7e3b5bdcf520..50382c98d7b3 100644
@@ -1639,6 +1639,15 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 	struct dm_io *io;
 	blk_status_t error = BLK_STS_OK;
 
+	if (unlikely(is_abnormal_io(bio))) {
+		/*
+		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+		 * otherwise associated queue_limits won't be imposed.
+		 */
+		blk_queue_split(&bio);
+		ci.is_abnormal_io = true;
+	}
+
 	init_clone_info(&ci, md, map, bio);
 	io = ci.io;
 
@@ -1646,13 +1655,6 @@ static void dm_split_and_process_bio(struct mapped_device *md,
 		__send_empty_flush(&ci);
 		/* dm_io_complete submits any data associated with flush */
 		goto out;
-	} else if (unlikely(is_abnormal_io(bio))) {
-		/*
-		 * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
-		 * otherwise associated queue_limits won't be imposed.
-		 */
-		blk_queue_split(&bio);
-		ci.is_abnormal_io = true;
 	}
 
 	error = __split_and_process_bio(&ci);
Signature · 3 lines
-- 
Best Regards,
Shin'ichiro Kawasaki

Thanks for your report, I'll sort it out but your patch doesn't seem right.

Quoted 2 lines

init_clone_info() will reset ci.is_abnormal_io so the bio won't be processed properly.

FYI, I will be pushing a rebased commit with the following patch folded in:

drivers/md/dm.c +1 -1
index 7e3b5bdcf520..9650ba2075b8 100644
@@ -1613,12 +1613,12 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci)
 }
 
 static void init_clone_info(struct clone_info *ci, struct mapped_device *md,
-			    struct dm_table *map, struct bio *bio)
+			    struct dm_table *map, struct bio *bio, bool is_abnormal)
 {
	ci->map = map;
	ci->io = alloc_io(md, bio);
	ci->bio = bio;
-	ci->is_abnormal_io = false;
+	ci->is_abnormal_io = is_abnormal;
	ci->submit_as_polled = false;
	ci->sector = bio->bi_iter.bi_sector;
	ci->sector_count = bio_sectors(bio);
@@ -1638,21 +1638,24 @@ static void dm_split_and_process_bio(struct mapped_device *md,
	struct clone_info ci;
	struct dm_io *io;
	blk_status_t error = BLK_STS_OK;

+ bool is_abnormal;

- init_clone_info(&ci, md, map, bio);
+ is_abnormal = is_abnormal_io(bio);
+ if (unlikely(is_abnormal)) {
+ /*
+ * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
+ * otherwise associated queue_limits won't be imposed.
+ */
+ blk_queue_split(&bio);
+ }
+

+	init_clone_info(&ci, md, map, bio, is_abnormal);
	io = ci.io;

	if (bio->bi_opf & REQ_PREFLUSH) {
		__send_empty_flush(&ci);
		/* dm_io_complete submits any data associated with flush */
		goto out;

- } else if (unlikely(is_abnormal_io(bio))) {
- /*
- * Use blk_queue_split() for abnormal IO (e.g. discard, etc)
- * otherwise associated queue_limits won't be imposed.
- */
- blk_queue_split(&bio);

-		ci.is_abnormal_io = true;
	}

	error = __split_and_process_bio(&ci);

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#25 lore raw

Re: [dm-devel] [dm-5.19 PATCH 21/21] dm: improve abnormal bio processing

Quoted 2 lines · Mike Snitzer

On Fri, Apr 22 2022 at 9:13P -0400,
Mike Snitzer <[email protected]> wrote:

[...]

Quoted 4 lines
Quoted 4 lines

Thanks for your report, I'll sort it out but your patch doesn't seem right.

init_clone_info() will reset ci.is_abnormal_io so the bio won't be processed properly.

Right, I missed that point.

drivers/md/dm.c

Quoted 6 lines

FYI, I will be pushing a rebased commit with the following patch folded in:

drivers/md/dm.c +0 -0
index 7e3b5bdcf520..9650ba2075b8 100644

[...]

I tested the dm-5.19 branch at git hash e7ea9556985f including the fix above, and observed it passed all of my test sets. Thanks!

Signature · 7 lines
Best Regards,
Shin'ichiro Kawasaki

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#26 lore raw
In reply to Mike Snitzer
Quoted 7 lines · Mike Snitzer

Hi,

This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)

Mike, thanks for posting. We will run this on zoned stuff to check. Note that patches 13 to 20 are empty...

Signature · 7 lines
Damien Le Moal
Western Digital Research

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
author maintainer
#27 lore raw
Quoted 10 lines · Damien Le Moal
Quoted 7 lines · Mike Snitzer

Hi,

This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)

Mike, thanks for posting. We will run this on zoned stuff to check.

OK, I appreciate it..

Quoted 1 line

Note that patches 13 to 20 are empty...

Not sure what's going on there... basically any patch that wasn't from me (so 1, 13-19) isn't showing up in patchwork or the dm-devel archive.

This is my first submission of patches using my [email protected] alias with redhat's smtp server that is really anal about not sending email that isn't from me. Yet it sent the email but seized on the following to have the body be empty:
"From: Ming Lei <[email protected]>"

Best part is my email account shows all the emails sent like all was perfect. Sorting this out will have to wait until Tuesday.

Sorry, but please just pull the dm-5.19 branch via git ;)

Thanks,
Mike

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

maintainer
#28 lore raw
Quoted 14 lines · Mike Snitzer
Quoted 10 lines · Damien Le Moal
Quoted 7 lines · Mike Snitzer

Hi,

This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)

Mike, thanks for posting. We will run this on zoned stuff to check.

Quoted 7 lines

OK, I appreciate it..

Quoted 1 line

Note that patches 13 to 20 are empty...

Not sure what's going on there... basically any patch that wasn't from me (so 1, 13-19) isn't showing up in patchwork or the dm-devel archive.

The patches I received look normal, fwiw.

Signature · 6 lines
Jens Axboe

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
maintainer Applied
#29 lore raw

Re: [dm-devel] (subset) [dm-5.19 PATCH 00/21] dm: changes staged for 5.19

In reply to Mike Snitzer
Quoted 10 lines · Mike Snitzer

This patchset reflects what I've staged in linux-dm.git's "dm-5.19" branch (also staged in "for-next" for linux-next):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

It build's on jens/for-5.19/block branch (which is based on v5.18-rc3)

I can still make changes or add Reviewed-by:s, etc. So please feel free to review.

[...]

Applied, thanks!

[01/21] block: change exported IO accounting interface from gendisk to bdev

        commit: 5f0614a55ecebdf55f1a17db0b5f6b787ed009f1

Best regards,

Signature · 7 lines
Jens Axboe


--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel