Skip to contentSkip to conversation

Block storage copy offloading

PATCH In review Bart Van Assche 2 participants 15 messages Apr 24 – May 22

Subsystems (7)
Documentation VFS SCSI Block layer +3 more subsystems
lore
Share & tools
lore
#0 lore raw

cover Block storage copy offloading

Hi Jens,

This patch series implements a new approach for copy offloading. Compared to Nitesh' approach, the differences are as follows:
- Two new limits have been introduced representing the maximum number of source and destination ranges. Support for multiple source ranges for the NVMe Copy command has been added.
- The blkdev_copy_offload() function can now submit multiple copy offload commands instead of only one.
- The implementation no longer depends on block layer plugging.

This patch series includes copy offloading support for the Linux kernel block layer core, the device mapper core, the null_blk and the NVMe and nvmet drivers. Support for the scsi_debug and SCSI core will follow later.

Test scripts are available here:
https://github.com/bvanassche/blktests/tree/copy-offloading

See also Bart Van Assche, [LSF/MM/BPF TOPIC] Block storage copy offloading, January 2026
(https://lore.kernel.org/all/[email protected]/).

See also Nitesh Shetty, Implement copy offload support, May 2024
(https://lore.kernel.org/linux-block/[email protected]/).

Anuj Gupta (1):
fs/read_write: Generalize generic_copy_file_checks()

Bart Van Assche (4):
block: Introduce blkdev_copy_offload()
block: Introduce accessor functions for copy offload bios
dm: Add support for copy offloading
dm-linear: Enable copy offloading

Nitesh Shetty (7):
block: Introduce queue limits for copy offloading
block: Add the REQ_OP_COPY_{SRC,DST} operations
block: Add an onloaded copy implementation
fs, block: Add copy_file_range() support for block devices
nvme: Add copy offloading support
nvmet: Support the Copy command
null_blk: Add support for REQ_OP_COPY_*

 Documentation/ABI/stable/sysfs-block |  24 +
 Documentation/block/null_blk.rst     |   4 +
 block/Makefile                       |   2 +-
 block/bio.c                          |   1 +
 block/blk-copy.c                     | 631 +++++++++++++++++++++++++++
 block/blk-core.c                     |  38 ++
 block/blk-merge.c                    |  13 +
 block/blk-settings.c                 |  36 ++
 block/blk-sysfs.c                    |  35 ++
 block/blk.h                          |   5 +
 block/fops.c                         |  54 +++
 drivers/block/null_blk/main.c        | 113 +++++
 drivers/block/null_blk/null_blk.h    |   1 +
 drivers/md/dm-linear.c               |   6 +
 drivers/md/dm-table.c                |   8 +
 drivers/nvme/host/constants.c        |   1 +
 drivers/nvme/host/core.c             | 106 +++++
 drivers/nvme/host/trace.c            |  21 +-
 drivers/nvme/target/admin-cmd.c      |  26 +-
 drivers/nvme/target/io-cmd-bdev.c    |  80 ++++
 drivers/nvme/target/io-cmd-file.c    |  59 ++-
 drivers/nvme/target/trace.c          |  19 +
 fs/read_write.c                      |   8 +-
 include/linux/blk-copy.h             |  50 +++
 include/linux/blk_types.h            |  57 +++
 include/linux/blkdev.h               |  20 +-
 include/linux/nvme.h                 |  47 +-
 27 files changed, 1449 insertions(+), 16 deletions(-)
 create mode 100644 block/blk-copy.c
 create mode 100644 include/linux/blk-copy.h
#1 lore raw

PATCH 1/12 block: Introduce queue limits for copy offloading

From: Nitesh Shetty <[email protected]>

Add the following request queue limits:
- max_copy_hw_sectors: the maximum number of sectors supported by the block driver for a single offloaded copy operation.
- max_copy_src_segments: the maximum number of source segments supported by the block driver for a single offloaded copy operation.
- max_copy_dst_segments: the maximum number of destination segments supported by the block driver for a single offloaded copy operation.
- max_user_copy_sectors: the maximum number of sectors configured by the user for a single offloaded copy operation.
- max_copy_sectors: the maximum number of sectors for a single offloaded copy operation. This is the minimum of the above two parameters.

The default value for all these new limits is zero which means that copy offloading is not supported unless if these limits are set by the block driver.

ake the following two limits available in sysfs:
- copy_max_bytes (RW)
- copy_max_hw_bytes (RO)

These limits will be used by the function that implements copy offloading to decide the bio size.

[ bvanassche: Added max_copy_{src,dst}_segments limits. Introduced
blk_validate_copy_limits(). Introduced BLK_FEAT_STACKING_COPY_OFFL. Modified patch description. ]

---
 Documentation/ABI/stable/sysfs-block | 24 +++++++++++++++++++
 block/blk-settings.c                 | 36 ++++++++++++++++++++++++++++
 block/blk-sysfs.c                    | 35 +++++++++++++++++++++++++++
 include/linux/blkdev.h               | 18 +++++++++++++-
 4 files changed, 112 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-block +24 -0
index 900b3fc4c72d..bec5e04085da 100644
@@ -239,6 +239,30 @@ Description:
 		last zone of the device which may be smaller.
 
 
+What:		/sys/block/<disk>/queue/copy_max_bytes
+Date:		May 2026
+Contact:	[email protected]
+Description:
+		[RW] This is the maximum number of bytes that the block layer
+		will allow for a copy request. This is always smaller or
+		equal to the maximum size allowed by the block driver.
+		Any value higher than 'copy_max_hw_bytes' will be reduced to
+		'copy_max_hw_bytes'. Writing '0' to this attribute will disable
+		copy offloading for this block device. If copy offloading is
+		disabled, copy requests will be translated into read and write
+		requests.
+
+
+What:		/sys/block/<disk>/queue/copy_max_hw_bytes
+Date:		May 2026
+Contact:	[email protected]
+Description:
+		[RO] This is the maximum number of bytes that is allowed for
+		a single data copy request. Set by the block driver. The value
+		zero indicates that the block device does not support copy
+		offloading.
+
+
 What:		/sys/block/<disk>/queue/crypto/
 Date:		February 2022
 Contact:	[email protected]
block/blk-settings.c +36 -0
index 78c83817b9d3..cb846ff2926e 100644
@@ -57,6 +57,11 @@ void blk_set_stacking_limits(struct queue_limits *lim)
 	lim->max_hw_zone_append_sectors = UINT_MAX;
 	lim->max_user_discard_sectors = UINT_MAX;
 	lim->atomic_write_hw_max = UINT_MAX;
+
+	lim->max_user_copy_sectors = UINT_MAX;
+	lim->max_copy_hw_sectors = UINT_MAX;
+	lim->max_copy_src_segments = U16_MAX;
+	lim->max_copy_dst_segments = U16_MAX;
 }
 EXPORT_SYMBOL(blk_set_stacking_limits);
 
@@ -333,6 +338,21 @@ static void blk_validate_atomic_write_limits(struct queue_limits *lim)
 	lim->atomic_write_unit_max = 0;
 }
 
+/*
+ * Check whether max_copy_hw_sectors and max_copy_{src,dst}_segments are
+ * either all nonzero or all zero.
+ */
+static int blk_validate_copy_limits(const struct queue_limits *lim)
+{
+	if (lim->max_copy_hw_sectors && lim->max_copy_src_segments &&
+	    lim->max_copy_dst_segments)
+		return 0;
+	if (!lim->max_copy_hw_sectors && !lim->max_copy_src_segments &&
+	    !lim->max_copy_dst_segments)
+		return 0;
+	return -EINVAL;
+}
+
 /*
  * Check that the limits in lim are valid, initialize defaults for unset
  * values, and cap values based on others where needed.
@@ -510,6 +530,13 @@ int blk_validate_limits(struct queue_limits *lim)
 	err = blk_validate_integrity_limits(lim);
 	if (err)
 		return err;
+
+	err = blk_validate_copy_limits(lim);
+	if (err)
+		return err;
+	lim->max_copy_sectors =
+		min(lim->max_copy_hw_sectors, lim->max_user_copy_sectors);
+
 	return blk_validate_zoned_limits(lim);
 }
 EXPORT_SYMBOL_GPL(blk_validate_limits);
@@ -528,6 +555,7 @@ int blk_set_default_limits(struct queue_limits *lim)
 	 */
 	lim->max_user_discard_sectors = UINT_MAX;
 	lim->max_user_wzeroes_unmap_sectors = UINT_MAX;
+	lim->max_user_copy_sectors = UINT_MAX;
 	return blk_validate_limits(lim);
 }
 
@@ -829,6 +857,14 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
 	t->max_segment_size = min_not_zero(t->max_segment_size,
 					   b->max_segment_size);
 
+	t->max_copy_hw_sectors =
+		min(t->max_copy_hw_sectors, b->max_copy_hw_sectors);
+	t->max_copy_src_segments =
+		min(t->max_copy_src_segments, b->max_copy_src_segments);
+	t->max_copy_dst_segments =
+		min(t->max_copy_dst_segments, b->max_copy_dst_segments);
+	t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors);
+
 	alignment = queue_limit_alignment_offset(b, start);
 
 	/* Bottom device has different alignment.  Check that it is
block/blk-sysfs.c +35 -0
index f22c1f253eb3..8e1e14d1682d 100644
@@ -325,6 +325,36 @@ queue_max_sectors_store(struct gendisk *disk, const char *page, size_t count,
 	return 0;
 }
 
+static ssize_t queue_copy_hw_max_show(struct gendisk *disk, char *page)
+{
+	return queue_var_show(
+		disk->queue->limits.max_copy_hw_sectors << SECTOR_SHIFT, page);
+}
+
+static ssize_t queue_copy_max_show(struct gendisk *disk, char *page)
+{
+	return queue_var_show(
+		disk->queue->limits.max_copy_sectors << SECTOR_SHIFT, page);
+}
+
+static int queue_copy_max_store(struct gendisk *disk, const char *page,
+				size_t count, struct queue_limits *lim)
+{
+	unsigned long max_copy_bytes;
+	ssize_t ret;
+
+	ret = queue_var_store(&max_copy_bytes, page, count);
+	if (ret < 0)
+		return ret;
+
+	if ((max_copy_bytes >> SECTOR_SHIFT) > UINT_MAX)
+		return -EINVAL;
+
+	lim->max_user_copy_sectors = max_copy_bytes >> SECTOR_SHIFT;
+
+	return 0;
+}
+
 static ssize_t queue_feature_store(struct gendisk *disk, const char *page,
 		size_t count, struct queue_limits *lim, blk_features_t feature)
 {
@@ -652,6 +682,9 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones");
 QUEUE_LIM_RO_ENTRY(queue_max_open_zones, "max_open_zones");
 QUEUE_LIM_RO_ENTRY(queue_max_active_zones, "max_active_zones");
 
+QUEUE_LIM_RO_ENTRY(queue_copy_hw_max, "copy_max_hw_bytes");
+QUEUE_LIM_RW_ENTRY(queue_copy_max, "copy_max_bytes");
+
 QUEUE_RW_ENTRY(queue_nomerges, "nomerges");
 QUEUE_LIM_RW_ENTRY(queue_iostats_passthrough, "iostats_passthrough");
 QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity");
@@ -760,6 +793,8 @@ static const struct attribute *const queue_attrs[] = {
 	&queue_max_hw_wzeroes_unmap_sectors_entry.attr,
 	&queue_max_wzeroes_unmap_sectors_entry.attr,
 	&queue_max_zone_append_sectors_entry.attr,
+	&queue_copy_hw_max_entry.attr,
+	&queue_copy_max_entry.attr,
 	&queue_zone_write_granularity_entry.attr,
 	&queue_rotational_entry.attr,
 	&queue_zoned_entry.attr,
include/linux/blkdev.h +17 -1
index 890128cdea1c..8ae64cc0546f 100644
@@ -353,13 +353,17 @@ typedef unsigned int __bitwise blk_features_t;
 #define BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE \
 	((__force blk_features_t)(1u << 15))
 
+/* block driver is a stacking block driver that supports copy offloading */
+#define BLK_FEAT_STACKING_COPY_OFFL	((__force blk_features_t)(1u << 16))
+
 /*
  * Flags automatically inherited when stacking limits.
  */
 #define BLK_FEAT_INHERIT_MASK \
 	(BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA | BLK_FEAT_ROTATIONAL | \
 	 BLK_FEAT_STABLE_WRITES | BLK_FEAT_ZONED | \
-	 BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE)
+	 BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE | \
+	 BLK_FEAT_STACKING_COPY_OFFL)
 
 /* internal flags in queue_limits.flags */
 typedef unsigned int __bitwise blk_flags_t;
@@ -415,6 +419,13 @@ struct queue_limits {
 	unsigned int		atomic_write_hw_unit_max;
 	unsigned int		atomic_write_unit_max;
 
+	/* copy offloading limits */
+	unsigned int		max_copy_hw_sectors;	/* set by block driver*/
+	uint16_t		max_copy_src_segments;	/* set by block driver*/
+	uint16_t		max_copy_dst_segments;	/* set by block driver*/
+	unsigned int		max_user_copy_sectors;	/* set via sysfs */
+	unsigned int		max_copy_sectors;	/* min() of the above */
+
 	unsigned short		max_segments;
 	unsigned short		max_integrity_segments;
 	unsigned short		max_discard_segments;
@@ -1454,6 +1465,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev)
 	return bdev_limits(bdev)->discard_granularity;
 }
 
+static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev)
+{
+	return bdev_get_queue(bdev)->limits.max_copy_sectors;
+}
+
 static inline unsigned int
 bdev_max_secure_erase_sectors(struct block_device *bdev)
 {
#2 lore raw

PATCH 2/12 block: Add the REQ_OP_COPY_{SRC,DST} operations

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

Introduce the REQ_OP_COPY_SRC and REQ_OP_COPY_DST operations. The source and destination LBA range information is in separate bios because any other approach would require a rewrite of the device mapper. These bios are associated with each other via the new bi_copy_ctx pointer. A new pointer has been introduced because the copy offloading context information must be preserved when cloning a bio and the bi_private bio member must not be copied when cloning a bio.

This patch supports the following approach for copy offloading:
1. Allocate a struct bio_copy_offload_ctx instance and set phase to BLKDEV_TRANSLATE_LBAS.
2. Allocate REQ_OP_COPY_SRC and REQ_OP_COPY_DST bios. Set the bi_copy_ctx member of these bios.
3. Set the bio_count member of struct bio_copy_offload_ctx.
4. Submit all REQ_OP_COPY_* bios.
5. In submit_bio(), do the following for REQ_OP_COPY_* bios:
- If bio->bi_bdev is a stacking device, submit the bio. This will send the bio to the device mapper. The device mapper will clone the bio, translate the LBAs and will submit the cloned bio. That will result in a recursive submit_bio() call.
- If bio->bi_bdev is not a stacking device, add the bio to the

     copy_ctx->bios list and decrement copy_ctx->bio_count.

6. Once copy_ctx->bio_count == 0, call copy_ctx->translation_complete().
7. In the implementation of copy_ctx->translation_complete(), change copy_ctx->phase from BLKDEV_TRANSLATE_LBAS into BLKDEV_COPY.
8. Submit the first REQ_OP_COPY_* bio of the copy_ctx->bios list.
9. Once this bio reaches the block driver associated with the bio, retrieve the other bios involved in the copy operation from the copy context data structure and convert all these bios into a copy offload operation.
10. Once this bio completes, also complete all the other bios involved in the copy offload operation.

This patch increases the size of struct bio from 104 to 112 bytes on 64-bit systems.

To be discussed further: whether adding a new member in struct bio is acceptable or whether the new pointer perhaps should be stored in front of the bio. bioset_init() supports front padding.

[ bvanassche: changed the approach of this patch from combining the
COPY_SRC and COPY_DST operations immediately to translating the LBA information first. ]

---
 block/bio.c               |  1 +
 block/blk-core.c          | 38 ++++++++++++++++++++++++++++++++
 block/blk-merge.c         | 13 +++++++++++
 block/blk.h               |  5 +++++
 include/linux/blk-copy.h  | 46 +++++++++++++++++++++++++++++++++++++++
 include/linux/blk_types.h | 17 +++++++++++++++
 6 files changed, 120 insertions(+)
 create mode 100644 include/linux/blk-copy.h
block/bio.c +1 -0
index b8972dba68a0..51480c9be27b 100644
@@ -852,6 +852,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 	bio->bi_write_hint = bio_src->bi_write_hint;
 	bio->bi_write_stream = bio_src->bi_write_stream;
 	bio->bi_iter = bio_src->bi_iter;
+	bio->bi_copy_ctx = bio_src->bi_copy_ctx;
 
 	if (bio->bi_bdev) {
 		if (bio->bi_bdev == bio_src->bi_bdev &&
block/blk-core.c +38 -0
index 17450058ea6d..37c01e717202 100644
@@ -16,6 +16,7 @@
 #include <linux/module.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
+#include <linux/blk-copy.h>
 #include <linux/blk-pm.h>
 #include <linux/blk-integrity.h>
 #include <linux/highmem.h>
@@ -108,6 +109,8 @@ static const char *const blk_op_name[] = {
 	REQ_OP_NAME(ZONE_FINISH),
 	REQ_OP_NAME(ZONE_APPEND),
 	REQ_OP_NAME(WRITE_ZEROES),
+	REQ_OP_NAME(COPY_SRC),
+	REQ_OP_NAME(COPY_DST),
 	REQ_OP_NAME(DRV_IN),
 	REQ_OP_NAME(DRV_OUT),
 };
@@ -782,6 +785,8 @@ void submit_bio_noacct(struct bio *bio)
 	struct block_device *bdev = bio->bi_bdev;
 	struct request_queue *q = bdev_get_queue(bdev);
 	blk_status_t status = BLK_STS_IOERR;
+	struct bio_copy_offload_ctx *copy_ctx;
+	u32 bio_count;
 
 	might_sleep();
 
@@ -875,6 +880,39 @@ void submit_bio_noacct(struct bio *bio)
 		 * requests.
 		 */
 		fallthrough;
+	case REQ_OP_COPY_SRC:
+	case REQ_OP_COPY_DST:
+		copy_ctx = bio->bi_copy_ctx;
+		WARN_ON_ONCE(copy_ctx->phase == BLKDEV_COPY_DONE);
+		if (copy_ctx->phase == BLKDEV_COPY)
+			break;
+		/* If copy offloading is not supported, fail the bio. */
+		if (!q->limits.max_copy_sectors) {
+			scoped_guard(spinlock_irqsave, &copy_ctx->lock)
+				copy_ctx->bio_count--;
+			goto not_supported;
+		}
+		/*
+		 * If the block driver is a stacking driver that supports copy
+		 * offloading, submit the bio.
+		 */
+		if (q->limits.features & BLK_FEAT_STACKING_COPY_OFFL)
+			break;
+		/*
+		 * Append the bio at the end of the bio->bi_copy_ctx->bios list.
+		 */
+		scoped_guard(spinlock_irqsave, &copy_ctx->lock) {
+			if (copy_ctx->biotail)
+				copy_ctx->biotail->bi_next = bio;
+			else
+				copy_ctx->bios = bio;
+			copy_ctx->biotail = bio;
+			bio_count = --copy_ctx->bio_count;
+		}
+		WARN_ON_ONCE(bio_count < 0);
+		if (bio_count == 0)
+			copy_ctx->translation_complete(copy_ctx);
+		return;
 	default:
 		goto not_supported;
 	}
block/blk-merge.c +13 -0
index fcf09325b22e..4678131650d2 100644
@@ -207,6 +207,19 @@ struct bio *bio_split_discard(struct bio *bio, const struct queue_limits *lim,
 	return __bio_split_discard(bio, lim, nsegs, max_sectors);
 }
 
+struct bio *bio_split_copy(struct bio *bio, const struct queue_limits *lim,
+			   unsigned int *nsegs)
+{
+	*nsegs = 1;
+	if (bio_sectors(bio) <= lim->max_copy_sectors)
+		return bio;
+
+	/* Splitting a REQ_OP_COPY_* bio is not supported. */
+	bio->bi_status = BLK_STS_NOTSUPP;
+	bio_endio(bio);
+	return NULL;
+}
+
 static inline unsigned int blk_boundary_sectors(const struct queue_limits *lim,
 						bool is_atomic)
 {
block/blk.h +5 -0
index b998a7761faf..274c226e87ee 100644
@@ -379,6 +379,8 @@ struct bio *bio_split_rw(struct bio *bio, const struct queue_limits *lim,
 		unsigned *nr_segs);
 struct bio *bio_split_zone_append(struct bio *bio,
 		const struct queue_limits *lim, unsigned *nr_segs);
+struct bio *bio_split_copy(struct bio *bio, const struct queue_limits *lim,
+			   unsigned int *nsegs);
 
 /*
  * All drivers must accept single-segments bios that are smaller than PAGE_SIZE.
@@ -435,6 +437,9 @@ static inline struct bio *__bio_split_to_limits(struct bio *bio,
 		return bio_split_discard(bio, lim, nr_segs);
 	case REQ_OP_WRITE_ZEROES:
 		return bio_split_write_zeroes(bio, lim, nr_segs);
+	case REQ_OP_COPY_SRC:
+	case REQ_OP_COPY_DST:
+		return bio_split_copy(bio, lim, nr_segs);
 	default:
 		/* other operations can't be split */
 		*nr_segs = 0;
include/linux/blk-copy.h new file +46 -0

46 lines not on this page · raw message

include/linux/blk_types.h +17 -0
index 8808ee76e73c..4e448e810b87 100644
@@ -284,6 +284,8 @@ struct bio {
 	atomic_t		__bi_cnt;	/* pin count */
 
 	struct bio_set		*bi_pool;
+
+	void			*bi_copy_ctx;
 };
 
 #define BIO_RESET_BYTES		offsetof(struct bio, bi_max_vecs)
@@ -370,6 +372,10 @@ enum req_op {
 	/** @REQ_OP_ZONE_RESET_ALL: reset all the zone present on the device */
 	REQ_OP_ZONE_RESET_ALL	= (__force blk_opf_t)19,
 
+	/* copy offload source and destination operations */
+	REQ_OP_COPY_SRC		= (__force blk_opf_t)20,
+	REQ_OP_COPY_DST		= (__force blk_opf_t)21,
+
 	/* Driver private requests */
 	/* private: */
 	REQ_OP_DRV_IN		= (__force blk_opf_t)34,
@@ -461,6 +467,17 @@ static inline bool op_is_write(blk_opf_t op)
 	return !!(op & (__force blk_opf_t)1);
 }
 
+static inline bool op_is_copy(blk_opf_t op)
+{
+	switch (op & REQ_OP_MASK) {
+	case REQ_OP_COPY_DST:
+	case REQ_OP_COPY_SRC:
+		return true;
+	default:
+		return false;
+	}
+}
+
 /*
  * Check if the bio or request is one that needs special treatment in the
  * flush state machine.
#3 lore raw

PATCH 3/12 block: Introduce blkdev_copy_offload()

In reply to Bart Van Assche

Introduce blkdev_copy_offload() for performing copy offloading. This function implements the algorithm explained the description of the previous patch. If the input parameters exceed what can be supported with a single copy offload operation, multiple copy offload operations are submitted.

---
 block/Makefile            |   2 +-
 block/blk-copy.c          | 355 ++++++++++++++++++++++++++++++++++++++
 include/linux/blk_types.h |  40 +++++
 include/linux/blkdev.h    |   1 +
 4 files changed, 397 insertions(+), 1 deletion(-)
 create mode 100644 block/blk-copy.c
block/Makefile +1 -1
index 7dce2e44276c..d99e8d4fda7d 100644
@@ -6,7 +6,7 @@
 obj-y		:= bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
 			blk-flush.o blk-settings.o blk-ioc.o blk-map.o \
 			blk-merge.o blk-timeout.o blk-lib.o blk-mq.o \
-			blk-mq-tag.o blk-mq-dma.o blk-stat.o \
+			blk-mq-tag.o blk-mq-dma.o blk-stat.o blk-copy.o \
 			blk-mq-sysfs.o blk-mq-cpumap.o blk-mq-sched.o ioctl.o \
 			genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \
 			disk-events.o blk-ia-ranges.o early-lookup.o
block/blk-copy.c new file +355 -0

355 lines not on this page · raw message

include/linux/blk_types.h +40 -0
index 4e448e810b87..27a0f92fc2cb 100644
@@ -535,4 +535,44 @@ struct blk_rq_stat {
 	u64 batch;
 };
 
+/* A single input or output segment descriptor. */
+struct blk_copy_seg {
+	loff_t pos;
+	loff_t len;
+};
+
+/**
+ * struct blk_copy_params - input parameters and internal parameters for copy
+ *	operations.
+ * @in_bdev: Input block device.
+ * @in_segs: Input LBA ranges.
+ * @in_nseg: Number of elements in @in_segs.
+ * @out_bdev: Output block device.
+ * @out_segs: Output LBA ranges.
+ * @out_nseg: Number of elements in @out_segs.
+ * @end_io: Called after copying data finished. If %NULL, copying data happens
+ *	synchronously instead of asynchronously.
+ * @private: May be used by @end_io. Not used directly.
+ * @len: Total number of bytes to copy. Set by blkdev_copy_offload() or
+ *	blkdev_copy_onload().
+ * @copy_ctxs: Number of in-flight copy contexts associated with copy offload
+ *	operations.
+ * @lock: Protects @status updates.
+ * @status: I/O completion status.
+ */
+struct blk_copy_params {
+	struct block_device *in_bdev;
+	struct blk_copy_seg *in_segs;
+	unsigned int in_nseg;
+	struct block_device *out_bdev;
+	struct blk_copy_seg *out_segs;
+	unsigned int out_nseg;
+	void (*end_io)(const struct blk_copy_params *params);
+	void *private;
+	loff_t len;
+	atomic_t copy_ctx_count;
+	spinlock_t lock;
+	blk_status_t status;
+};
+
 #endif /* __LINUX_BLK_TYPES_H */
include/linux/blkdev.h +1 -0
index 8ae64cc0546f..fea296150cda 100644
@@ -1283,6 +1283,7 @@ void __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp_mask, struct bio **biop);
 int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp);
+int blkdev_copy_offload(struct blk_copy_params *params);
 
 #define BLKDEV_ZERO_NOUNMAP	(1 << 0)  /* do not free blocks */
 #define BLKDEV_ZERO_NOFALLBACK	(1 << 1)  /* don't write explicit zeroes */
#4 lore raw

PATCH 4/12 block: Add an onloaded copy implementation

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

For the devices which do not support copy offloading, add a function that copies data by submitting READ and WRITE operations.

Onloaded copying is implemented by reading from the source block device into memory and by writing this data to the destination block device.

---
 block/blk-copy.c       | 229 +++++++++++++++++++++++++++++++++++++++++
 include/linux/blkdev.h |   1 +
 2 files changed, 230 insertions(+)
block/blk-copy.c +229 -0
index 8ac8879442f7..459ed8581efc 100644
@@ -7,6 +7,26 @@
 #include <linux/blk-copy.h>
 #include <linux/blk-mq.h>
 
+/**
+ * Tracks the state of a single onloaded copy operation.
+ * @params: Data copy parameters.
+ * @read_work: For scheduling read work.
+ * @write_work: For scheduling write work.
+ * @buf: Data buffer.
+ * @buf_len: Length in bytes of @buf.
+ * @offset: Current copying offset. Range: [0, @len[.
+ * @chunk: Size in bytes of the chunk of data that is being copied.
+ */
+struct blkdev_copy_onload_ctx {
+	struct blk_copy_params *params;
+	struct work_struct read_work;
+	struct work_struct write_work;
+	void *buf;
+	ssize_t buf_len;
+	loff_t offset;
+	loff_t chunk;
+};
+
 /* End all bios in the @ctx->bios list with status @ctx->status. */
 static void blkdev_end_bios(struct bio_copy_offload_ctx *ctx)
 {
@@ -353,3 +373,212 @@ int blkdev_copy_offload(struct blk_copy_params *params)
 	return -EIOCBQUEUED;
 }
 EXPORT_SYMBOL_GPL(blkdev_copy_offload);
+
+static void *blkdev_copy_alloc_buf(size_t req_size, size_t *alloc_size)
+{
+	unsigned int min_size = PAGE_SIZE;
+	char *buf;
+
+	while (req_size >= min_size) {
+		buf = kmalloc(req_size, GFP_NOIO | __GFP_NOWARN);
+		if (buf) {
+			*alloc_size = req_size;
+			return buf;
+		}
+		req_size >>= 1;
+	}
+
+	return NULL;
+}
+
+static struct bio *bio_map_buf(void *buf, unsigned int len)
+{
+	struct page *page;
+	struct bio *bio;
+	static const uint16_t nr_vecs = 1;
+
+	bio = bio_kmalloc(nr_vecs, GFP_NOIO);
+	if (!bio)
+		return NULL;
+	bio_init_inline(bio, /*bdev=*/NULL, /*max_vecs=*/nr_vecs, /*opf=*/0);
+
+	page = virt_to_page(buf);
+	if (bio_add_page(bio, page, len, offset_in_page(buf)) < len) {
+		/* we don't support partial mappings */
+		bio_uninit(bio);
+		kfree(bio);
+		WARN_ON_ONCE(true);
+		return NULL;
+	}
+
+	return bio;
+}
+
+static void blkdev_write_done(struct bio *bio)
+{
+	struct blkdev_copy_onload_ctx *ctx = bio->bi_copy_ctx;
+	struct blk_copy_params *params = ctx->params;
+	blk_status_t sts = bio->bi_status;
+
+	kfree(bio);
+
+	if (sts) {
+		params->status = sts;
+		params->end_io(params);
+		return;
+	}
+
+	ctx->offset += ctx->chunk;
+
+	schedule_work(&ctx->read_work);
+}
+
+static sector_t blkdev_offset_to_out_pos(const struct blk_copy_params *params,
+					 loff_t offset)
+{
+	for (int i = 0; i < params->out_nseg; i++) {
+		loff_t rem = params->out_segs[i].len - offset;
+
+		if (rem > 0)
+			return params->out_segs[i].pos + offset;
+		offset -= params->out_segs[i].len;
+	}
+	return 0;
+}
+
+static void blkdev_write_work(struct work_struct *work)
+{
+	struct blkdev_copy_onload_ctx *ctx =
+		container_of(work, typeof(*ctx), read_work);
+	struct blk_copy_params *params = ctx->params;
+	struct bio *bio;
+	loff_t out_pos;
+
+	out_pos = blkdev_offset_to_out_pos(params, ctx->offset);
+
+	bio = bio_map_buf(ctx->buf, ctx->buf_len);
+	if (!bio) {
+		params->status = BLK_STS_AGAIN;
+		params->end_io(params);
+		return;
+	}
+	bio->bi_opf = REQ_OP_WRITE;
+	bio_set_dev(bio, params->out_bdev);
+	bio->bi_iter.bi_sector = out_pos >> SECTOR_SHIFT;
+	bio->bi_iter.bi_size = ctx->chunk;
+	bio->bi_end_io = blkdev_write_done;
+	bio->bi_copy_ctx = ctx;
+	submit_bio(bio);
+}
+
+static void blkdev_read_done(struct bio *bio)
+{
+	struct blkdev_copy_onload_ctx *ctx = bio->bi_copy_ctx;
+	struct blk_copy_params *params = ctx->params;
+	blk_status_t sts = bio->bi_status;
+
+	kfree(bio);
+
+	if (sts) {
+		params->status = sts;
+		params->end_io(params);
+		return;
+	}
+
+	schedule_work(&ctx->write_work);
+}
+
+static sector_t blkdev_offset_to_in_pos(const struct blk_copy_params *params,
+					loff_t offset, loff_t *chunk)
+{
+	for (int i = 0; i < params->in_nseg; i++) {
+		loff_t rem = params->in_segs[i].len - offset;
+
+		if (rem > 0) {
+			if (*chunk > rem)
+				*chunk = rem;
+			return params->in_segs[i].pos + offset;
+		}
+		offset -= params->in_segs[i].len;
+	}
+	*chunk = 0;
+	return 0;
+}
+
+static void blkdev_read_work(struct work_struct *work)
+{
+	struct blkdev_copy_onload_ctx *ctx =
+		container_of(work, typeof(*ctx), read_work);
+	struct blk_copy_params *params = ctx->params;
+	loff_t offset = ctx->offset;
+	sector_t in_pos;
+	struct bio *bio;
+
+	ctx->chunk = min(ctx->buf_len, params->len - offset);
+	if (ctx->chunk)
+		in_pos = blkdev_offset_to_in_pos(params, offset, &ctx->chunk);
+	if (ctx->chunk == 0) {
+		params->end_io(params);
+		return;
+	}
+
+	bio = bio_map_buf(ctx->buf, ctx->buf_len);
+	if (!bio) {
+		params->status = BLK_STS_AGAIN;
+		params->end_io(params);
+		return;
+	}
+	bio->bi_opf = REQ_OP_READ;
+	bio_set_dev(bio, params->in_bdev);
+	bio->bi_iter.bi_sector = in_pos >> SECTOR_SHIFT;
+	bio->bi_iter.bi_size = ctx->chunk;
+	bio->bi_end_io = blkdev_read_done;
+	bio->bi_copy_ctx = ctx;
+	submit_bio(bio);
+}
+
+/**
+ * blkdev_copy_onload - asynchronously copy data between two block devices using
+ *	read and write operations.
+ * @params: Input and output block devices, input and output ranges and
+ *	completion callback pointer.
+ * Return: 0 upon success; -EIOCBQUEUED if the completion callback function will
+ *	be called or has already been called.
+ */
+int blkdev_copy_onload(struct blk_copy_params *params)
+{
+	loff_t max_hw_bytes =
+		min(queue_max_hw_sectors(params->in_bdev->bd_queue),
+		    queue_max_hw_sectors(params->out_bdev->bd_queue)) <<
+		SECTOR_SHIFT;
+	struct blkdev_copy_onload_ctx *ctx;
+	loff_t len;
+	int ret;
+
+	ret = blkdev_copy_check_params(params, &len);
+	if (ret)
+		return ret;
+
+	params->len = len;
+
+	ctx = kzalloc_obj(*ctx);
+	if (!ctx)
+		return -ENOMEM;
+
+	INIT_WORK(&ctx->read_work, blkdev_read_work);
+	INIT_WORK(&ctx->write_work, blkdev_write_work);
+	ctx->params = params;
+
+	ctx->buf = blkdev_copy_alloc_buf(min(max_hw_bytes, len), &ctx->buf_len);
+	if (!ctx->buf)
+		goto err;
+
+	blkdev_read_work(&ctx->read_work);
+
+	return -EIOCBQUEUED;
+
+err:
+	kfree(ctx);
+	return -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(blkdev_copy_onload);
include/linux/blkdev.h +1 -0
index fea296150cda..817eeba2f207 100644
@@ -1284,6 +1284,7 @@ void __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
 int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp);
 int blkdev_copy_offload(struct blk_copy_params *params);
+int blkdev_copy_onload(struct blk_copy_params *params);
 
 #define BLKDEV_ZERO_NOUNMAP	(1 << 0)  /* do not free blocks */
 #define BLKDEV_ZERO_NOFALLBACK	(1 << 1)  /* don't write explicit zeroes */
#5 lore raw

PATCH 5/12 block: Introduce accessor functions for copy offload bios

In reply to Bart Van Assche

Make it easy for block drivers to iterate over the copy offload bios by providing accessor functions for the copy offloading bios.

---
 block/blk-copy.c         | 47 ++++++++++++++++++++++++++++++++++++++++
 include/linux/blk-copy.h |  4 ++++
 2 files changed, 51 insertions(+)
block/blk-copy.c +47 -0
index 459ed8581efc..f49a5f835b4a 100644
@@ -7,6 +7,53 @@
 #include <linux/blk-copy.h>
 #include <linux/blk-mq.h>
 
+static struct bio *__blk_next_copy_bio(struct request *rq, struct bio *prev_bio,
+				       enum req_op op)
+{
+	struct bio *bio;
+
+	if (prev_bio) {
+		bio = prev_bio->bi_next;
+	} else {
+		struct bio_copy_offload_ctx *copy_ctx = rq->bio->bi_copy_ctx;
+
+		bio = copy_ctx->bios;
+	}
+
+	for (; bio && bio_op(bio) != op; bio = bio->bi_next)
+		;
+	return bio;
+}
+
+struct bio *blk_first_copy_bio(struct request *rq, enum req_op op)
+{
+	struct bio *bio = rq->bio;
+
+	if (bio_op(bio) == op)
+		return bio;
+
+	return __blk_next_copy_bio(rq, NULL, op);
+}
+EXPORT_SYMBOL_GPL(blk_first_copy_bio);
+
+struct bio *blk_next_copy_bio(struct bio *bio)
+{
+	return __blk_next_copy_bio(NULL, bio, bio_op(bio));
+}
+EXPORT_SYMBOL_GPL(blk_next_copy_bio);
+
+unsigned int blk_copy_bio_count(struct request *rq, enum req_op op)
+{
+	unsigned int count = 0;
+
+	for (struct bio *bio = blk_first_copy_bio(rq, op); bio;
+	     bio = blk_next_copy_bio(bio))
+		count++;
+
+	return count;
+}
+EXPORT_SYMBOL_GPL(blk_copy_bio_count);
+
 /**
  * Tracks the state of a single onloaded copy operation.
  * @params: Data copy parameters.
include/linux/blk-copy.h +4 -0
index 5e38cfc14a71..4c8435312752 100644
@@ -43,4 +43,8 @@ struct bio_copy_offload_ctx {
 	void (*translation_complete)(struct bio_copy_offload_ctx *ctx);
 };
 
+struct bio *blk_first_copy_bio(struct request *rq, enum req_op op);
+struct bio *blk_next_copy_bio(struct bio *bio);
+unsigned int blk_copy_bio_count(struct request *rq, enum req_op op);
+
 #endif /* __LINUX_BLK_COPY_H */
#6 lore raw

PATCH 6/12 fs/read_write: Generalize generic_copy_file_checks()

In reply to Bart Van Assche

From: Anuj Gupta <[email protected]>

Prepare for adding copy_file_range() support for block devices by making the following changes:
- Change file_inode(file) into file->f_mapping->host. Although only one inode is associated with regular files, two inodes are associated with block devices. file->f_mapping->host is the primary block device inode.
- Change S_ISREG() into S_ISREG() || S_ISBLK().
- Add an inode->i_mode & S_IFMT check that verifies that source and destination have the same type (block device or regular file).

[ bvanassche: rewrote patch description ]
---
 fs/read_write.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
fs/read_write.c +5 -3
index 50bff7edc91f..d6fba5afff94 100644
@@ -1484,8 +1484,8 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in,
 				    struct file *file_out, loff_t pos_out,
 				    size_t *req_count, unsigned int flags)
 {
-	struct inode *inode_in = file_inode(file_in);
-	struct inode *inode_out = file_inode(file_out);
+	struct inode *inode_in = file_in->f_mapping->host;
+	struct inode *inode_out = file_out->f_mapping->host;
 	uint64_t count = *req_count;
 	loff_t size_in;
 	int ret;
@@ -1791,7 +1791,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out)
 	/* Don't copy dirs, pipes, sockets... */
 	if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode))
 		return -EISDIR;
-	if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode))
+	if (!S_ISREG(inode_in->i_mode) && !S_ISBLK(inode_in->i_mode))
+		return -EINVAL;
+	if ((inode_in->i_mode & S_IFMT) != (inode_out->i_mode & S_IFMT))
 		return -EINVAL;
 
 	if (!(file_in->f_mode & FMODE_READ) ||
#7 lore raw

PATCH 7/12 fs, block: Add copy_file_range() support for block devices

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

Add copy_file_range() support for block devices. If input and output block devices have been opened with O_DIRECT and if copy offloading is supported use blkdev_copy_offload(). Otherwise use splice_copy_file_range().

---
 block/fops.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
block/fops.c +54 -0
index bb6642b45937..f438503f1b77 100644
@@ -19,6 +19,7 @@
 #include <linux/iomap.h>
 #include <linux/module.h>
 #include <linux/io_uring/cmd.h>
+#include <linux/splice.h>
 #include "blk.h"
 
 static inline struct inode *bdev_file_inode(struct file *file)
@@ -861,6 +862,58 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	return ret;
 }
 
+static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in,
+				      struct file *file_out, loff_t pos_out,
+				      size_t len, unsigned int flags)
+{
+	struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in));
+	struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out));
+	loff_t in_end, out_end;
+	int err;
+
+	if (check_add_overflow(pos_in, len, &in_end) ||
+	    PAGE_ALIGN(in_end) < in_end ||
+	    check_add_overflow(pos_out, len, &out_end) ||
+	    PAGE_ALIGN(out_end) < out_end)
+		return -EINVAL;
+
+	/*
+	 * filemap_write_and_wait_range() and filemap_invalidate_inode() expect
+	 * that the 'end' argument is rounded up to the next multiple of
+	 * PAGE_SIZE.
+	 */
+	in_end = PAGE_ALIGN(in_end);
+	out_end = PAGE_ALIGN(out_end);
+
+	if (bdev_max_copy_sectors(in_bdev) && bdev_max_copy_sectors(out_bdev) &&
+	    file_in->f_iocb_flags & file_out->f_iocb_flags & IOCB_DIRECT) {
+		struct blk_copy_seg in_seg = { .pos = pos_in, .len = len };
+		struct blk_copy_seg out_seg = { .pos = pos_out, .len = len };
+		struct blk_copy_params params = {
+			.in_bdev = in_bdev,
+			.out_bdev = out_bdev,
+			.in_nseg = 1,
+			.in_segs = &in_seg,
+			.out_nseg = 1,
+			.out_segs = &out_seg,
+		};
+		err = filemap_write_and_wait_range(file_in->f_mapping, pos_in,
+						   in_end);
+		if (err)
+			return err;
+		err = filemap_invalidate_inode(bdev_file_inode(file_out),
+					       /*flush=*/false,
+					       pos_out, out_end);
+		if (err)
+			return err;
+		if (blkdev_copy_offload(&params) == 0)
+			return len;
+		/* If copy offloading fails, fall back to onloading. */
+	}
+
+	return splice_copy_file_range(file_in, pos_in, file_out, pos_out, len);
+}
+
 #define	BLKDEV_FALLOC_FL_SUPPORTED					\
 		(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |		\
 		 FALLOC_FL_ZERO_RANGE | FALLOC_FL_WRITE_ZEROES)
@@ -967,6 +1020,7 @@ const struct file_operations def_blk_fops = {
 	.fallocate	= blkdev_fallocate,
 	.uring_cmd	= blkdev_uring_cmd,
 	.fop_flags	= FOP_BUFFER_RASYNC,
+	.copy_file_range = blkdev_copy_file_range,
 };
 
 static __init int blkdev_init(void)
#8 lore raw

PATCH 8/12 nvme: Add copy offloading support

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

Add support for the NVMe Copy command. This command supports a single destination range and up to 256 source ranges.

Add trace event support for nvme_copy_cmd.

[ bvanassche: generalized Copy support from one to 256 source ranges; fixed
an endianness issue in nvme_config_copy(); renamed rsvd91 into rsvd81 and verified the offset with pahole ]

---
 drivers/nvme/host/constants.c |   1 +
 drivers/nvme/host/core.c      | 106 ++++++++++++++++++++++++++++++++++
 drivers/nvme/host/trace.c     |  19 ++++++
 include/linux/nvme.h          |  46 ++++++++++++++-
 4 files changed, 169 insertions(+), 3 deletions(-)
drivers/nvme/host/constants.c +1 -0
index dc90df9e13a2..b80c7c7fb629 100644
@@ -19,6 +19,7 @@ static const char * const nvme_ops[] = {
 	[nvme_cmd_resv_report] = "Reservation Report",
 	[nvme_cmd_resv_acquire] = "Reservation Acquire",
 	[nvme_cmd_resv_release] = "Reservation Release",
+	[nvme_cmd_copy] = "Copy Offload",
 	[nvme_cmd_zone_mgmt_send] = "Zone Management Send",
 	[nvme_cmd_zone_mgmt_recv] = "Zone Management Receive",
 	[nvme_cmd_zone_append] = "Zone Append",
drivers/nvme/host/core.c +106 -0

148 lines not on this page · raw message

drivers/nvme/host/trace.c +19 -0
index ad25ad1e4041..7096ade7740c 100644
@@ -153,6 +153,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10)
 	return ret;
 }
 
+static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u64 sdlba = get_unaligned_le64(cdw10);
+	u8 nr_range = get_unaligned_le16(cdw10 + 8);
+	u16 control = get_unaligned_le16(cdw10 + 10);
+	u32 dsmgmt = get_unaligned_le32(cdw10 + 12);
+	u32 reftag = get_unaligned_le32(cdw10 + 16);
+
+	trace_seq_printf(p,
+		"sdlba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u",
+		sdlba, nr_range, control, dsmgmt, reftag);
+	trace_seq_putc(p, 0);
+
+	return ret;
+}
+
 static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -386,6 +403,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p,
 		return nvme_trace_resv_rel(p, cdw10);
 	case nvme_cmd_resv_report:
 		return nvme_trace_resv_report(p, cdw10);
+	case nvme_cmd_copy:
+		return nvme_trace_copy(p, cdw10);
 	default:
 		return nvme_trace_common(p, cdw10);
 	}
include/linux/nvme.h +43 -3

88 lines not on this page · raw message

#9 lore raw

PATCH 9/12 nvmet: Support the Copy command

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

Support the Copy command for namespaces backed by a block device or by a file. For namespaces backed by a block device, we call blkdev_copy_offload() and fall back to blkdev_copy_onload() if necessary. For namespaces backed by a file we call vfs_copy_file_range().

nvmet always reports that the Copy command is supported.

Tracing support is added for the Copy command.

[ bvanassche: Increased namespace limits. ]
---
 drivers/nvme/host/trace.c         |  2 +-
 drivers/nvme/target/admin-cmd.c   | 26 +++++++++-
 drivers/nvme/target/io-cmd-bdev.c | 80 +++++++++++++++++++++++++++++++
 drivers/nvme/target/io-cmd-file.c | 59 +++++++++++++++++++++--
 drivers/nvme/target/trace.c       | 19 ++++++++
 include/linux/nvme.h              |  1 +
 6 files changed, 179 insertions(+), 8 deletions(-)
drivers/nvme/host/trace.c +1 -1
index 7096ade7740c..fd49363f8516 100644
@@ -143,7 +143,7 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10)
 	u16 length = get_unaligned_le16(cdw10 + 8);
 	u16 control = get_unaligned_le16(cdw10 + 10);
 	u32 dsmgmt = get_unaligned_le32(cdw10 + 12);
-	u32 reftag = get_unaligned_le32(cdw10 +  16);
+	u32 reftag = get_unaligned_le32(cdw10 + 16);
 
 	trace_seq_printf(p,
 			 "slba=%llu, len=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u",
drivers/nvme/target/admin-cmd.c +24 -2
index e4fd1caadfb0..1e404df6ad84 100644
@@ -733,8 +733,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
 			NVME_CTRL_ONCS_WRITE_ZEROES |
-			NVME_CTRL_ONCS_RESERVATIONS);
-
+			NVME_CTRL_ONCS_RESERVATIONS | NVME_CTRL_ONCS_COPY);
 	/* XXX: don't report vwc if the underlying device is write through */
 	id->vwc = NVME_CTRL_VWC_PRESENT;
 
@@ -797,6 +796,27 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
+static void nvmet_set_copy_limits(struct nvme_id_ns *id)
+{
+	/*
+	 * MSRC = Maximum Source Range Count - the maximum number of
+	 * source ranges that may be used to specify source data in a
+	 * Copy command. 0's based.
+	 */
+	id->msrc = 256 - 1;
+	/*
+	 * MSSRL = Maximum Single Source Range Length - the maximum number
+	 * of logical blocks that may be specified in the Number of Logical
+	 * Blocks field in each valid Source Range Entries Descriptor.
+	 */
+	id->mssrl = cpu_to_le16(U16_MAX);
+	/*
+	 * MCL = Maximum Copy Length - the maximum number of logical
+	 * blocks that may be specified in a Copy command.
+	 */
+	id->mcl = cpu_to_le32(U32_MAX);
+}
+
 static void nvmet_execute_identify_ns(struct nvmet_req *req)
 {
 	struct nvme_id_ns *id;
@@ -845,6 +865,8 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 	if (req->ns->bdev)
 		nvmet_bdev_set_limits(req->ns->bdev, id);
 
+	nvmet_set_copy_limits(id);
+
 	/*
 	 * We just provide a single LBA format that matches what the
 	 * underlying device reports.
drivers/nvme/target/io-cmd-bdev.c +80 -0

92 lines not on this page · raw message

drivers/nvme/target/io-cmd-file.c +54 -5

83 lines not on this page · raw message

drivers/nvme/target/trace.c +19 -0
index 6dbc7036f2e4..2baef7294491 100644
@@ -92,6 +92,23 @@ static const char *nvmet_trace_dsm(struct trace_seq *p, u8 *cdw10)
 	return ret;
 }
 
+static const char *nvmet_trace_copy(struct trace_seq *p, u8 *cdw10)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u64 sdlba = get_unaligned_le64(cdw10);
+	u8 nr_range = get_unaligned_le16(cdw10 + 8);
+	u16 control = get_unaligned_le16(cdw10 + 10);
+	u32 dsmgmt = get_unaligned_le32(cdw10 + 12);
+	u32 reftag = get_unaligned_le32(cdw10 +  16);
+
+	trace_seq_printf(p,
+		"sdlba=%llu, nr_range=%u, ctrl=1x%x, dsmgmt=%u, reftag=%u",
+		sdlba, nr_range, control, dsmgmt, reftag);
+	trace_seq_putc(p, 0);
+
+	return ret;
+}
+
 static const char *nvmet_trace_common(struct trace_seq *p, u8 *cdw10)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -303,6 +320,8 @@ const char *nvmet_trace_parse_nvm_cmd(struct trace_seq *p,
 		return nvmet_trace_resv_rel(p, cdw10);
 	case nvme_cmd_resv_report:
 		return nvmet_trace_resv_report(p, cdw10);
+	case nvme_cmd_copy:
+		return nvmet_trace_copy(p, cdw10);
 	default:
 		return nvmet_trace_common(p, cdw10);
 	}
include/linux/nvme.h +1 -0
index ead8e5128e3b..c6325aeb13a0 100644
@@ -2220,6 +2220,7 @@ enum {
 	NVME_SC_PMR_SAN_PROHIBITED	= 0x123,
 	NVME_SC_ANA_GROUP_ID_INVALID	= 0x124,
 	NVME_SC_ANA_ATTACH_FAILED	= 0x125,
+	NVME_SC_COMMAND_SIZE_LIMIT_EXC	= 0x183,
 
 	/*
 	 * I/O Command Set Specific - NVM commands:
#10 lore raw

PATCH 10/12 dm: Add support for copy offloading

In reply to Bart Van Assche

In dm_calculate_queue_limits(), clear the copy offload limits if the device mapper driver does not support copy offloading. This is necessary since blk_set_stacking_limits() sets the copy offload limits to their maximum.

---
 drivers/md/dm-table.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/md/dm-table.c +8 -0
index dc2eff6b739d..888c5bdca5f1 100644
@@ -1816,6 +1816,14 @@ int dm_calculate_queue_limits(struct dm_table *t,
 			return -EINVAL;
 
 combine_limits:
+		if (!(ti_limits.features & BLK_FEAT_STACKING_COPY_OFFL)) {
+			ti_limits.max_copy_hw_sectors = 0;
+			ti_limits.max_copy_src_segments = 0;
+			ti_limits.max_copy_dst_segments = 0;
+			ti_limits.max_user_copy_sectors = 0;
+			ti_limits.max_copy_sectors = 0;
+		}
+
 		/*
 		 * Merge this target's queue limits into the overall limits
 		 * for the table.
#11 lore raw

PATCH 11/12 dm-linear: Enable copy offloading

In reply to Bart Van Assche

Set BLK_FEAT_STACKING_COPY_OFFL and max_copy_hw_sectors to enable copy offloading.

---
 drivers/md/dm-linear.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/md/dm-linear.c +6 -0
index 38c17846deb0..3de8bf5f11fb 100644
@@ -119,6 +119,11 @@ static void linear_status(struct dm_target *ti, status_type_t type,
 	}
 }
 
+static void linear_io_hints(struct dm_target *ti, struct queue_limits *limits)
+{
+	limits->features |= BLK_FEAT_STACKING_COPY_OFFL;
+}
+
 static int linear_prepare_ioctl(struct dm_target *ti, struct block_device **bdev,
 				unsigned int cmd, unsigned long arg,
 				bool *forward)
@@ -211,6 +216,7 @@ static struct target_type linear_target = {
 	.dtr    = linear_dtr,
 	.map    = linear_map,
 	.status = linear_status,
+	.io_hints = linear_io_hints,
 	.prepare_ioctl = linear_prepare_ioctl,
 	.iterate_devices = linear_iterate_devices,
 	.direct_access = linear_dax_direct_access,
#12 lore raw

PATCH 12/12 null_blk: Add support for REQ_OP_COPY_*

In reply to Bart Van Assche

From: Nitesh Shetty <[email protected]>

Implementation is based on existing read and write infrastructure.
copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Only request based queue mode will support for copy offload. Added tracefs support to copy IO tracing.

[ bvanassche: Split nullb_do_copy() into two functions. Added a
cond_resched() call inside nullb_do_copy(). ]

---
 Documentation/block/null_blk.rst  |   4 ++
 drivers/block/null_blk/main.c     | 113 ++++++++++++++++++++++++++++++
 drivers/block/null_blk/null_blk.h |   1 +
 3 files changed, 118 insertions(+)
Documentation/block/null_blk.rst +4 -0
index 4dd78f24d10a..ea0616dbf7f3 100644
@@ -149,3 +149,7 @@ zone_size=[MB]: Default: 256
 zone_nr_conv=[nr_conv]: Default: 0
   The number of conventional zones to create when block device is zoned.  If
   zone_nr_conv >= nr_zones, it will be reduced to nr_zones - 1.
+
+max_copy_bytes=[size in bytes]: Default: UINT_MAX
+  A module and configfs parameter which can be used to set hardware/driver
+  supported maximum copy offload limit.
drivers/block/null_blk/main.c +113 -0

173 lines not on this page · raw message

drivers/block/null_blk/null_blk.h +1 -0
index 6c4c4bbe7dad..c15c319ed91b 100644
@@ -93,6 +93,7 @@ struct nullb_device {
 	unsigned int queue_mode; /* block interface */
 	unsigned int blocksize; /* block size */
 	unsigned int max_sectors; /* Max sectors per command */
+	unsigned long max_copy_bytes; /* Max copy offload length in bytes */
 	unsigned int irqmode; /* IRQ completion handler */
 	unsigned int hw_queue_depth; /* queue depth */
 	unsigned int index; /* index of the disk, only valid with a disk */
#13 lore raw
In reply to Bart Van Assche
Quoted 18 lines · Bart Van Assche

Hi Jens,

This patch series implements a new approach for copy offloading. Compared to Nitesh' approach, the differences are as follows:
- Two new limits have been introduced representing the maximum number of source and destination ranges. Support for multiple source ranges for the NVMe Copy command has been added.
- The blkdev_copy_offload() function can now submit multiple copy offload commands instead of only one.
- The implementation no longer depends on block layer plugging.

This patch series includes copy offloading support for the Linux kernel block layer core, the device mapper core, the null_blk and the NVMe and nvmet drivers. Support for the scsi_debug and SCSI core will follow later.

Test scripts are available here:
https://github.com/bvanassche/blktests/tree/copy-offloading

Quoted 6 lines

See also Bart Van Assche, [LSF/MM/BPF TOPIC] Block storage copy offloading, January 2026
(https://lore.kernel.org/all/[email protected]/).

See also Nitesh Shetty, Implement copy offload support, May 2024
(https://lore.kernel.org/linux-block/[email protected]/).

FYI, blktests CI trial run detected that this patch series triggers nvme/018 failure. I manually applied this series on top of the v7.1-rc4 kernel and observed the failure is recreated in stable manner.

nvme/018 (tr=loop) (unit test NVMe-oF out of range access on a file backend) [failed]

    runtime  1.208s  ...  1.189s
    --- tests/nvme/018.out      2025-04-22 13:13:27.738873155 +0900
    +++ /home/shin/Blktests/blktests/results/nodev_tr_loop/nvme/018.out.bad     2026-05-22 20:57:31.060000000 +0900
    @@ -1,3 +1,4 @@
     Running nvme/018
    +ERROR: nvme read for out of range LBA was not rejected
     disconnected 1 controller(s)
     Test complete
#14 lore raw
Quoted 5 lines · Shin'ichiro Kawasaki

FYI, blktests CI trial run detected that this patch series triggers nvme/018 failure. I manually applied this series on top of the v7.1-rc4 kernel and observed the failure is recreated in stable manner.

nvme/018 (tr=loop) (unit test NVMe-oF out of range access on a file backend) [failed]

Quoted 8 lines
     runtime  1.208s  ...  1.189s
     --- tests/nvme/018.out      2025-04-22 13:13:27.738873155 +0900
     +++ /home/shin/Blktests/blktests/results/nodev_tr_loop/nvme/018.out.bad     2026-05-22 20:57:31.060000000 +0900
     @@ -1,3 +1,4 @@
      Running nvme/018
     +ERROR: nvme read for out of range LBA was not rejected
      disconnected 1 controller(s)
      Test complete

Thanks Shin'ichiro for having reported this. I plan to include a fix when I publish v2 of this patch series.

Bart.