scsi: target: Add target_core_uring, an io_uring based user backstore
Subsystems (5)
cover scsi: target: Add target_core_uring, an io_uring based user backstore
TCM/USER is how a userspace process serves a LIO device, for instance to export distributed storage over iSCSI or vhost-scsi. All commands of a device go through one command ring under one mutex, the data of each command is copied into and out of a shared data area, and the server has to parse every SCSI CDB itself.
This series adds target_core_uring, a second user backstore next to TCM/USER, built the way ublk is:
- The kernel emulates the SCSI commands. The server only sees reads, writes, flushes, UNMAP and WRITE SAME as an operation, an offset and a length.
- A device has several queues. Each queue is served through io_uring commands on its own ring, with no lock shared between queues.
- Data is copied once, with pread()/pwrite() on the character device, or not at all when the server registers the pages of a command as an io_uring fixed buffer.
Patches 1-4 create the device, its queues and the character device. Patches 5-8 pass reads and writes to the server. Patches 9-11 handle a server that exits or hangs, and aborts. Patches 12-14 add flush, UNMAP and WRITE SAME, patch 15 adds fixed buffers. Patches 16-18 add a small server, a selftest and the documentation.
Not covered yet: protection information, and a way for existing tcmu-runner handlers to move over without changes, as they implement SCSI rather than block commands.
Exporting a 2 GiB device through tcm_loop to fio (io_uring, direct I/O, iodepth 32 for 4k and 8 for 1M, 6 seconds per job) on a 32 CPU VM, with TCM/USER and a single-threaded handler, then with this series, 32 queues and one server thread per queue. Both servers keep the data in memory. A single job is slower; the gain comes with concurrency and with large I/O. Three alternating runs (median, min-max in parentheses):
workload TCM/USER (MB/s) uring (MB/s) change
4k randread, 1 job 426 (409-463) 369 (366-384) -13%
4k randread, 4 jobs 1165 (1072-1253) 1524 (1413-1541) +31% 4k randread, 16 jobs 920 (916-1162) 6193 (5845-6222) +574% 4k randwrite, 16 jobs 793 (711-886) 5701 (5257-5706) +619% 1M read, 4 jobs 6019 (5857-6047) 49460 (46706-57997) +722% 1M write, 4 jobs 2791 (2649-2907) 26692 (25958-27560) +856%
---
Qiliang Yuan (18):
scsi: target: uring: Introduce an io_uring based user backstore
scsi: target: uring: Allocate per-queue command descriptors
scsi: target: uring: Create a character device per backstore device
scsi: target: uring: Map the command descriptors into the server
scsi: target: uring: Queue commands until a server fetches them
scsi: target: uring: Hand commands out through TCMUR_IO_FETCH
scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH
scsi: target: uring: Copy command data with pread() and pwrite()
scsi: target: uring: Fail in-flight commands when the server exits
scsi: target: uring: Time out commands the server does not complete
scsi: target: uring: Drop aborted commands that are still queued
scsi: target: uring: Support SYNCHRONIZE CACHE
scsi: target: uring: Implement UNMAP
scsi: target: uring: Handle WRITE SAME
scsi: target: uring: Register command data as io_uring fixed buffers
selftests: target_core_uring: Add a minimal server
selftests: target_core_uring: Test a device exported through tcm_loop
Documentation: target: Describe the uring backstore
Documentation/target/index.rst | 1 + Documentation/target/tcm-uring.rst | 84 ++ MAINTAINERS | 2 + drivers/target/Kconfig | 10 + drivers/target/Makefile | 1 + drivers/target/target_core_uring.c | 1126 ++++++++++++++++++++ include/uapi/linux/target_core_uring.h | 126 +++ tools/testing/selftests/Makefile | 1 + .../testing/selftests/target_core_uring/.gitignore | 1 + tools/testing/selftests/target_core_uring/Makefile | 8 + tools/testing/selftests/target_core_uring/config | 5 + .../selftests/target_core_uring/tcmur_server.c | 310 ++++++ .../selftests/target_core_uring/tcmur_test.sh | 137 +++ 13 files changed, 1812 insertions(+)
---
base-commit: 23c62304dde29abe92013d85910d03a77dada35d
change-id: 20261115-feat-target-core-uring-d1351dfe5fe1
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
RFC PATCH 1/18 scsi: target: uring: Introduce an io_uring based user backstore
TCM/USER lets a userspace process serve the commands of a LIO device, which is how distributed storage is exported over iSCSI or vhost-scsi.
All commands of a TCM/USER device go through one command ring under one mutex, so throughput does not scale with the number of submitters. The data of each command is copied into and out of a shared data area, and the server has to parse every SCSI CDB itself.
Start a new backstore, target_core_uring, that will hand block commands to the server through io_uring commands on several queues. Register the backstore and let configfs create devices with a size and a block size. Let the core emulate the SCSI commands through sbc_parse_cdb(), and fail reads and writes until a server can pick them up.
--- drivers/target/Kconfig | 10 ++ drivers/target/Makefile | 1 + drivers/target/target_core_uring.c | 214 +++++++++++++++++++++++++++++++++++++ 3 files changed, 225 insertions(+)
drivers/target/Kconfig +10 -0
@@ -43,6 +43,16 @@ config TCM_USER2 process to handle requests. This is version 2 of the ABI; version 1 is obsolete. +config TCM_URING + tristate "TCM/URING Subsystem Plugin for Linux" + depends on IO_URING + help + Say Y here to enable the TCM/URING subsystem plugin, which lets a + userspace server handle the block commands of a device through + io_uring commands. Unlike TCM/USER, the kernel emulates the SCSI + commands and the server only reads, writes, flushes and unmaps + blocks, on several queues and without copying data twice. + source "drivers/target/loopback/Kconfig" source "drivers/target/tcm_fc/Kconfig" source "drivers/target/iscsi/Kconfig"
drivers/target/Makefile +1 -0
@@ -24,6 +24,7 @@ obj-$(CONFIG_TCM_IBLOCK) += target_core_iblock.o obj-$(CONFIG_TCM_FILEIO) += target_core_file.o obj-$(CONFIG_TCM_PSCSI) += target_core_pscsi.o obj-$(CONFIG_TCM_USER2) += target_core_user.o +obj-$(CONFIG_TCM_URING) += target_core_uring.o # Fabric modules obj-$(CONFIG_LOOPBACK_TARGET) += loopback/
drivers/target/target_core_uring.c new file +214 -0
214 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 01/18] scsi: target: uring: Introduce an io_uring based user backstore
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] scsi: target: missing target_core_uring autoload in transport_subsystem_check_init() - [Medium]…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] scsi: target: missing target_core_uring autoload in transport_subsystem_check_init()
- [Medium] scsi: target: uring: block_size desync and underflow in tcmur_get_blocks()
--
commit 61188ead36f715e3a08be0d083f957ddb97dafd3
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Introduce an io_uring based user backstore
This commit adds the initial target_core_uring LIO backstore plugin with configfs support for configuring device size and block size, and emulates SCSI commands via sbc_parse_cdb().
Quoted 8 lines
drivers/target/Kconfig +0
-0@@ -43,6 +43,16 @@ config TCM_USER2 process to handle requests. This is version 2 of the ABI; version 1 is obsolete.
drivers/target/Kconfig config TCM_USER2 · line 55
Quoted 10 lines
drivers/target/Kconfig +10
-0+config TCM_URING + tristate "TCM/URING Subsystem Plugin for Linux" + depends on IO_URING + help + Say Y here to enable the TCM/URING subsystem plugin, which lets a + userspace server handle the block commands of a device through + io_uring commands. Unlike TCM/USER, the kernel emulates the SCSI + commands and the server only reads, writes, flushes and unmaps + blocks, on several queues and without copying data twice. +
[Severity: Medium]
Should transport_subsystem_check_init() also autoload target_core_uring when CONFIG_TCM_URING=m?
When creating an HBA directory in configfs, target_core_call_addhbatotarget() calls transport_subsystem_check_init(), which autoloads the other modular backstores (target_core_iblock, target_core_file, target_core_pscsi, and target_core_user) via request_module() before core_alloc_hba() searches backend_list. Without an entry for target_core_uring, creating a uring_<id> HBA in configfs fails with -EINVAL unless the module was loaded manually beforehand.
Quoted 5 lines
drivers/target/target_core_uring.c new file +0
-0
[ ... ]
Quoted 21 lines
drivers/target/target_core_uring.c new file +21
-0+switch (token) { +case Opt_block_size: +if (val <512 || val >PAGE_SIZE || !is_power_of_2 (val)) + ret = -EINVAL ; +else + udev->block_size = val; +break ; + } + } + +kfree (orig); +return ret ? ret : count; +} + +static ssize_t tcmur_show_configfs_dev_params (struct se_device *dev,char *b) +{ +struct tcmur_dev *udev =TCMUR_DEV (dev); +ssize_t bl =0 ; + + bl +=sprintf (b + bl,"Size: %llu Block size: %u" , udev->dev_size, + udev->block_size);
drivers/target/target_core_uring.c line 167
Quoted 10 lines
drivers/target/target_core_uring.c new file +10
-0+ bl +=sprintf (b + bl,"\n" ); +return bl; +} + +static sector_t tcmur_get_blocks (struct se_device *dev) +{ +struct tcmur_dev *udev =TCMUR_DEV (dev); + +return div_u64 (udev->dev_size, dev->dev_attrib.block_size) -1 ; +}
[Severity: Medium]
Can tcmur_get_blocks() underflow to ULLONG_MAX and leave udev->block_size out of sync when attrib/block_size is modified after device configuration?
Because tcmur_ops sets tb_dev_attrib_attrs = sbc_attrib_attrs, userspace can write to attrib/block_size after target_configure_device() has run. block_size_store() updates dev->dev_attrib.block_size without updating udev->block_size or revalidating udev->dev_size. If a device is configured with dev_size=512 and block_size=512 and attrib/block_size is then set to 4096, div_u64(udev->dev_size, dev->dev_attrib.block_size) returns 0 and tcmur_get_blocks() wraps to ULLONG_MAX, wrapping end_lba to 0 in sbc_parse_cdb() and reporting 0xffffffffffffffff blocks in
READ CAPACITY (16).
Also, Opt_block_size accepts powers of two up to PAGE_SIZE, which on 16K or 64K page kernels permits block sizes above 4096 that LIO core does not support in block_size_store() or target_to_linux_sector().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
RFC PATCH 2/18 scsi: target: uring: Allocate per-queue command descriptors
In reply to Qiliang YuanA target_core_uring device only has a size and a block size so far.
The server needs to learn what each command asks for, and commands submitted on different CPUs should not share state.
Add the nr_queues and queue_depth parameters and allocate that many queues when the device is configured. Give each queue an array of struct tcmur_io_desc, one per tag, that describes a command as an operation, an offset and a length. Define the descriptor in a new uAPI header and allocate the array with vmalloc_user() so that it can be mapped later.
--- MAINTAINERS | 1 + drivers/target/target_core_uring.c | 90 +++++++++++++++++++++++++++++++++- include/uapi/linux/target_core_uring.h | 43 ++++++++++++++++ 3 files changed, 133 insertions(+), 1 deletion(-)
MAINTAINERS +1 -0
@@ -25330,6 +25330,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git F: Documentation/target/ F: drivers/target/ F: include/target/ +F: include/uapi/linux/target_core_uring.h SCTP PROTOCOL M: Marcelo Ricardo Leitner <[email protected]>
drivers/target/target_core_uring.c +89 -1
165 lines not on this page · raw message
include/uapi/linux/target_core_uring.h new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.43.0
RFC PATCH 3/18 scsi: target: uring: Create a character device per backstore device
In reply to Qiliang YuanA target_core_uring device has queues and command descriptors, but userspace has no handle on it.
The server needs a file to issue io_uring commands on, and the state of the device has to stay around for as long as that file is open, which can be longer than the configfs item lives.
Create /dev/tcmurN when a device is configured and show its name in the info file. Allow one opener at a time, with CAP_SYS_ADMIN. Tie the lifetime of the device to the embedded struct device, so that the queues are freed when the last reference to the character device goes. Report the geometry of the device through the TCMUR_GET_DEV_INFO ioctl.
--- drivers/target/target_core_uring.c | 138 ++++++++++++++++++++++++++++++++- include/uapi/linux/target_core_uring.h | 27 ++++++- 2 files changed, 160 insertions(+), 5 deletions(-)
drivers/target/target_core_uring.c +134 -4
@@ -7,6 +7,9 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <linux/module.h> +#include <linux/cdev.h> +#include <linux/device.h> +#include <linux/idr.h> #include <linux/mm.h> #include <linux/parser.h> #include <linux/slab.h>
@@ -16,10 +19,14 @@#include <uapi/linux/target_core_uring.h> #define TCMUR_VERSION "1.0" +#define TCMUR_MINORS (1U << MINORBITS)#define TCMUR_MAX_IO_SIZE (4U << 20)#define TCMUR_DEF_QUEUE_DEPTH 128#define TCMUR_DEF_BLOCK_SIZE 512 +/* Set while the file is open, to allow one server at a time */ +#define TCMUR_DEV_OPEN 0 +struct tcmur_dev ;/**
@@ -34,6 +41,10 @@ struct tcmur_queue {struct tcmur_dev {struct se_device se_dev; +struct device cdev_dev; +struct cdev cdev; +int dev_id; +unsigned long flags;u64 dev_size;u32 block_size;
@@ -43,6 +54,13 @@ struct tcmur_dev {struct tcmur_queue *queues; }; +static DEFINE_IDA (tcmur_ida); +static dev_t tcmur_devt; + +static const struct class tcmur_class = { + .name ="tcm-uring" , +}; +static struct tcmur_dev *TCMUR_DEV (struct se_device *dev) { return container_of(dev, struct tcmur_dev, se_dev);
@@ -61,6 +79,56 @@ static size_t tcmur_desc_area_size(struct tcmur_dev *udev)return PAGE_ALIGN (udev->queue_depth *sizeof (struct tcmur_io_desc )); } +static long tcmur_ch_ioctl (struct file *filp,unsigned int cmd, +unsigned long arg) +{ +struct tcmur_dev *udev = filp->private_data; +struct tcmur_dev_info info = { + .dev_size = udev->dev_size, + .block_size = udev->block_size, + .max_io_size =TCMUR_MAX_IO_SIZE , + .nr_queues = udev->nr_queues, + .queue_depth = udev->queue_depth, + }; + +if (cmd !=TCMUR_GET_DEV_INFO ) +return -ENOTTY ; + +if (copy_to_user ((void __user *)arg, &info,sizeof (info))) +return -EFAULT ; +return 0 ; +} + +static int tcmur_ch_open (struct inode *inode,struct file *filp) +{ + struct tcmur_dev *udev = container_of(inode->i_cdev, struct tcmur_dev, + cdev); + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; +if (test_and_set_bit (TCMUR_DEV_OPEN , &udev->flags)) +return -EBUSY ; + + filp->private_data = udev; +return 0 ; +} + +static int tcmur_ch_release (struct inode *inode,struct file *filp) +{ +struct tcmur_dev *udev = filp->private_data; + +clear_bit (TCMUR_DEV_OPEN , &udev->flags); +return 0 ; +} + +static const struct file_operations tcmur_ch_fops = { + .owner =THIS_MODULE , + .open = tcmur_ch_open, + .release = tcmur_ch_release, + .unlocked_ioctl = tcmur_ch_ioctl, + .compat_ioctl = compat_ptr_ioctl, +}; +static void tcmur_free_queues (struct tcmur_dev *udev) {unsigned int i;
@@ -104,6 +172,16 @@ static void tcmur_dev_call_rcu(struct rcu_head *p)kfree (TCMUR_DEV (dev)); } +static void tcmur_cdev_release (struct device *dev) +{ + struct tcmur_dev *udev = container_of(dev, struct tcmur_dev, cdev_dev); + +tcmur_free_queues (udev); +if (udev->dev_id >=0 ) +ida_free (&tcmur_ida, udev->dev_id); +call_rcu (&udev->se_dev.rcu_head, tcmur_dev_call_rcu); +} +static int tcmur_attach_hba (struct se_hba *hba,u32 host_id) {return 0 ;
@@ -122,10 +200,15 @@ static struct se_device *tcmur_alloc_device(struct se_hba *hba,if (!udev)return NULL ; + udev->dev_id =-1 ; udev->nr_queues =1 ; udev->queue_depth =TCMUR_DEF_QUEUE_DEPTH ; udev->block_size =TCMUR_DEF_BLOCK_SIZE ; +device_initialize (&udev->cdev_dev); + udev->cdev_dev.class = &tcmur_class; + udev->cdev_dev.release = tcmur_cdev_release; +return &udev->se_dev; }
@@ -143,24 +226,47 @@ static int tcmur_configure_device(struct se_device *dev)if (ret)return ret; + ret =ida_alloc_max (&tcmur_ida,TCMUR_MINORS -1 ,GFP_KERNEL ); +if (ret <0 ) + goto free_queues; + udev->dev_id = ret; + dev->dev_attrib.hw_block_size = udev->block_size; dev->dev_attrib.hw_max_sectors =TCMUR_MAX_IO_SIZE / udev->block_size; dev->dev_attrib.hw_queue_depth = udev->nr_queues * udev->queue_depth; dev->dev_attrib.is_nonrot =1 ; + udev->cdev_dev.devt =MKDEV (MAJOR (tcmur_devt), udev->dev_id); + ret =dev_set_name (&udev->cdev_dev,"tcmur%d" , udev->dev_id); +if (ret) + goto free_id; + +cdev_init (&udev->cdev, &tcmur_ch_fops); + udev->cdev.owner =THIS_MODULE ; + ret =cdev_device_add (&udev->cdev, &udev->cdev_dev); +if (ret) + goto free_id; +return 0 ; + +free_id: +ida_free (&tcmur_ida, udev->dev_id); + udev->dev_id =-1 ; +free_queues: +tcmur_free_queues (udev); +return ret; }static void tcmur_destroy_device (struct se_device *dev) {struct tcmur_dev *udev =TCMUR_DEV (dev); -tcmur_free_queues (udev); +cdev_device_del (&udev->cdev, &udev->cdev_dev); }static void tcmur_free_device (struct se_device *dev) { -call_rcu (&dev->rcu_head, tcmur_dev_call_rcu); +put_device (&TCMUR_DEV (dev)->cdev_dev); }enum {
@@ -239,6 +345,8 @@ static ssize_t tcmur_show_configfs_dev_params(struct se_device *dev, char *b)struct tcmur_dev *udev =TCMUR_DEV (dev);ssize_t bl =0 ; +if (udev->dev_id >=0 ) + bl +=sprintf (b + bl,"Device: /dev/tcmur%d " , udev->dev_id); bl +=sprintf (b + bl,"Size: %llu Block size: %u" , udev->dev_size, udev->block_size); bl += sprintf(b + bl, " Queues: %u Queue depth: %u", udev->nr_queues,
@@ -284,13 +392,35 @@ static const struct target_backend_ops tcmur_ops = {static int __init tcmur_module_init (void ) { -return transport_backend_register (&tcmur_ops); +int ret; + + ret =alloc_chrdev_region (&tcmur_devt,0 ,TCMUR_MINORS ,"tcm-uring" ); +if (ret) +return ret; + + ret =class_register (&tcmur_class); +if (ret) + goto unregister_chrdev; + + ret =transport_backend_register (&tcmur_ops); +if (ret) + goto unregister_class; + +return 0 ; + +unregister_class: +class_unregister (&tcmur_class); +unregister_chrdev: +unregister_chrdev_region (tcmur_devt,TCMUR_MINORS ); +return ret; } static void __exit tcmur_module_exit(void) {target_backend_unregister (&tcmur_ops); -/* Wait for the devices that tcmur_free_device() frees with RCU */ +class_unregister (&tcmur_class); +unregister_chrdev_region (tcmur_devt,TCMUR_MINORS ); +/* Wait for the devices that tcmur_cdev_release() frees with RCU */ rcu_barrier (); }
include/uapi/linux/target_core_uring.h +26 -1
@@ -3,15 +3,40 @@#define _UAPI_LINUX_TARGET_CORE_URING_H#include <linux/types.h> +#include <linux/ioctl.h> /* * target_core_uring hands the block commands of a LIO device to a userspace -* server. +* server. The server opens /dev/tcmurN to drive the device. */ #define TCMUR_MAX_NR_QUEUES 512#define TCMUR_MAX_QUEUE_DEPTH 4096 +/** +* struct tcmur_dev_info - Geometry of a device +* @dev_size: Size of the device in bytes +* @block_size: Logical block size in bytes +* @max_io_size: Largest amount of data in one command, in bytes +* @nr_queues: Number of queues +* @queue_depth: Number of tags per queue +* @reserved0: Zero +* @features: Zero for now +* @reserved1: Zero +*/ +struct tcmur_dev_info { +__u64 dev_size; +__u32 block_size; +__u32 max_io_size; +__u16 nr_queues; +__u16 queue_depth; +__u32 reserved0; +__u64 features; +__u64 reserved1[4 ]; +}; + +#define TCMUR_GET_DEV_INFO _IOR(0xBB, 0x00, struct tcmur_dev_info) +enum tcmur_io_op {TCMUR_OP_READ ,TCMUR_OP_WRITE ,
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 03/18] scsi: target: uring: Create a character device per backstore device
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] scsi: target: uring: incomplete rollback in tcmur_destroy_device() - [Medium] scsi: target: uring…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] scsi: target: uring: incomplete rollback in tcmur_destroy_device()
- [Medium] scsi: target: uring: unsafe error unwind in tcmur_configure_device()
--
commit d9ed44c0dc0f5b30e64edec0424a89f4739a2cf8
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Create a character device per backstore device
This commit creates a /dev/tcmurN character device when a target_core_uring device is configured, ties the device and queue lifetimes to the embedded struct device, and adds the TCMUR_GET_DEV_INFO ioctl.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
Quoted 24 lines
drivers/target/target_core_uring.c +15
-0@@ -143,24 +226,47 @@ static int tcmur_configure_device(struct se_device *dev)if (ret)return ret; + ret =ida_alloc_max (&tcmur_ida,TCMUR_MINORS -1 ,GFP_KERNEL ); +if (ret <0 ) + goto free_queues; + udev->dev_id = ret; + dev->dev_attrib.hw_block_size = udev->block_size; dev->dev_attrib.hw_max_sectors =TCMUR_MAX_IO_SIZE / udev->block_size; dev->dev_attrib.hw_queue_depth = udev->nr_queues * udev->queue_depth; dev->dev_attrib.is_nonrot =1 ; + udev->cdev_dev.devt =MKDEV (MAJOR (tcmur_devt), udev->dev_id); + ret =dev_set_name (&udev->cdev_dev,"tcmur%d" , udev->dev_id); +if (ret) + goto free_id; + +cdev_init (&udev->cdev, &tcmur_ch_fops); + udev->cdev.owner =THIS_MODULE ; + ret =cdev_device_add (&udev->cdev, &udev->cdev_dev); +if (ret) + goto free_id;
drivers/target/target_core_uring.c static int tcmur_configure_device(struct se_device *dev) · line 258
Quoted 10 lines
drivers/target/target_core_uring.c +8
-0+return 0 ; + +free_id: +ida_free (&tcmur_ida, udev->dev_id); + udev->dev_id =-1 ; +free_queues: +tcmur_free_queues (udev); +return ret; }
[Severity: Medium]
Can the free_id error path free udev->queues and udev->dev_id while a file is still open, or re-register udev->cdev_dev if enabling the device is retried?
cdev_device_add() publishes udev->cdev via cdev_add() before calling device_add(). If device_add() fails, a process may already have opened the character device before cdev_del() runs, so freeing udev->dev_id and udev->queues here releases resources while that opener still holds a file referencing udev. In addition, because DF_CONFIGURED is not set on failure, a later write to the configfs enable attribute calls tcmur_configure_device() again on the same udev, running cdev_init() and device_add() a second time on the embedded udev->cdev and udev->cdev_dev.
drivers/target/target_core_uring.c static int tcmur_configure_device(struct se_device *dev) · line 265
Quoted 8 lines
drivers/target/target_core_uring.c +1
-1static void tcmur_destroy_device (struct se_device *dev) {struct tcmur_dev *udev =TCMUR_DEV (dev); -tcmur_free_queues (udev); +cdev_device_del (&udev->cdev, &udev->cdev_dev); }
[Severity: Medium]
What cleans up udev->queues and udev->dev_id if target_configure_device() fails after tcmur_configure_device() succeeds?
drivers/target/target_core_device.c:target_configure_device() {
...
ret = core_setup_alua(dev);
if (ret)
goto out_destroy_device;
...
out_destroy_device:
dev->transport->destroy_device(dev);
...
}When core_setup_alua() fails, target_configure_device() calls tcmur_destroy_device() without setting DF_CONFIGURED or calling tcmur_free_device(), so udev->cdev_dev keeps its initial reference and tcmur_cdev_release() does not run. The unconfigured se_device stays alive in configfs with udev->queues still allocated, udev->dev_id still set, and udev->cdev_dev already passed through device_add() and device_del().
If the user then increases nr_queues via tcmur_set_configfs_dev_params() before removing the device, tcmur_free_queues() reads past the end of the old udev->queues array and passes out-of-bounds pointers to vfree(). If the user instead retries enabling the device, tcmur_configure_device() leaks the previous udev->queues and udev->dev_id and calls cdev_init() and device_add() a second time on the embedded udev->cdev and udev->cdev_dev.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
RFC PATCH 4/18 scsi: target: uring: Map the command descriptors into the server
In reply to Qiliang YuanEach queue of a target_core_uring device has an array of command descriptors that the kernel fills.
The server cannot read the descriptors yet.
Let the server mmap() the descriptors of queue q_id at offset q_id * TCMUR_DESC_AREA_SIZE of the character device. Refuse writable mappings, as only the kernel writes descriptors.
--- drivers/target/target_core_uring.c | 22 +++++++++++++++++++++- include/uapi/linux/target_core_uring.h | 6 +++++- 2 files changed, 26 insertions(+), 2 deletions(-)
drivers/target/target_core_uring.c +21 -1
@@ -31,7 +31,7 @@ struct tcmur_dev;/** * struct tcmur_queue - One command queue -* @descs: Command descriptors, indexed by tag +* @descs: Command descriptors, indexed by tag and mapped by the server * @udev: Device the queue belongs to */ struct tcmur_queue {
@@ -79,6 +79,25 @@ static size_t tcmur_desc_area_size(struct tcmur_dev *udev)return PAGE_ALIGN (udev->queue_depth *sizeof (struct tcmur_io_desc )); } +static int tcmur_ch_mmap (struct file *filp,struct vm_area_struct *vma) +{ +struct tcmur_dev *udev = filp->private_data; +u64 off = (u64 )vma->vm_pgoff <<PAGE_SHIFT ; +u32 rem; +u64 q_id =div_u64_rem (off,TCMUR_DESC_AREA_SIZE , &rem); + +if (rem || q_id >= udev->nr_queues || + vma->vm_end - vma->vm_start !=tcmur_desc_area_size (udev)) +return -EINVAL ; + +/* Refuse writable mappings, as only the kernel writes descriptors */ +if (vma->vm_flags &VM_WRITE ) +return -EPERM ; +vm_flags_clear (vma,VM_MAYWRITE ); + +return remap_vmalloc_range (vma, udev->queues[q_id].descs,0 ); +} +static long tcmur_ch_ioctl (struct file *filp,unsigned int cmd,unsigned long arg) {
@@ -125,6 +144,7 @@ static const struct file_operations tcmur_ch_fops = { .owner = THIS_MODULE, .open = tcmur_ch_open, .release = tcmur_ch_release, + .mmap = tcmur_ch_mmap, .unlocked_ioctl = tcmur_ch_ioctl, .compat_ioctl = compat_ptr_ioctl, };
include/uapi/linux/target_core_uring.h +5 -1
@@ -54,7 +54,8 @@ enum tcmur_io_op {* @data_len: Bytes of data to transfer * @reserved: Zero * -* The descriptors of a queue form an array indexed by tag. +* The descriptors of a queue form an array indexed by tag, which the server +* maps read-only at offset q_id * TCMUR_DESC_AREA_SIZE of the file. */ struct tcmur_io_desc {__u32 op;
@@ -65,4 +66,7 @@ struct tcmur_io_desc { __u32 reserved; }; +#define TCMUR_DESC_AREA_SIZE \ + (TCMUR_MAX_QUEUE_DEPTH * sizeof(struct tcmur_io_desc)) +#endif /* _UAPI_LINUX_TARGET_CORE_URING_H */
Signature · 1 line
2.43.0
RFC PATCH 5/18 scsi: target: uring: Queue commands until a server fetches them
In reply to Qiliang Yuantarget_core_uring fails every read and write, as nothing can serve them.
A command can arrive before the server is ready for it: at startup, while all tags of a queue are busy, or while the server restarts.
Allocate a struct tcmur_cmd for each read and write, fill its descriptor and put it on the pending list of the queue of the submitting CPU. The commands wait there until a later patch lets a server fetch them.
--- drivers/target/target_core_uring.c | 81 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 78 insertions(+), 3 deletions(-)
drivers/target/target_core_uring.c +78 -3
@@ -12,6 +12,7 @@#include <linux/idr.h> #include <linux/mm.h> #include <linux/parser.h> +#include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> #include <target/target_core_base.h>
@@ -29,12 +30,32 @@struct tcmur_dev ; +/** +* struct tcmur_cmd - One command for the server +* @node: Entry in tcmur_queue.pending or in a local list +* @se_cmd: Target command this belongs to +* @sgl: Data pages of the command +* @sgl_nents: Number of entries in @sgl +* @desc: What the server is asked to do +*/ +struct tcmur_cmd { +struct list_head node; +struct se_cmd *se_cmd; +struct scatterlist *sgl; +u32 sgl_nents; +struct tcmur_io_desc desc; +}; +/** * struct tcmur_queue - One command queue +* @lock: Protect the list +* @pending: Commands waiting for a tag * @descs: Command descriptors, indexed by tag and mapped by the server * @udev: Device the queue belongs to */ struct tcmur_queue { +spinlock_t lock; +struct list_head pending;struct tcmur_io_desc *descs;struct tcmur_dev *udev; }____cacheline_aligned_in_smp ;
@@ -54,6 +75,7 @@ struct tcmur_dev {struct tcmur_queue *queues; }; +static struct kmem_cache *tcmur_cmd_cache;static DEFINE_IDA (tcmur_ida);static dev_t tcmur_devt;
@@ -66,12 +88,56 @@ static struct tcmur_dev *TCMUR_DEV(struct se_device *dev) return container_of(dev, struct tcmur_dev, se_dev); } +static struct tcmur_queue *tcmur_cmd_queue (struct tcmur_dev *udev, +struct se_cmd *se_cmd) +{ +return &udev->queues[(unsigned int )se_cmd->cpuid % udev->nr_queues]; +} + +static void tcmur_queue_cmd (struct tcmur_dev *udev,struct tcmur_cmd *tcmd) +{ +struct tcmur_queue *q =tcmur_cmd_queue (udev, tcmd->se_cmd); + +spin_lock (&q->lock); +list_add_tail (&tcmd->node, &q->pending); +spin_unlock (&q->lock); +} + +static sense_reason_t tcmur_submit (struct se_cmd *se_cmd,u32 op,u64 offset, +u64 len,struct scatterlist *sgl, +u32 sgl_nents) +{ +struct tcmur_dev *udev =TCMUR_DEV (se_cmd->se_dev); +struct tcmur_cmd *tcmd; + + tcmd =kmem_cache_zalloc (tcmur_cmd_cache,GFP_NOIO ); +if (!tcmd) +return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; + + tcmd->se_cmd = se_cmd; + tcmd->sgl = sgl; + tcmd->sgl_nents = sgl_nents; + tcmd->desc.op = op; + tcmd->desc.offset = offset; + tcmd->desc.len = len; + tcmd->desc.data_len = se_cmd->data_length; +if (op ==TCMUR_OP_WRITE && (se_cmd->se_cmd_flags &SCF_FUA )) + tcmd->desc.flags |=TCMUR_IO_F_FUA ; + +tcmur_queue_cmd (udev, tcmd); +return 0 ; +} +static sense_reason_t tcmur_execute_rw (struct se_cmd *se_cmd,struct scatterlist *sgl,u32 sgl_nents,enum dma_data_direction dir) { -/* Fail the commands until a server can pick them up */ -return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +return tcmur_submit (se_cmd, dir ==DMA_FROM_DEVICE ?TCMUR_OP_READ : +TCMUR_OP_WRITE , + se_cmd->t_task_lba * block_size, + se_cmd->data_length, sgl, sgl_nents); }static size_t tcmur_desc_area_size (struct tcmur_dev *udev)
@@ -175,6 +241,8 @@ static int tcmur_alloc_queues(struct tcmur_dev *udev)for (i =0 ; i < udev->nr_queues; i++) { q = &udev->queues[i]; +spin_lock_init (&q->lock); +INIT_LIST_HEAD (&q->pending); q->udev = udev; q->descs =vmalloc_user (tcmur_desc_area_size (udev));if (!q->descs) {
@@ -414,9 +482,13 @@ static int __init tcmur_module_init(void) {int ret; + tcmur_cmd_cache =KMEM_CACHE (tcmur_cmd,0 ); +if (!tcmur_cmd_cache) +return -ENOMEM ; + ret =alloc_chrdev_region (&tcmur_devt,0 ,TCMUR_MINORS ,"tcm-uring" );if (ret) -return ret; + goto destroy_cache; ret =class_register (&tcmur_class);if (ret)
@@ -432,6 +504,8 @@ static int __init tcmur_module_init(void)class_unregister (&tcmur_class); unregister_chrdev:unregister_chrdev_region (tcmur_devt,TCMUR_MINORS ); +destroy_cache: +kmem_cache_destroy (tcmur_cmd_cache);return ret; }
@@ -442,6 +516,7 @@ static void __exit tcmur_module_exit(void)unregister_chrdev_region (tcmur_devt,TCMUR_MINORS );/* Wait for the devices that tcmur_cdev_release() frees with RCU */ rcu_barrier (); +kmem_cache_destroy (tcmur_cmd_cache); }module_init (tcmur_module_init);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 05/18] scsi: target: uring: Queue commands until a server fetches them
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: target: uring: missing TCMUR_MAX_IO_SIZE check in tcmur_execute_rw() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: target: uring: missing TCMUR_MAX_IO_SIZE check in tcmur_execute_rw()
--
commit aaa41f59a5c1c48ba973ecbd7b4b910b4c814276
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Queue commands until a server fetches them
This commit allocates a struct tcmur_cmd for each read and write command, populates its I/O descriptor, and queues it on the pending list of the submitting CPU's queue.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
Quoted 47 lines
drivers/target/target_core_uring.c +40
-0@@ -66,12 +88,56 @@ static struct tcmur_dev *TCMUR_DEV(struct se_device *dev) return container_of(dev, struct tcmur_dev, se_dev); } +static struct tcmur_queue *tcmur_cmd_queue (struct tcmur_dev *udev, +struct se_cmd *se_cmd) +{ +return &udev->queues[(unsigned int )se_cmd->cpuid % udev->nr_queues]; +} + +static void tcmur_queue_cmd (struct tcmur_dev *udev,struct tcmur_cmd *tcmd) +{ +struct tcmur_queue *q =tcmur_cmd_queue (udev, tcmd->se_cmd); + +spin_lock (&q->lock); +list_add_tail (&tcmd->node, &q->pending); +spin_unlock (&q->lock); +} + +static sense_reason_t tcmur_submit (struct se_cmd *se_cmd,u32 op,u64 offset, +u64 len,struct scatterlist *sgl, +u32 sgl_nents) +{ +struct tcmur_dev *udev =TCMUR_DEV (se_cmd->se_dev); +struct tcmur_cmd *tcmd; + + tcmd =kmem_cache_zalloc (tcmur_cmd_cache,GFP_NOIO ); +if (!tcmd) +return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; + + tcmd->se_cmd = se_cmd; + tcmd->sgl = sgl; + tcmd->sgl_nents = sgl_nents; + tcmd->desc.op = op; + tcmd->desc.offset = offset; + tcmd->desc.len = len; + tcmd->desc.data_len = se_cmd->data_length; +if (op ==TCMUR_OP_WRITE && (se_cmd->se_cmd_flags &SCF_FUA )) + tcmd->desc.flags |=TCMUR_IO_F_FUA ; + +tcmur_queue_cmd (udev, tcmd); +return 0 ; +} +static sense_reason_t tcmur_execute_rw (struct se_cmd *se_cmd,struct scatterlist *sgl,u32 sgl_nents,enum dma_data_direction dir)
drivers/target/target_core_uring.c static struct tcmur_dev *TCMUR_DEV(struct se_device *dev) · line 141
Quoted 10 lines
drivers/target/target_core_uring.c +6
-2{ -/* Fail the commands until a server can pick them up */ -return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +return tcmur_submit (se_cmd, dir ==DMA_FROM_DEVICE ?TCMUR_OP_READ : +TCMUR_OP_WRITE , + se_cmd->t_task_lba * block_size, + se_cmd->data_length, sgl, sgl_nents); }
[Severity: High]
Can tcmur_execute_rw() queue a command with tcmd->desc.data_len larger than the TCMUR_MAX_IO_SIZE limit advertised by TCMUR_GET_DEV_INFO in info.max_io_size?
Although tcmur_configure_device() sets dev->dev_attrib.hw_max_sectors to TCMUR_MAX_IO_SIZE / udev->block_size, the target core only uses hw_max_sectors to report the advisory maximum transfer length in Block Limits VPD page 0xb0 and does not reject READ or WRITE commands that exceed it (unlike fd_execute_rw(), which checks cmd->data_length > FD_MAX_BYTES). An initiator can send a READ or WRITE command larger than 4 MiB and cause a userspace server that sizes its I/O buffer to info.max_io_size (such as tcmur_server.c in this series) to overflow its buffer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
RFC PATCH 6/18 scsi: target: uring: Hand commands out through TCMUR_IO_FETCH
In reply to Qiliang YuanReads and writes of a target_core_uring device wait on the pending list of their queue.
Nothing takes commands off that list.
Add the TCMUR_IO_FETCH io_uring command. The server issues one per tag; it completes once the descriptor of the tag holds a command. If a command is pending, hand it out right away. Otherwise park the fetch command on the idle list of the queue and complete it from task work when a command arrives. Complete parked fetch commands with -ECANCELED when io_uring cancels them, and with -ENODEV when the device goes away.
--- drivers/target/target_core_uring.c | 216 ++++++++++++++++++++++++++++++++- include/uapi/linux/target_core_uring.h | 24 +++- 2 files changed, 235 insertions(+), 5 deletions(-)
drivers/target/target_core_uring.c +212 -4
@@ -2,6 +2,10 @@/* * LIO backstore that hands block commands to a userspace server through * io_uring commands. +* +* Each device has a character device and a set of queues. A queue holds +* queue_depth tags; the server owns a tag from the moment the kernel +* completes its fetch command until it commits the result. */ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt
@@ -10,6 +14,7 @@#include <linux/cdev.h> #include <linux/device.h> #include <linux/idr.h> +#include <linux/io_uring/cmd.h> #include <linux/mm.h> #include <linux/parser.h> #include <linux/scatterlist.h>
@@ -46,16 +51,34 @@ struct tcmur_cmd {struct tcmur_io_desc desc; }; +/** +* struct tcmur_io - State of one tag +* @node: Entry in tcmur_queue.idle while @ucmd waits for a command +* @ucmd: Fetch command of the server +* @cmd: Command the server is working on +* @owned: The server owns the tag and has to commit it +*/ +struct tcmur_io { +struct list_head node; +struct io_uring_cmd *ucmd; +struct tcmur_cmd *cmd; +bool owned; +}; +/** * struct tcmur_queue - One command queue -* @lock: Protect the list +* @lock: Protect the lists and the tags +* @idle: Tags with a fetch command waiting for a command * @pending: Commands waiting for a tag +* @ios: Tags of the queue * @descs: Command descriptors, indexed by tag and mapped by the server * @udev: Device the queue belongs to */ struct tcmur_queue {spinlock_t lock; +struct list_head idle;struct list_head pending; +struct tcmur_io *ios;struct tcmur_io_desc *descs;struct tcmur_dev *udev; }____cacheline_aligned_in_smp ;
@@ -66,6 +89,7 @@ struct tcmur_dev {struct cdev cdev;int dev_id;unsigned long flags; +bool dead;u64 dev_size;u32 block_size;
@@ -75,6 +99,18 @@ struct tcmur_dev {struct tcmur_queue *queues; }; +/** +* struct tcmur_uring_pdu - State of a parked fetch command +* @q: Queue of the tag +* @io: Tag the command fetches for +* @res: Result to complete the command with +*/ +struct tcmur_uring_pdu { +struct tcmur_queue *q; +struct tcmur_io *io; +int res; +}; +static struct kmem_cache *tcmur_cmd_cache;static DEFINE_IDA (tcmur_ida);static dev_t tcmur_devt;
@@ -94,13 +130,46 @@ static struct tcmur_queue *tcmur_cmd_queue(struct tcmur_dev *udev,return &udev->queues[(unsigned int )se_cmd->cpuid % udev->nr_queues]; } +/* Hand @tcmd to the server through the tag @io */ +static void tcmur_io_start (struct tcmur_queue *q,struct tcmur_io *io, +struct tcmur_cmd *tcmd) +{ +lockdep_assert_held (&q->lock); + + io->cmd = tcmd; + io->owned = true; + q->descs[io - q->ios] = tcmd->desc; +} + +static void tcmur_fetch_tw_cb (struct io_tw_req tw_req,io_tw_token_t tw) +{ +struct io_uring_cmd *ucmd =io_uring_cmd_from_tw (tw_req); +struct tcmur_uring_pdu *pdu =io_uring_cmd_to_pdu (ucmd, + struct tcmur_uring_pdu); + +io_uring_cmd_done (ucmd, pdu->res,IO_URING_CMD_TASK_WORK_ISSUE_FLAGS ); +} +static void tcmur_queue_cmd (struct tcmur_dev *udev,struct tcmur_cmd *tcmd) {struct tcmur_queue *q =tcmur_cmd_queue (udev, tcmd->se_cmd); +struct io_uring_cmd *ucmd =NULL ; +struct tcmur_io *io;spin_lock (&q->lock); -list_add_tail (&tcmd->node, &q->pending); + io = list_first_entry_or_null(&q->idle, struct tcmur_io, node); +if (io) { +list_del (&io->node); + ucmd = io->ucmd; + io->ucmd =NULL ; +tcmur_io_start (q, io, tcmd); + }else { +list_add_tail (&tcmd->node, &q->pending); + }spin_unlock (&q->lock); + +if (ucmd) +io_uring_cmd_complete_in_task (ucmd, tcmur_fetch_tw_cb); } static sense_reason_t tcmur_submit(struct se_cmd *se_cmd, u32 op, u64 offset,
@@ -140,6 +209,113 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static void tcmur_cancel_fetch (struct io_uring_cmd *ucmd, +unsigned int issue_flags) +{ +struct tcmur_uring_pdu *pdu =io_uring_cmd_to_pdu (ucmd, + struct tcmur_uring_pdu); +struct tcmur_queue *q = pdu->q; +struct tcmur_io *io = pdu->io; +bool found = false; + +spin_lock (&q->lock); +if (io->ucmd == ucmd) { +list_del (&io->node); + io->ucmd =NULL ; + found = true; + } +spin_unlock (&q->lock); + +if (found) +io_uring_cmd_done (ucmd, -ECANCELED , issue_flags); +} + +static int tcmur_fetch (struct tcmur_queue *q,struct tcmur_io *io, +struct io_uring_cmd *ucmd,unsigned int issue_flags) +{ +struct tcmur_uring_pdu *pdu =io_uring_cmd_to_pdu (ucmd, + struct tcmur_uring_pdu); +struct tcmur_cmd *tcmd; +int ret =0 ; + +spin_lock (&q->lock); +if (q->udev->dead) { + ret = -ENODEV ; + }else if (io->ucmd || io->owned) { + ret = -EBUSY ; + }else if (!list_empty (&q->pending)) { + tcmd = list_first_entry(&q->pending, struct tcmur_cmd, node); +list_del (&tcmd->node); +tcmur_io_start (q, io, tcmd); + }else { + pdu->q = q; + pdu->io = io; + pdu->res =0 ; + io->ucmd = ucmd; +list_add (&io->node, &q->idle); + ret = -EIOCBQUEUED ; + } +spin_unlock (&q->lock); + +if (ret == -EIOCBQUEUED ) +io_uring_cmd_mark_cancelable (ucmd, issue_flags); +return ret; +} + +static int tcmur_ch_uring_cmd_local (struct io_uring_cmd *ucmd, +unsigned int issue_flags) +{ +const struct tcmur_io_cmd *ioc =io_uring_sqe_cmd (ucmd->sqe, + struct tcmur_io_cmd); +struct tcmur_dev *udev = ucmd->file->private_data; +u16 q_id =READ_ONCE (ioc->q_id); +u16 tag =READ_ONCE (ioc->tag); +struct tcmur_queue *q; +struct tcmur_io *io; + +if (q_id >= udev->nr_queues || tag >= udev->queue_depth) +return -EINVAL ; + + q = &udev->queues[q_id]; + io = &q->ios[tag]; + +switch (ucmd->cmd_op) { +case TCMUR_IO_FETCH : +return tcmur_fetch (q, io, ucmd, issue_flags); +default : +return -EOPNOTSUPP ; + } +} + +static void tcmur_ch_uring_cmd_tw_cb (struct io_tw_req tw_req,io_tw_token_t tw) +{ +unsigned int issue_flags =IO_URING_CMD_TASK_WORK_ISSUE_FLAGS ; +struct io_uring_cmd *ucmd =io_uring_cmd_from_tw (tw_req); +int ret = -ECANCELED ; + +if (!tw.cancel) + ret =tcmur_ch_uring_cmd_local (ucmd, issue_flags); +if (ret != -EIOCBQUEUED ) +io_uring_cmd_done (ucmd, ret, issue_flags); +} + +static int tcmur_ch_uring_cmd (struct io_uring_cmd *ucmd, +unsigned int issue_flags) +{ +if (unlikely (issue_flags &IO_URING_F_CANCEL )) { +tcmur_cancel_fetch (ucmd, issue_flags); +return 0 ; + } + +/* Run in the submitting task, where fetch commands are completed */ +if (unlikely (issue_flags &IO_URING_F_UNLOCKED )) { +io_uring_cmd_complete_in_task (ucmd, tcmur_ch_uring_cmd_tw_cb); +return -EIOCBQUEUED ; + } + +return tcmur_ch_uring_cmd_local (ucmd, issue_flags); +} +static size_t tcmur_desc_area_size (struct tcmur_dev *udev) {return PAGE_ALIGN (udev->queue_depth *sizeof (struct tcmur_io_desc ));
@@ -210,6 +386,7 @@ static const struct file_operations tcmur_ch_fops = { .owner = THIS_MODULE, .open = tcmur_ch_open, .release = tcmur_ch_release, + .uring_cmd = tcmur_ch_uring_cmd, .mmap = tcmur_ch_mmap, .unlocked_ioctl = tcmur_ch_ioctl, .compat_ioctl = compat_ptr_ioctl,
@@ -222,8 +399,10 @@ static void tcmur_free_queues(struct tcmur_dev *udev)if (!udev->queues)return ; -for (i =0 ; i < udev->nr_queues; i++) +for (i =0 ; i < udev->nr_queues; i++) {vfree (udev->queues[i].descs); +kvfree (udev->queues[i].ios); + }kvfree (udev->queues); udev->queues =NULL ; }
@@ -242,10 +421,13 @@ static int tcmur_alloc_queues(struct tcmur_dev *udev) q = &udev->queues[i];spin_lock_init (&q->lock); +INIT_LIST_HEAD (&q->idle);INIT_LIST_HEAD (&q->pending); q->udev = udev; + q->ios =kvcalloc (udev->queue_depth,sizeof (*q->ios), +GFP_KERNEL ); q->descs =vmalloc_user (tcmur_desc_area_size (udev)); -if (!q->descs) { +if (!q->ios || !q->descs) {tcmur_free_queues (udev);return -ENOMEM ; }
@@ -348,8 +530,34 @@ static int tcmur_configure_device(struct se_device *dev)static void tcmur_destroy_device (struct se_device *dev) {struct tcmur_dev *udev =TCMUR_DEV (dev); +struct tcmur_uring_pdu *pdu; +struct io_uring_cmd *ucmd; +struct tcmur_queue *q; +struct tcmur_io *io; +unsigned int i;cdev_device_del (&udev->cdev, &udev->cdev_dev); + +/* +* Tell the server that the device is gone. Complete the fetch +* commands from task work, like tcmur_queue_cmd() does. +*/ +for (i =0 ; i < udev->nr_queues; i++) { + q = &udev->queues[i]; + +spin_lock (&q->lock); + udev->dead = true; +while (!list_empty (&q->idle)) { + io = list_first_entry(&q->idle, struct tcmur_io, node); +list_del (&io->node); + ucmd = io->ucmd; + io->ucmd =NULL ; + pdu = io_uring_cmd_to_pdu(ucmd, struct tcmur_uring_pdu); + pdu->res = -ENODEV ; +io_uring_cmd_complete_in_task (ucmd, tcmur_fetch_tw_cb); + } +spin_unlock (&q->lock); + } }static void tcmur_free_device (struct se_device *dev)
include/uapi/linux/target_core_uring.h +23 -1
@@ -7,7 +7,9 @@/* * target_core_uring hands the block commands of a LIO device to a userspace -* server. The server opens /dev/tcmurN to drive the device. +* server. The server opens /dev/tcmurN, maps the command descriptors of +* each queue and exchanges commands with the kernel through io_uring +* commands (IORING_OP_URING_CMD) issued on that file. */ #define TCMUR_MAX_NR_QUEUES 512
@@ -37,6 +39,26 @@ struct tcmur_dev_info {#define TCMUR_GET_DEV_INFO _IOR(0xBB, 0x00, struct tcmur_dev_info) +/** +* struct tcmur_io_cmd - Payload of the io_uring commands, placed in sqe->cmd +* @q_id: Queue of the tag +* @tag: Tag the command is for +* @reserved: Zero +* @reserved1: Zero +*/ +struct tcmur_io_cmd { +__u16 q_id; +__u16 tag; +__s32 reserved; +__u64 reserved1; +}; + +/* +* TCMUR_IO_FETCH: wait for a command on (q_id, tag). The CQE result is 0 +* once the descriptor of the tag holds a command. +*/ +#define TCMUR_IO_FETCH _IOWR(0xBB, 0x08, struct tcmur_io_cmd) +enum tcmur_io_op {TCMUR_OP_READ ,TCMUR_OP_WRITE ,
Signature · 1 line
2.43.0
RFC PATCH 7/18 scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH
In reply to Qiliang YuanThe server of a target_core_uring device can fetch a command for each tag.
It cannot report the result, so the command never completes and the tag is never used again.
Add the TCMUR_IO_COMMIT_AND_FETCH io_uring command, which completes the command of a tag with GOOD or CHECK CONDITION status and fetches the next one in the same call. Count references on struct tcmur_cmd and complete the target command when the last one goes, as later patches add users that can outlive the commit.
--- drivers/target/target_core_uring.c | 47 ++++++++++++++++++++++++++++++++++ include/uapi/linux/target_core_uring.h | 9 +++++-- 2 files changed, 54 insertions(+), 2 deletions(-)
drivers/target/target_core_uring.c +47 -0
@@ -20,6 +20,7 @@#include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> +#include <scsi/scsi_proto.h> #include <target/target_core_base.h> #include <target/target_core_backend.h> #include <uapi/linux/target_core_uring.h>
@@ -42,6 +43,8 @@ struct tcmur_dev;* @sgl: Data pages of the command * @sgl_nents: Number of entries in @sgl * @desc: What the server is asked to do +* @ref: Held by the tag or the pending list +* @status: SAM status to complete @se_cmd with */ struct tcmur_cmd {struct list_head node;
@@ -49,6 +52,8 @@ struct tcmur_cmd {struct scatterlist *sgl;u32 sgl_nents;struct tcmur_io_desc desc; +refcount_t ref; +u8 status; };/**
@@ -130,6 +135,22 @@ static struct tcmur_queue *tcmur_cmd_queue(struct tcmur_dev *udev,return &udev->queues[(unsigned int )se_cmd->cpuid % udev->nr_queues]; } +static void tcmur_cmd_put (struct tcmur_cmd *tcmd) +{ +if (!refcount_dec_and_test (&tcmd->ref)) +return ; + +target_complete_cmd (tcmd->se_cmd, tcmd->status); +kmem_cache_free (tcmur_cmd_cache, tcmd); +} + +/* Drop the reference of the tag or of the pending list and set the result */ +static void tcmur_cmd_end (struct tcmur_cmd *tcmd,u8 status) +{ + tcmd->status = status; +tcmur_cmd_put (tcmd); +} +/* Hand @tcmd to the server through the tag @io */ static void tcmur_io_start (struct tcmur_queue *q,struct tcmur_io *io,struct tcmur_cmd *tcmd)
@@ -156,6 +177,8 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd)struct io_uring_cmd *ucmd =NULL ;struct tcmur_io *io; +refcount_set (&tcmd->ref,1 ); +spin_lock (&q->lock); io = list_first_entry_or_null(&q->idle, struct tcmur_io, node);if (io) {
@@ -262,6 +285,24 @@ static int tcmur_fetch(struct tcmur_queue *q, struct tcmur_io *io,return ret; } +static int tcmur_commit (struct tcmur_queue *q,struct tcmur_io *io,int result) +{ +struct tcmur_cmd *tcmd; + +spin_lock (&q->lock); +if (!io->owned) { +spin_unlock (&q->lock); +return -EINVAL ; + } + tcmd = io->cmd; + io->cmd =NULL ; + io->owned = false; +spin_unlock (&q->lock); + +tcmur_cmd_end (tcmd, result ?SAM_STAT_CHECK_CONDITION :SAM_STAT_GOOD ); +return 0 ; +} +static int tcmur_ch_uring_cmd_local (struct io_uring_cmd *ucmd,unsigned int issue_flags) {
@@ -272,6 +313,7 @@ static int tcmur_ch_uring_cmd_local(struct io_uring_cmd *ucmd,u16 tag =READ_ONCE (ioc->tag);struct tcmur_queue *q;struct tcmur_io *io; +int ret;if (q_id >= udev->nr_queues || tag >= udev->queue_depth)return -EINVAL ;
@@ -282,6 +324,11 @@ static int tcmur_ch_uring_cmd_local(struct io_uring_cmd *ucmd,switch (ucmd->cmd_op) {case TCMUR_IO_FETCH :return tcmur_fetch (q, io, ucmd, issue_flags); +case TCMUR_IO_COMMIT_AND_FETCH : + ret =tcmur_commit (q, io,READ_ONCE (ioc->result)); +if (ret) +return ret; +return tcmur_fetch (q, io, ucmd, issue_flags);default :return -EOPNOTSUPP ; }
include/uapi/linux/target_core_uring.h +7 -2
@@ -43,21 +43,26 @@ struct tcmur_dev_info {* struct tcmur_io_cmd - Payload of the io_uring commands, placed in sqe->cmd * @q_id: Queue of the tag * @tag: Tag the command is for -* @reserved: Zero +* @result: TCMUR_IO_COMMIT_AND_FETCH: 0, or a negative errno if the +* command failed * @reserved1: Zero */ struct tcmur_io_cmd {__u16 q_id;__u16 tag; -__s32 reserved; +__s32 result;__u64 reserved1; };/* * TCMUR_IO_FETCH: wait for a command on (q_id, tag). The CQE result is 0 * once the descriptor of the tag holds a command. +* +* TCMUR_IO_COMMIT_AND_FETCH: complete the command of (q_id, tag) with +* @result, then wait for the next one. */ #define TCMUR_IO_FETCH _IOWR(0xBB, 0x08, struct tcmur_io_cmd) +#define TCMUR_IO_COMMIT_AND_FETCH _IOWR(0xBB, 0x09, struct tcmur_io_cmd)enum tcmur_io_op {TCMUR_OP_READ ,
Signature · 1 line
2.43.0
RFC PATCH 8/18 scsi: target: uring: Copy command data with pread() and pwrite()
In reply to Qiliang YuanThe server of a target_core_uring device fetches commands and commits their results.
It has no access to the data of reads and writes.
Let the server pwrite() the data of a read command and pread() the data of a write command on the character device, at a position that encodes the queue, the tag and the offset in the command. Copy straight between the pages of the command and the buffer of the server, so that the data is copied once. Hold a reference on the command during the copy, so that it does not complete underneath.
Exporting a 2 GiB device through tcm_loop to fio (io_uring, direct I/O, iodepth 32 for 4k and 8 for 1M, 6 seconds per job) on a 32 CPU VM, with TCM/USER and a single-threaded handler, then with this backstore, 32 queues and one server thread per queue. Both servers keep the data in memory. A single job is slower; the gain comes with concurrency and with large I/O. Three alternating runs (median, min-max in parentheses):
workload TCM/USER (MB/s) uring (MB/s) change
4k randread, 1 job 426 (409-463) 354 (348-378) -17%
4k randread, 4 jobs 1165 (1072-1253) 1466 (1451-1518) +26% 4k randread, 16 jobs 920 (916-1162) 6500 (6173-6769) +607% 4k randwrite, 16 jobs 793 (711-886) 6070 (5966-6216) +665% 1M read, 4 jobs 6019 (5857-6047) 47687 (32175-49464) +692% 1M write, 4 jobs 2791 (2649-2907) 28295 (27152-29050) +914%
--- drivers/target/target_core_uring.c | 89 +++++++++++++++++++++++++++++++++- include/uapi/linux/target_core_uring.h | 15 ++++++ 2 files changed, 103 insertions(+), 1 deletion(-)
drivers/target/target_core_uring.c +88 -1
@@ -19,6 +19,7 @@#include <linux/parser.h> #include <linux/scatterlist.h> #include <linux/slab.h> +#include <linux/uio.h> #include <linux/vmalloc.h> #include <scsi/scsi_proto.h> #include <target/target_core_base.h>
@@ -43,7 +44,8 @@ struct tcmur_dev;* @sgl: Data pages of the command * @sgl_nents: Number of entries in @sgl * @desc: What the server is asked to do -* @ref: Held by the tag or the pending list +* @ref: Held by the tag or the pending list and by data copies in +* progress * @status: SAM status to complete @se_cmd with */ struct tcmur_cmd {
@@ -232,6 +234,26 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +/* Take a reference on the command the server works on in (@q_id, @tag) */ +static struct tcmur_cmd *tcmur_get_cmd (struct tcmur_dev *udev,u16 q_id, +u16 tag) +{ +struct tcmur_cmd *tcmd; +struct tcmur_queue *q; + +if (q_id >= udev->nr_queues || tag >= udev->queue_depth) +return NULL ; + + q = &udev->queues[q_id]; +spin_lock (&q->lock); + tcmd = q->ios[tag].cmd; +if (tcmd) +refcount_inc (&tcmd->ref); +spin_unlock (&q->lock); + +return tcmd; +} +static void tcmur_cancel_fetch (struct io_uring_cmd *ucmd,unsigned int issue_flags) {
@@ -363,6 +385,69 @@ static int tcmur_ch_uring_cmd(struct io_uring_cmd *ucmd,return tcmur_ch_uring_cmd_local (ucmd, issue_flags); } +/* Copy between the data pages of a command and the buffer of the server */ +static ssize_t tcmur_ch_copy (struct kiocb *iocb,struct iov_iter *iter, +bool to_cmd) +{ +struct tcmur_dev *udev = iocb->ki_filp->private_data; +u32 off = iocb->ki_pos &TCMUR_BUF_OFF_MASK ; +u16 tag = iocb->ki_pos >>TCMUR_BUF_TAG_SHIFT ; +u16 q_id = iocb->ki_pos >>TCMUR_BUF_QID_SHIFT ; +size_t count, n, copied; +struct tcmur_cmd *tcmd; +struct scatterlist *sg; +ssize_t done =0 ; +int i; + + tcmd =tcmur_get_cmd (udev, q_id, tag); +if (!tcmd) +return -EINVAL ; + +if (to_cmd != (tcmd->desc.op ==TCMUR_OP_READ ) || + off > tcmd->desc.data_len) { + done = -EINVAL ; + goto put_cmd; + } + + count =min (iov_iter_count (iter), tcmd->desc.data_len - off); +for_each_sg (tcmd->sgl, sg, tcmd->sgl_nents, i) { +if (!count) +break ; +if (off >= sg->length) { + off -= sg->length; +continue ; + } + + n =min (sg->length - off, count); +if (to_cmd) + copied =copy_page_from_iter (sg_page (sg), + sg->offset + off, n, iter); +else + copied =copy_page_to_iter (sg_page (sg), + sg->offset + off, n, iter); + done += copied; + count -= copied; +if (copied != n) +break ; + off =0 ; + } +if (!done && count) + done = -EFAULT ; +put_cmd: +tcmur_cmd_put (tcmd); +return done; +} + +static ssize_t tcmur_ch_read_iter (struct kiocb *iocb,struct iov_iter *to) +{ +return tcmur_ch_copy (iocb, to, false); +} + +static ssize_t tcmur_ch_write_iter (struct kiocb *iocb,struct iov_iter *from) +{ +return tcmur_ch_copy (iocb, from, true); +} +static size_t tcmur_desc_area_size (struct tcmur_dev *udev) {return PAGE_ALIGN (udev->queue_depth *sizeof (struct tcmur_io_desc ));
@@ -433,6 +518,8 @@ static const struct file_operations tcmur_ch_fops = { .owner = THIS_MODULE, .open = tcmur_ch_open, .release = tcmur_ch_release, + .read_iter = tcmur_ch_read_iter, + .write_iter = tcmur_ch_write_iter, .uring_cmd = tcmur_ch_uring_cmd, .mmap = tcmur_ch_mmap, .unlocked_ioctl = tcmur_ch_ioctl,
include/uapi/linux/target_core_uring.h +15 -0
@@ -96,4 +96,19 @@ struct tcmur_io_desc {#define TCMUR_DESC_AREA_SIZE \ (TCMUR_MAX_QUEUE_DEPTH * sizeof(struct tcmur_io_desc)) +/* +* The data of a command is read (TCMUR_OP_WRITE) or written +* (TCMUR_OP_READ) with pread()/pwrite() on the file, at the position +* returned by tcmur_buf_pos(). +*/ +#define TCMUR_BUF_TAG_SHIFT 32 +#define TCMUR_BUF_QID_SHIFT 48 +#define TCMUR_BUF_OFF_MASK ((1ULL << TCMUR_BUF_TAG_SHIFT) - 1) + +static inline __u64 tcmur_buf_pos (__u16 q_id,__u16 tag,__u32 off) +{ +return ((__u64 )q_id <<TCMUR_BUF_QID_SHIFT ) | + ((__u64 )tag <<TCMUR_BUF_TAG_SHIFT ) | off; +} +#endif /* _UAPI_LINUX_TARGET_CORE_URING_H */
Signature · 1 line
2.43.0
RFC PATCH 9/18 scsi: target: uring: Fail in-flight commands when the server exits
In reply to Qiliang YuanA tag of a target_core_uring device belongs to the server from the moment it fetches a command until it commits the result.
If the server exits or crashes in between, the command never completes and the tag stays owned, so a new server cannot fetch on it.
Fail the commands the server was working on when it closes the character device, and give the tags back. Commands that were not fetched yet stay on the pending lists for the next server.
--- drivers/target/target_core_uring.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
drivers/target/target_core_uring.c +31 -0
@@ -153,6 +153,16 @@ static void tcmur_cmd_end(struct tcmur_cmd *tcmd, u8 status)tcmur_cmd_put (tcmd); } +static void tcmur_end_cmds (struct list_head *cmds,u8 status) +{ +struct tcmur_cmd *tcmd, *tmp; + +list_for_each_entry_safe (tcmd, tmp, cmds, node) { +list_del (&tcmd->node); +tcmur_cmd_end (tcmd, status); + } +} +/* Hand @tcmd to the server through the tag @io */ static void tcmur_io_start (struct tcmur_queue *q,struct tcmur_io *io,struct tcmur_cmd *tcmd)
@@ -509,6 +519,27 @@ static int tcmur_ch_open(struct inode *inode, struct file *filp)static int tcmur_ch_release (struct inode *inode,struct file *filp) {struct tcmur_dev *udev = filp->private_data; +struct tcmur_queue *q; +struct tcmur_io *io; +LIST_HEAD (failed); +unsigned int i, j; + +/* Fail the commands that the server did not commit */ +for (i =0 ; i < udev->nr_queues; i++) { + q = &udev->queues[i]; + +spin_lock (&q->lock); +for (j =0 ; j < udev->queue_depth; j++) { + io = &q->ios[j]; + io->owned = false; +if (!io->cmd) +continue ; +list_add_tail (&io->cmd->node, &failed); + io->cmd =NULL ; + } +spin_unlock (&q->lock); + } +tcmur_end_cmds (&failed,SAM_STAT_CHECK_CONDITION );clear_bit (TCMUR_DEV_OPEN , &udev->flags);return 0 ;
Signature · 1 line
2.43.0
RFC PATCH 10/18 scsi: target: uring: Time out commands the server does not complete
In reply to Qiliang YuanCommands of a target_core_uring device wait on a pending list until a server fetches them, and then for the server to commit them.
Without a server, or with one that hangs, they wait forever, and so does everything that waits for them, such as removing the LUN.
Add the cmd_time_out parameter, 30 seconds by default, and fail the commands that are older. A delayed work scans the queues when the oldest command is due. A timed out command that the server works on is completed, but the server keeps the tag until it commits it.
--- drivers/target/target_core_uring.c | 80 +++++++++++++++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 2 deletions(-)
drivers/target/target_core_uring.c +78 -2
@@ -21,6 +21,7 @@#include <linux/slab.h> #include <linux/uio.h> #include <linux/vmalloc.h> +#include <linux/workqueue.h> #include <scsi/scsi_proto.h> #include <target/target_core_base.h> #include <target/target_core_backend.h>
@@ -31,6 +32,7 @@#define TCMUR_MAX_IO_SIZE (4U << 20)#define TCMUR_DEF_QUEUE_DEPTH 128#define TCMUR_DEF_BLOCK_SIZE 512 +#define TCMUR_DEF_CMD_TIMEOUT 30/* Set while the file is open, to allow one server at a time */ #define TCMUR_DEV_OPEN 0
@@ -46,6 +48,7 @@ struct tcmur_dev;* @desc: What the server is asked to do * @ref: Held by the tag or the pending list and by data copies in * progress +* @deadline: When to give up waiting for the server * @status: SAM status to complete @se_cmd with */ struct tcmur_cmd {
@@ -55,6 +58,7 @@ struct tcmur_cmd {u32 sgl_nents;struct tcmur_io_desc desc;refcount_t ref; +unsigned long deadline;u8 status; };
@@ -102,8 +106,10 @@ struct tcmur_dev {u32 block_size;u16 nr_queues;u16 queue_depth; +unsigned int cmd_timeout;struct tcmur_queue *queues; +struct delayed_work timeout_work; };/**
@@ -190,6 +196,8 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd)struct tcmur_io *io;refcount_set (&tcmd->ref,1 ); +if (udev->cmd_timeout) + tcmd->deadline = jiffies + udev->cmd_timeout *HZ ;spin_lock (&q->lock); io = list_first_entry_or_null(&q->idle, struct tcmur_io, node);
@@ -205,6 +213,9 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd)if (ucmd)io_uring_cmd_complete_in_task (ucmd, tcmur_fetch_tw_cb); +if (udev->cmd_timeout) +schedule_delayed_work (&udev->timeout_work, + udev->cmd_timeout *HZ ); } static sense_reason_t tcmur_submit(struct se_cmd *se_cmd, u32 op, u64 offset,
@@ -244,6 +255,58 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static void tcmur_timeout_work (struct work_struct *work) +{ + struct tcmur_dev *udev = container_of(work, struct tcmur_dev, + timeout_work.work); +unsigned long now = jiffies, next =0 ; +struct tcmur_cmd *tcmd, *tmp; +struct tcmur_queue *q; +struct tcmur_io *io; +bool rearm = false; +LIST_HEAD (expired); +unsigned int i, j; + +for (i =0 ; i < udev->nr_queues; i++) { + q = &udev->queues[i]; + +spin_lock (&q->lock); +list_for_each_entry_safe (tcmd, tmp, &q->pending, node) { +if (time_after (tcmd->deadline, now)) { +if (!rearm ||time_before (tcmd->deadline, next)) + next = tcmd->deadline; + rearm = true; +break ; + } +list_move_tail (&tcmd->node, &expired); + } +for (j =0 ; j < udev->queue_depth; j++) { + io = &q->ios[j]; + tcmd = io->cmd; +if (!tcmd) +continue ; +if (time_after (tcmd->deadline, now)) { +if (!rearm ||time_before (tcmd->deadline, next)) + next = tcmd->deadline; + rearm = true; +continue ; + } +/* Leave the tag to the server until it commits it */ + io->cmd =NULL ; +list_add_tail (&tcmd->node, &expired); + } +spin_unlock (&q->lock); + } + +if (!list_empty (&expired)) +pr_warn_ratelimited ("tcmur%d: command timed out\n" , + udev->dev_id); +tcmur_end_cmds (&expired,SAM_STAT_CHECK_CONDITION ); + +if (rearm) +schedule_delayed_work (&udev->timeout_work, next - now); +} +/* Take a reference on the command the server works on in (@q_id, @tag) */ static struct tcmur_cmd *tcmur_get_cmd (struct tcmur_dev *udev,u16 q_id,u16 tag)
@@ -331,7 +394,10 @@ static int tcmur_commit(struct tcmur_queue *q, struct tcmur_io *io, int result) io->owned = false;spin_unlock (&q->lock); -tcmur_cmd_end (tcmd, result ?SAM_STAT_CHECK_CONDITION :SAM_STAT_GOOD ); +/* Skip a command that timed out already */ +if (tcmd) +tcmur_cmd_end (tcmd, result ?SAM_STAT_CHECK_CONDITION : +SAM_STAT_GOOD );return 0 ; }
@@ -639,6 +705,8 @@ static struct se_device *tcmur_alloc_device(struct se_hba *hba, udev->nr_queues =1 ; udev->queue_depth =TCMUR_DEF_QUEUE_DEPTH ; udev->block_size =TCMUR_DEF_BLOCK_SIZE ; + udev->cmd_timeout =TCMUR_DEF_CMD_TIMEOUT ; +INIT_DELAYED_WORK (&udev->timeout_work, tcmur_timeout_work);device_initialize (&udev->cdev_dev); udev->cdev_dev.class = &tcmur_class;
@@ -723,6 +791,8 @@ static void tcmur_destroy_device(struct se_device *dev) }spin_unlock (&q->lock); } + +cancel_delayed_work_sync (&udev->timeout_work); }static void tcmur_free_device (struct se_device *dev)
@@ -731,7 +801,8 @@ static void tcmur_free_device(struct se_device *dev) }enum { - Opt_dev_size, Opt_block_size, Opt_nr_queues, Opt_queue_depth, Opt_err, + Opt_dev_size, Opt_block_size, Opt_nr_queues, Opt_queue_depth, + Opt_cmd_time_out, Opt_err, };static const match_table_t tcmur_tokens = {
@@ -739,6 +810,7 @@ static const match_table_t tcmur_tokens = { { Opt_block_size, "block_size=%u" }, { Opt_nr_queues, "nr_queues=%u" }, { Opt_queue_depth, "queue_depth=%u" }, + { Opt_cmd_time_out, "cmd_time_out=%u" }, { Opt_err, NULL } };
@@ -794,6 +866,9 @@ static ssize_t tcmur_set_configfs_dev_params(struct se_device *dev,else udev->queue_depth = val;break ; +case Opt_cmd_time_out: + udev->cmd_timeout = val; +break ; } }
@@ -812,6 +887,7 @@ static ssize_t tcmur_show_configfs_dev_params(struct se_device *dev, char *b) udev->block_size); bl +=sprintf (b + bl," Queues: %u Queue depth: %u" , udev->nr_queues, udev->queue_depth); + bl +=sprintf (b + bl," Command timeout: %u" , udev->cmd_timeout); bl +=sprintf (b + bl,"\n" );return bl; }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 10/18] scsi: target: uring: Time out commands the server does not complete
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider: - [Medium] scsi: target: uring: schedule_delayed_work() race in tcmur_timeout_work() - [Medium] scsi: target…
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider:
- [Medium] scsi: target: uring: schedule_delayed_work() race in tcmur_timeout_work()
- [Medium] scsi: target: uring: UAF race scheduling timeout_work in tcmur_queue_cmd()
- [Low] scsi: target: uring: 32-bit overflow in cmd_timeout * HZ in tcmur_queue_cmd()
- [Low] scsi: target: uring: unlocked deadline calculation in tcmur_queue_cmd()
--
commit 2624a7f3c24be9a1330c0e703d8b3102658c07d2
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Time out commands the server does not complete
This commit adds a configurable command timeout (cmd_time_out, 30 seconds by default) and a delayed work item to fail pending or in-flight commands that the userspace server does not complete in time.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
drivers/target/target_core_uring.c static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd) · line 200
Quoted 6 lines
drivers/target/target_core_uring.c +2
-0@@ -190,6 +196,8 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd)struct tcmur_io *io;refcount_set (&tcmd->ref,1 ); +if (udev->cmd_timeout) + tcmd->deadline = jiffies + udev->cmd_timeout *HZ ;
[Severity: Low]
Can udev->cmd_timeout * HZ overflow 32 bits here and in the schedule_delayed_work() call below?
tcmur_set_configfs_dev_params() accepts any non-negative int for cmd_time_out without an upper bound, and both udev->cmd_timeout and HZ are 32-bit integers. A value above UINT_MAX / HZ wraps in 32-bit unsigned arithmetic before being widened to unsigned long (and on 32-bit architectures a value above LONG_MAX / HZ wraps time_after()), causing commands to time out prematurely or immediately.
drivers/target/target_core_uring.c static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd) · line 219
Quoted 11 lines
drivers/target/target_core_uring.c +3
-0spin_lock (&q->lock); io = list_first_entry_or_null(&q->idle, struct tcmur_io, node);@@ -205,6 +213,9 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd)if (ucmd)io_uring_cmd_complete_in_task (ucmd, tcmur_fetch_tw_cb); +if (udev->cmd_timeout) +schedule_delayed_work (&udev->timeout_work, + udev->cmd_timeout *HZ ); }
[Severity: Medium]
Can scheduling udev->timeout_work after dropping q->lock race with command completion and device teardown, causing a use-after-free on udev?
Once q->lock is released, the server can complete tcmd immediately on another CPU, dropping the last LUN and device references while this thread is preempted:
CPU0 CPU1tcmur_queue_cmd()
spin_lock(&q->lock)
tcmur_io_start(q, io, tcmd)
spin_unlock(&q->lock)
tcmur_commit()
tcmur_cmd_end(tcmd, ...)
...
target_complete_cmd()
tcmur_destroy_device()
cancel_delayed_work_sync(
&udev->timeout_work)
tcmur_free_device()
...
kfree(udev)
if (udev->cmd_timeout)
schedule_delayed_work(
&udev->timeout_work, ...) <-- use-after-free / re-armed after cancelScheduling udev->timeout_work while holding q->lock avoids this race.
[ ... ]
Quoted 19 lines
drivers/target/target_core_uring.c +15
-0@@ -244,6 +255,58 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static void tcmur_timeout_work (struct work_struct *work) +{ + struct tcmur_dev *udev = container_of(work, struct tcmur_dev, + timeout_work.work); +unsigned long now = jiffies, next =0 ; +struct tcmur_cmd *tcmd, *tmp; +struct tcmur_queue *q; +struct tcmur_io *io; +bool rearm = false; +LIST_HEAD (expired); +unsigned int i, j; + +for (i =0 ; i < udev->nr_queues; i++) { + q = &udev->queues[i]; +
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, · line 282
Quoted 10 lines
drivers/target/target_core_uring.c +10
-0+spin_lock (&q->lock); +list_for_each_entry_safe (tcmd, tmp, &q->pending, node) { +if (time_after (tcmd->deadline, now)) { +if (!rearm ||time_before (tcmd->deadline, next)) + next = tcmd->deadline; + rearm = true; +break ; + } +list_move_tail (&tcmd->node, &expired); + }
[Severity: Low]
Does breaking on the first non-expired command in q->pending skip expired commands queued behind it?
In tcmur_queue_cmd(), tcmd->deadline is computed before acquiring q->lock. If one CPU computes an earlier deadline and is delayed before taking q->lock while another CPU computes a later deadline and appends to q->pending first, q->pending is no longer ordered by deadline and tcmur_timeout_work() stops at the later deadline until it expires.
Quoted 16 lines
drivers/target/target_core_uring.c +16
-0+for (j =0 ; j < udev->queue_depth; j++) { + io = &q->ios[j]; + tcmd = io->cmd; +if (!tcmd) +continue ; +if (time_after (tcmd->deadline, now)) { +if (!rearm ||time_before (tcmd->deadline, next)) + next = tcmd->deadline; + rearm = true; +continue ; + } +/* Leave the tag to the server until it commits it */ + io->cmd =NULL ; +list_add_tail (&tcmd->node, &expired); + } +spin_unlock (&q->lock);
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, · line 308
Quoted 10 lines
drivers/target/target_core_uring.c +10
-0+ } + +if (!list_empty (&expired)) +pr_warn_ratelimited ("tcmur%d: command timed out\n" , + udev->dev_id); +tcmur_end_cmds (&expired,SAM_STAT_CHECK_CONDITION ); + +if (rearm) +schedule_delayed_work (&udev->timeout_work, next - now); +}
[Severity: Medium]
Can a concurrent tcmur_queue_cmd() cause this schedule_delayed_work() call to be ignored, delaying expiration of the oldest command by an extra cmd_timeout period?
CPU0 (kworker) CPU1tcmur_timeout_work()
/* sees command due at next */
rearm = true
tcmur_queue_cmd()
schedule_delayed_work(
&udev->timeout_work,
udev->cmd_timeout * HZ) schedule_delayed_work(
&udev->timeout_work,
next - now) <-- ignored (already pending for cmd_timeout * HZ)
Because WORK_STRUCT_PENDING_BIT is cleared when tcmur_timeout_work() starts, CPU1's schedule_delayed_work() arms the timer for the full cmd_timeout * HZ, and CPU0's schedule_delayed_work() returns false without shortening the timer to next - now. Using mod_delayed_work() when rearming would update the timer to the earlier deadline.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
RFC PATCH 11/18 scsi: target: uring: Drop aborted commands that are still queued
In reply to Qiliang YuanWhen a command is aborted, the core waits for the backstore to complete it.
A target_core_uring command that no server has fetched only completes when it times out, which delays the abort by up to cmd_time_out.
Implement tmr_notify() and complete the aborted commands that are still on a pending list with TASK ABORTED status. Track them through se_cmd->priv, which is set only while a command is pending.
--- drivers/target/target_core_uring.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/target/target_core_uring.c +29 -0
@@ -175,6 +175,7 @@ static void tcmur_io_start(struct tcmur_queue *q, struct tcmur_io *io, {lockdep_assert_held (&q->lock); + tcmd->se_cmd->priv =NULL ; io->cmd = tcmd; io->owned = true; q->descs[io - q->ios] = tcmd->desc;
@@ -207,6 +208,7 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd) io->ucmd =NULL ;tcmur_io_start (q, io, tcmd); }else { + tcmd->se_cmd->priv = tcmd;list_add_tail (&tcmd->node, &q->pending); }spin_unlock (&q->lock);
@@ -255,6 +257,31 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static void tcmur_tmr_notify (struct se_device *dev,enum tcm_tmreq_table tmf, +struct list_head *aborted_cmds) +{ +struct tcmur_dev *udev =TCMUR_DEV (dev); +struct tcmur_cmd *tcmd; +struct tcmur_queue *q; +struct se_cmd *se_cmd; + +/* Drop the aborted commands that no server has picked up yet */ +list_for_each_entry (se_cmd, aborted_cmds, state_list) { + q =tcmur_cmd_queue (udev, se_cmd); + +spin_lock (&q->lock); + tcmd = se_cmd->priv; +if (tcmd) { + se_cmd->priv =NULL ; +list_del (&tcmd->node); + } +spin_unlock (&q->lock); + +if (tcmd) +tcmur_cmd_end (tcmd,SAM_STAT_TASK_ABORTED ); + } +} +static void tcmur_timeout_work (struct work_struct *work) { struct tcmur_dev *udev = container_of(work, struct tcmur_dev,
@@ -278,6 +305,7 @@ static void tcmur_timeout_work(struct work_struct *work) rearm = true;break ; } + tcmd->se_cmd->priv =NULL ;list_move_tail (&tcmd->node, &expired); }for (j =0 ; j < udev->queue_depth; j++) {
@@ -920,6 +948,7 @@ static const struct target_backend_ops tcmur_ops = { .destroy_device = tcmur_destroy_device, .free_device = tcmur_free_device, .parse_cdb = tcmur_parse_cdb, + .tmr_notify = tcmur_tmr_notify, .set_configfs_dev_params = tcmur_set_configfs_dev_params, .show_configfs_dev_params = tcmur_show_configfs_dev_params, .get_device_type = sbc_get_device_type,
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 11/18] scsi: target: uring: Drop aborted commands that are still queued
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: target: uring: missing CMD_T_ABORTED check in tcmur_queue_cmd() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: target: uring: missing CMD_T_ABORTED check in tcmur_queue_cmd()
--
commit 84783430b27d7a40ed74a32ae69a8126b2da14ae
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Drop aborted commands that are still queued
This commit implements tmr_notify() for target_core_uring so that aborted commands still waiting on a queue's pending list are removed and completed immediately with SAM_STAT_TASK_ABORTED instead of waiting to time out.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
Quoted 26 lines
drivers/target/target_core_uring.c +15
-0@@ -207,6 +208,7 @@ static void tcmur_queue_cmd(struct tcmur_dev *udev, struct tcmur_cmd *tcmd) io->ucmd =NULL ;tcmur_io_start (q, io, tcmd); }else { + tcmd->se_cmd->priv = tcmd;list_add_tail (&tcmd->node, &q->pending); }spin_unlock (&q->lock);@@ -255,6 +257,31 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static void tcmur_tmr_notify (struct se_device *dev,enum tcm_tmreq_table tmf, +struct list_head *aborted_cmds) +{ +struct tcmur_dev *udev =TCMUR_DEV (dev); +struct tcmur_cmd *tcmd; +struct tcmur_queue *q; +struct se_cmd *se_cmd; + +/* Drop the aborted commands that no server has picked up yet */ +list_for_each_entry (se_cmd, aborted_cmds, state_list) { + q =tcmur_cmd_queue (udev, se_cmd); + +spin_lock (&q->lock); + tcmd = se_cmd->priv;
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, · line 283
Quoted 10 lines
drivers/target/target_core_uring.c +10
-0+if (tcmd) { + se_cmd->priv =NULL ; +list_del (&tcmd->node); + } +spin_unlock (&q->lock); + +if (tcmd) +tcmur_cmd_end (tcmd,SAM_STAT_TASK_ABORTED ); + } +}
[Severity: High]
Can tcmur_queue_cmd() queue a command onto q->pending after tcmur_tmr_notify() has already run for it?
transport_generic_new_cmd() places se_cmd on state_list before the command reaches tcmur_queue_cmd():
CPU0 CPU1transport_generic_new_cmd()
target_add_to_state_list(se_cmd)
target_execute_cmd(se_cmd)
...
core_tmr_abort_task()
__target_check_io_state()
se_cmd->transport_state |=
CMD_T_ABORTED
list_move_tail(..., &aborted)
tcmur_tmr_notify()
spin_lock(&q->lock)
tcmd = se_cmd->priv <-- NULL
spin_unlock(&q->lock)
tcmur_queue_cmd()
spin_lock(&q->lock)
tcmd->se_cmd->priv = tcmd
list_add_tail(&tcmd->node,
&q->pending) <-- queued after tmr_notify()
spin_unlock(&q->lock)
target_put_cmd_and_wait()The same ordering also happens without a narrow race when a command on dev->delayed_cmd_list is aborted while se_cmd->priv is still NULL and is later dispatched by target_do_delayed_work() -> __target_execute_cmd().
Once tcmd is placed on q->pending after tcmur_tmr_notify() finishes, se_cmd is no longer on state_list, so nothing removes tcmd if no server fetches it. target_put_cmd_and_wait() then blocks for up to cmd_timeout seconds, or hangs indefinitely when cmd_time_out is 0.
Should tcmur_queue_cmd() check se_cmd->transport_state & CMD_T_ABORTED under q->lock before starting or queueing tcmd, similar to tcmu_queue_cmd() in target_core_user.c?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
RFC PATCH 12/18 scsi: target: uring: Support SYNCHRONIZE CACHE
In reply to Qiliang Yuantarget_core_uring only passes reads and writes to the server.
A server that caches writes has no way to learn that the initiator wants them on stable storage.
Add TCMUR_OP_FLUSH and send it for SYNCHRONIZE CACHE. The descriptor carries the range to flush; a zero length means up to the end of the device.
--- drivers/target/target_core_uring.c | 14 +++++++++++++- include/uapi/linux/target_core_uring.h | 1 + 2 files changed, 14 insertions(+), 1 deletion(-)
drivers/target/target_core_uring.c +13 -1
@@ -237,7 +237,8 @@ static sense_reason_t tcmur_submit(struct se_cmd *se_cmd, u32 op, u64 offset, tcmd->desc.op = op; tcmd->desc.offset = offset; tcmd->desc.len = len; - tcmd->desc.data_len = se_cmd->data_length; +if (sgl) + tcmd->desc.data_len = se_cmd->data_length;if (op ==TCMUR_OP_WRITE && (se_cmd->se_cmd_flags &SCF_FUA )) tcmd->desc.flags |=TCMUR_IO_F_FUA ;
@@ -257,6 +258,16 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static sense_reason_t tcmur_execute_sync_cache (struct se_cmd *se_cmd) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +/* Pass a zero length to flush up to the end of the device */ +return tcmur_submit (se_cmd,TCMUR_OP_FLUSH , + se_cmd->t_task_lba * block_size, + se_cmd->data_length,NULL ,0 ); +} +static void tcmur_tmr_notify (struct se_device *dev,enum tcm_tmreq_table tmf,struct list_head *aborted_cmds) {
@@ -929,6 +940,7 @@ static sector_t tcmur_get_blocks(struct se_device *dev)static struct exec_cmd_ops tcmur_exec_cmd_ops = { .execute_rw = tcmur_execute_rw, + .execute_sync_cache = tcmur_execute_sync_cache, };static sense_reason_t tcmur_parse_cdb (struct se_cmd *se_cmd)
include/uapi/linux/target_core_uring.h +1 -0
@@ -67,6 +67,7 @@ struct tcmur_io_cmd {enum tcmur_io_op {TCMUR_OP_READ ,TCMUR_OP_WRITE , +TCMUR_OP_FLUSH , };/* Put the write on stable storage before completing it */
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 12/18] scsi: target: uring: Support SYNCHRONIZE CACHE
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: target: uring: wrong flush length in tcmur_execute_sync_cache() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: target: uring: wrong flush length in tcmur_execute_sync_cache()
--
commit 36370455252a7c6edc07d1415bceee399ffd1ebe
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Support SYNCHRONIZE CACHE
This commit adds TCMUR_OP_FLUSH and wires up .execute_sync_cache in target_core_uring so SYNCHRONIZE CACHE commands are forwarded to the userspace server.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, · line 269
Quoted 13 lines
drivers/target/target_core_uring.c +9
-0@@ -257,6 +258,16 @@ static sense_reason_t tcmur_execute_rw(struct se_cmd *se_cmd, se_cmd->data_length, sgl, sgl_nents); } +static sense_reason_t tcmur_execute_sync_cache (struct se_cmd *se_cmd) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +/* Pass a zero length to flush up to the end of the device */ +return tcmur_submit (se_cmd,TCMUR_OP_FLUSH , + se_cmd->t_task_lba * block_size, + se_cmd->data_length,NULL ,0 ); +}
[Severity: High]
Does tcmur_execute_sync_cache() pass the wrong flush length in tcmd->desc.len by reading se_cmd->data_length?
SYNCHRONIZE CACHE (10) and SYNCHRONIZE CACHE (16) are non-data commands (DMA_NONE), so fabrics where cmd->unknown_data_length is 0 initialize cmd->data_length to the transport expected transfer length of 0. When .execute_sync_cache is set, sbc_parse_cdb() jumps to check_lba, which computes size = sbc_get_size(cmd, sectors) even though SCF_SCSI_DATA_CDB is not set, and passes that size to target_cmd_size_check():
drivers/target/target_core_transport.c:target_cmd_size_check() {
if (cmd->unknown_data_length) {
cmd->data_length = size;
} else if (size != cmd->data_length) {
...
if (size > cmd->data_length) {
cmd->se_cmd_flags |= SCF_OVERFLOW_BIT;
cmd->residual_count = (size - cmd->data_length);
} else {
cmd->se_cmd_flags |= SCF_UNDERFLOW_BIT;
cmd->residual_count = (cmd->data_length - size);
if (cmd->data_direction == DMA_FROM_DEVICE) {
cmd->data_length = size;
}
}
}When an initiator specifies a non-zero block count on a normal fabric, target_cmd_size_check() logs a transfer length mismatch warning, sets SCF_OVERFLOW_BIT with a bogus residual_count, and leaves se_cmd->data_length at 0, so tcmd->desc.len is always 0 instead of the requested range.
On fabrics that set cmd->unknown_data_length (such as USB gadget f_tcm), setting cmd->data_length to size causes transport_generic_new_cmd() to allocate size bytes of unused scatterlist pages via target_alloc_sgl() for a DMA_NONE command. In addition, both sbc_get_size() and se_cmd->data_length are 32-bit u32 values, which truncates 32-bit SYNCHRONIZE_CACHE_16 block counts when the byte range reaches 4 GiB.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
RFC PATCH 13/18 scsi: target: uring: Implement UNMAP
In reply to Qiliang Yuantarget_core_uring passes reads, writes and flushes to the server.
Thin provisioned storage also needs to know which blocks the initiator no longer uses.
Add TCMUR_OP_UNMAP and send one for each range of an UNMAP command or of a WRITE SAME with the UNMAP bit. The core expects execute_unmap() to return once the range is unmapped, so queue a struct tcmur_cmd that lives on the stack and wait for the server to commit it.
--- drivers/target/target_core_uring.c | 39 ++++++++++++++++++++++++++++++++++ include/uapi/linux/target_core_uring.h | 1 + 2 files changed, 40 insertions(+)
drivers/target/target_core_uring.c +39 -0
@@ -49,6 +49,7 @@ struct tcmur_dev;* @ref: Held by the tag or the pending list and by data copies in * progress * @deadline: When to give up waiting for the server +* @done: Complete this instead of @se_cmd when the last reference goes * @status: SAM status to complete @se_cmd with */ struct tcmur_cmd {
@@ -59,6 +60,7 @@ struct tcmur_cmd {struct tcmur_io_desc desc;refcount_t ref;unsigned long deadline; +struct completion *done;u8 status; };
@@ -148,6 +150,11 @@ static void tcmur_cmd_put(struct tcmur_cmd *tcmd)if (!refcount_dec_and_test (&tcmd->ref))return ; +if (tcmd->done) { +complete (tcmd->done); +return ; + } +target_complete_cmd (tcmd->se_cmd, tcmd->status);kmem_cache_free (tcmur_cmd_cache, tcmd); }
@@ -268,6 +275,36 @@ static sense_reason_t tcmur_execute_sync_cache(struct se_cmd *se_cmd) se_cmd->data_length, NULL, 0); } +static sense_reason_t tcmur_execute_unmap (struct se_cmd *se_cmd,sector_t lba, +sector_t nolb) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; +DECLARE_COMPLETION_ONSTACK (done); +struct tcmur_cmd tcmd = { + .se_cmd = se_cmd, + .done = &done, + .desc.op =TCMUR_OP_UNMAP , + .desc.offset = lba * block_size, + .desc.len = nolb * block_size, + }; + +/* Wait here, as the core completes the command after all ranges */ +tcmur_queue_cmd (TCMUR_DEV (se_cmd->se_dev), &tcmd); +wait_for_completion (&done); + +return tcmd.status ?TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE :0 ; +} + +static bool tcmur_configure_unmap (struct se_device *dev) +{ + dev->dev_attrib.max_unmap_lba_count =UINT_MAX / + dev->dev_attrib.hw_block_size; + dev->dev_attrib.max_unmap_block_desc_count =1 ; + dev->dev_attrib.unmap_granularity =1 ; + dev->dev_attrib.unmap_granularity_alignment =0 ; +return true; +} +static void tcmur_tmr_notify (struct se_device *dev,enum tcm_tmreq_table tmf,struct list_head *aborted_cmds) {
@@ -941,6 +978,7 @@ static sector_t tcmur_get_blocks(struct se_device *dev)static struct exec_cmd_ops tcmur_exec_cmd_ops = { .execute_rw = tcmur_execute_rw, .execute_sync_cache = tcmur_execute_sync_cache, + .execute_unmap = tcmur_execute_unmap, };static sense_reason_t tcmur_parse_cdb (struct se_cmd *se_cmd)
@@ -959,6 +997,7 @@ static const struct target_backend_ops tcmur_ops = { .configure_device = tcmur_configure_device, .destroy_device = tcmur_destroy_device, .free_device = tcmur_free_device, + .configure_unmap = tcmur_configure_unmap, .parse_cdb = tcmur_parse_cdb, .tmr_notify = tcmur_tmr_notify, .set_configfs_dev_params = tcmur_set_configfs_dev_params,
include/uapi/linux/target_core_uring.h +1 -0
@@ -68,6 +68,7 @@ enum tcmur_io_op {TCMUR_OP_READ ,TCMUR_OP_WRITE ,TCMUR_OP_FLUSH , + TCMUR_OP_UNMAP, };/* Put the write on stable storage before completing it */
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 13/18] scsi: target: uring: Implement UNMAP
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: target: uring: submission workqueue stall and TMR hang in tcmur_execute_unmap() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: target: uring: submission workqueue stall and TMR hang in tcmur_execute_unmap()
--
commit a69183b9652241c1e0d7dcfe7c2618562f084113
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Implement UNMAP
This commit adds UNMAP support to target_core_uring by introducing TCMUR_OP_UNMAP and synchronously waiting for each unmap range to complete in tcmur_execute_unmap().
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
Quoted 10 lines
drivers/target/target_core_uring.c +6
-0@@ -268,6 +275,36 @@ static sense_reason_t tcmur_execute_sync_cache(struct se_cmd *se_cmd) se_cmd->data_length, NULL, 0); } +static sense_reason_t tcmur_execute_unmap (struct se_cmd *se_cmd,sector_t lba, +sector_t nolb) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; +DECLARE_COMPLETION_ONSTACK (done); +struct tcmur_cmd tcmd = {
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_sync_cache(struct se_cmd *se_cmd) · line 293
Quoted 10 lines
drivers/target/target_core_uring.c +10
-0+ .se_cmd = se_cmd, + .done = &done, + .desc.op = TCMUR_OP_UNMAP, + .desc.offset = lba * block_size, + .desc.len = nolb * block_size, + }; + +/* Wait here, as the core completes the command after all ranges */ +tcmur_queue_cmd (TCMUR_DEV (se_cmd->se_dev), &tcmd); +wait_for_completion (&done);
[Severity: High]
Can waiting on done here stall the submission workqueue and hang task management?
On fabrics using TARGET_QUEUE_SUBMIT (such as vhost-scsi or tcm_loop), target_execute_cmd() runs inside target_queued_submit_work() on dev->queues[i].sq.work. Sleeping in wait_for_completion(&done) blocks further commands on that queue until the server completes the unmap.
If no idle tag is available, tcmur_queue_cmd() places tcmd on q->pending. When an ABORT_TASK or LUN_RESET arrives, core_tmr_abort_task() and core_tmr_drain_state_list() flush sq.work before calling dev->transport->tmr_notify() (tcmur_tmr_notify()):
sq.work tmr_wqtarget_queued_submit_work()
...
tcmur_execute_unmap()
tcmur_queue_cmd()
wait_for_completion(&done)
core_tmr_abort_task()
flush_work(&dev->queues[i].sq.work)
<-- waits for sq.work
...
dev->transport->tmr_notify()Because flush_work() waits for tcmur_execute_unmap() to return, tcmur_tmr_notify() never runs to remove tcmd from q->pending and complete done, stalling task management until cmd_timeout expires (or forever when cmd_time_out is 0).
Even when target_execute_cmd() runs outside sq.work, if a TMR aborts se_cmd before tcmur_queue_cmd() queues tcmd on q->pending, tcmur_tmr_notify() has already run and removed se_cmd from state_list, so tcmd is never aborted and target_put_cmd_and_wait() hangs.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
RFC PATCH 14/18 scsi: target: uring: Handle WRITE SAME
In reply to Qiliang Yuantarget_core_uring does not set execute_write_same(), so the core rejects WRITE SAME without the UNMAP bit.
Initiators use it to zero or initialize ranges without sending the data for every block.
Add TCMUR_OP_WRITE_SAME. The descriptor carries the range to fill, and the data of the command is the one block to repeat. Accept up to 65536 blocks per command.
--- drivers/target/target_core_uring.c | 14 ++++++++++++++ include/uapi/linux/target_core_uring.h | 9 +++++---- 2 files changed, 19 insertions(+), 4 deletions(-)
drivers/target/target_core_uring.c +14 -0
@@ -30,6 +30,8 @@#define TCMUR_VERSION "1.0"#define TCMUR_MINORS (1U << MINORBITS)#define TCMUR_MAX_IO_SIZE (4U << 20) +/* Limit one WRITE SAME to this many blocks */ +#define TCMUR_MAX_WRITE_SAME_LEN 0x10000#define TCMUR_DEF_QUEUE_DEPTH 128#define TCMUR_DEF_BLOCK_SIZE 512#define TCMUR_DEF_CMD_TIMEOUT 30
@@ -295,6 +297,16 @@ static sense_reason_t tcmur_execute_unmap(struct se_cmd *se_cmd, sector_t lba,return tcmd.status ?TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE :0 ; } +static sense_reason_t tcmur_execute_write_same (struct se_cmd *se_cmd) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +return tcmur_submit (se_cmd,TCMUR_OP_WRITE_SAME , + se_cmd->t_task_lba * block_size, + (u64 )sbc_get_write_same_sectors (se_cmd) * block_size, + se_cmd->t_data_sg, se_cmd->t_data_nents); +} +static bool tcmur_configure_unmap (struct se_device *dev) { dev->dev_attrib.max_unmap_lba_count = UINT_MAX /
@@ -813,6 +825,7 @@ static int tcmur_configure_device(struct se_device *dev) dev->dev_attrib.hw_block_size = udev->block_size; dev->dev_attrib.hw_max_sectors =TCMUR_MAX_IO_SIZE / udev->block_size; dev->dev_attrib.hw_queue_depth = udev->nr_queues * udev->queue_depth; + dev->dev_attrib.max_write_same_len =TCMUR_MAX_WRITE_SAME_LEN ; dev->dev_attrib.is_nonrot =1 ; udev->cdev_dev.devt =MKDEV (MAJOR (tcmur_devt), udev->dev_id);
@@ -979,6 +992,7 @@ static struct exec_cmd_ops tcmur_exec_cmd_ops = { .execute_rw = tcmur_execute_rw, .execute_sync_cache = tcmur_execute_sync_cache, .execute_unmap = tcmur_execute_unmap, + .execute_write_same = tcmur_execute_write_same, };static sense_reason_t tcmur_parse_cdb (struct se_cmd *se_cmd)
include/uapi/linux/target_core_uring.h +5 -4
@@ -69,6 +69,7 @@ enum tcmur_io_op {TCMUR_OP_WRITE ,TCMUR_OP_FLUSH ,TCMUR_OP_UNMAP , + TCMUR_OP_WRITE_SAME, };/* Put the write on stable storage before completing it */
@@ -80,7 +81,7 @@ enum tcmur_io_op {* @flags: TCMUR_IO_F_* * @offset: Start of the range in bytes * @len: Length of the range in bytes -* @data_len: Bytes of data to transfer +* @data_len: Bytes of data to transfer; one block for TCMUR_OP_WRITE_SAME * @reserved: Zero * * The descriptors of a queue form an array indexed by tag, which the server
@@ -99,9 +100,9 @@ struct tcmur_io_desc { (TCMUR_MAX_QUEUE_DEPTH *sizeof (struct tcmur_io_desc ))/* -* The data of a command is read (TCMUR_OP_WRITE) or written -* (TCMUR_OP_READ) with pread()/pwrite() on the file, at the position -* returned by tcmur_buf_pos(). +* The data of a command is read (TCMUR_OP_WRITE, TCMUR_OP_WRITE_SAME) or +* written (TCMUR_OP_READ) with pread()/pwrite() on the file, at the +* position returned by tcmur_buf_pos(). */#define TCMUR_BUF_TAG_SHIFT 32#define TCMUR_BUF_QID_SHIFT 48
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 14/18] scsi: target: uring: Handle WRITE SAME
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: target: uring: missing payload checks in tcmur_execute_write_same() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: target: uring: missing payload checks in tcmur_execute_write_same()
--
commit 194f0b295175d883a8fb6c1f553cf215c122630e
Author: Qiliang Yuan <[email protected]>
scsi: target: uring: Handle WRITE SAME
This commit adds WRITE SAME support to the target_core_uring backend via a new TCMUR_OP_WRITE_SAME operation.
Quoted 4 lines
drivers/target/target_core_uring.c +0
-0
[ ... ]
drivers/target/target_core_uring.c static sense_reason_t tcmur_execute_unmap(struct se_cmd *se_cmd, sector_t lba, · line 308
Quoted 13 lines
drivers/target/target_core_uring.c +9
-0@@ -295,6 +297,16 @@ static sense_reason_t tcmur_execute_unmap(struct se_cmd *se_cmd, sector_t lba,return tcmd.status ?TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE :0 ; } +static sense_reason_t tcmur_execute_write_same (struct se_cmd *se_cmd) +{ +u32 block_size = se_cmd->se_dev->dev_attrib.block_size; + +return tcmur_submit (se_cmd,TCMUR_OP_WRITE_SAME , + se_cmd->t_task_lba * block_size, + (u64 )sbc_get_write_same_sectors (se_cmd) * block_size, + se_cmd->t_data_sg, se_cmd->t_data_nents); +}
[Severity: High]
Can tcmur_execute_write_same() submit a TCMUR_OP_WRITE_SAME descriptor whose data_len is 0 or larger than block_size?
Because sbc_parse_cdb() does not set SCF_SCSI_DATA_CDB for WRITE SAME, target_cmd_size_check() does not reject commands when an initiator sends no data buffer (leaving t_data_sg NULL and t_data_nents 0) or a transfer length larger than one block, nor when protection information sets se_cmd->prot_op. tcmur_submit() then sets desc.data_len to 0 or se_cmd->data_length, breaking the UAPI guarantee that data_len is one block and causing a userspace server to spin forever on a zero step or overflow a single-block buffer.
Should tcmur_execute_write_same() validate se_cmd->prot_op, se_cmd->t_data_nents == 1, and se_cmd->t_data_sg[0].length == block_size the same way fd_execute_write_same() and iblock_execute_write_same() do?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
RFC PATCH 15/18 scsi: target: uring: Register command data as io_uring fixed buffers
In reply to Qiliang YuanThe server of a target_core_uring device copies the data of each command with pread() or pwrite().
For large commands that copy dominates, although the server often only passes the data on to a file, a block device or a socket.
Add TCMUR_IO_REGISTER_BUF, which registers the pages of a command as a fixed buffer of the server's io_uring, and TCMUR_IO_UNREGISTER_BUF. The server then uses the buffer in fixed read and write requests on its backing store. Let the buffer hold a reference on the command, so that the command completes only after the buffer is released.
Exporting a 2 GiB device through tcm_loop to fio (io_uring, direct I/O, iodepth 32 for 4k and 8 for 1M, 6 seconds per job) on a 32 CPU VM, with 32 queues and a server that keeps the data in a tmpfs file, copying through a bounce buffer or using fixed buffers. Small commands are slower with fixed buffers, as each one takes two more io_uring requests; the server can choose per command. Three alternating runs (median, min-max in parentheses):
workload copy (MB/s) fixed buffer (MB/s) change
4k randread, 16 jobs 5808 (5463-5944) 5217 (5090-5463) -10% 4k randwrite, 16 jobs 2023 (1831-2101) 1845 (1571-1850) -9%
1M read, 4 jobs 21516 (10137-22138) 44973 (31086-45013) +109%
1M write, 4 jobs 5997 (5669-7364) 7430 (5970-8002) +24%
--- drivers/target/target_core_uring.c | 60 ++++++++++++++++++++++++++++++++-- include/uapi/linux/target_core_uring.h | 13 ++++++-- 2 files changed, 69 insertions(+), 4 deletions(-)
drivers/target/target_core_uring.c +58 -2
@@ -48,8 +48,8 @@ struct tcmur_dev;* @sgl: Data pages of the command * @sgl_nents: Number of entries in @sgl * @desc: What the server is asked to do -* @ref: Held by the tag or the pending list and by data copies in -* progress +* @ref: Held by the tag or the pending list, by data copies in +* progress and by registered buffers * @deadline: When to give up waiting for the server * @done: Complete this instead of @se_cmd when the last reference goes * @status: SAM status to complete @se_cmd with
@@ -489,6 +489,56 @@ static int tcmur_commit(struct tcmur_queue *q, struct tcmur_io *io, int result)return 0 ; } +static void tcmur_buf_release (void *priv) +{ +tcmur_cmd_put (priv); +} + +static int tcmur_register_buf (struct tcmur_dev *udev,u16 q_id,u16 tag, +struct io_uring_cmd *ucmd,unsigned int index, +unsigned int issue_flags) +{ +struct tcmur_cmd *tcmd =tcmur_get_cmd (udev, q_id, tag); +unsigned int len, n =0 ; +struct scatterlist *sg; +struct bio_vec *bvs; +int ret = -EINVAL ; +u32 left; +int i; + +if (!tcmd) +return -EINVAL ; +if (!tcmd->desc.data_len) + goto put_cmd; + + bvs =kmalloc_array (tcmd->sgl_nents,sizeof (*bvs),GFP_KERNEL ); +if (!bvs) { + ret = -ENOMEM ; + goto put_cmd; + } + + left = tcmd->desc.data_len; +for_each_sg (tcmd->sgl, sg, tcmd->sgl_nents, i) { + len =min (sg->length, left); +bvec_set_page (&bvs[n++],sg_page (sg), len, sg->offset); + left -= len; +if (!left) +break ; + } + +/* Keep the command reference until the buffer is released */ + ret =io_buffer_register_bvec (ucmd, bvs, n, tcmur_buf_release, tcmd, + tcmd->desc.op ==TCMUR_OP_READ ? +IO_BUF_DEST :IO_BUF_SOURCE , + index, issue_flags); +kfree (bvs); +if (!ret) +return 0 ; +put_cmd: +tcmur_cmd_put (tcmd); +return ret; +} +static int tcmur_ch_uring_cmd_local (struct io_uring_cmd *ucmd,unsigned int issue_flags) {
@@ -515,6 +565,12 @@ static int tcmur_ch_uring_cmd_local(struct io_uring_cmd *ucmd,if (ret)return ret;return tcmur_fetch (q, io, ucmd, issue_flags); +case TCMUR_IO_REGISTER_BUF : +return tcmur_register_buf (udev, q_id, tag, ucmd, +READ_ONCE (ioc->addr), issue_flags); +case TCMUR_IO_UNREGISTER_BUF : +return io_buffer_unregister (ucmd,READ_ONCE (ioc->addr), + issue_flags);default :return -EOPNOTSUPP ; }
include/uapi/linux/target_core_uring.h +11 -2
@@ -45,13 +45,14 @@ struct tcmur_dev_info {* @tag: Tag the command is for * @result: TCMUR_IO_COMMIT_AND_FETCH: 0, or a negative errno if the * command failed -* @reserved1: Zero +* @addr: TCMUR_IO_REGISTER_BUF and TCMUR_IO_UNREGISTER_BUF: index in the +* buffer table of the ring */ struct tcmur_io_cmd {__u16 q_id;__u16 tag;__s32 result; -__u64 reserved1; +__u64 addr; };/*
@@ -64,6 +65,14 @@ struct tcmur_io_cmd {#define TCMUR_IO_FETCH _IOWR(0xBB, 0x08, struct tcmur_io_cmd)#define TCMUR_IO_COMMIT_AND_FETCH _IOWR(0xBB, 0x09, struct tcmur_io_cmd) +/* +* TCMUR_IO_REGISTER_BUF: register the data pages of the command of +* (q_id, tag) as fixed buffer @addr of the ring, so that the server can +* do I/O on them without copying. TCMUR_IO_UNREGISTER_BUF removes it. +*/ +#define TCMUR_IO_REGISTER_BUF _IOWR(0xBB, 0x0a, struct tcmur_io_cmd) +#define TCMUR_IO_UNREGISTER_BUF _IOWR(0xBB, 0x0b, struct tcmur_io_cmd) +enum tcmur_io_op {TCMUR_OP_READ ,TCMUR_OP_WRITE ,
Signature · 1 line
2.43.0
RFC PATCH 16/18 selftests: target_core_uring: Add a minimal server
In reply to Qiliang Yuantarget_core_uring needs a userspace server to do anything.
There is none to test the backstore with, and no example of how to use its interface.
Add tcmur_server, which serves a device from a backing file, from memory or as a null device. It runs one thread and one io_uring per queue. With -z it registers the data pages of reads and writes as fixed buffers and does the I/O on the backing file without copying.
--- MAINTAINERS | 1 + .../testing/selftests/target_core_uring/.gitignore | 1 + tools/testing/selftests/target_core_uring/Makefile | 7 + .../selftests/target_core_uring/tcmur_server.c | 310 +++++++++++++++++++++ 4 files changed, 319 insertions(+)
MAINTAINERS +1 -0
@@ -25331,6 +25331,7 @@ F: Documentation/target/ F: drivers/target/ F: include/target/ F: include/uapi/linux/target_core_uring.h +F: tools/testing/selftests/target_core_uring/ SCTP PROTOCOL M: Marcelo Ricardo Leitner <[email protected]>
tools/testing/selftests/target_core_uring/.gitignore new file +1 -0
@@ -0,0 +1 @@ +tcmur_server
tools/testing/selftests/target_core_uring/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +CFLAGS += -O2 -Wall -D_GNU_SOURCE $(KHDR_INCLUDES) +LDLIBS += -luring -lpthread + +TEST_GEN_PROGS_EXTENDED := tcmur_server + +include ../lib.mk
tools/testing/selftests/target_core_uring/tcmur_server.c new file +310 -0
310 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 16/18] selftests: target_core_uring: Add a minimal server
Thank you for your contribution! Sashiko AI review found 7 potential issues to consider: - [Medium] selftests: target_core_uring: missing FUA sync in zero-copy handle() - [Medium] selftests…
Thank you for your contribution! Sashiko AI review found 7 potential issues to consider:
- [Medium] selftests: target_core_uring: missing FUA sync in zero-copy handle()
- [Medium] selftests: target_core_uring: tag leak on UD_REG failure in queue_fn()
- [Medium] selftests: target_core_uring: commit before UD_UNREG in queue_fn()
- [Medium] selftests: target_core_uring: 32-bit off_t truncation in copy_data()
- [Low] selftests: target_core_uring: 0-length I/O passed to start_zero_copy()
- [Low] selftests: target_core_uring: perror() after liburing calls in queue_fn()
- [Low] selftests: target_core_uring: unchecked calloc() for queues in main()
--
commit a8d885b7b600468b558a9ffcad7e5d1e7afa97ce
Author: Qiliang Yuan <[email protected]>
selftests: target_core_uring: Add a minimal server
This commit adds tcmur_server, a minimal userspace server for target_core_uring devices supporting null, RAM, and file backends with optional zero-copy fixed-buffer I/O.
Quoted 18 lines
tools/testing/selftests/target_core_uring/Makefile new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +CFLAGS += -O2 -Wall -D_GNU_SOURCE $(KHDR_INCLUDES) +LDLIBS += -luring -lpthread + +TEST_GEN_PROGS_EXTENDED := tcmur_server + +include ../lib.mktools/testing/selftests/target_core_uring/tcmur_server.c new file +0
-0
[ ... ]
tools/testing/selftests/target_core_uring/tcmur_server.c line 83
Quoted 12 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +12
-0+static int copy_data (struct queue *q,int tag,void *buf,uint32_t len, +bool to_cmd) +{ +uint64_t pos =tcmur_buf_pos (q->q_id, tag,0 ); +ssize_t ret; + +if (to_cmd) + ret =pwrite (dev_fd, buf, len, pos); +else + ret =pread (dev_fd, buf, len, pos); +return ret == len ?0 : -EIO ; +}
[Severity: Medium]
Does passing pos to pread() and pwrite() truncate the upper 32 bits on 32-bit architectures?
tcmur_buf_pos() encodes q_id in bits 48..63 and tag in bits 32..47, but neither the Makefile nor tcmur_server.c defines _FILE_OFFSET_BITS=64, so off_t is 32-bit on 32-bit targets and every non-zero q_id or tag is lost. Also, passing -D_GNU_SOURCE in the Makefile conflicts with -D_GNU_SOURCE= in ../lib.mk, triggering a macro redefinition warning during the build.
[ ... ]
Quoted 31 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +31
-0+/* Start the I/O on the data pages of the command, without copying them */ +static void start_zero_copy (struct queue *q,int tag,struct tcmur_io_desc *d) +{ +struct io_uring_sqe *sqe; + + sqe =prep_cmd (q,TCMUR_IO_REGISTER_BUF , tag,0 ,UD_REG ); + sqe->flags |=IOSQE_IO_LINK |IOSQE_CQE_SKIP_SUCCESS ; + + sqe =get_sqe (q); +if (d->op ==TCMUR_OP_READ ) +io_uring_prep_read_fixed (sqe, backing_fd,NULL , d->data_len, + d->offset, tag); +else +io_uring_prep_write_fixed (sqe, backing_fd,NULL , d->data_len, + d->offset, tag); + sqe->flags |=IOSQE_IO_HARDLINK ; + sqe->user_data = (uint64_t )UD_RW <<16 | tag; + + sqe =prep_cmd (q,TCMUR_IO_UNREGISTER_BUF , tag,0 ,UD_UNREG ); + sqe->flags |=IOSQE_CQE_SKIP_SUCCESS ; +} + +/* Return true if the command completes later */ +static bool handle (struct queue *q,int tag,int *result) +{ +struct tcmur_io_desc *d = &q->descs[tag]; +bool rd = d->op ==TCMUR_OP_READ ; +int ret =0 ; + +switch (d->op) { +case TCMUR_OP_READ :
tools/testing/selftests/target_core_uring/tcmur_server.c line 156
Quoted 10 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +10
-0+case TCMUR_OP_WRITE : +if (d->offset + d->len > info.dev_size) { + ret = -EINVAL ; + }else if (backend ==BACKEND_RAM ) { + ret =copy_data (q, tag, ram + d->offset, d->data_len, + rd); + }else if (backend ==BACKEND_FILE && zero_copy) { +start_zero_copy (q, tag, d); +return true; + }
[Severity: Low]
What happens in zero-copy mode when a valid 0-sector SCSI READ or WRITE arrives with d->data_len == 0?
handle() still calls start_zero_copy(), which submits TCMUR_IO_REGISTER_BUF, and tcmur_register_buf() in drivers/target/target_core_uring.c rejects !tcmd->desc.data_len with -EINVAL instead of completing the 0-length transfer with 0 as the non-zero-copy paths do.
Quoted 7 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +7
-0+ }else if (backend ==BACKEND_FILE && rd) { +if (pread (backing_fd, q->buf, d->data_len, d->offset) != + d->data_len) + ret = -EIO ; +else + ret =copy_data (q, tag, q->buf, d->data_len, + true);
tools/testing/selftests/target_core_uring/tcmur_server.c line 160
Quoted 10 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +10
-0+ }else if (backend ==BACKEND_FILE ) { + ret =copy_data (q, tag, q->buf, d->data_len, false); +if (!ret &&pwrite (backing_fd, q->buf, d->data_len, + d->offset) != d->data_len) + ret = -EIO ; + } +if (!ret && backend ==BACKEND_FILE && + (d->flags &TCMUR_IO_F_FUA ) &&fdatasync (backing_fd)) + ret = -errno; +break ;
[Severity: Medium]
Does zero-copy mode ignore TCMUR_IO_F_FUA on writes?
When zero_copy is true, handle() returns early after start_zero_copy() and skips the fdatasync(backing_fd) check above, while neither start_zero_copy() (for instance via RWF_DSYNC on sqe->rw_flags) nor the UD_RW completion path in queue_fn() syncs backing_fd before committing the command.
[ ... ]
Quoted 16 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +16
-0+static void *queue_fn (void *arg) +{ +struct io_uring_params p = { + .flags =IORING_SETUP_SINGLE_ISSUER |IORING_SETUP_DEFER_TASKRUN , + }; +struct queue *q = arg; +struct io_uring_cqe *cqe; +unsigned int head, n; +int tag, kind, result; + +/* Create the ring here, as only its creator may submit to it */ +if (io_uring_queue_init_params (info.queue_depth *4 , &q->ring, &p) || + (zero_copy && +io_uring_register_buffers_sparse (&q->ring, info.queue_depth))) { +perror ("io_uring setup" ); +exit (1 );
tools/testing/selftests/target_core_uring/tcmur_server.c line 210
Quoted 10 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +10
-0+ } + +for (tag =0 ; tag < info.queue_depth; tag++) +prep_cmd (q,TCMUR_IO_FETCH , tag,0 ,UD_FETCH ); + +for (;;) { +if (io_uring_submit_and_wait (&q->ring,1 ) <0 ) { +perror ("io_uring_submit_and_wait" ); +exit (1 ); + }
[Severity: Low]
Will perror() print a meaningful error after these liburing calls fail? io_uring_queue_init_params(), io_uring_register_buffers_sparse(), and io_uring_submit_and_wait() return negative errno values directly without setting errno.
Quoted 12 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +12
-0+ + n =0 ; +io_uring_for_each_cqe (&q->ring, head, cqe) { + n++; + tag = cqe->user_data &0xffff ; + kind = cqe->user_data >>16 ; + +switch (kind) { +case UD_FETCH : +if (cqe->res <0 ) + goto out; +if (handle (q, tag, &result))
tools/testing/selftests/target_core_uring/tcmur_server.c line 232
Quoted 10 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +10
-0+continue ; +break ; +case UD_RW : + result = cqe->res == q->descs[tag].data_len ? +0 : -EIO ; +break ; +default : +/* Report failures through the I/O that follows */ +continue ; + }
[Severity: Medium]
If TCMUR_IO_REGISTER_BUF (UD_REG) fails, does this ignore the only CQE and leak the tag?
In start_zero_copy(), UD_REG is submitted withIOSQE_IO_LINK | IOSQE_CQE_SKIP_SUCCESS. When a request with IOSQE_CQE_SKIP_SUCCESS fails and uses a soft link (IOSQE_IO_LINK), req_set_fail() sets REQ_F_SKIP_LINK_CQES and io_fail_links() suppresses CQEs for all subsequent linked requests (both UD_RW and UD_UNREG). Because the default branch ignores the UD_REG error CQE waiting for a UD_RW CQE that will never arrive, TCMUR_IO_COMMIT_AND_FETCH is never issued for tag.
tools/testing/selftests/target_core_uring/tcmur_server.c line 235
Quoted 3 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +3
-0+prep_cmd (q,TCMUR_IO_COMMIT_AND_FETCH , tag, result, +UD_FETCH ); + }
[Severity: Medium]
Can submitting TCMUR_IO_COMMIT_AND_FETCH on the UD_RW CQE race ahead of the linked TCMUR_IO_UNREGISTER_BUF (UD_UNREG) request?
In start_zero_copy(), UD_UNREG is linked after UD_RW with IOSQE_CQE_SKIP_SUCCESS, so UD_RW's CQE is posted to the CQ ring before UD_UNREG runs:
kernel queue_fn()
------ ----------UD_RW completes (e.g. via io-wq)
posts UD_RW CQE to CQ ring
io_uring_submit_and_wait(..., 1)
reaps UD_RW CQE
prep_cmd(TCMUR_IO_COMMIT_AND_FETCH)
io_uring_submit_and_wait(..., 1)__sys_io_uring_enter()
io_submit_sqes()
TCMUR_IO_COMMIT_AND_FETCH completes
fetches new command on same tag
reaps UD_FETCH CQE start_zero_copy()
prep_cmd(TCMUR_IO_REGISTER_BUF)
io_kernel_buffer_init()
returns -EBUSY <-- slot tag still occupied before UD_UNREG runs
Because __sys_io_uring_enter() processes new SQEs before draining pending local task_work (or while UD_UNREG is still queued from io-wq), the tag is committed and reused while fixed-buffer slot tag is still registered. Linking TCMUR_IO_COMMIT_AND_FETCH after UD_UNREG, or waiting for UD_UNREG before committing, would avoid this ordering issue.
[ ... ]
tools/testing/selftests/target_core_uring/tcmur_server.c line 295
Quoted 7 lines
tools/testing/selftests/target_core_uring/tcmur_server.c new file +7
-0+ desc_size = info.queue_depth *sizeof (struct tcmur_io_desc ); + desc_size = (desc_size +getpagesize () -1 ) & ~(getpagesize () -1 ); + queues =calloc (info.nr_queues,sizeof (*queues)); +for (i =0 ; i < info.nr_queues; i++) { +struct queue *q = &queues[i]; + + q->q_id = i;
[Severity: Low]
Can calloc() return NULL here and cause a NULL pointer dereference at q->q_id = i?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
RFC PATCH 17/18 selftests: target_core_uring: Test a device exported through tcm_loop
In reply to Qiliang Yuantcmur_server can serve a target_core_uring device from a backing file.
Nothing checks that the backstore and the server work together.
Add a test that exports a device through tcm_loop and compares the SCSI disk with the backing file after writes, reads, flushes, UNMAP and WRITE SAME, with and without zero copy. Also check that a new server takes over after the first one is killed, and that commands time out without a server.
--- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/target_core_uring/Makefile | 1 + tools/testing/selftests/target_core_uring/config | 5 + .../selftests/target_core_uring/tcmur_test.sh | 137 +++++++++++++++++++++ 4 files changed, 144 insertions(+)
tools/testing/selftests/Makefile +1 -0
@@ -131,6 +131,7 @@ TARGETS += static_keys TARGETS += sync TARGETS += syscall_user_dispatch TARGETS += sysctl +TARGETS += target_core_uring TARGETS += tc-testing TARGETS += tdx TARGETS += thermal/intel/power_floor
tools/testing/selftests/target_core_uring/Makefile +1 -0
@@ -2,6 +2,7 @@ CFLAGS += -O2 -Wall -D_GNU_SOURCE $(KHDR_INCLUDES) LDLIBS += -luring -lpthread +TEST_PROGS := tcmur_test.sh TEST_GEN_PROGS_EXTENDED := tcmur_server include ../lib.mk
tools/testing/selftests/target_core_uring/config new file +5 -0
@@ -0,0 +1,5 @@ +CONFIG_IO_URING=y +CONFIG_TARGET_CORE=m +CONFIG_TCM_URING=m +CONFIG_LOOPBACK_TARGET=m +CONFIG_BLK_DEV_SD=m
tools/testing/selftests/target_core_uring/tcmur_test.sh new file +137 -0
137 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH RFC 17/18] selftests: target_core_uring: Test a device exported through tcm_loop
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] selftests: target_core_uring: missing CONFIG_SCSI, sd_mod load, and dev check - [Low] selftests…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] selftests: target_core_uring: missing CONFIG_SCSI, sd_mod load, and dev check
- [Low] selftests: target_core_uring: mktemp leak on skip and missing EXIT trap
- [Low] selftests: target_core_uring: hand-rolled TAP output in tcmur_test.sh
--
commit 6f0f6d18ce018c3449526ab2ffc205257d98be91
Author: Qiliang Yuan <[email protected]>
selftests: target_core_uring: Test a device exported through tcm_loop
This commit adds a kselftest script and kernel config fragment that export a target_core_uring device via tcm_loop, serve it with tcmur_server, and verify I/O, server restart, and command timeouts against the backing file.
Quoted 18 lines
tools/testing/selftests/target_core_uring/tcmur_test.sh new file +12
-0@@ -0,0 +1,137 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Export a target_core_uring device through tcm_loop, serve it with +# tcmur_server and check the resulting SCSI disk against the backing file. + +DIR=$(dirname "$(readlink -f "$0")") +KSFT_SKIP=4 +T=/sys/kernel/config/target +C=$T/core/uring_70/tcmur_test +WWN=naa.5001405074636d75 +L=$T/loopback/$WWN/tpgt_1
tools/testing/selftests/target_core_uring/tcmur_test.sh line 22
Quoted 10 lines
tools/testing/selftests/target_core_uring/tcmur_test.sh new file +10
-0+SIZE=$((256 << 20)) +IMG=$(mktemp /tmp/tcmur_test.XXXXXX) +PAT=$(mktemp /tmp/tcmur_test.XXXXXX) +server= +dev= +fail=0 + +[ "$(id -u)" = 0 ] || { echo "SKIP: need root"; exit $KSFT_SKIP; } +modprobe -q target_core_uring && modprobe -q tcm_loop || + { echo "SKIP: need target_core_uring and tcm_loop"; exit $KSFT_SKIP; }
[Severity: Low]
Does exiting on the root or module skip checks leak the IMG and PAT files in /tmp? IMG and PAT are created with mktemp before the prerequisite checks, and rm -f "$IMG" "$PAT" only runs at the end of the script. Without an EXIT trap, an interrupted or timed-out test run also leaves IMG, PAT, the background tcmur_server process, and the configfs entries behind.
tools/testing/selftests/target_core_uring/tcmur_test.sh line 33
Quoted 11 lines
tools/testing/selftests/target_core_uring/tcmur_test.sh new file +11
-0+ +check() { # <name> <command...> + local name=$1; shift + + if "$@"; then + echo "ok $name" + else + echo "not ok $name" + fail=1 + fi +}
[Severity: Low]
This isn't a bug, but could tcmur_test.sh source tools/testing/selftests/kselftest/ktap_helpers.sh instead of hand-rolling KSFT_SKIP, check(), and the skip messages? Using ktap_print_header(), ktap_set_plan(), ktap_skip_all(), ktap_test_result(), ktap_test_skip(), and ktap_finished() would emit standard KTAP headers, test numbers, and summary totals.
[ ... ]
tools/testing/selftests/target_core_uring/tcmur_test.sh line 67
Quoted 13 lines
tools/testing/selftests/target_core_uring/tcmur_test.sh new file +13
-0+ before=$(lsblk -dno NAME | sort) + mkdir -p $L + echo naa.5001405074636d76 > $L/nexus + mkdir $L/lun/lun_0 + ln -s $C $L/lun/lun_0/tcmur_test + for _ in $(seq 40); do + dev=$(comm -13 <(echo "$before") <(lsblk -dno NAME | sort) | head -1) + [ -n "$dev" ] && break + sleep 0.5 + done + udevadm settle + dev=/dev/$dev +}
[Severity: Medium]
What happens in setup() if the SCSI disk never appears in lsblk?
After polling for 20 seconds, dev remains empty and dev=/dev/$dev sets dev to /dev/, causing each of the four setup() calls to stall for 20 seconds (exceeding the default 45-second kselftest timeout) before running dd and blkdiscard against /dev/.
This is easy to hit in minimal test environments for two reasons:tools/testing/selftests/target_core_uring/config sets CONFIG_LOOPBACK_TARGET=m and CONFIG_BLK_DEV_SD=m without enabling CONFIG_SCSI, which both symbols depend on (CONFIG_TARGET_CORE only selects CONFIG_SCSI_COMMON), so Kconfig drops them when merging onto a base config without CONFIG_SCSI. In addition, the prerequisite check at the top of the script loads target_core_uring and tcm_loop, but does not load sd_mod (for CONFIG_BLK_DEV_SD=m) or verify that configfs is mounted at /sys/kernel/config/target.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
RFC PATCH 18/18 Documentation: target: Describe the uring backstore
In reply to Qiliang Yuantarget_core_uring is a new LIO backstore served by a userspace process.
Its configfs parameters and the protocol between the kernel and the server are only described by the uAPI header and the code.
Add tcm-uring.rst, which explains how the backstore differs from TCM/USER, how to create a device and what a server has to do.
--- Documentation/target/index.rst | 1 + Documentation/target/tcm-uring.rst | 84 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)
Documentation/target/index.rst +1 -0
@@ -8,5 +8,6 @@ TCM Virtual Device :maxdepth: 1 tcmu-design + tcm-uring tcm_mod_builder scripts
Documentation/target/tcm-uring.rst new file +84 -0
84 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 9 lines · Qiliang Yuan
TCM/USER is how a userspace process serves a LIO device, for instance to export distributed storage over iSCSI or vhost-scsi. All commands of a device go through one command ring under one mutex, the data of each command is copied into and out of a shared data area, and the server has to parse every SCSI CDB itself.
This series adds
target_core_uring, a second user backstore next to TCM/USER, built the way ublk is:
Quoted 8 lines
- The kernel emulates the SCSI commands. The server only sees reads, writes, flushes, UNMAP and WRITE SAME as an operation, an offset and a length.
- A device has several queues. Each queue is served throughio_uringcommands on its own ring, with no lock shared between queues.
- Data is copied once, with pread()/pwrite() on the character device, or not at all when the server registers the pages of a command as anio_uringfixed buffer.
This is ublk's uapi re-implemented inside LIO: FETCH/COMMIT_AND_FETCH per tag, pread()/pwrite() with (qid, tag) in the offset, and buffer registration via io_buffer_register_bvec(). It starts without what ublk added afterwards (queue affinity, recovery, auto buffer registration, batching), and the sashiko reviews already found the same abort and lifetime races ublk fixed years ago.
Follows two alternatives, especially with zero kernel change:
1) If LIO is required (vhost-scsi, FC, SRP, iSER), serve the device with ublk and export it via target_core_iblock. Same per-queue io_uring, same user copy and zero copy path, no new uapi. Please benchmark that configuration, with UBLK_F_USER_COPY and with UBLK_F_AUTO_BUF_REG, same 32 queues and threads.
2) If the goal is iSCSI for userspace storage, keep the kernel out of the data path entirely. A write here still goes NIC -> iscsit -> LIO -> uring_cmd -> pread() -> storage client, and a distributed storage client needs the data in its own memory anyway, so the zero copy path only helps file and block backends, which iblock and fileio already cover.
A userspace iSCSI target on io_uring receives straight into the server's buffer, with one copy fewer and no new kernel ABI. ioutgt
https://github.com/io-target/ioutgt
is that design: one ring per queue thread for socket and backend I/O, lock-free data path, NVMe/TCP and NVMe/RDMA today, iSCSI on the roadmap. It already beats the kernel target by a wide margin: about 2x nvmet on 4k NVMe/TCP random read and write, and 2x on 64k over both TCP and RDMA, on 100GbE. Adding iSCSI there is a bounded userspace project.
Quoted 6 lines
Patches 1-4 create the device, its queues and the character device. Patches 5-8 pass reads and writes to the server. Patches 9-11 handle a server that exits or hangs, and aborts. Patches 12-14 add flush, UNMAP and WRITE SAME, patch 15 adds fixed buffers. Patches 16-18 add a small server, a selftest and the documentation.
Quoted 8 lines
Not covered yet: protection information, and a way for existing tcmu-runner handlers to move over without changes, as they implement SCSI rather than block commands.
Exporting a 2 GiB device through
tcm_loopto fio (io_uring, direct I/O, iodepth 32 for 4k and 8 for 1M, 6 seconds per job) on a 32 CPU VM, with TCM/USER and a single-threaded handler, then with this series, 32 queues and one server thread per queue. Both servers keep the data in memory.
32 threads against one, over tcm_loop with data in memory, measures only the hand-off. Please run a real fabric and explain the -13% single job result.
Thanks,
Ming
Quoted 3 lines · Ming Lei
This is ublk's uapi re-implemented inside LIO: FETCH/COMMIT_AND_FETCH per tag, pread()/pwrite() with (qid, tag) in the offset, and buffer registration via
io_buffer_register_bvec().
Yes, the interface is modelled on ublk on purpose.
Quoted 5 lines
1) If LIO is required (vhost-scsi, FC, SRP, iSER), serve the device with ublk and export it via
target_core_iblock. Same per-queueio_uring, same user copy and zero copy path, no new uapi. Please benchmark that configuration, withUBLK_F_USER_COPYand withUBLK_F_AUTO_BUF_REG, same 32 queues and threads.
Done, with kublk from selftests/ublk (loop target, -q 32 -d 128, "-u" and "--auto_zc") behind target_core_iblock, against this series with 32 queues. Both serve the same 2 GiB file on tmpfs. Same VM (32 CPUs) and fio jobs as in the cover letter, three alternating runs, median.
Over tcm_loop:
User copy (UBLK_F_USER_COPY against pread()/pwrite()):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 201 358 +78%
4k randread, 4 jobs 614 1440 +135% 4k randread, 16 jobs 4456 5899 +32% 4k randwrite, 16 jobs 1206 2077 +72%
1M read, 4 jobs 6444 15465 +140%
1M write, 4 jobs 5638 6941 +23%
Zero copy (UBLK_F_AUTO_BUF_REG against fixed buffers):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 195 330 +69%
4k randread, 4 jobs 621 1674 +170% 4k randread, 16 jobs 4051 5773 +43% 4k randwrite, 16 jobs 1140 1952 +71%
1M read, 4 jobs 27568 49666 +80%
1M write, 4 jobs 5326 7262 +36%
Over iSCSI on 127.0.0.1 (iscsi_target_mod and iscsi_tcp in the same VM, 16 sessions, one fio job per session):
User copy (UBLK_F_USER_COPY against pread()/pwrite()):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 136 154 +13% 4k randread, 4 jobs 578 698 +21% 4k randread, 16 jobs 2516 2412 -4% 4k randwrite, 16 jobs 1512 1739 +15%
1M read, 4 jobs 613 716 +17%
1M write, 4 jobs 3747 5218 +39%
Zero copy (UBLK_F_AUTO_BUF_REG against fixed buffers):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 145 153 +6% 4k randread, 4 jobs 576 679 +18% 4k randread, 16 jobs 2740 2222 -19% 4k randwrite, 16 jobs 1716 1508 -12%
1M read, 4 jobs 778 805 +3%
1M write, 4 jobs 5235 5262 +1%
Through vhost-scsi to a nested KVM guest (16 vCPUs, 16 virtqueues, worker_per_virtqueue=on), fio running in the guest:
User copy (UBLK_F_USER_COPY against pread()/pwrite()):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 335 213 -36% 4k randread, 4 jobs 1436 844 -41% 4k randread, 16 jobs 2341 2651 +13% 4k randwrite, 16 jobs 1467 1640 +12%
1M read, 4 jobs 8077 21991 +172%
1M write, 4 jobs 5006 6943 +39%
Zero copy (UBLK_F_AUTO_BUF_REG against fixed buffers):
workload ublk + iblock (MB/s) this series (MB/s) change
4k randread, 1 job 339 306 -10% 4k randread, 4 jobs 1521 1030 -32% 4k randread, 16 jobs 2589 1987 -23% 4k randwrite, 16 jobs 1518 1669 +10%
1M read, 4 jobs 37494 39978 +7%
1M write, 4 jobs 4519 8240 +82%
So this series is ahead over tcm_loop, about even over iSCSI, and behind ublk + iblock for 4k with few jobs through vhost-scsi. I have not found out yet where the time goes in the vhost-scsi case. With zero copy and 16 jobs ublk is ahead on both fabrics, which I attribute to auto buffer registration saving the two extra commands per I/O that this series needs.
Quoted 3 lines
32 threads against one, over
tcm_loopwith data in memory, measures only the hand-off. Please run a real fabric and explain the -13% single job result.
Same fabrics, data in memory, TCM/USER against this series with one queue (one server thread) and with 32. Over iSCSI:
workload TCM/USER 1 queue 32 queues
(MB/s) (MB/s) change (MB/s) change
4k randread, 1 job 138 157 +14% 155 +12% 4k randread, 4 jobs 538 679 +26% 685 +27% 4k randread, 16 jobs 720 1101 +53% 2631 +265% 4k randwrite, 16 jobs 859 1228 +43% 2161 +152% 1M read, 4 jobs 756 799 +6% 765 +1%
1M write, 4 jobs 2263 6075 +168% 5379 +138%
1M reads over iSCSI are limited by the loopback TCP connection for every backend here (retransmits and zero window stalls), so that row says nothing about the backstores.
And through vhost-scsi:
workload TCM/USER 1 queue 32 queues
(MB/s) (MB/s) change (MB/s) change
4k randread, 1 job 201 237 +18% 238 +18% 4k randread, 4 jobs 1095 902 -18% 843 -23% 4k randread, 16 jobs 957 1119 +17% 2865 +199% 4k randwrite, 16 jobs 749 1098 +47% 2599 +247% 1M read, 4 jobs 5373 15317 +185% 56532 +952% 1M write, 4 jobs 2064 10948 +430% 27074 +1212%
On the single job over tcm_loop: one fio job submits from one CPU, so only one queue and one server thread are in use and it is one thread against one. The TCM/USER handler finds the data area in its mmap and takes a whole batch per wakeup, while the server here issues one pwrite() per read on top of the io_uring_enter(). Counted over the whole system during that job, that is 1.1 system calls and 0.27 context switches per I/O for TCM/USER against 2.2 and 0.39 for this series.
Quoted 2 lines
2) If the goal is iSCSI for userspace storage, keep the kernel out of the data path entirely.
The goal is the fabrics LIO already has, vhost-scsi and tcm_loop as much as iSCSI, for setups that run TCM/USER today.
Thanks,
Qiliang
Quoted 13 lines · Qiliang Yuan
Quoted 3 lines · Ming Lei
This is ublk's uapi re-implemented inside LIO: FETCH/COMMIT_AND_FETCH per tag, pread()/pwrite() with (qid, tag) in the offset, and buffer registration via
io_buffer_register_bvec().Yes, the interface is modelled on ublk on purpose.
Quoted 5 lines
1) If LIO is required (vhost-scsi, FC, SRP, iSER), serve the device with ublk and export it via
target_core_iblock. Same per-queueio_uring, same user copy and zero copy path, no new uapi. Please benchmark that configuration, withUBLK_F_USER_COPYand withUBLK_F_AUTO_BUF_REG, same 32 queues and threads.
Quoted 4 lines
Done, with kublk from selftests/ublk (loop target, -q 32 -d 128, "-u" and "--auto_zc") behind
target_core_iblock, against this series with 32 queues. Both serve the same 2 GiB file on tmpfs. Same VM (32 CPUs) and fio jobs as in the cover letter, three alternating runs, median.
Can you share your setup? It is supposed that you run fio from VM, otherwise you can just compare ublk with scsi backed with tcmu.
But if you run fio from VM, I believe you can setup virtio-scsi backed by ublk block device directly, which should cover your requirement, or maybe I miss something?
...
Quoted 22 lines
Quoted 3 lines
32 threads against one, over
tcm_loopwith data in memory, measures only the hand-off. Please run a real fabric and explain the -13% single job result.Same fabrics, data in memory, TCM/USER against this series with one queue (one server thread) and with 32. Over iSCSI:
workload TCM/USER 1 queue 32 queues
(MB/s) (MB/s) change (MB/s) change4k randread, 1 job 138 157 +14% 155 +12% 4k randread, 4 jobs 538 679 +26% 685 +27% 4k randread, 16 jobs 720 1101 +53% 2631 +265% 4k randwrite, 16 jobs 859 1228 +43% 2161 +152% 1M read, 4 jobs 756 799 +6% 765 +1%1M write, 4 jobs 2263 6075 +168% 5379 +138%
1M reads over iSCSI are limited by the loopback TCP connection for every backend here (retransmits and zero window stalls), so that row says nothing about the backstores.
And through vhost-scsi:
Quoted 8 lines
workload TCM/USER 1 queue 32 queues
(MB/s) (MB/s) change (MB/s) change4k randread, 1 job 201 237 +18% 238 +18% 4k randread, 4 jobs 1095 902 -18% 843 -23% 4k randread, 16 jobs 957 1119 +17% 2865 +199% 4k randwrite, 16 jobs 749 1098 +47% 2599 +247% 1M read, 4 jobs 5373 15317 +185% 56532 +952% 1M write, 4 jobs 2064 10948 +430% 27074 +1212%
My question is actually why there is regression wrt. 4k randread:
workload TCM/USER (MB/s) uring (MB/s) change
4k randread, 1 job 426 (409-463) 369 (366-384) -13%
Maybe you can focus on single job perf only, which is easy to compare and investigate.
Quoted 9 lines
On the single job over
tcm_loop: one fio job submits from one CPU, so only one queue and one server thread are in use and it is one thread against one. The TCM/USER handler finds the data area in its mmap and takes a whole batch per wakeup, while the server here issues onepwrite()per read on top of theio_uring_enter(). Counted over the whole system during that job, that is 1.1 system calls and 0.27 context switches per I/O for TCM/USER against 2.2 and 0.39 for this series.
Quoted 5 lines
Quoted 2 lines
2) If the goal is iSCSI for userspace storage, keep the kernel out of the data path entirely.
The goal is the fabrics LIO already has, vhost-scsi and
tcm_loopas much as iSCSI, for setups that run TCM/USER today.
When userspace target becomes more efficient, I'd suggest you to think about why these applications can't be improved or refactored for the potential (bigger)perf requirement.
Thanks,
Ming
Quoted 6 lines · Ming Lei
Can you share your setup? It is supposed that you run fio from VM, otherwise you can just compare ublk with scsi backed with tcmu.
But if you run fio from VM, I believe you can setup virtio-scsi backed by ublk block device directly, which should cover your requirement, or maybe I miss something?
Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16 sessions over 127.0.0.1), fio runs in that guest on the initiator disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the vhost-scsi disk, with the target and the backstore server in the outer guest. In the ublk runs, kublk serves the file as /dev/ublkbN and target_core_iblock exports that through the same fabric.
You are not missing anything. This backstore has block semantics: the kernel emulates the SCSI commands, PR and ALUA included, exactly as for iblock, so ublk behind target_core_iblock offers the same function on every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of the ublk device covers it as well. What the series adds over that is only performance, and outside tcm_loop the numbers I sent do not show a consistent gain.
[...]
Quoted 7 lines
My question is actually why there is regression wrt. 4k randread:
workload TCM/USER (MB/s) uring (MB/s) change
4k randread, 1 job 426 (409-463) 369 (366-384) -13%Maybe you can focus on single job perf only, which is easy to compare and investigate.
It comes from the scheduler moving fio between CPUs. The pwrite() I pointed to in my last mail is not it: doing that copy inside io_uring_enter() instead makes no difference.
tcm_loop queues the LIO submission and completion work on the CPU that submitted the command, so with one job, fio's CPU always has bound kworkers and ksoftirqd runnable next to fio. The load balancer cannot move those and moves fio instead. This backstore gives it more chances:
the completion work on fio's CPU handles 14 commands per run here against 24 with TCM/USER, so per 1000 I/Os the completion kworker and ksoftirqd run about 57 times against 35, fio is preempted as often, and it is moved two to three times as often. With balancing on, fio's own CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us with TCM/USER, spread over the whole submission path rather than in one function.
Turning load balancing off for fio's CPUs and nothing else reverses the result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run, five alternating runs, median (min-max). fio runs in a cpuset of CPUs 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either a normal member (balanced) or an isolated partition (not balanced):
fio CPUs 4-31, backstore bandwidth (MB/s) change
balanced, TCM/USER 439 (398-462) balanced, uring 379 (371-388) -13.7% not balanced, TCM/USER 662 (544-710) not balanced, uring 760 (575-798) +14.8%
Booting the guest with idle=poll removes the gap as well: with fio free to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s here.
That points at what each switch costs in a guest. An idle vCPU executes HLT, which exits to the host and puts the vCPU thread to sleep; waking it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host wakeup and a VM entry, and the host may run the thread on another physical CPU, so the guest code then runs with cold caches. With idle=poll the vCPUs spin instead, stay on their physical CPUs and take the IPI without an exit. This backstore makes more of those switches per I/O, so it pays more of that cost, and fio's slowdown spread over the whole submission path looks like cold caches.
What is measured is that idle=poll removes the gap and that the gap follows the switches and migrations. Which part of the cost is the exit and wakeup latency and which part is the vCPU thread landing on another physical CPU is inferred: I cannot trace the host side here. Waking an idle CPU is much cheaper on bare metal, so I expect the gap to be smaller there, but I have not measured it. idle=poll is only a diagnostic, it burns a host CPU per idle vCPU.
[...]
Quoted 3 lines
When userspace target becomes more efficient, I'd suggest you to think about why these applications can't be improved or refactored for the potential (bigger)perf requirement.
Agreed. Since this backstore only takes block commands, an application that can move to it can just as well move to ublk, and one that needs SCSI semantics in userspace is not served by it either.
Thanks,
Qiliang
Quoted 15 lines · Qiliang Yuan
Quoted 6 lines · Ming Lei
Can you share your setup? It is supposed that you run fio from VM, otherwise you can just compare ublk with scsi backed with tcmu.
But if you run fio from VM, I believe you can setup virtio-scsi backed by ublk block device directly, which should cover your requirement, or maybe I miss something?
Everything runs in one 32 CPU guest. For
tcm_loopand iSCSI (16 sessions over 127.0.0.1), fio runs in that guest on the initiator disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the vhost-scsi disk, with the target and the backstore server in the outer guest. In the ublk runs, kublk serves the file as/dev/ublkbNandtarget_core_iblockexports that through the same fabric.
Quoted 7 lines
You are not missing anything. This backstore has block semantics: the kernel emulates the SCSI commands, PR and ALUA included, exactly as for iblock, so ublk behind
target_core_iblockoffers the same function on every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of the ublk device covers it as well. What the series adds over that is only performance, and outsidetcm_loopthe numbers I sent do not show a consistent gain.
But in your last reply, not see the setup(virtio-scsi over ublk) and perf number/compare with your scsi:target optimization.
Quoted 15 lines
[...]
Quoted 7 lines
My question is actually why there is regression wrt. 4k randread:
workload TCM/USER (MB/s) uring (MB/s) change
4k randread, 1 job 426 (409-463) 369 (366-384) -13%Maybe you can focus on single job perf only, which is easy to compare and investigate.
It comes from the scheduler moving fio between CPUs. The
pwrite()I pointed to in my last mail is not it: doing that copy insideio_uring_enter()instead makes no difference.
Quoted 11 lines
tcm_loopqueues the LIO submission and completion work on the CPU that submitted the command, so with one job, fio's CPU always has bound kworkers and ksoftirqd runnable next to fio. The load balancer cannot move those and moves fio instead. This backstore gives it more chances:
the completion work on fio's CPU handles 14 commands per run here against 24 with TCM/USER, so per 1000 I/Os the completion kworker and ksoftirqd run about 57 times against 35, fio is preempted as often, and it is moved two to three times as often. With balancing on, fio's own CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us with TCM/USER, spread over the whole submission path rather than in one function.
That is one disadvantage for running IO from kernel space, and it is also one reason why ioutgt ( https://github.com/io-target/ioutgt ) performs much better than nvmet, especially for nvmet/tcp, not mention ioutgt doesn't implement io_uring rxzc yet, and there are also other network optimizations available, all can be applied flexibly.
Quoted 37 lines
Turning load balancing off for fio's CPUs and nothing else reverses the result. Single job, 4k randread at iodepth 32 over
tcm_loop, 6 s per run, five alternating runs, median (min-max). fio runs in a cpuset of CPUs 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either a normal member (balanced) or an isolated partition (not balanced):fio CPUs 4-31, backstore bandwidth (MB/s) change
balanced, TCM/USER 439 (398-462) balanced, uring 379 (371-388) -13.7% not balanced, TCM/USER 662 (544-710) not balanced, uring 760 (575-798) +14.8%Booting the guest with idle=poll removes the gap as well: with fio free to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s here.
That points at what each switch costs in a guest. An idle vCPU executes HLT, which exits to the host and puts the vCPU thread to sleep; waking it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host wakeup and a VM entry, and the host may run the thread on another physical CPU, so the guest code then runs with cold caches. With idle=poll the vCPUs spin instead, stay on their physical CPUs and take the IPI without an exit. This backstore makes more of those switches per I/O, so it pays more of that cost, and fio's slowdown spread over the whole submission path looks like cold caches.
What is measured is that idle=poll removes the gap and that the gap follows the switches and migrations. Which part of the cost is the exit and wakeup latency and which part is the vCPU thread landing on another physical CPU is inferred: I cannot trace the host side here. Waking an idle CPU is much cheaper on bare metal, so I expect the gap to be smaller there, but I have not measured it. idle=poll is only a diagnostic, it burns a host CPU per idle vCPU.
[...]
Quoted 7 lines
Quoted 3 lines
When userspace target becomes more efficient, I'd suggest you to think about why these applications can't be improved or refactored for the potential (bigger)perf requirement.
Agreed. Since this backstore only takes block commands, an application that can move to it can just as well move to ublk, and one that needs SCSI semantics in userspace is not served by it either.
Why? SCSI semantics can be served in userspace, just like existing virtio-scsi user backend.
Such as, uring-scsi may be supported in guest IO emulation application(qemu), or one uring-virtqueue bridge, the later may be done in both userspace or vhost_scsi, I guess.
Thanks,
Ming