scst: Add SRP target driver
Subsystems (3)
PATCH 4/7 scst: Add SRP target driver
This patch adds the kernel module ib_srpt, which is a SCSI RDMA Protocol (SRP) target implementation. This driver uses the InfiniBand stack and the SCST core.
It is a high performance driver capable of handling 600K+ 4K random write IOPS by a single target as well as 2.5+ GB/s sequential throughput over a single QDR IB port.
This driver was originally developed by Vu Pham (Mellanox) and has been optimized by Bart Van Assche.
--- Documentation/scst/README.srpt | 112 ++ drivers/scst/Kconfig | 1 + drivers/scst/Makefile | 2 +- drivers/scst/srpt/Kconfig | 12 + drivers/scst/srpt/Makefile | 1 + drivers/scst/srpt/ib_dm_mad.h | 139 ++ drivers/scst/srpt/ib_srpt.c | 3613 ++++++++++++++++++++++++++++++++++++++++ drivers/scst/srpt/ib_srpt.h | 352 ++++ 8 files changed, 4231 insertions(+), 1 deletions(-) create mode 100644 Documentation/scst/README.srpt create mode 100644 drivers/scst/srpt/Kconfig create mode 100644 drivers/scst/srpt/Makefile create mode 100644 drivers/scst/srpt/ib_dm_mad.h create mode 100644 drivers/scst/srpt/ib_srpt.c create mode 100644 drivers/scst/srpt/ib_srpt.h
Documentation/scst/README.srpt new file +112 -0
@@ -0,0 +1,112 @@ +SCSI RDMA Protocol (SRP) Target driver for Linux +================================================= + +The SRP Target driver is designed to work directly on top of the +OpenFabrics OFED-1.x software stack (http://www.openfabrics.org) or +the Infiniband drivers in the Linux kernel tree +(http://www.kernel.org). The SRP target driver also interfaces with +the generic SCSI target mid-level driver called SCST +(http://scst.sourceforge.net). + +How-to run +----------- + +A. On srp target machine +1. Please refer to SCST's README for loading scst driver and its +dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...) + +Example 1: working with real back-end scsi disks +a. modprobe scst +b. modprobe scst_disk +c. cat /proc/scsi_tgt/scsi_tgt + +ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt +Device (host:ch:id:lun or name) Device handler +0:0:0:0 dev_disk +4:0:0:0 dev_disk +5:0:0:0 dev_disk +6:0:0:0 dev_disk +7:0:0:0 dev_disk + +Now you want to exclude the first scsi disk and expose the last 4 scsi disks as +IB/SRP luns for I/O +echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices +echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices +echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices +echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices + +Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file) +a. modprobe scst +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk +e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices + +Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0) +a. modprobe scst +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices +h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices + +2. modprobe ib_srpt + + +B. On initiator machines you can manualy do the following steps: +1. modprobe ib_srp +2. ibsrpdm -c (to discover new SRP target) +3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target +4. fdisk -l (will show new discovered scsi disks) + +Example: +Assume that you use port 1 of first HCA in the system ie. mthca0 + +[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 +id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 +[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > +/sys/class/infiniband_srp/srp-mthca0-1/add_target + +OR + ++ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon +automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes ++ To set up and use high availability feature you need dm-multipath driver +and multipath tool ++ Please refer to OFED-1.x SRP's user manual for more in-details instructions +on how-to enable/use HA feature + +To minimize QUEUE_FULL conditions, you can apply scst_increase_max_tgt_cmds +patch from SRPT package from http://sourceforge.net/project/showfiles.php?group_id=110471 + + +Performance notes +----------------- + +In some cases, for instance working with SSD devices, which consume 100% +of a single CPU load for data transfers in their internal threads, to +maximize IOPS it can be needed to assign for those threads dedicated +CPUs using Linux CPU affinity facilities. No IRQ processing should be +done on those CPUs. Check that using /proc/interrupts. See taskset +command and Documentation/IRQ-affinity.txt in your kernel's source tree +for how to assign CPU affinity to tasks and IRQs. + +The reason for that is that processing of coming commands in SIRQ context +can be done on the same CPUs as SSD devices' threads doing data +transfers. As the result, those threads won't receive all the CPU power +and perform worse. + +Alternatively to CPU affinity assignment, you can try to enable SRP +target's internal thread. It will allows Linux CPU scheduler to better +distribute load among available CPUs. To enable SRP target driver's +internal thread you should load ib_srpt module with parameter +"thread=1". + + +Send questions about this driver to [email protected], CC: +Vu Pham <[email protected]> and Bart Van Assche <[email protected]>.
drivers/scst/Kconfig +1 -0
@@ -249,5 +249,6 @@ config SCST_MEASURE_LATENCY If unsure, say "N". source "drivers/scst/scst_local/Kconfig" +source "drivers/scst/srpt/Kconfig" endmenu
drivers/scst/Makefile +1 -1
@@ -8,4 +8,4 @@ scst-y += scst_sysfs.o scst-y += scst_mem.o scst-y += scst_debug.o -obj-$(CONFIG_SCST) += scst.o dev_handlers/ scst_local/ +obj-$(CONFIG_SCST) += scst.o dev_handlers/ scst_local/ srpt/
drivers/scst/srpt/Kconfig new file +12 -0
@@ -0,0 +1,12 @@ +config SCST_SRPT + tristate "InfiniBand SCSI RDMA Protocol target support" + depends on INFINIBAND && SCST + ---help--- + + Support for the SCSI RDMA Protocol (SRP) Target driver. The + SRP protocol is a protocol that allows an initiator to access + a block storage device on another host (target) over a network + that supports the RDMA protocol. Currently the RDMA protocol is + supported by InfiniBand and by iWarp network hardware. More + information about the SRP protocol can be found on the website + of the INCITS T10 technical committee (http://www.t10.org/).
drivers/scst/srpt/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_SCST_SRPT) += ib_srpt.o
drivers/scst/srpt/ib_dm_mad.h new file +139 -0
139 lines not on this page · raw message
drivers/scst/srpt/ib_srpt.c new file +3613 -0
2,725 lines not on this page · raw message
drivers/scst/srpt/ib_srpt.h new file +352 -0
352 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.7.1
Part of the diff is not shown here. View the raw message.
Quoted 2 lines
This patch adds the kernel module
ib_srpt, which is a SCSI RDMA Protocol
(SRP)
Quoted 1 line
target implementation. This driver uses the InfiniBand stack and the SCST
core.
Quoted 55 lines
It is a high performance driver capable of handling 600K+ 4K random write IOPS by a single target as well as 2.5+ GB/s sequential throughput over a single QDR IB port.
This driver was originally developed by Vu Pham (Mellanox) and has been optimized by Bart Van Assche.
Signed-off-by Bart Van Assche <[email protected]>Signed-off-by Vladislav Bolkhovitin <[email protected]>Cc Vu Pham <[email protected]>Cc Roland Dreier <[email protected]>Cc David Dillow <[email protected]>--- Documentation/scst/README.srpt | 112 ++ drivers/scst/Kconfig | 1 + drivers/scst/Makefile | 2 +- drivers/scst/srpt/Kconfig | 12 + drivers/scst/srpt/Makefile | 1 + drivers/scst/srpt/ib_dm_mad.h | 139 ++ drivers/scst/srpt/ib_srpt.c | 3613 ++++++++++++++++++++++++++++++++++++++++ drivers/scst/srpt/ib_srpt.h | 352 ++++ 8 files changed, 4231 insertions(+), 1 deletions(-) create mode 100644 Documentation/scst/README.srpt create mode 100644 drivers/scst/srpt/Kconfig create mode 100644 drivers/scst/srpt/Makefile create mode 100644 drivers/scst/srpt/ib_dm_mad.h create mode 100644 drivers/scst/srpt/ib_srpt.c create mode 100644 drivers/scst/srpt/ib_srpt.hDocumentation/scst/README.srpt new file +16
-0@@ -0,0 +1,112 @@ +SCSI RDMA Protocol (SRP) Target driver for Linux +================================================= + +The SRP Target driver is designed to work directly on top of the +OpenFabrics OFED-1.x software stack (http://www.openfabrics.org) or +the Infiniband drivers in the Linux kernel tree +(http://www.kernel.org). The SRP target driver also interfaces with +the generic SCSI target mid-level driver called SCST +(http://scst.sourceforge.net). + +How-to run +----------- + +A. On srp target machine +1. Please refer to SCST's README for loading scst driver and its +dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode,nullio, ...)
Documentation/scst/README.srpt
Quoted 8 lines
Documentation/scst/README.srpt new file +8
-0+ +Example 1: working with real back-end scsi disks +a. modprobe scst +b. modprobe scst_disk +c. cat /proc/scsi_tgt/scsi_tgt + +ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt +Device (host:ch:id:lun or name) Device
handler
Documentation/scst/README.srpt
Quoted 7 lines
+0:0:0:0 dev_disk +4:0:0:0 dev_disk +5:0:0:0 dev_disk +6:0:0:0 dev_disk +7:0:0:0 dev_diskDocumentation/scst/README.srpt new file +2
-0+ +Now you want to exclude the first scsi disk and expose the last 4 scsi
disks
Documentation/scst/README.srpt
Quoted 8 lines
as
Documentation/scst/README.srpt new file +7
-0+IB/SRP luns for I/O +echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices +echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices +echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices +echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices + +Example 2: working with VDISK FILEIO mode (using md0 device and file
10G-file)
Quoted 9 lines
Documentation/scst/README.srpt new file +8
-0+a. modprobe scst +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk +e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices + +Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, andcciss/c1d0)
Documentation/scst/README.srpt
Quoted 5 lines
Documentation/scst/README.srpt new file +5
-0+a. modprobe scst +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" >
/proc/scsi_tgt/vdisk/vdisk
Documentation/scst/README.srpt line 55
Quoted 4 lines
Documentation/scst/README.srpt new file +4
-0+f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices +h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices +
[Jack] This README looks should update to new sysfs interface too.
A message that is not in the archive (sent off-list or not yet received): 544F03C80B314396A5517F4CD1A848B6-aA8Fybl81pBv1O+Z8WTAqQ@public.gmane.org
Quoted 9 lines · Jack Wang
Quoted 3 lines
This patch adds the kernel module
ib_srpt, which is a SCSI RDMA Protocol(SRP)
Quoted 1 line
target implementation. This driver uses the InfiniBand stack and the SCST
core.
[ ... ][Jack] This README looks should update to new sysfs interface too.
That's correct - thanks for the feedback. If I do not receive further feedback, I will apply the patch below:
Subject: [PATCH] [SCSI] scst/ib_srpt: Updated documentation
Make the documentation more clear, update it for the new sysfs interface and add detailed information about the ib_srpt kernel module parameters.
--- Documentation/scst/README.srpt | 235 +++++++++++++++++++++++----------------- 1 files changed, 136 insertions(+), 99 deletions(-)
Documentation/scst/README.srpt +1 -44
@@ -1,112 +1,149 @@ -SCSI RDMA Protocol (SRP) Target driver for Linux +SCSI RDMA Protocol (SRP) Target Driver for Linux ================================================= -The SRP Target driver is designed to work directly on top of the -OpenFabrics OFED-1.x software stack (http://www.openfabrics.org) or -the Infiniband drivers in the Linux kernel tree -(http://www.kernel.org). The SRP target driver also interfaces with -the generic SCSI target mid-level driver called SCST -(http://scst.sourceforge.net). - -How-to run ------------ - -A. On srp target machine -1. Please refer to SCST's README for loading scst driver and its -dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...) - -Example 1: working with real back-end scsi disks -a. modprobe scst -b. modprobe scst_disk -c. cat /proc/scsi_tgt/scsi_tgt - -ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt -Device (host:ch:id:lun or name) Device handler -0:0:0:0 dev_disk -4:0:0:0 dev_disk -5:0:0:0 dev_disk -6:0:0:0 dev_disk -7:0:0:0 dev_disk - -Now you want to exclude the first scsi disk and expose the last 4 scsi disks as -IB/SRP luns for I/O -echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices -echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices -echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices -echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices - -Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file) -a. modprobe scst -b. modprobe scst_vdisk -c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk -d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk -e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices -f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices - -Example 3: working with VDISK BLOCKIO mode (using md0 device, sda,
and cciss/c1d0)
-a. modprobe scst
-b. modprobe scst_vdisk
-c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices -g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices -h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
-
-2. modprobe ib_srpt
-
-
-B. On initiator machines you can manualy do the following steps:
-1. modprobe ib_srp
-2. ibsrpdm -c (to discover new SRP target)
-3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target -4. fdisk -l (will show new discovered scsi disks)
-
-Example:
-Assume that you use port 1 of first HCA in the system ie. mthca0 +The SRP target driver ib_srpt is based on the generic SCSI target +infrastructure called SCST. It supports both the InfiniBand drivers included +with the Linux kernel and the OpenFabrics InfiniBand software stack.
-[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 -id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, -dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 -[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, -dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > -/sys/class/infiniband_srp/srp-mthca0-1/add_target
+Installation
+------------
+
+A. SRP target configuration
+
+1. Load the ib_srpt kernel module
+
+Add ib_srpt to the SCST_MODULES variable in /etc/init.d/scst such that ib_srpt +is loaded automatically upon startup. Next, load the ib_srpt kernel module +e.g. as follows:
+
+ touch /etc/scst.conf
+ /etc/init.d/scst start
+
+2. Configure SCST
+
+How to configure SCST is explained in detail in Documentation/scst/README.scst. +Once you have finished configuring SCST, save the new configuration to +/etc/scst.conf:
-OR
+ scstadmin -write_config /etc/scst.conf
-+ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon -automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes
-+ To set up and use high availability feature you need dm-multipath driver -and multipath tool
-+ Please refer to OFED-1.x SRP's user manual for more in-details instructions -on how-to enable/use HA feature
-To minimize QUEUE_FULL conditions, you can apply scst_increase_max_tgt_cmds -patch from SRPT package from
http://sourceforge.net/project/showfiles.php?group_id=110471
+B. SRP initiator configuration
+Configure the initiator as follows:
-Performance notes
------------------
+1. Verify whether the InfiniBand subnet manager is operational, e.g. as follows:
+ ping <IBoIB address of SRP target>
-In some cases, for instance working with SSD devices, which consume 100% -of a single CPU load for data transfers in their internal threads, to -maximize IOPS it can be needed to assign for those threads dedicated -CPUs using Linux CPU affinity facilities. No IRQ processing should be -done on those CPUs. Check that using /proc/interrupts. See taskset -command and Documentation/IRQ-affinity.txt in your kernel's source tree -for how to assign CPU affinity to tasks and IRQs.
+2. Load the SRP initator kernel module.
+ modprobe ib_srp
-The reason for that is that processing of coming commands in SIRQ context -can be done on the same CPUs as SSD devices' threads doing data -transfers. As the result, those threads won't receive all the CPU power -and perform worse.
+3. Run ibsrpdm to obtain a list of available SRP target systems.
+ ibsrpdm -c
-Alternatively to CPU affinity assignment, you can try to enable SRP -target's internal thread. It will allows Linux CPU scheduler to better -distribute load among available CPUs. To enable SRP target driver's -internal thread you should load ib_srpt module with parameter -"thread=1".
+4. Tell the SRP initiator to log in to the SRP target.
+ echo <target info> > /sys/class/infiniband_srp/${SRP_HCA_NAME}/add_target
+
+5. Verify whether login succeeded, e.g. as follows:
+ lsscsi
+
+ SRP targets can be recognized in the lsscsi output by looking for
+ the disk names assigned to the SCST target ("disk01" in the example below):
+
+ [8:0:0:0] disk SCST_FIO disk01 102 /dev/sdb
+
+An example:
+
+[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 +id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 +[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > +/sys/class/infiniband_srp/srp-mthca0-1/add_target
-Send questions about this driver to [email protected], CC:
-Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]> and Bart Van Assche <[email protected]>. +C. High Availability
+
+If there are redundant paths in the IB network between initiator and target,
+automatic path failover can be set up on the initiator as follows:
+* Edit /etc/infiniband/openib.conf to load the SRP driver and SRP HA daemon
+ automatically: set SRP_LOAD=yes and SRPHA_ENABLE=yes.
+* To set up and use the high availability feature you need the dm-multipath
+ driver and multipath tool.
+* Please refer to the OFED-1.x user manual for more detailed instructions
+ on how to enable and how to use the HA feature. See e.g.
+ http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED%20_Linux_user
+_manual_1_5_1_2.pdf.
+
+A setup with automatic failover between redundant targets is possible by
+installing and configuring replication software such as DRBD on both
+targets. If the initiator system supports mirroring (e.g. Linux), you can use
+the following approach:
+* Configure the replication software in Active/Active mode.
+* Configure the initiator(s) for mirroring between the redundant targets.
+
+If the initiator system does not support mirroring (e.g. VMware ESX), you
+can use the following approach:
+* Configure DRBD in Active/Passive mode and enable STONITH mode in the
+ Heartbeat software.
+
+
+D. Notes
+
+For workloads with large I/O depths increasing the SCST_MAX_TGT_DEV_COMMANDS
+constant in drivers/scst/scst_priv.h may improve performance.
+
+For latency sensitive applications, using the noop scheduler at the initiator
+side can give significantly better results than with other schedulers.
+
+The following initiator-side parameters have a small but measurable impact on
+SRP performance:
+ * /sys/class/block/${dev}/queue/rotational
+ * /sys/class/block/${dev}/queue/rq_affinity
+ * /proc/irq/${ib_int_no}/smp_affinity
+
+The ib_srpt kernel module supports the following parameters:
+* srp_max_req_size (number)
+ Maximum size of an SRP control message in bytes. Examples of SRP control
+ messages are: login request, logout request, data transfer request, ...
+ The larger this parameter, the more scatter/gather list elements can be
+ sent at once. Use the following formula to compute an appropriate value
+ for this parameter: 68 + 16 * (sg_tablesize). The default value of
+ this parameter is 2116, which corresponds to an sg table size of 128.
+* srp_max_rsp_size (number)
+ Maximum size of an SRP response message in bytes. Sense data is sent back
+ via these messages towards the initiator. The default size is 256 bytes.
+ With this value there remains (256-36) = 220 bytes for sense data.
+* srp_max_rdma_size (number)
+ Maximum number of bytes that may be transferred at once via RDMA. Defaults
+ to 65536 bytes, which is sufficient to use the full bandwidth of low-latency
+ HCAs. Increasing this value may decrease latency for applications
+ transferring large amounts of data at once.
+* srpt_srq_size (number, default 4095)
+ ib_srpt uses a shared receive queue (SRQ) for processing incoming SRP
+ requests. This number may have to be increased when a large number of
+ initiator systems is accessing a single SRP target system.
+* srpt_sq_size (number, default 4096)
+ Per-channel InfiniBand send queue size. The default setting is sufficient
+ for a credit limit of 128. Changing this parameter to a smaller value may
+ cause RDMA requests to be retried and hence may slow down data transfer
+ severely.
+* thread (0, 1 or 2, default 1)
+ Defines the context on which SRP requests are processed:
+ * thread=0: do as much processing in IRQ context as possible. Results in
+ lower latency than the other two modes but may trigger soft lockup
+ complaints when multiple initiators are simultaneously processing
+ workloads with large I/O depths. Scalability of this mode is limited
+ - it exploits only a fraction of the power available on multiprocessor
+ systems.+ * thread=1: dedicates one kernel thread per initiator. Scales well on
+ multiprocessor systems. This is the recommended mode when multiple
+ initiator systems are accessing the same target system simultaneously.
+ * thread=2: makes one CPU process all IB completions and defer further
+ processing to kernel thread context. Scales better than mode thread=0 but + not as good as mode thread=1. May trigger soft lockup complaints when + multiple initiators are simultaneously processing workloads with large I/O + depths.
+* trace_flag (unsigned integer, only available in debug builds)
+ The individual bits of the trace_flag parameter define which categories of
+ trace messages should be sent to the kernel log and which ones not.
Signature · 5 lines
1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
So, just looking over some of the atomic_t usage here (which is usually one of the first places I review):
drivers/scst/srpt/ib_srpt.h line 212
Quoted 3 lines
drivers/scst/srpt/ib_srpt.h new file +2
-0+struct srpt_send_ioctx { +atomic_t state;
this seems to be accessed only with atomic_read(), atomic_set() and atomic_cmpxchg() (without any memory barriers). It's rather hard to see that this is correct (and indeed without memory barriers I suspect it's not; also since atomic_read() does not protect the state from changing immediately afterwards there may be other races).
I think it might be better to handle this in a simpler, more naive way -- just have a lock (probably an existing one) that protects the contents of state and not use cmpxchg().
In any case since no "real" atomic operations are used, I suspect it would be better to just code this in terms of unsigned int and regular cmpxchg().
Also, there is processing_compl:
drivers/scst/srpt/ib_srpt.c line 1817
Quoted 6 lines
drivers/scst/srpt/ib_srpt.c new file +6
-0+static void srpt_completion (struct ib_cq *cq,void *ctx) +{ +struct srpt_rdma_ch *ch = ctx; + +BUG_ON (!ch); +atomic_inc (&ch->processing_compl);
and
drivers/scst/srpt/ib_srpt.c line 1970
Quoted 4 lines
drivers/scst/srpt/ib_srpt.c new file +3
-0+static void srpt_unregister_channel(struct srpt_rdma_ch *ch) + while (atomic_read(&ch->processing_compl)) + ;
this seems racy to me -- I don't see any reason why we couldn't have:
srpt_completion() srpt_unregister_channel() processing_compl == 0, continue atomic_inc(&ch->processing_compl); finish unregistering channel use unregistered channel
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · Roland Dreier
So, just looking over some of the
atomic_tusage here (which is usually one of the first places I review):> +struct
srpt_send_ioctx{
> ...
> + atomic_t state;this seems to be accessed only with
atomic_read(),atomic_set()andatomic_cmpxchg()(without any memory barriers). It's rather hard to see that this is correct (and indeed without memory barriers I suspect it's not; also sinceatomic_read()does not protect the state from changing immediately afterwards there may be other races).
Quoted 7 lines
I think it might be better to handle this in a simpler, more naive way -- just have a lock (probably an existing one) that protects the contents of state and not use
cmpxchg().In any case since no "real" atomic operations are used, I suspect it would be better to just code this in terms of unsigned int and regular
cmpxchg().
Hello Roland,
Thanks for the feedback.
I'll have a look at converting these atomic operations into regular locking. The current implementation should be fine though. Manipulating I/O context state only happens inside callback functions. These callback functions are either invoked by the SCST core or from inside an IB completion callback. For each I/O context at most one such callback function is active at any time. Even if different callback functions would be invoked on different CPU cores, it is the responsibility of either the SCST core or the IB core / driver to provide proper synchronization (memory barrier).
There is one exception though: the function srpt_pending_cmd_timeout() can in theory proceed concurrently with an IB completion callback. This is why I/O context state manipulation was implemented via atomic operations. That race is unlikely to cause any harm though: the callback srpt_pending_cmd_timeout() is only invoked if 60 seconds (srpt_template.max_hw_pending_time) after an IB send was posted no IB completion has been received. This timeout should be a magnitude larger than the time needed to receive an IB completion indicating either success or failure.
Please note that with IB firmware that complies to the IB specs the function srpt_pending_cmd_timeout() will never be invoked because before that function gets invoked an IB completion will already have been received.
Quoted 24 lines
Also, there is
processing_compl:> +static void
srpt_completion(struct ib_cq *cq, void *ctx)> +{
> + structsrpt_rdma_ch*ch = ctx;
> +
> + BUG_ON(!ch);
> + atomic_inc(&ch->processing_compl);and
> +static void
srpt_unregister_channel(struct srpt_rdma_ch *ch)> ...
> + while (atomic_read(&ch->processing_compl))
> + ;this seems racy to me -- I don't see any reason why we couldn't have:
srpt_completion()
srpt_unregister_channel()
processing_compl== 0, continue
Quoted 5 lines
atomic_inc(&ch->processing_compl);
finish unregistering channel
use unregistered channel
I'm not sure what the above pseudo-code should do ?
Regarding the current implementation: there is a hard requirement in SCST that no new commands are queued for a given session after scst_unregister_session() has been invoked. So scst_unregister_session() must only be invoked after the IB queue pair has been reset *and* srpt_completion() has finished. It would be great if that could be implemented without using one or another kind of counter. I'm not sure however whether it is possible to eliminate the "processing_compl" counter entirely.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
I'll have a look at converting these atomic operations into regular locking. The current implementation should be fine though.
I believe that the current implementation is correct. However it is much harder for someone naive like me to understand, since using cmpxchg is much subtler than just using a lock to protect data. If this isn't on the hottest of hot paths then I think cmpxchg is over-optimization -- much better to have less fancy code.
Quoted 24 lines
Quoted 24 lines
Also, there is
processing_compl:> +static void
srpt_completion(struct ib_cq *cq, void *ctx)> +{
> + structsrpt_rdma_ch*ch = ctx;
> +
> + BUG_ON(!ch);
> + atomic_inc(&ch->processing_compl);and
> +static void
srpt_unregister_channel(struct srpt_rdma_ch *ch)> ...
> + while (atomic_read(&ch->processing_compl))
> + ;this seems racy to me -- I don't see any reason why we couldn't have:
srpt_completion()
srpt_unregister_channel()
processing_compl== 0, continue
Quoted 7 lines
Quoted 5 lines
atomic_inc(&ch->processing_compl);
finish unregistering channel
use unregistered channel
I'm not sure what the above pseudo-code should do ?
I'm trying to illustrate the race by showing what two different contexts might be doing -- the left column is one context, the right column is the other context.
In this specific case, if the while loop testing processing_compl actually ever does anything then I don't see how it could be safe -- if srpt_completion() could be executing during the while loop, then I don't see anything that prevents the while loop from finishing before srpt_completion does its atomic_inc().
Quoted 8 lines
Regarding the current implementation: there is a hard requirement in SCST that no new commands are queued for a given session after
scst_unregister_session()has been invoked. Soscst_unregister_session()must only be invoked after the IB queue pair has been reset *and*srpt_completion()has finished. It would be great if that could be implemented without using one or another kind of counter. I'm not sure however whether it is possible to eliminate the "processing_compl" counter entirely.
I don't pretend to understand the flow of code here. But it seems the only safe way to implement this cleanup code is to stop posting work requests and then wait until all the requests you've posted are complete. Trying to wait for the completion handler to stop executing does not seem possible to implement in a safe way.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html