Add an SRP test for the SoftiWARP driver
v2 cover Add an SRP test for the SoftiWARP driver
Hi Omar,
Recently a new low-level RDMA driver went upstream, namely the SoftiWARP driver. That driver implements RDMA over TCP. Support has been added in the SRP initiator and target drivers for iWARP. This patch series adds a test for SRP over SoftiWARP. Please consider integration of this patch series in the official blktests repository.
Thanks,
Bart.
Changes compared to v1:
- Only run the new test if the kernel version is at least 5.5 (the version in which iWARP support was added to the SRP drivers) and if "rdma link" is supported.
Bart Van Assche (4):
common/multipath-over-rdma: Fix expand_ipv6_addr() common/multipath-over-rdma: Rename two functions common/multipath-over-rdma, tests/srp: Make it easy to use siw instead
of rdma_rxe
tests/srp/015: Add a test that uses the SoftiWARP (siw) driver
common/multipath-over-rdma | 58 +++++++++++++++++----- common/rc | 28 +++++++++++ tests/srp/015 | 45 +++++++++++++++++ tests/srp/015.out | 2 + tests/srp/rc | 98 ++++++++++++++++++++++---------------- 5 files changed, 179 insertions(+), 52 deletions(-) create mode 100755 tests/srp/015 create mode 100644 tests/srp/015.out
PATCH v2 1/4 common/multipath-over-rdma: Fix expand_ipv6_addr()
For IPv6 address ::1, instead of returning
0:0000:0000:0000:0000:0000:0000:0000:0001, return
0000:0000:0000:0000:0000:0000:0000:0001.
--- common/multipath-over-rdma | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
common/multipath-over-rdma +1 -1
@@ -61,7 +61,7 @@ get_ipv4_addr() { # Convert e.g. ::1 into 0000:0000:0000:0000:0000:0000:0000:0001. expand_ipv6_addr() { - awk -F : 'BEGIN{left=1} { for(i=1;i<=NF;i++) { a=substr("0000", 1+length($i)) $i; if ($i == "") left=0; else if (left) pre = pre ":" a; else suf = suf ":" a }; mid=substr(":0000:0000:0000:0000:0000:0000:0000:0000", (pre!="")+length(pre)+length(suf)); print substr(pre,2) mid suf}' + awk -F : '{ left=1; for(i=1;i<=NF;i++) { a=substr("0000", 1+length($i)) $i; if ($i == "") left=0; else if (left) pre = pre ":" a; else suf = suf ":" a }; mid=substr("0000:0000:0000:0000:0000:0000:0000:0000", (pre=="")+length(pre)+length(suf)); print substr(pre,2) mid suf}' } get_ipv6_addr() {
PATCH v2 2/4 common/multipath-over-rdma: Rename two functions
In reply to Bart Van AsscheSince the meaning of this functions will change, make sure that the function name will match the new meaning.
--- common/multipath-over-rdma | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
common/multipath-over-rdma +4 -4
@@ -409,7 +409,7 @@ has_rdma_rxe() { } # Load the rdma_rxe kernel module and associate it with all network interfaces. -start_rdma_rxe() { +start_soft_rdma() { { modprobe rdma_rxe || return $? (
@@ -425,7 +425,7 @@ start_rdma_rxe() { # Dissociate the rdma_rxe kernel module from all network interfaces and unload # the rdma_rxe kernel module. -stop_rdma_rxe() { +stop_soft_rdma() { ( cd /sys/class/net && for i in *; do
@@ -608,7 +608,7 @@ unload_null_blk() { } setup_rdma() { - start_rdma_rxe + start_soft_rdma ( echo "RDMA interfaces:" cd /sys/class/infiniband &&
@@ -627,7 +627,7 @@ teardown_uncond() { killall -9 multipathd >&/dev/null rm -f /etc/multipath.conf stop_target - stop_rdma_rxe + stop_soft_rdma unload_null_blk }
PATCH v2 3/4 common/multipath-over-rdma, tests/srp: Make it easy to use siw instead of rdma_rxe
In reply to Bart Van AsscheMake all_primary_gids() return GIDs with zero suffix since SoftiWARP GIDs have such a suffix. Add SoftiWARP support in start_soft_rdma() and stop_soft_rdma(). Make do_rdma_cm_login() submit the source IP address and scope ID to the SRP initiator driver since that information is essential to establish an IPv6 connection between link-local addresses. Iterate over RDMA ports instead of UMAD device nodes in log_in() since no UMAD device node is associated with SoftiWARP ports. In start_lio_srpt(), use the GID format to identify SoftiWARP ports since the GID suffix for these ports is zero.
--- common/multipath-over-rdma | 48 ++++++++++++++++--- tests/srp/rc | 98 ++++++++++++++++++++++---------------- 2 files changed, 99 insertions(+), 47 deletions(-)
common/multipath-over-rdma +41 -7
@@ -11,6 +11,7 @@ filesystem_type=ext4 fio_aux_path=/tmp/fio-state-files memtotal=$(sed -n 's/^MemTotal:[[:blank:]]*\([0-9]*\)[[:blank:]]*kB$/\1/p' /proc/meminfo) max_ramdisk_size=$((1<<25)) +use_siw= ramdisk_size=$((memtotal*(1024/16))) # in bytes if [ $ramdisk_size -gt $max_ramdisk_size ]; then ramdisk_size=$max_ramdisk_size
@@ -389,9 +390,13 @@ function mountpoint() { # All primary RDMA GIDs all_primary_gids() { - find /sys/devices -name infiniband | while read -r p; do - cat "$p"/*/ports/*/gids/0 - done | grep -v ':0000:0000:0000:0000$' + local gid p + + for p in /sys/class/infiniband/*/ports/*/gids/0; do + gid="$(<"$p")" + [ "$gid" != 0000:0000:0000:0000:0000:0000:0000:0000 ] && + echo "$gid" + done } # Check whether or not an rdma_rxe instance has been associated with network
@@ -408,24 +413,40 @@ has_rdma_rxe() { return 1 } -# Load the rdma_rxe kernel module and associate it with all network interfaces. +# Load the rdma_rxe or siw kernel module and associate it with all network +# interfaces. start_soft_rdma() { { + if [ -n "$use_siw" ]; then + modprobe siw || return $? + ( + cd /sys/class/net && + for i in *; do + [ -e "$i" ] || continue + [ -e "/sys/class/infiniband/${i}_siw" ] && continue + rdma link add "${i}_siw" type siw netdev "$i" || + echo "Failed to bind the siw driver to $i" + done + ) + else modprobe rdma_rxe || return $? ( cd /sys/class/net && for i in *; do if [ -e "$i" ] && ! has_rdma_rxe "$i"; then - echo "$i" > /sys/module/rdma_rxe/parameters/add + echo "$i" > /sys/module/rdma_rxe/parameters/add || + echo "Failed to bind the rdma_rxe driver to $i" fi done ) + fi } >>"$FULL" } -# Dissociate the rdma_rxe kernel module from all network interfaces and unload -# the rdma_rxe kernel module. +# Dissociate the rdma_rxe or siw kernel module from all network interfaces and +# unload the rdma_rxe kernel module. stop_soft_rdma() { + { ( cd /sys/class/net && for i in *; do
@@ -439,6 +460,19 @@ stop_soft_rdma() { echo "Unloading rdma_rxe failed" return 1 fi + ( + cd /sys/class/net && + for i in *_siw; do + [ -e "$i" ] || continue + rdma link del "${i}" || + echo "Failed to unbind the siw driver from ${i%_siw}" + done + ) + if ! unload_module siw 10; then + echo "Unloading siw failed" + return 1 + fi + } >>"$FULL" } # /dev/sd... device node assigned to the scsi_debug kernel module.
tests/srp/rc +58 -40
143 lines not on this page · raw message
PATCH v2 4/4 tests/srp/015: Add a test that uses the SoftiWARP (siw) driver
In reply to Bart Van AsscheRecently support has been added in the SRP initiator and target drivers for the SoftiWARP driver. Add a test for SRP over SoftiWARP.
--- common/rc | 28 ++++++++++++++++++++++++++++ tests/srp/015 | 45 +++++++++++++++++++++++++++++++++++++++++++++ tests/srp/015.out | 2 ++ 3 files changed, 75 insertions(+) create mode 100755 tests/srp/015 create mode 100644 tests/srp/015.out
common/rc +28 -0
@@ -87,6 +87,21 @@ _have_program() { return 1 } +# Check whether the iproute2 snapshot is greater than or equal to $1. +_have_iproute2() { + local snapshot + + if ! snapshot=$(ip -V | sed 's/ip utility, iproute2-ss//'); then + SKIP_REASON="ip utility not found" + return 1 + fi + if [ "$snapshot" -lt "$1" ]; then + SKIP_REASON="ip utility too old" + return 1 + fi + return 0 +} + _have_src_program() { if [[ ! -x "$SRCDIR/$1" ]]; then SKIP_REASON="$1 was not built; run \`make\`"
@@ -134,6 +149,19 @@ _have_kernel_option() { return 1 } +# Check whether the version of the running kernel is greater than or equal to +# $1.$2.$3 +_have_kver() { + local d=$1 e=$2 f=$3 + + IFS='.' read -r a b c < <(uname -r | sed 's/-.*//') + if [ $((a * 65536 + b * 256 + c)) -lt $((d * 65536 + e * 256 + f)) ]; + then + SKIP_REASON="Kernel version too old" + return 1 + fi +} + _have_tracefs() { stat /sys/kernel/debug/tracing/trace > /dev/null 2>&1 if ! findmnt -t tracefs /sys/kernel/debug/tracing >/dev/null; then
tests/srp/015 new file +45 -0
45 lines not on this page · raw message
tests/srp/015.out new file +2 -0
@@ -0,0 +1,2 @@ +Configured SRP target driver +Passed
Quoted 8 lines · Bart Van Assche
Hi Omar,
Recently a new low-level RDMA driver went upstream, namely the SoftiWARP driver. That driver implements RDMA over TCP. Support has been added in the SRP initiator and target drivers for iWARP. This patch series adds a test for SRP over SoftiWARP. Please consider integration of this patch series in the official blktests repository.
Quoted 8 lines
Thanks,
Bart.
Changes compared to v1:
- Only run the new test if the kernel version is at least 5.5 (the version in which iWARP support was added to the SRP drivers) and if "rdma link" is supported.
Hi, Bart,
Is there no way to detect this feature other than checking the kernel version?
Thanks.
Quoted 7 lines · Omar Sandoval
Quoted 6 lines · Bart Van Assche
Recently a new low-level RDMA driver went upstream, namely the SoftiWARP driver. That driver implements RDMA over TCP. Support has been added in the SRP initiator and target drivers for iWARP. This patch series adds a test for SRP over SoftiWARP. Please consider integration of this patch series in the official blktests repository.
Quoted 7 lines
Quoted 4 lines
Changes compared to v1:
- Only run the new test if the kernel version is at least 5.5 (the version in which iWARP support was added to the SRP drivers) and if "rdma link" is supported.Is there no way to detect this feature other than checking the kernel version?
Hi Omar,
The only other way I can think of to verify whether the SRP initiator and target drivers support iWARP is by loading the SRP drivers, loading the iWARP driver, by configuring LIO + ib_srpt and by attempting to set up a connection. However, I assume that's way more than what a _have() test should do?
Bart.
Quoted 16 lines · Bart Van Assche
Quoted 7 lines · Omar Sandoval
Quoted 6 lines · Bart Van Assche
Recently a new low-level RDMA driver went upstream, namely the SoftiWARP driver. That driver implements RDMA over TCP. Support has been added in the SRP initiator and target drivers for iWARP. This patch series adds a test for SRP over SoftiWARP. Please consider integration of this patch series in the official blktests repository.
Quoted 7 lines
Quoted 4 lines
Changes compared to v1:
- Only run the new test if the kernel version is at least 5.5 (the version in which iWARP support was added to the SRP drivers) and if "rdma link" is supported.Is there no way to detect this feature other than checking the kernel version?
Quoted 7 lines
Hi Omar,
The only other way I can think of to verify whether the SRP initiator and target drivers support iWARP is by loading the SRP drivers, loading the iWARP driver, by configuring LIO +
ib_srptand by attempting to set up a connection. However, I assume that's way more than what a_have()test should do?
Hmm, we should probably add the ability to skip a test from test() in order to handle these cases where you need to do a bunch of setup before you know whether the test can run. For now, the kernel version check is fine.