Three SRP patches
cover Three SRP patches
Hi Omar,
The three patches in this series improve the SRP tests. Please consider these patches for inclusion in the upstream blktests repository.
Thanks,
Bart.
Bart Van Assche (3):
tests/block/030: Make this test less noisy tests/srp/rc: Improve reliability of stop_lio_srpt()
rdma: Use rdma link instead of /sys/class/infiniband/*/parent
common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/block/030 | 2 +- tests/srp/rc | 52 ++++++----------- 3 files changed, 48 insertions(+), 117 deletions(-)
PATCH 1/3 tests/block/030: Make this test less noisy
Since test block/030 injects blk_mq_realloc_hw_ctxs() failures, it is expected that writes into the 'submit_queues' attribute can fail. Send the 'nproc: write error: Cannot allocate memory' failures to $FULL instead of stderr. See also commit a668c61064f2 ("Add a test that triggers the blk_mq_realloc_hw_ctxs() error path").
--- tests/block/030 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/block/030 +1 -1
@@ -42,7 +42,7 @@ test() { if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then for ((i = 0; i < 100; i++)); do echo 1 > $sq - nproc > $sq + { nproc > $sq; } >>"$FULL" 2>&1 done else SKIP_REASON="Skipping test because $sq cannot be modified"
PATCH 2/3 tests/srp/rc: Improve reliability of stop_lio_srpt()
In reply to Bart Van AsscheRemove the 'np' directory if it exists. Unload the iscsi_target_mod kernel module if it has been loaded.
--- tests/srp/rc | 43 +++++++++++++++---------------------------- 1 file changed, 15 insertions(+), 28 deletions(-)
tests/srp/rc +15 -28
@@ -545,34 +545,21 @@ stop_lio_srpt() { fi done - if [ -e /sys/kernel/config/target/srpt ]; then - ( - cd /sys/kernel/config/target/srpt && ( - for d in */*/acls/*/*/lun*; do [ -L "$d" ] && rm "$d"; done - for d in */*/acls/*/lun*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*/acls/*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*/lun/lun*/*; do [ -L "$d" ] && rm "$d"; done - for d in */*/lun/lun*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*; do [ -e "$d/lun" ] && rmdir "$d"; done - for d in *; do [ -e "$d/fabric_statistics" ] && rmdir "$d"; done - true - ) && - cd .. && - for ((i=0;i<10;i++)); do - rmdir srpt - [ -e srpt ] || break - sleep .1 - done && - [ ! -e srpt ] && - unload_module ib_srpt 10 - ) || return $? - fi - - rmdir /sys/kernel/config/target/core/*/* >&/dev/null - rmdir /sys/kernel/config/target/core/* >&/dev/null - - for m in ib_srpt target_core_pscsi target_core_iblock target_core_file \ - target_core_stgt target_core_user target_core_mod + rmdir /sys/kernel/config/target/*/*/*/np/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/np >&/dev/null + rm -f /sys/kernel/config/target/*/*/*/acls/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/acls/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/acls/* >&/dev/null + rm -f /sys/kernel/config/target/*/*/*/lun/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/lun/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/* >&/dev/null + rmdir /sys/kernel/config/target/* >&/dev/null + + for m in ib_srpt iscsi_target_mod target_core_pscsi target_core_iblock \ + target_core_file target_core_stgt target_core_user \ + target_core_mod do unload_module $m 10 || return $? done
PATCH 3/3 rdma: Use rdma link instead of /sys/class/infiniband/*/parent
In reply to Bart Van AsscheThe approach of verifying whether or not an RDMA interface is associated with the rdma_rxe interface by looking up its parent device is deprecated and will be removed soon from the Linux kernel. Hence this patch that uses the rdma link command instead.
--- common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/srp/rc | 9 +-- 2 files changed, 32 insertions(+), 88 deletions(-)
common/multipath-over-rdma +31 -80
@@ -57,6 +57,9 @@ _multipathd_version_ge() { } get_ipv4_addr() { + if [ ! -e "/sys/class/net/$1" ]; then + echo "get_ipv4_addr(): $1 is not a network interface" 1>&2 + fi ip -4 -o addr show dev "$1" | sed -n 's/.*[[:blank:]]inet[[:blank:]]*\([^[:blank:]/]*\).*/\1/p' }
@@ -76,41 +79,11 @@ is_number() { [ "$1" -eq "0$1" ] 2>/dev/null } -# Check whether a device is an RDMA device. An example argument: -# /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0 -is_rdma_device() { - local d i inode1 inode2 - - inode1=$(stat -c %i "$1") - # echo "inode1 = $inode1" - for i in /sys/class/infiniband/*; do - d=/sys/class/infiniband/"$(readlink "$i")" - d=$(dirname "$(dirname "$d")") - inode2=$(stat -c %i "$d") - # echo "inode2 = $inode2" - if [ "$inode1" = "$inode2" ]; then - return - fi - done - false -} - # Lists RDMA capable network interface names, e.g. ib0 ib1. rdma_network_interfaces() { - ( - cd /sys/class/net && - for i in *; do - [ -e "$i" ] || continue - # Skip IPoIB (ARPHRD_INFINIBAND) network - # interfaces. - [ "$(<"$i"/type)" = 32 ] && continue - [ -L "$i/device" ] || continue - d=$(readlink "$i/device" 2>/dev/null) - if [ -n "$d" ] && is_rdma_device "$i/$d"; then - echo "$i" - fi - done - ) + rdma link show | + sed -n 's/^.*[[:blank:]]netdev[[:blank:]]\+\([^[:blank:]]*\)[[:blank:]]*/\1/p' | + sort -u } # Check whether any stacked block device holds block device $1. If so, echo
@@ -411,47 +384,36 @@ all_primary_gids() { done } -# Check whether or not an rdma_rxe instance has been associated with network -# interface $1. -has_rdma_rxe() { - local f - - for f in /sys/class/infiniband/*/parent; do - if [ -e "$f" ] && [ "$(<"$f")" = "$1" ]; then - return 0 - fi - done - - return 1 +# Check whether or not an rdma_rxe or siw instance has been associated with +# network interface $1. +has_soft_rdma() { + rdma link | grep -q " netdev $1[[:blank:]]*\$" } # Load the rdma_rxe or siw kernel module and associate it with all network # interfaces. start_soft_rdma() { + local type + { if [ -n "$use_siw" ]; then modprobe siw || return $? - ( - cd /sys/class/net && - for i in *; do - [ -e "$i" ] || continue - [ -e "/sys/class/infiniband/${i}_siw" ] && continue - rdma link add "${i}_siw" type siw netdev "$i" || - echo "Failed to bind the siw driver to $i" - done - ) + type=siw else modprobe rdma_rxe || return $? - ( - cd /sys/class/net && - for i in *; do - if [ -e "$i" ] && ! has_rdma_rxe "$i"; then - echo "$i" > /sys/module/rdma_rxe/parameters/add || - echo "Failed to bind the rdma_rxe driver to $i" - fi - done - ) + type=rxe fi + ( + cd /sys/class/net && + for i in *; do + [ -e "$i" ] || continue + [ "$i" = "lo" ] && continue + [ "$(<"$i/addr_len")" = 6 ] || continue + has_soft_rdma "$i" && continue + rdma link add "${i}_$type" type $type netdev "$i" || + echo "Failed to bind the $type driver to $i" + done + ) } >>"$FULL" }
@@ -459,27 +421,16 @@ start_soft_rdma() { # unload the rdma_rxe kernel module. stop_soft_rdma() { { - ( - cd /sys/class/net && - for i in *; do - if [ -e "$i" ] && has_rdma_rxe "$i"; then - { echo "$i" > /sys/module/rdma_rxe/parameters/remove; } \ - 2>/dev/null - fi - done - ) + rdma link | + sed -n 's,^link[[:blank:]]*\([^/]*\)/.* netdev .*,\1,p' | + while read -r i; do + echo "$i ..." + rdma link del "${i}" || echo "Failed to remove ${i}" + done if ! unload_module rdma_rxe 10; then echo "Unloading rdma_rxe failed" return 1 fi - ( - cd /sys/class/net && - for i in *_siw; do - [ -e "$i" ] || continue - rdma link del "${i}" || - echo "Failed to unbind the siw driver from ${i%_siw}" - done - ) if ! unload_module siw 10; then echo "Unloading siw failed" return 1
tests/srp/rc +1 -8
@@ -142,14 +142,7 @@ do_ib_cm_login() { } rdma_dev_to_net_dev() { - local b d rdma_dev=$1 - - b=/sys/class/infiniband/$rdma_dev/parent - if [ -e "$b" ]; then - echo "$(<"$b")" - else - echo "${rdma_dev%_siw}" - fi + rdma link show "$1/1" | sed 's/.* netdev //;s/[[:blank:]]*$//' } # Tell the SRP initiator to log in to an SRP target using the RDMA/CM.
Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent
Quoted 5 lines · Bart Van Assche
The approach of verifying whether or not an RDMA interface is associated with the
rdma_rxeinterface by looking up its parent device is deprecated and will be removed soon from the Linux kernel. Hence this patch that uses the rdma link command instead.
Quoted 7 lines
Cc Jason Gunthorpe <[email protected]>Cc Yi Zhang <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/srp/rc | 9 +-- 2 files changed, 32 insertions(+), 88 deletions(-)
I think we need to add _have_program rdma checks to srp and nvmeof-mp, right? The first two patches look fine, I'll merge those.
Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent
Quoted 14 lines · Omar Sandoval
Quoted 5 lines · Bart Van Assche
The approach of verifying whether or not an RDMA interface is associated with the
rdma_rxeinterface by looking up its parent device is deprecated and will be removed soon from the Linux kernel. Hence this patch that uses the rdma link command instead.Quoted 7 lines
Cc Jason Gunthorpe <[email protected]>Cc Yi Zhang <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/srp/rc | 9 +-- 2 files changed, 32 insertions(+), 88 deletions(-)
Quoted 2 lines
I think we need to add
_have_programrdma checks to srp and nvmeof-mp, right? The first two patches look fine, I'll merge those.
Sure, I will add a _have_program rdma check and repost this patch.
Thanks,
Bart.
Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent
Hello Bart
Currently blktests was broken on latest kernel, could you resend this updated patch. :)
Thanks
Yi
Quoted 23 lines · Bart Van Assche
Quoted 13 lines · Omar Sandoval
Quoted 5 lines · Bart Van Assche
The approach of verifying whether or not an RDMA interface is associated with the
rdma_rxeinterface by looking up its parent device is deprecated and will be removed soon from the Linux kernel. Hence this patch that uses the rdma link command instead.Quoted 7 lines
Cc Jason Gunthorpe <[email protected]>Cc Yi Zhang <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/srp/rc | 9 +-- 2 files changed, 32 insertions(+), 88 deletions(-)Quoted 2 lines
I think we need to add
_have_programrdma checks to srp and nvmeof-mp, right? The first two patches look fine, I'll merge those.Sure, I will add a
_have_programrdma check and repost this patch.Thanks,
Bart.
Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent
Quoted 2 lines · Yi Zhang
Currently blktests was broken on latest kernel, could you resend this updated patch. :)
Thanks for the reminder. A new version of this patch has just been posted.
Bart.