Skip to contentSkip to conversation

nvmeof-mp/srp/nvme-rdma misc fix and enhancement

PATCH Superseded Yi Zhang 2 participants 10 messages Nov 25, 2020 – Nov 26, 2020

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Yi Zhang

author
#0 lore raw

v2 cover nvmeof-mp/srp/nvme-rdma misc fix and enhancement

Hi

This patch series addressed some failures when I run nvmeof-mp/srp test and also add suport to use siw for nvme-rdma/nvmeof-mp/srp testing from cmdline, like this:

$ use_siw=1 nvme-trtype=rdma ./check nvme
$ use_siw=1 ./check nvmeof-mp
$ use_siw-1 ./check srp

V2:
Update the ib_srpt module path in patch 1, avoid to use ls Update the SKIP_REASON in patch 2
Introduce get_scheduler_list, fix nvmeof-mp/012 and srp/012 in patch 3 Typo fix in patch 4

Yi Zhang (5):
tests/srp/rc: update the ib_srpt module name tests/nvmeof-mp/rc: run nvmeof-mp tests if we set multipath=N nvmeof-mp/012, srp/012: fix the scheduler list common/rc: _have_iproute2 fix for "ip -V" change common/multipath-over-rdma: allow to set use_siw

 common/multipath-over-rdma | 14 +++++++++++++-
 common/rc                  |  2 +-
 tests/nvmeof-mp/012        | 10 ++++++----
 tests/nvmeof-mp/rc         |  8 +++++---
 tests/srp/012              | 10 ++++++----
 tests/srp/rc               |  4 ++--
 6 files changed, 33 insertions(+), 15 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v2 1/5 tests/srp/rc: update the ib_srpt module name

Fix the ib_srpt module insmod failure as the module in some distros are end with .xz, like bellow on fedora:
/lib/modules/$(uname -r)/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko.xz

---
 tests/srp/rc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/srp/rc +2 -2
index 7fc094b..1f665a2 100755
@@ -465,7 +465,7 @@ configure_target_ports() {
 
 # Load LIO and configure the SRP target driver and LUNs.
 start_lio_srpt() {
-	local b d gid i ini_ids=() opts p target_ids=() vdev
+	local b d gid i ini_ids=() opts p target_ids=()
 
 	for gid in $(all_primary_gids); do
 		if [ "${gid#fe8}" != "$gid" ]; then
@@ -500,7 +500,7 @@ start_lio_srpt() {
 	if modinfo ib_srpt | grep -q '^parm:[[:blank:]]*rdma_cm_port:'; then
 		opts+=("rdma_cm_port=${srp_rdma_cm_port}")
 	fi
-	insmod "/lib/modules/$(uname -r)/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko" "${opts[@]}" || return $?
+	insmod "/lib/modules/$(uname -r)/kernel/drivers/infiniband/ulp/srpt/ib_srpt."* "${opts[@]}" || return $?
 	i=0
 	for r in "${vdev_path[@]}"; do
 		if [ -b "$(readlink -f "$r")" ]; then
Signature · 1 line
2.21.0
Reviewed-by on v2 1/5
#2 lore raw

Re: [PATCH V2 blktests 1/5] tests/srp/rc: update the ib_srpt module name

Quoted 3 lines · Yi Zhang

Fix the ib_srpt module insmod failure as the module in some distros are end with .xz, like bellow on fedora:
/lib/modules/$(uname -r)/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko.xz

author
#3 lore raw

PATCH v2 2/5 tests/nvmeof-mp/rc: run nvmeof-mp tests if we set multipath=N

In reply to Yi Zhang

To enable it, just do bellow step before we run it:

$ echo "options nvme_core multipath=N" >/etc/modprobe.d/nvme.conf
---
 tests/nvmeof-mp/rc | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tests/nvmeof-mp/rc +5 -3
index d7a7c87..fc06856 100755
@@ -16,9 +16,11 @@ group_requires() {
 
 	# Since the nvmeof-mp tests are based on the dm-mpath driver, these
 	# tests are incompatible with the NVME_MULTIPATH kernel configuration
-	# option.
-	if _have_kernel_option NVME_MULTIPATH; then
-		SKIP_REASON="CONFIG_NVME_MULTIPATH has been set in .config"
+	# option with multipathing enabled in the nvme_core kernel module.
+	if _have_kernel_option NVME_MULTIPATH && \
+		_have_module_param_value nvme_core multipath Y; then
+		SKIP_REASON="CONFIG_NVME_MULTIPATH has been set in .config \
+	and multipathing has been enabled in the nvme_core kernel module"
 		return
 	fi
 
Signature · 1 line
2.21.0
Reviewed-by on v2 2/5
#4 lore raw

Re: [PATCH V2 blktests 2/5] tests/nvmeof-mp/rc: run nvmeof-mp tests if we set multipath=N

Quoted 2 lines · Yi Zhang

To enable it, just do bellow step before we run it:

$ echo "options nvme_core multipath=N" >/etc/modprobe.d/nvme.conf
author
#5 lore raw

PATCH v2 3/5 nvmeof-mp/012, srp/012: fix the scheduler list

In reply to Yi Zhang

There is no cfq scheduler and new added kyber scheduler in lastest kernel, introduce get_scheduler_list and fix nvmeof-mp/012 srp/012

To reproduce it:

$ ./check nvmeof-mp/012
nvmeof-mp/012 (dm-mpath on top of multiple I/O schedulers)   [passed]
    runtime  5.922s  ...  8.804s
$ cat results/nodev/nvmeof-mp/012.full  | grep -n "Changing scheduler"
31:Changing scheduler of dm-3 from mq-deadline kyber [bfq] none into cfq failed
47:Changing scheduler of dm-3 from mq-deadline kyber [bfq] none into cfq failed
---
 common/multipath-over-rdma | 12 ++++++++++++
 tests/nvmeof-mp/012        | 10 ++++++----
 tests/srp/012              | 10 ++++++----
 3 files changed, 24 insertions(+), 8 deletions(-)
common/multipath-over-rdma +12 -0
index ebc5939..1bbefb7 100644
@@ -326,6 +326,18 @@ set_scheduler() {
 	fi
 }
 
+# Get block dev scheduler list
+get_scheduler_list() {
+	local b=$1 p
+	p=/sys/block/"$b"/queue/scheduler
+	if [ -e "$p" ]; then
+		scheds=$(sed 's/[][]//g' /sys/block/"$b"/queue/scheduler)
+		echo "$scheds"
+	else
+		return 1
+	fi
+}
+
 # Get a /dev/... path that points at dm device number $1. Set its I/O scheduler
 # to $2 and its timeout to $3. The shell script that includes this file must
 # define a function get_bdev_path() that translates device number $1 into a
tests/nvmeof-mp/012 +6 -4
index ae6a92c..8b2e918 100755
@@ -8,7 +8,7 @@ DESCRIPTION="dm-mpath on top of multiple I/O schedulers"
 QUICK=1
 
 test_io_schedulers() {
-	local dev m
+	local dev m dm scheds
 
 	# Load all I/O scheduler kernel modules
 	for m in "/lib/modules/$(uname -r)/kernel/block/"*.ko; do
@@ -17,15 +17,17 @@ test_io_schedulers() {
 	for mq in y n; do
 		use_blk_mq ${mq} || return $?
 		dev=$(get_bdev 0) || return $?
-		for sched in noop deadline bfq cfq; do
-			set_scheduler "$(basename "$(readlink -f "${dev}")")" $sched \
+		dm=$(basename "$(readlink -f "${dev}")") || return $?
+		scheds="$(get_scheduler_list "$dm")" || return $?
+		for sched in $scheds; do
+			set_scheduler "$dm" "$sched" \
 				      >>"$FULL" 2>&1 || continue
 			echo "I/O scheduler: $sched; use_blk_mq: $mq" >>"$FULL"
 			run_fio --verify=md5 --rw=randwrite --bs=4K --size=64K \
 				--ioengine=libaio --iodepth=64 \
 				--iodepth_batch=32 --group_reporting --sync=1 \
 				--direct=1 --filename="$dev" \
-				--name=${sched} --thread --numjobs=1 \
+				--name="${sched}" --thread --numjobs=1 \
 				--output="${RESULTS_DIR}/nvmeof-mp/012-${sched}-${mq}.txt" \
 				>>"$FULL" ||
 				return $?
tests/srp/012 +6 -4
index 20c6daf..ffd56ef 100755
@@ -8,7 +8,7 @@ DESCRIPTION="dm-mpath on top of multiple I/O schedulers"
 QUICK=1
 
 test_io_schedulers() {
-	local dev m
+	local dev m dm scheds
 
 	# Load all I/O scheduler kernel modules
 	for m in "/lib/modules/$(uname -r)/kernel/block/"*.ko; do
@@ -18,15 +18,17 @@ test_io_schedulers() {
 		[ $mq = n ] && ! _have_legacy_dm && continue
 		use_blk_mq ${mq} ${mq} || return $?
 		dev=$(get_bdev 0) || return $?
-		for sched in noop deadline bfq cfq; do
-			set_scheduler "$(basename "$(readlink -f "${dev}")")" $sched \
+		dm=$(basename "$(readlink -f "${dev}")") || return $?
+		scheds="$(get_scheduler_list "$dm")" || return $?
+		for sched in $scheds; do
+			set_scheduler "$dm" "$sched" \
 				      >>"$FULL" 2>&1 || continue
 			echo "I/O scheduler: $sched; use_blk_mq: $mq" >>"$FULL"
 			run_fio --verify=md5 --rw=randwrite --bs=4K --size=64K \
 				--ioengine=libaio --iodepth=64 \
 				--iodepth_batch=32 --group_reporting --sync=1 \
 				--direct=1 --filename="$dev" \
-				--name=${sched} --thread --numjobs=1 \
+				--name="${sched}" --thread --numjobs=1 \
 				--output="${RESULTS_DIR}/srp/012-${sched}-${mq}.txt" \
 				>>"$FULL" ||
 				return $?
Signature · 1 line
2.21.0
#6 lore raw

Re: [PATCH V2 blktests 3/5] nvmeof-mp/012, srp/012: fix the scheduler list

common/multipath-over-rdma set_scheduler() { · line 339

Quoted 11 lines · Yi Zhang
common/multipath-over-rdma +11 -0
+# Get block dev scheduler list
+get_scheduler_list() {
+	local b=$1 p
+	p=/sys/block/"$b"/queue/scheduler
+	if [ -e "$p" ]; then
+		scheds=$(sed 's/[][]//g' /sys/block/"$b"/queue/scheduler)
+		echo "$scheds"
+	else
+		return 1
+	fi
+}

Can the echo statement and the 'scheds' assignment be left out? This is what I have in mind:

	sed 's/[][]//g' /sys/block/"$b"/queue/scheduler

Otherwise this patch looks good to me.

Thanks,

Bart.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#7 lore raw

Re: [PATCH V2 blktests 3/5] nvmeof-mp/012, srp/012: fix the scheduler list

Quoted 14 lines · Bart Van Assche

common/multipath-over-rdma set_scheduler() { · line 339

Quoted 11 lines · Yi Zhang
common/multipath-over-rdma +11 -0
+# Get block dev scheduler list
+get_scheduler_list() {
+	local b=$1 p
+	p=/sys/block/"$b"/queue/scheduler
+	if [ -e "$p" ]; then
+		scheds=$(sed 's/[][]//g' /sys/block/"$b"/queue/scheduler)
+		echo "$scheds"
+	else
+		return 1
+	fi
+}

Can the echo statement and the 'scheds' assignment be left out? This is what I have in mind:

Sure, will update with V3.

Quoted 8 lines
	sed 's/[][]//g' /sys/block/"$b"/queue/scheduler

Otherwise this patch looks good to me.

Thanks,

Bart.

author
#8 lore raw

PATCH v2 4/5 common/rc: _have_iproute2 fix for "ip -V" change

In reply to Yi Zhang

With below commit, the version will be updated base on the tag fbef6555 replace SNAPSHOT with auto-generated version string

To reproduce it:

$ ./check srp/015
common/rc: line 98: [: ip utility, iproute2-5.9.0: integer expression expected
---
 common/rc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
common/rc +1 -1
index 0d7a3cd..37eb873 100644
@@ -95,7 +95,7 @@ _have_iproute2() {
 		SKIP_REASON="ip utility not found"
 		return 1
 	fi
-	if [ "$snapshot" -lt "$1" ]; then
+	if [[ "$snapshot" =~ ^[0-9]+$ && "$snapshot" -lt "$1" ]]; then
 		SKIP_REASON="ip utility too old"
 		return 1
 	fi
Signature · 1 line
2.21.0
author
#9 lore raw

PATCH v2 5/5 common/multipath-over-rdma: allow to set use_siw

In reply to Yi Zhang

With this change, we can change to use siw for nvme-rdma/nvmeof-mp/srp testing from cmdline:

$ use_siw=1 nvme-trtype=rdma ./check nvme/
$ use_siw=1 ./check nvmeof-mp/
$ use_siw=1 ./check srp/
---
 common/multipath-over-rdma | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
common/multipath-over-rdma +1 -1
index 1bbefb7..d156a12 100644
@@ -12,7 +12,7 @@ filesystem_type=ext4
 fio_aux_path=/tmp/fio-state-files
 memtotal=$(sed -n 's/^MemTotal:[[:blank:]]*\([0-9]*\)[[:blank:]]*kB$/\1/p' /proc/meminfo)
 max_ramdisk_size=$((1<<25))
-use_siw=
+use_siw=${use_siw:-""}
 ramdisk_size=$((memtotal*(1024/16)))  # in bytes
 if [ $ramdisk_size -gt $max_ramdisk_size ]; then
 	ramdisk_size=$max_ramdisk_size
Signature · 1 line
2.21.0