Skip to contentSkip to conversation

Add a test that triggers the blk_mq_realloc_hw_ctxs() error path

PATCH Superseded Bart Van Assche 2 participants 14 messages Mar 20, 2020 – Mar 25, 2020 Block layer

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add a test that triggers the blk_mq_realloc_hw_ctxs() error path

Hi Omar,

Please consider this patch series for the upstream blktests repository.

Thanks,

Bart.

Changes compared to v2:
- Addressed Omar's review comments.

Changes between v1 and v2:
- Added three patches that refactor null_blk loading, unloading and configuration (Chaitanya).

Bart Van Assche (4):
Make _exit_null_blk remove all null_blk device instances Use _{init,exit}_null_blk instead of open-coding these functions Introduce the function _configure_null_blk()
Add a test that triggers the blk_mq_realloc_hw_ctxs() error path

 common/multipath-over-rdma | 29 ++++++++------------------
 common/null_blk            | 21 ++++++++++++++++++-
 tests/block/022            |  3 ---
 tests/block/029            | 17 ++-------------
 tests/block/030            | 42 ++++++++++++++++++++++++++++++++++++++
 tests/block/030.out        |  1 +
 tests/nvmeof-mp/rc         |  2 +-
 7 files changed, 74 insertions(+), 41 deletions(-)
 create mode 100755 tests/block/030
 create mode 100644 tests/block/030.out
#1 lore raw

PATCH v3 1/4 Make _exit_null_blk remove all null_blk device instances

Instead of making every test remove null_blk device instances before calling _exit_null_blk(), move the null_blk device instance removal code into _exit_null_blk().

---
 common/null_blk | 7 ++++++-
 tests/block/022 | 3 ---
 tests/block/029 | 1 -
 3 files changed, 6 insertions(+), 5 deletions(-)
common/null_blk +6 -1
index 2e300c20bbc7..a4140e365955 100644
@@ -8,11 +8,15 @@ _have_null_blk() {
 	_have_modules null_blk
 }
 
-_init_null_blk() {
+_remove_null_blk_devices() {
 	if [[ -d /sys/kernel/config/nullb ]]; then
 		find /sys/kernel/config/nullb -mindepth 1 -maxdepth 1 \
 		     -type d -delete
 	fi
+}
+
+_init_null_blk() {
+	_remove_null_blk_devices
 
 	local zoned=""
 	if (( RUN_FOR_ZONED )); then zoned="zoned=1"; fi
@@ -26,6 +30,7 @@ _init_null_blk() {
 }
 
 _exit_null_blk() {
+	_remove_null_blk_devices
 	udevadm settle
 	modprobe -r null_blk
 }
tests/block/022 +0 -3
index 1404aacef295..b2c53e266d81 100755
@@ -50,9 +50,6 @@ test() {
 		wait $pid1
 	} 2>/dev/null
 
-	rmdir /sys/kernel/config/nullb/1
-	rmdir /sys/kernel/config/nullb/0
-
 	_exit_null_blk
 	echo "Test complete"
 }
tests/block/029 +0 -1
index d298bac8db5c..0d521edb0cf6 100755
@@ -58,7 +58,6 @@ test() {
 	else
 		echo "Skipping test because $sq cannot be modified" >>"$FULL"
 	fi
-	rmdir /sys/kernel/config/nullb/nullb0
 	_exit_null_blk
 	echo Passed
 }
Reviewed-by on v3 1/4
#2 lore raw

Re: [PATCH blktests v3 1/4] Make _exit_null_blk remove all null_blk device instances

Quoted 6 lines · Bart Van Assche

Instead of making every test remove null_blk device instances before calling _exit_null_blk(), move the null_blk device instance removal code into _exit_null_blk().

#3 lore raw

PATCH v3 2/4 Use _{init,exit}_null_blk instead of open-coding these functions

In reply to Bart Van Assche

This patch reduces code duplication.

---
 common/multipath-over-rdma | 12 +++---------
 tests/nvmeof-mp/rc         |  2 +-
 2 files changed, 4 insertions(+), 10 deletions(-)
common/multipath-over-rdma +3 -9
index 40efc4b3aa2e..a56e7a8269db 100644
@@ -5,6 +5,7 @@
 # Functions and global variables used by both the srp and nvmeof-mp tests.
 
 . common/shellcheck
+. common/null_blk
 
 debug=
 filesystem_type=ext4
@@ -634,13 +635,6 @@ configure_null_blk() {
 	ls -l /dev/nullb* &>>"$FULL"
 }
 
-unload_null_blk() {
-	local d
-
-	for d in /sys/kernel/config/nullb/*; do [ -d "$d" ] && rmdir "$d"; done
-	unload_module null_blk
-}
-
 setup_rdma() {
 	start_soft_rdma
 	(
@@ -662,7 +656,7 @@ teardown_uncond() {
 	rm -f /etc/multipath.conf
 	stop_target
 	stop_soft_rdma
-	unload_null_blk
+	_exit_null_blk
 }
 
 teardown() {
@@ -698,7 +692,7 @@ setup_test() {
 		[ -e "/sys/module/$m" ] || modprobe "$m" || return $?
 	done
 
-	modprobe null_blk nr_devices=0 || return $?
+	_init_null_blk nr_devices=0 || return $?
 
 	configure_null_blk || return $?
 
tests/nvmeof-mp/rc +1 -1
index 1fd631445921..136163bc73ad 100755
@@ -278,7 +278,7 @@ stop_nvme_target() {
 	)
 	unload_module nvmet_rdma &&
 		unload_module nvmet &&
-		unload_null_blk
+		_exit_null_blk
 }
 
 start_target() {
#5 lore raw

PATCH v3 3/4 Introduce the function _configure_null_blk()

In reply to Bart Van Assche

Introduce a function for creating a null_blk device instance through configfs.

---
 common/multipath-over-rdma | 17 +++++------------
 common/null_blk            | 14 ++++++++++++++
 tests/block/029            | 16 ++--------------
 3 files changed, 21 insertions(+), 26 deletions(-)
common/multipath-over-rdma +5 -12
index a56e7a8269db..7e610a0ccbbb 100644
@@ -620,18 +620,11 @@ run_fio() {
 configure_null_blk() {
 	local i
 
-	(
-		cd /sys/kernel/config/nullb || return $?
-		for i in nullb0 nullb1; do (
-			{ mkdir -p $i &&
-				  cd $i &&
-				  echo 0 > completion_nsec &&
-				  echo 512 > blocksize &&
-				  echo $((ramdisk_size>>20)) > size &&
-				  echo 1 > memory_backed &&
-				  echo 1 > power; } || exit $?
-		) done
-	)
+	for i in nullb0 nullb1; do
+		_configure_null_blk nullb$i completion_nsec=0 blocksize=512 \
+				    size=$((ramdisk_size>>20)) memory_backed=1 \
+				    power=1 || return $?
+	done
 	ls -l /dev/nullb* &>>"$FULL"
 }
 
common/null_blk +14 -0
index a4140e365955..08008fca8eae 100644
@@ -29,6 +29,20 @@ _init_null_blk() {
 	return 0
 }
 
+# Configure one null_blk instance with name $1 and parameters $2..${$#}.
+_configure_null_blk() {
+	local nullb=/sys/kernel/config/nullb/$1 param val
+
+	shift
+	mkdir "$nullb" || return $?
+	while [ -n "$1" ]; do
+		param="${1%%=*}"
+		val="${1#*=}"
+		shift
+		echo "$val" > "$nullb/$param" || return $?
+	done
+}
+
 _exit_null_blk() {
 	_remove_null_blk_devices
 	udevadm settle
tests/block/029 +2 -14
index 0d521edb0cf6..dbb582eab473 100755
@@ -14,19 +14,6 @@ requires() {
 	_have_null_blk
 }
 
-# Configure one null_blk instance.
-configure_null_blk() {
-	local nullb0="/sys/kernel/config/nullb/nullb0"
-
-	mkdir "$nullb0" &&
-	echo 0 > "$nullb0/completion_nsec" &&
-	echo 512 > "$nullb0/blocksize" &&
-	echo 16 > "$nullb0/size" &&
-	echo 1 > "$nullb0/memory_backed" &&
-	echo 1 > "$nullb0/power" &&
-	ls -l /dev/nullb* &>>"$FULL"
-}
-
 modify_nr_hw_queues() {
 	local deadline num_cpus
 
@@ -45,7 +32,8 @@ test() {
 
 	: "${TIMEOUT:=30}"
 	_init_null_blk nr_devices=0 queue_mode=2 &&
-	configure_null_blk
+	_configure_null_blk nullb0 completion_nsec=0 blocksize=512 \
+			    size=16 memory_backed=1 power=1 &&
 	if { echo 1 >$sq; } 2>/dev/null; then
 		modify_nr_hw_queues &
 		fio --rw=randwrite --bs=4K --loops=$((10**6)) \
#6 lore raw

Re: [PATCH blktests v3 3/4] Introduce the function _configure_null_blk()

Hi Bart,

Quoted 24 lines · Bart Van Assche

Introduce a function for creating a null_blk device instance through configfs.

---
 common/multipath-over-rdma | 17 +++++------------
 common/null_blk            | 14 ++++++++++++++
 tests/block/029            | 16 ++--------------
 3 files changed, 21 insertions(+), 26 deletions(-)
common/multipath-over-rdma +0 -4
index a56e7a8269db..7e610a0ccbbb 100644
@@ -620,18 +620,11 @@ run_fio() {
 configure_null_blk() {
 	local i
 
-	(
-		cd /sys/kernel/config/nullb || return $?
-		for i in nullb0 nullb1; do (
-			{ mkdir -p $i &&

common/multipath-over-rdma run_fio() { · line 624

Quoted 10 lines
common/multipath-over-rdma +2 -8
-				  cd $i &&
-				  echo 0 > completion_nsec &&
-				  echo 512 > blocksize &&
-				  echo $((ramdisk_size>>20)) > size &&
-				  echo 1 > memory_backed &&
-				  echo 1 > power; } || exit $?
-		) done
-	)
+	for i in nullb0 nullb1; do
+		_configure_null_blk nullb$i completion_nsec=0 blocksize=512 \

I think this should be:

		_configure_null_blk $i completion_nsec=0 blocksize=512 \

Rest looks good, feel free do add my Reviewed-by when spinning a new version.

Thanks,
Daniel

#7 lore raw

Re: [PATCH blktests v3 3/4] Introduce the function _configure_null_blk()

Quoted 26 lines · Daniel Wagner
Quoted 12 lines · Bart Van Assche
common/multipath-over-rdma +0 -4
index a56e7a8269db..7e610a0ccbbb 100644
@@ -620,18 +620,11 @@ run_fio() {
 configure_null_blk() {
 	local i
 
-	(
-		cd /sys/kernel/config/nullb || return $?
-		for i in nullb0 nullb1; do (
-			{ mkdir -p $i &&

common/multipath-over-rdma run_fio() { · line 624

Quoted 10 lines
common/multipath-over-rdma +2 -8
-				  cd $i &&
-				  echo 0 > completion_nsec &&
-				  echo 512 > blocksize &&
-				  echo $((ramdisk_size>>20)) > size &&
-				  echo 1 > memory_backed &&
-				  echo 1 > power; } || exit $?
-		) done
-	)
+	for i in nullb0 nullb1; do
+		_configure_null_blk nullb$i completion_nsec=0 blocksize=512 \

I think this should be:

Quoted 1 line
		_configure_null_blk $i completion_nsec=0 blocksize=512 \

Good catch, will fix.

Thanks,

Bart.

#8 lore raw

PATCH v3 4/4 Add a test that triggers the blk_mq_realloc_hw_ctxs() error path

In reply to Bart Van Assche

Add a test that triggers the code touched by commit d0930bb8f46b ("blk-mq:
Fix a recently introduced regression in blk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit 596444e75705 ("null_blk: Add support for init_hctx() fault injection").

---
 tests/block/030     | 42 ++++++++++++++++++++++++++++++++++++++++++
 tests/block/030.out |  1 +
 2 files changed, 43 insertions(+)
 create mode 100755 tests/block/030
 create mode 100644 tests/block/030.out
tests/block/030 new file +42 -0
new file mode 100755
index 000000000000..4a17550ab8eb
@@ -0,0 +1,42 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright 2020 Google LLC
+#
+# Trigger the blk_mq_realloc_hw_ctxs() error path.
+
+. tests/block/rc
+. common/null_blk
+
+DESCRIPTION="trigger the blk_mq_realloc_hw_ctxs() error path"
+QUICK=1
+
+requires() {
+	_have_null_blk || return $?
+	_have_module_param null_blk init_hctx || return $?
+}
+
+test() {
+	local i sq=/sys/kernel/config/nullb/nullb0/submit_queues
+
+	: "${TIMEOUT:=30}"
+	if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then
+		echo "Loading null_blk failed"
+		return 1
+	fi
+	if ! _configure_null_blk nullb0 completion_nsec=0 blocksize=512 size=16\
+	     submit_queues="$(nproc)" memory_backed=1 power=1; then
+		echo "Configuring null_blk failed"
+		return 1
+	fi
+	if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then
+		for ((i=0;i<100;i++)); do
+			echo 1 >$sq
+			nproc >$sq
+		done
+	else
+		SKIP_REASON="Skipping test because $sq cannot be modified"
+	fi
+	rmdir /sys/kernel/config/nullb/nullb0
+	_exit_null_blk
+	echo Passed
+}
tests/block/030.out new file +1 -0
new file mode 100644
index 000000000000..863339fb8ced
@@ -0,0 +1 @@
+Passed
#9 lore raw

Hi Bart,

Quoted 5 lines · Bart Van Assche

Add a test that triggers the code touched by commit d0930bb8f46b ("blk-mq:
Fix a recently introduced regression in blk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit 596444e75705 ("null_blk: Add support for init_hctx() fault injection").

[...]

tests/block/030 line 22

Quoted 5 lines
tests/block/030 new file +5 -0
+test() {
+	local i sq=/sys/kernel/config/nullb/nullb0/submit_queues
+
+	: "${TIMEOUT:=30}"
+	if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then

Doesn't make the $(nproc) the test subtil depending on the execution environment?

Thanks,
Daniel

#10 lore raw
Quoted 9 lines · Daniel Wagner
Quoted 5 lines · Bart Van Assche

Add a test that triggers the code touched by commit d0930bb8f46b ("blk-mq:
Fix a recently introduced regression in blk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit 596444e75705 ("null_blk: Add support for init_hctx() fault injection").

[...]

Quoted 8 lines

tests/block/030 line 22

Quoted 5 lines
tests/block/030 new file +5 -0
+test() {
+	local i sq=/sys/kernel/config/nullb/nullb0/submit_queues
+
+	: "${TIMEOUT:=30}"
+	if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then

Doesn't make the $(nproc) the test subtil depending on the execution environment?

Hi Daniel,

The value $(nproc) has been chosen on purpose. The following code from the test script:

+ echo 1 >$sq
+ nproc >$sq

triggers (nproc + 1) calls to null_init_hctx(). So injecting a failure after (nproc) null_init_hctx() calls triggers the following pattern:
* The first blk_mq_realloc_hw_ctxs() call fails after (nproc - 1) null_init_hctx() calls.
* The second blk_mq_realloc_hw_ctxs() call fails after (nproc - 2) null_init_hctx() calls.
...
* The (nproc) th blk_mq_realloc_hw_ctxs() call fails after one null_init_hctx() call.
* The (nproc + 1) th blk_mq_realloc_hw_ctxs() call succeeds.

I'm not sure to trigger this behavior without using the $(nproc) value?

Thanks,

Bart.

#11 lore raw

Hi Bart,

Quoted 7 lines · Bart Van Assche
Quoted 6 lines · Daniel Wagner

tests/block/030 line 22

Quoted 5 lines · Bart Van Assche
tests/block/030 new file +5 -0
+test() {
+	local i sq=/sys/kernel/config/nullb/nullb0/submit_queues
+
+	: "${TIMEOUT:=30}"
+	if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then

From the kernel code:

	/* "<interval>,<probability>,<space>,<times>" */

Don't you need to set the times attribute to -1 in order to inject the everytime the interval is reached? If I understood it correctly, with times=0 no failure is injected.

BTW, I've had to change it to init_hctx=$(($(nproc)+1)) to pass the initial __configure_null_blk call before the first fail hits.

Quoted 19 lines
Quoted 2 lines

Doesn't make the $(nproc) the test subtil depending on the execution environment?

The value $(nproc) has been chosen on purpose. The following code from the test script:

tests/block/030 new file +2 -0
+			echo 1 >$sq
+			nproc >$sq

triggers (nproc + 1) calls to null_init_hctx().So injecting a failure after (nproc) null_init_hctx() calls triggers the following pattern:
* The first blk_mq_realloc_hw_ctxs() call fails after (nproc - 1) null_init_hctx() calls.
* The second blk_mq_realloc_hw_ctxs() call fails after (nproc - 2) null_init_hctx() calls.
...
* The (nproc) th blk_mq_realloc_hw_ctxs() call fails after one null_init_hctx() call.
* The (nproc + 1) th blk_mq_realloc_hw_ctxs() call succeeds.

Quoted 1 line

I'm not sure to trigger this behavior without using the $(nproc) value?

Okay, I get the idea how you want to test.

Is the dependency on nproc because null_blk expects submit_queue <= online cpus?

Though why the 100?

		for ((i=0;i<100;i++)); do
			echo 1 >$sq
			nproc >$sq
		done

And shouldn't be there a test for error?

Thanks,
Daniel

#12 lore raw
Quoted 13 lines · Daniel Wagner

Hi Bart,

Quoted 9 lines · Bart Van Assche
Quoted 6 lines · Daniel Wagner

tests/block/030 line 22

Quoted 5 lines · Bart Van Assche
tests/block/030 new file +5 -0
+test() {
+	local i sq=/sys/kernel/config/nullb/nullb0/submit_queues
+
+	: "${TIMEOUT:=30}"
+	if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then

From the kernel code:

Quoted 8 lines
	/* "<interval>,<probability>,<space>,<times>" */

Don't you need to set the times attribute to -1 in order to inject the everytime the interval is reached? If I understood it correctly, with times=0 no failure is injected.

BTW, I've had to change it to init_hctx=$(($(nproc)+1)) to pass the initial __configure_null_blk call before the first fail hits.

Hi Daniel,

I will make both changes in the init_hctx string. Not sure how this escaped from my attention.

Quoted 19 lines
Quoted 19 lines
Quoted 2 lines

Doesn't make the $(nproc) the test subtil depending on the execution environment?

The value $(nproc) has been chosen on purpose. The following code from the test script:

tests/block/030 new file +2 -0
+			echo 1 >$sq
+			nproc >$sq

triggers (nproc + 1) calls to null_init_hctx().So injecting a failure after (nproc) null_init_hctx() calls triggers the following pattern:
* The first blk_mq_realloc_hw_ctxs() call fails after (nproc - 1) null_init_hctx() calls.
* The second blk_mq_realloc_hw_ctxs() call fails after (nproc - 2) null_init_hctx() calls.
...
* The (nproc) th blk_mq_realloc_hw_ctxs() call fails after one null_init_hctx() call.
* The (nproc + 1) th blk_mq_realloc_hw_ctxs() call succeeds.

Quoted 6 lines
Quoted 1 line

I'm not sure to trigger this behavior without using the $(nproc) value?

Okay, I get the idea how you want to test.

Is the dependency on nproc because null_blk expects submit_queue <= online cpus?

That's correct. I want to test with the maximum number of submit queues allowed, hence the use of $(nproc).

Quoted 6 lines

Though why the 100?

		for ((i=0;i<100;i++)); do
			echo 1 >$sq
			nproc >$sq
		done

No particular reason other than "a significant number of iterations".

Quoted 1 line

And shouldn't be there a test for error?

All I want to test is the absence of kernel crashes. The blktests framework already inspects dmesg for the absence of kernel crashes. So I don't think that I have to check whether or not the quoted sysfs writes succeed.

Thanks,

Bart.

#13 lore raw

Hi Bart,

Quoted 5 lines
Quoted 2 lines

Is the dependency on nproc because null_blk expects submit_queue <= online cpus?

That's correct. I want to test with the maximum number of submit queues allowed, hence the use of $(nproc).

Ok, I'd probably add this info as comment or have it in the commit message. Just in case someone in the future wonders why nproc.

Quoted 10 lines
Quoted 6 lines

Though why the 100?

		for ((i=0;i<100;i++)); do
			echo 1 >$sq
			nproc >$sq
		done

No particular reason other than "a significant number of iterations".

Quoted 1 line

And shouldn't be there a test for error?

Ah ok. I was getting paranoid about numbers :)

Quoted 4 lines

All I want to test is the absence of kernel crashes. The blktests framework already inspects dmesg for the absence of kernel crashes. So I don't think that I have to check whether or not the quoted sysfs writes succeed.

Maybe adding this info to the commit message what this test tries to cover wouldn't hurt.

Thanks,
Daniel