Add scsi-stress-remove to blktests

This test exposed a race condition with shutting down a request_queue and the new blkg association. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE when trying to reference it. So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran.

Signed-off-by Dennis Zhou <[email protected]>

Cc Ming Lei <[email protected]>

---
 tests/block/022     | 96 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/022.out |  2 +
 2 files changed, 98 insertions(+)
 create mode 100755 tests/block/022
 create mode 100644 tests/block/022.out

tests/block/022 new file +49 -0

new file mode 100755
index 0000000..45bfff7
@@ -0,0 +1,96 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for patch "blkcg: handle dying request_queue when associating
+# a blkg"
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	scsi_debug_path="/sys/bus/pseudo/drivers/scsi_debug"
+	count=21
+
+	runtime=12
+	nr_fio_jobs=8
+	scsi_dbg_ndelay=10000
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	#figure out the CAN_QUEUE
+	can_queue=$(((count + 1) * (count / 2) / 2))
+
+	rmmod scsi_debug > /dev/null 2>&1
+	modprobe scsi_debug virtual_gb=128 max_luns=$count \
+		ndelay=$scsi_dbg_ndelay max_queue=$can_queue
+
+	# figure out scsi_debug disks
+	hosts=$(ls -d /sys/bus/pseudo/drivers/scsi_debug/adapter0/host*)
+	hostname=$(basename "$hosts")
+	host=$(echo "$hostname" | grep -o -E '[0-9]+')
+
+	sdisks=$(ls -d $scsi_debug_path/adapter*/"$hostname"/target*/*/block/*)
+	disks=""
+	for sd in $sdisks; do

Quoted 71 lines · Dennis Zhou

This test exposed a race condition with shutting down a request_queue and the new blkg association. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE when trying to reference it. So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran.

Signed-off-by Dennis Zhou <[email protected]>

Cc Ming Lei <[email protected]>

---
 tests/block/022     | 96 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/022.out |  2 +
 2 files changed, 98 insertions(+)
 create mode 100755 tests/block/022
 create mode 100644 tests/block/022.out

tests/block/022 new file +49 -0

new file mode 100755
index 0000000..45bfff7
@@ -0,0 +1,96 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for patch "blkcg: handle dying request_queue when associating
+# a blkg"
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	scsi_debug_path="/sys/bus/pseudo/drivers/scsi_debug"
+	count=21
+
+	runtime=12
+	nr_fio_jobs=8
+	scsi_dbg_ndelay=10000
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	#figure out the CAN_QUEUE
+	can_queue=$(((count + 1) * (count / 2) / 2))
+
+	rmmod scsi_debug > /dev/null 2>&1
+	modprobe scsi_debug virtual_gb=128 max_luns=$count \
+		ndelay=$scsi_dbg_ndelay max_queue=$can_queue
+
+	# figure out scsi_debug disks
+	hosts=$(ls -d /sys/bus/pseudo/drivers/scsi_debug/adapter0/host*)
+	hostname=$(basename "$hosts")
+	host=$(echo "$hostname" | grep -o -E '[0-9]+')
+
+	sdisks=$(ls -d $scsi_debug_path/adapter*/"$hostname"/target*/*/block/*)
+	disks=""
+	for sd in $sdisks; do

tests/block/022 line 59

Quoted 10 lines

tests/block/022 new file +10 -0

+		disks+="/dev/"$(basename "$sd")
+		disks+=" "
+	done
+
+	use_mq=$(cat /sys/module/scsi_mod/parameters/use_blk_mq)
+	if [[ $use_mq = "Y" ]]; then
+		scheds=("none" "mq-deadline" "kyber")
+	else
+		scheds=("noop" "deadline" "cfq")
+	fi

You may use the following to figure out all supported io schedulers, especially we have removed all legacy io schedulers.

IOSCHEDS=sed 's/[][]//g' $Q_PATH/scheduler

Thanks,
Ming

This test exposed a race condition with shutting down a request_queue and the new blkg association. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE when trying to reference it. So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran.

Signed-off-by Dennis Zhou <[email protected]>

Cc Ming Lei <[email protected]>

---
v2:
- Change scheduler retrieving logic based on Ming's comment

 tests/block/022     | 90 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/022.out |  2 +
 2 files changed, 92 insertions(+)
 create mode 100755 tests/block/022
 create mode 100644 tests/block/022.out

tests/block/022 new file +90 -0

new file mode 100755
index 0000000..84336e0
@@ -0,0 +1,90 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for patch "blkcg: handle dying request_queue when associating
+# a blkg"
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	scsi_debug_path="/sys/bus/pseudo/drivers/scsi_debug"
+	count=21
+
+	runtime=12
+	nr_fio_jobs=8
+	scsi_dbg_ndelay=10000
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	#figure out the CAN_QUEUE
+	can_queue=$(((count + 1) * (count / 2) / 2))
+
+	rmmod scsi_debug > /dev/null 2>&1
+	modprobe scsi_debug virtual_gb=128 max_luns=$count \
+		ndelay=$scsi_dbg_ndelay max_queue=$can_queue
+
+	# figure out scsi_debug disks
+	hosts=$(ls -d /sys/bus/pseudo/drivers/scsi_debug/adapter0/host*)
+	hostname=$(basename "$hosts")
+	host=$(echo "$hostname" | grep -o -E '[0-9]+')
+
+	sdisks=$(ls -d $scsi_debug_path/adapter*/"$hostname"/target*/*/block/*)
+	disks=""
+	for sd in $sdisks; do
+		disks+="/dev/"$(basename "$sd")
+		disks+=" "
+	done
+
+	fio_jobs=""
+	cnt=0
+	for sd in $disks; do
+		cnt=$((cnt+1))
+		fio_jobs=$fio_jobs" --name=job1 --filename=$sd: "
+		dev_name=$(basename "$sd")
+		q_path=/sys/block/$dev_name/queue
+
+		scheds=($(sed 's/[][]//g' "$q_path/scheduler"))
+		sched_idx=$((cnt % ${#scheds[@]}))
+		echo "${scheds[$sched_idx]}" > "$q_path/scheduler"
+		echo $cnt > "$q_path/../device/queue_depth"
+	done
+
+	fio --rw=randread --size=128G --direct=1 --ioengine=libaio \
+		--iodepth=2048 --numjobs=$nr_fio_jobs --bs=4k \
+		--group_reporting=1 --group_reporting=1 --runtime=$runtime \
+		--loops=10000 "$fio_jobs" > "$FULL" 2>&1 &
+
+	sleep 7
+	for sd in $disks; do
+		dev_name=$(basename "$sd")
+		dpath=/sys/block/$dev_name/device
+		[ -f "$dpath/delete" ] && echo 1 > "$dpath/delete"
+	done
+
+	wait
+}
+
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	scsi_debug_stress_remove
+
+	echo "Test complete"
+}

tests/block/022.out new file +2 -0

new file mode 100644
index 0000000..14d43cb
@@ -0,0 +1,2 @@
+Running block/022
+Test complete

Signature · 1 line

2.17.1

This is a prep patch for a new test that will race blkg association and request_queue cleanup. As blkg association is a underlying cgroup io controller feature, we need the ability to check if the controller is available.

Signed-off-by Dennis Zhou <[email protected]>

---
 common/cgroup | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

common/cgroup +4 -5

index d445093..3481458 100644
@@ -37,19 +37,27 @@ _have_cgroup2()
 	return 0
 }
 
-_have_cgroup2_controller_file()
+_have_cgroup2_controller()
 {
-	_have_cgroup2 || return 1
-
 	local controller="$1"
-	local file="$2"
-	local dir
+
+	_have_cgroup2 || return 1
 
 	dir="$(_cgroup2_base_dir)"
+
 	if ! grep -q "$controller" "$dir/cgroup.controllers"; then
 		SKIP_REASON="no support for $controller cgroup controller; if it is enabled, you may need to boot with cgroup_no_v1=$controller"
 		return 1

common/cgroup +0 -0

 	mkdir "$dir/blktests"
 	echo "+$controller" > "$dir/cgroup.subtree_control"

Signature · 1 line

2.17.1

This test exposed a race condition with shutting down a request_queue and the new blkg association. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE when trying to reference it. So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran.

Signed-off-by Dennis Zhou <[email protected]>

Cc Ming Lei <[email protected]>

---
v2:
- Change scheduler retrieving logic based on Ming's comment

 tests/block/022     | 90 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/022.out |  2 +
 2 files changed, 92 insertions(+)
 create mode 100755 tests/block/022
 create mode 100644 tests/block/022.out

tests/block/022 new file +42 -0

new file mode 100755
index 0000000..84336e0
@@ -0,0 +1,90 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for patch "blkcg: handle dying request_queue when associating
+# a blkg"
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	scsi_debug_path="/sys/bus/pseudo/drivers/scsi_debug"
+	count=21
+
+	runtime=12
+	nr_fio_jobs=8
+	scsi_dbg_ndelay=10000
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	#figure out the CAN_QUEUE
+	can_queue=$(((count + 1) * (count / 2) / 2))
+
+	rmmod scsi_debug > /dev/null 2>&1
+	modprobe scsi_debug virtual_gb=128 max_luns=$count \
+		ndelay=$scsi_dbg_ndelay max_queue=$can_queue
+
+	# figure out scsi_debug disks

Quoted 74 lines · Dennis Zhou

This test exposed a race condition with shutting down a request_queue and the new blkg association. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE when trying to reference it. So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran.

Signed-off-by Dennis Zhou <[email protected]>

Cc Ming Lei <[email protected]>

---
v2:
- Change scheduler retrieving logic based on Ming's comment

 tests/block/022     | 90 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/022.out |  2 +
 2 files changed, 92 insertions(+)
 create mode 100755 tests/block/022
 create mode 100644 tests/block/022.out

tests/block/022 new file +49 -0

new file mode 100755
index 0000000..84336e0
@@ -0,0 +1,90 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for patch "blkcg: handle dying request_queue when associating
+# a blkg"
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	scsi_debug_path="/sys/bus/pseudo/drivers/scsi_debug"
+	count=21
+
+	runtime=12
+	nr_fio_jobs=8
+	scsi_dbg_ndelay=10000
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	#figure out the CAN_QUEUE
+	can_queue=$(((count + 1) * (count / 2) / 2))
+
+	rmmod scsi_debug > /dev/null 2>&1
+	modprobe scsi_debug virtual_gb=128 max_luns=$count \
+		ndelay=$scsi_dbg_ndelay max_queue=$can_queue
+
+	# figure out scsi_debug disks
+	hosts=$(ls -d /sys/bus/pseudo/drivers/scsi_debug/adapter0/host*)
+	hostname=$(basename "$hosts")
+	host=$(echo "$hostname" | grep -o -E '[0-9]+')
+
+	sdisks=$(ls -d $scsi_debug_path/adapter*/"$hostname"/target*/*/block/*)
+	disks=""
+	for sd in $sdisks; do

tests/block/022 line 52

Quoted 3 lines

tests/block/022 new file +3 -0

+		disks+="/dev/"$(basename "$sd")
+		disks+=" "
+	done

blktests has _init_scsi_debug which does all of this for you. And, block/001 is very similar to this test, just without the fio workload or changing schedulers. Could you please rework this to be based on block/001?