Skip to contentSkip to conversation

Add scsi-stress-remove to blktests

PATCH In review Dennis Zhou 2 participants 4 messages Dec 20, 2018 Block layer

lore
Share & tools
lore
#0 lore raw

v2 cover Add scsi-stress-remove to blktests

Hi,

v2:
There was a minor typo in 0001 pointed out by Omar which I fixed here. 0002 was refactored to use common/scsi_debug. I also rebased onto the top of osandov#josef changing the test from block/022 -> block/027.

From v1:
Ming Lei's scsi-stress-remove test found a bug in blkg destruction [1] where bios being created when the request_queue was being cleaned up threw a NPE in blkg association. The fix is currently being discussed in [2]. To make this test more accessible, I've ported it to blktests with Ming Lei's copyright. I've tested this in my qemu instance and verified we do not see the NPE on a fixed kernel.

Ming, please let me know if you have any objections.

[1] https://lore.kernel.org/linux-block/CACVXFVO_QXipD3cmPvpLyBYSiEcWPN_ThQ=0pO9AwLqN-Lv93w@mail.gmail.com
[2] https://lore.kernel.org/lkml/[email protected]/

This patchset is ontop of osandov#josef 98db2e11c97f.

diffstats below:

Dennis Zhou (2):
blktests: split out cgroup2 controller and file check
blktests: add Ming Lei's scsi-stress-remove

 common/cgroup       | 18 +++++++----
 tests/block/027     | 73 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/027.out |  2 ++
 3 files changed, 88 insertions(+), 5 deletions(-)
 create mode 100755 tests/block/027
 create mode 100644 tests/block/027.out

Thanks,
Dennis

#1 lore raw

PATCH v2 1/2 blktests: split out cgroup2 controller and file check

This is a prep patch for a new test that will race blkg association and request_queue cleanup. As blkg association is a underlying cgroup io controller feature, we need the ability to check if the controller is available.

---
v2:
- fixed minor typo.

 common/cgroup | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)
common/cgroup +13 -5
index d445093..48e546f 100644
@@ -37,19 +37,27 @@ _have_cgroup2()
 	return 0
 }
 
-_have_cgroup2_controller_file()
+_have_cgroup2_controller()
 {
-	_have_cgroup2 || return 1
-
 	local controller="$1"
-	local file="$2"
-	local dir
+
+	_have_cgroup2 || return 1
 
 	dir="$(_cgroup2_base_dir)"
+
 	if ! grep -q "$controller" "$dir/cgroup.controllers"; then
 		SKIP_REASON="no support for $controller cgroup controller; if it is enabled, you may need to boot with cgroup_no_v1=$controller"
 		return 1
 	fi
+}
+
+_have_cgroup2_controller_file()
+{
+	local controller="$1"
+	local file="$2"
+	local dir
+
+	_have_cgroup2_controller "$controller" || return 1
 
 	mkdir "$dir/blktests"
 	echo "+$controller" > "$dir/cgroup.subtree_control"
Signature · 1 line
2.17.1
#2 lore raw

PATCH v2 2/2 blktests: add Ming Lei's scsi-stress-remove

In reply to Dennis Zhou

This test exposed a race condiiton when shutting down a request_queue with active IO against it and blkg association for the IOs [1]. The issue ended up being that while the request_queue will just start failing requests, blkg destruction sets the q->root_blkg to %NULL. This caused a NPE. This was fixed in [2].

So to help prevent this from happening again, integrate Ming's test into blktests so that it can more easily be ran. Here I've ported it to fit better into the blktests framework.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

---
v2:
- Used helpers in common/scsi_debug per Omar's suggestion.

 tests/block/027     | 73 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/027.out |  2 ++
 2 files changed, 75 insertions(+)
 create mode 100755 tests/block/027
 create mode 100644 tests/block/027.out
tests/block/027 new file +73 -0
new file mode 100755
index 0000000..d5e1463
@@ -0,0 +1,73 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for commit 0273ac349f08 ("blkcg: handle dying request_queue
+# when associating a blkg")
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+	_have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+	if ! _init_scsi_debug "$@"; then
+		return
+	fi
+
+	# set higher aio limit
+	echo 524288 > /proc/sys/fs/aio-max-nr
+
+	local dev fio_jobs scheds
+	local cnt=1
+	for dev in "${SCSI_DEBUG_DEVICES[@]}"; do
+		# create fio job
+		fio_jobs=$fio_jobs" --name=job1 --filename=/dev/$dev "
+
+		# set queue scheduler and queue_depth
+		queue_path=/sys/block/$dev/queue
+		scheds=($(sed 's/[][]//g' "$queue_path/scheduler"))
+		sched_idx=$((cnt % ${#scheds[@]}))
+		echo "${scheds[$sched_idx]}" > "$queue_path/scheduler"
+		echo $cnt > "$queue_path/../device/queue_depth"
+		cnt=$((cnt+1))
+	done
+
+	local num_jobs=4 runtime=21
+	fio --rw=randread --size=128G --direct=1 --ioengine=libaio \
+		--iodepth=2048 --numjobs=$num_jobs --bs=4k \
+		--group_reporting=1 --group_reporting=1 --runtime=$runtime \
+		--loops=10000 "$fio_jobs" > "$FULL" 2>&1 &
+
+	sleep 7
+	local device_path
+	for dev in "${SCSI_DEBUG_DEVICES[@]}"; do
+		# shutdown devices in progress
+		device_path=/sys/block/$dev/device
+		[ -f "$device_path/delete" ] && echo 1 > "$device_path/delete"
+	done
+
+	wait
+
+	_exit_scsi_debug
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	scsi_debug_stress_remove virtual_gb=128 max_luns=21 ndelay=10000 \
+		max_queue=110
+
+	echo "Test complete"
+}
tests/block/027.out new file +2 -0
new file mode 100644
index 0000000..b03498f
@@ -0,0 +1,2 @@
+Running block/027
+Test complete
Signature · 1 line
2.17.1
#3 lore raw
In reply to Dennis Zhou
Quoted 28 lines · Dennis Zhou

Hi,

v2:
There was a minor typo in 0001 pointed out by Omar which I fixed here. 0002 was refactored to use common/scsi_debug. I also rebased onto the top of osandov#josef changing the test from block/022 -> block/027.

From v1:
Ming Lei's scsi-stress-remove test found a bug in blkg destruction [1] where bios being created when the request_queue was being cleaned up threw a NPE in blkg association. The fix is currently being discussed in [2]. To make this test more accessible, I've ported it to blktests with Ming Lei's copyright. I've tested this in my qemu instance and verified we do not see the NPE on a fixed kernel.

Ming, please let me know if you have any objections.

[1] https://lore.kernel.org/linux-block/CACVXFVO_QXipD3cmPvpLyBYSiEcWPN_ThQ=0pO9AwLqN-Lv93w@mail.gmail.com
[2] https://lore.kernel.org/lkml/[email protected]/

This patchset is ontop of osandov#josef 98db2e11c97f.

diffstats below:

Dennis Zhou (2):
blktests: split out cgroup2 controller and file check
blktests: add Ming Lei's scsi-stress-remove

Quoted 6 lines
 common/cgroup       | 18 +++++++----
 tests/block/027     | 73 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/027.out |  2 ++
 3 files changed, 88 insertions(+), 5 deletions(-)
 create mode 100755 tests/block/027
 create mode 100644 tests/block/027.out

Thanks, Dennis, applied along with Josef's cgroup infrastructure patch.