Skip to contentSkip to conversation

Add QD=1 and gap zone tests

PATCH In review Bart Van Assche 2 participants 11 messages Apr 27, 2022 – May 9, 2022 Block layer

lore
Share & tools
lore
#0 lore raw

cover Add QD=1 and gap zone tests

Hi Omar,

This patch series adds two tests for queue depth one and two tests for SCSI devices with gap zones. Please consider these patches for inclusion in the official blktests repository.

Thanks,

Bart.

Bart Van Assche (3):
Introduce the io_schedulers() function
Add I/O scheduler tests for queue depth 1 tests/scsi: Add tests for SCSI devices with gap zones

 common/iosched             | 12 +++++++
 common/multipath-over-rdma | 11 ------
 common/rc                  |  5 +++
 tests/block/005            |  4 +--
 tests/block/014            |  7 ++--
 tests/block/015            |  7 ++--
 tests/block/020            |  7 ++--
 tests/block/021            |  6 ++--
 tests/block/032            | 62 +++++++++++++++++++++++++++++++++
 tests/block/032.out        |  2 ++
 tests/nvmeof-mp/012        |  3 +-
 tests/scsi/008             | 63 ++++++++++++++++++++++++++++++++++
 tests/scsi/008.out         |  2 ++
 tests/scsi/009             | 56 ++++++++++++++++++++++++++++++
 tests/scsi/009.out         |  2 ++
 tests/scsi/010             | 70 ++++++++++++++++++++++++++++++++++++++
 tests/scsi/010.out         |  2 ++
 tests/srp/012              |  3 +-
 18 files changed, 290 insertions(+), 34 deletions(-)
 create mode 100644 common/iosched
 create mode 100755 tests/block/032
 create mode 100644 tests/block/032.out
 create mode 100755 tests/scsi/008
 create mode 100644 tests/scsi/008.out
 create mode 100755 tests/scsi/009
 create mode 100644 tests/scsi/009.out
 create mode 100644 tests/scsi/010
 create mode 100644 tests/scsi/010.out
#1 lore raw

PATCH 1/3 Introduce the io_schedulers() function

The functionality for retrieving the I/O schedulers supported by a request queue occurs multiple times. Hence introduce a function for retrieving the supported I/O schedulers.

---
 common/iosched             | 12 ++++++++++++
 common/multipath-over-rdma | 11 -----------
 tests/block/005            |  4 ++--
 tests/block/014            |  7 ++-----
 tests/block/015            |  7 ++-----
 tests/block/020            |  7 ++-----
 tests/block/021            |  6 ++----
 tests/nvmeof-mp/012        |  3 ++-
 tests/srp/012              |  3 ++-
 9 files changed, 26 insertions(+), 34 deletions(-)
 create mode 100644 common/iosched
common/iosched new file +12 -0
new file mode 100644
index 000000000000..5dd46216afcb
@@ -0,0 +1,12 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 Google LLC
+
+# Prints a space-separated list with the names of all I/O schedulers supported
+# by block device $1.
+io_schedulers() {
+	local path=/sys/class/block/$1/queue/scheduler
+
+	[ -e "${path}" ] || return 1
+	sed 's/[][]//g' "${path}"
+}
common/multipath-over-rdma +0 -11
index cef05ec92d29..75a5c832d6da 100644
@@ -298,17 +298,6 @@ set_scheduler() {
 	fi
 }
 
-# Get block dev scheduler list
-get_scheduler_list() {
-	local b=$1 p
-	p=/sys/block/"$b"/queue/scheduler
-	if [ -e "$p" ]; then
-		sed 's/[][]//g' /sys/block/"$b"/queue/scheduler
-	else
-		return 1
-	fi
-}
-
 # Get a /dev/... path that points at dm device number $1. Set its I/O scheduler
 # to $2 and its timeout to $3. The shell script that includes this file must
 # define a function get_bdev_path() that translates device number $1 into a
tests/block/005 +2 -2
index 77b9e2f57203..383c8f5b7d2b 100755
@@ -5,6 +5,7 @@
 # Threads doing IO to a device, while we switch schedulers
 
 . tests/block/rc
+. common/iosched
 
 DESCRIPTION="switch schedulers while doing IO"
 TIMED=1
@@ -17,9 +18,8 @@ requires() {
 test_device() {
 	echo "Running ${TEST_NAME}"
 
-	local scheds
 	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' "${TEST_DEV_SYSFS}/queue/scheduler"))
+	local scheds=($(io_schedulers "${TEST_DEV_SYSFS}"))
 
 	if _test_dev_is_rotational; then
 		size="32m"
tests/block/014 +2 -5
index 04c34fa12b5c..78e90269389e 100755
@@ -6,6 +6,7 @@
 
 . tests/block/rc
 . common/null_blk
+. common/iosched
 
 DESCRIPTION="run null-blk with blk-mq and timeout injection configured"
 
@@ -22,11 +23,7 @@ test() {
 		return 1
 	fi
 
-	local scheds
-	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' /sys/block/nullb0/queue/scheduler))
-
-	for sched in "${scheds[@]}"; do
+	for sched in $(io_schedulers nullb0); do
 		echo "Testing $sched" >> "$FULL"
 		echo "$sched" > /sys/block/nullb0/queue/scheduler
 		# Do a bunch of I/Os which will timeout and then complete. The
tests/block/015 +2 -5
index 79102a2bea54..5eb954b4906c 100755
@@ -7,6 +7,7 @@
 
 . tests/block/rc
 . common/null_blk
+. common/iosched
 
 DESCRIPTION="run null-blk on different schedulers with requeue injection configured"
 QUICK=1
@@ -24,11 +25,7 @@ test() {
 		return 1
 	fi
 
-	local scheds
-	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' /sys/block/nullb0/queue/scheduler))
-
-	for sched in "${scheds[@]}"; do
+	for sched in $(io_schedulers nullb0); do
 		echo "Testing $sched" >> "$FULL"
 		echo "$sched" > /sys/block/nullb0/queue/scheduler
 		dd if=/dev/nullb0 of=/dev/null bs=4K count=$((512 * 1024)) \
tests/block/020 +2 -5
index b4887a26ff0a..0301129e1c1d 100755
@@ -8,6 +8,7 @@
 
 . tests/block/rc
 . common/null_blk
+. common/iosched
 
 DESCRIPTION="run null-blk on different schedulers with only one hardware tag"
 QUICK=1
@@ -25,16 +26,12 @@ test() {
 		return 1
 	fi
 
-	local scheds
-	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' /sys/block/nullb0/queue/scheduler))
-
 	local max_iodepth=$(($(cat /proc/sys/fs/aio-max-nr) / $(nproc)))
 	local iodepth=1024
 	if (( iodepth > max_iodepth )); then
 		iodepth=$max_iodepth
 	fi
-	for sched in "${scheds[@]}"; do
+	for sched in $(io_schedulers nullb0); do
 		echo "Testing $sched" >> "$FULL"
 		echo "$sched" > /sys/block/nullb0/queue/scheduler
 		_fio_perf --bs=4k --ioengine=libaio --iodepth=$iodepth \
tests/block/021 +2 -4
index a1bbf45a3bc9..7b562de0f99c 100755
@@ -8,6 +8,7 @@
 
 . tests/block/rc
 . common/null_blk
+. common/iosched
 
 DESCRIPTION="read/write nr_requests on null-blk with different schedulers"
 QUICK=1
@@ -26,11 +27,8 @@ test() {
 
 	local max_nr
 	local nr
-	local scheds
-	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' /sys/block/nullb0/queue/scheduler))
 
-	for sched in "${scheds[@]}"; do
+	for sched in $(io_schedulers nullb0); do
 		echo "Testing $sched" >> "$FULL"
 		echo "$sched" > /sys/block/nullb0/queue/scheduler
 		max_nr="$(cat /sys/block/nullb0/queue/nr_requests)"
tests/nvmeof-mp/012 +2 -1
index 8b2e918773e5..0b56dfb46d1e 100755
@@ -3,6 +3,7 @@
 # Copyright (c) 2018 Western Digital Corporation or its affiliates
 
 . tests/nvmeof-mp/rc
+. common/iosched
 
 DESCRIPTION="dm-mpath on top of multiple I/O schedulers"
 QUICK=1
@@ -18,7 +19,7 @@ test_io_schedulers() {
 		use_blk_mq ${mq} || return $?
 		dev=$(get_bdev 0) || return $?
 		dm=$(basename "$(readlink -f "${dev}")") || return $?
-		scheds="$(get_scheduler_list "$dm")" || return $?
+		scheds="$(io_schedulers "$dm")" || return $?
 		for sched in $scheds; do
 			set_scheduler "$dm" "$sched" \
 				      >>"$FULL" 2>&1 || continue
tests/srp/012 +2 -1
index 1a2fc6d2dc2f..7c72288b773b 100755
@@ -3,6 +3,7 @@
 # Copyright (c) 2018 Western Digital Corporation or its affiliates
 
 . tests/srp/rc
+. common/iosched
 
 DESCRIPTION="dm-mpath on top of multiple I/O schedulers"
 QUICK=1
@@ -22,7 +23,7 @@ test_io_schedulers() {
 		use_blk_mq ${mq} ${mq} || return $?
 		dev=$(get_bdev 0) || return $?
 		dm=$(basename "$(readlink -f "${dev}")") || return $?
-		scheds="$(get_scheduler_list "$dm")" || return $?
+		scheds="$(io_schedulers "$dm")" || return $?
 		for sched in $scheds; do
 			set_scheduler "$dm" "$sched" \
 				      >>"$FULL" 2>&1 || continue
#2 lore raw

Re: [PATCH blktests 1/3] Introduce the io_schedulers() function

Quoted 5 lines · Bart Van Assche

The functionality for retrieving the I/O schedulers supported by a request queue occurs multiple times. Hence introduce a function for retrieving the supported I/O schedulers.

[...]

Quoted 13 lines
tests/block/005 +1 -0
index 77b9e2f57203..383c8f5b7d2b 100755
@@ -5,6 +5,7 @@
 # Threads doing IO to a device, while we switch schedulers
 
 . tests/block/rc
+. common/iosched
 
 DESCRIPTION="switch schedulers while doing IO"
 TIMED=1

tests/block/005 requires() { · line 22

Quoted 7 lines
tests/block/005 +1 -2
@@ -17,9 +18,8 @@ requires() {
 test_device() {
 	echo "Running ${TEST_NAME}"
 
-	local scheds
 	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' "${TEST_DEV_SYSFS}/queue/scheduler"))
+	local scheds=($(io_schedulers "${TEST_DEV_SYSFS}"))

I ran block/005 with this patch and observed it fails without failure message. To fix it, the line above should be:

        local scheds=($(io_schedulers "$(basename "${TEST_DEV}")"))
Signature · 2 lines
Best Regards,
Shin'ichiro Kawasaki
#3 lore raw

Re: [PATCH blktests 1/3] Introduce the io_schedulers() function

Quoted 10 lines · Shinichiro Kawasaki

tests/block/005 requires() { · line 22

Quoted 4 lines · Bart Van Assche
tests/block/005 +1 -2
-	local scheds
  	# shellcheck disable=SC2207
-	scheds=($(sed 's/[][]//g' "${TEST_DEV_SYSFS}/queue/scheduler"))
+	local scheds=($(io_schedulers "${TEST_DEV_SYSFS}"))

I ran block/005 with this patch and observed it fails without failure message. To fix it, the line above should be:

         local scheds=($(io_schedulers "$(basename "${TEST_DEV}")"))

This is something I should have noticed myself. I will fix this.

Thanks,

Bart.

#4 lore raw

PATCH 2/3 Add I/O scheduler tests for queue depth 1

In reply to Bart Van Assche

Some block devices, e.g. USB sticks, only support queue depth 1. The QD=1 code paths do not get tested routinely. Hence add tests for the QD=1 use case.

---
 common/rc           |  5 ++++
 tests/block/032     | 62 ++++++++++++++++++++++++++++++++++++++++++++
 tests/block/032.out |  2 ++
 tests/scsi/008      | 63 +++++++++++++++++++++++++++++++++++++++++++++
 tests/scsi/008.out  |  2 ++
 5 files changed, 134 insertions(+)
 create mode 100755 tests/block/032
 create mode 100644 tests/block/032.out
 create mode 100755 tests/scsi/008
 create mode 100644 tests/scsi/008.out
common/rc +5 -0
index 0528ce808256..2377e223aea7 100644
@@ -321,6 +321,11 @@ _uptime_s() {
 	awk '{ print int($1) }' /proc/uptime
 }
 
+# System uptime in centiseconds.
+_uptime_cs() {
+	sed 's/\.//;s/ .*//' /proc/uptime
+}
+
 # Arguments: module to unload ($1) and retry count ($2).
 unload_module() {
 	local i m=$1 rc=${2:-1}
tests/block/032 new file +62 -0

62 lines not on this page · raw message

tests/block/032.out new file +2 -0
new file mode 100644
index 000000000000..3604e9e1e751
@@ -0,0 +1,2 @@
+Running block/032
+Test complete
tests/scsi/008 new file +63 -0

63 lines not on this page · raw message

tests/scsi/008.out new file +2 -0
new file mode 100644
index 000000000000..135bd5ae4b2d
@@ -0,0 +1,2 @@
+Running scsi/008
+Test complete
#5 lore raw

Re: [PATCH blktests 2/3] Add I/O scheduler tests for queue depth 1

Quoted 5 lines · Bart Van Assche

Some block devices, e.g. USB sticks, only support queue depth 1. The QD=1 code paths do not get tested routinely. Hence add tests for the QD=1 use case.

I tried this new test case on recent Fedora kernel 5.16.20-200.fc35.x86_64 and Intel Core i9 machine, then observed failure:

$ sudo ./check block/032
block/032 (test I/O scheduler performance of null_blk with queue_depth 1) [failed]
    bfq          679 vs 243: fail  ...  679 vs 252: fail
    kyber        542 vs 243: fail  ...  551 vs 252: fail
    mq-deadline  577 vs 243: fail  ...  572 vs 252: fail
    runtime      20.514s           ...  20.660s
    --- tests/block/032.out     2022-04-27 22:02:46.602861565 -0700
    +++ /home/shin/kts/kernel-test-suite/src/blktests/results/nodev/block/032.out.bad2022-04-27 23:18:48.470170788 -0700
    @@ -1,2 +1,2 @@
     Running block/032
    -Test complete
    +Test failed

I tried v5.18-rcX kernel versions and machines (QEMU or VMware) but the test case failed on all of the trials. Do I miss anything to make the test case pass?

Signature · 2 lines
Best Regards,
Shin'ichiro Kawasaki
#6 lore raw

Re: [PATCH blktests 2/3] Add I/O scheduler tests for queue depth 1

Quoted 23 lines · Shinichiro Kawasaki
Quoted 5 lines · Bart Van Assche

Some block devices, e.g. USB sticks, only support queue depth 1. The QD=1 code paths do not get tested routinely. Hence add tests for the QD=1 use case.

I tried this new test case on recent Fedora kernel 5.16.20-200.fc35.x86_64 and Intel Core i9 machine, then observed failure:

$ sudo ./check block/032
block/032 (test I/O scheduler performance of null_blk with queue_depth 1) [failed]
     bfq          679 vs 243: fail  ...  679 vs 252: fail
     kyber        542 vs 243: fail  ...  551 vs 252: fail
     mq-deadline  577 vs 243: fail  ...  572 vs 252: fail
     runtime      20.514s           ...  20.660s
     --- tests/block/032.out     2022-04-27 22:02:46.602861565 -0700
     +++ /home/shin/kts/kernel-test-suite/src/blktests/results/nodev/block/032.out.bad2022-04-27 23:18:48.470170788 -0700
     @@ -1,2 +1,2 @@
      Running block/032
     -Test complete
     +Test failed
Quoted 2 lines

I tried v5.18-rcX kernel versions and machines (QEMU or VMware) but the test case failed on all of the trials. Do I miss anything to make the test case pass?

Hi Shinichiro,

The two tests added by this patch pass when using the legacy block layer (kernel v4.19) but not when using blk-mq. I see this as a (performance) bug in blk-mq. With blk-mq an excessive number of queue runs is triggered for QD=1 if multiple processes try to submit I/O concurrently.

Thanks,

Bart.

#7 lore raw

Re: [PATCH blktests 2/3] Add I/O scheduler tests for queue depth 1

Quoted 27 lines · Bart Van Assche
Quoted 23 lines · Shinichiro Kawasaki
Quoted 5 lines · Bart Van Assche

Some block devices, e.g. USB sticks, only support queue depth 1. The QD=1 code paths do not get tested routinely. Hence add tests for the QD=1 use case.

I tried this new test case on recent Fedora kernel 5.16.20-200.fc35.x86_64 and Intel Core i9 machine, then observed failure:

$ sudo ./check block/032
block/032 (test I/O scheduler performance of null_blk with queue_depth 1) [failed]
     bfq          679 vs 243: fail  ...  679 vs 252: fail
     kyber        542 vs 243: fail  ...  551 vs 252: fail
     mq-deadline  577 vs 243: fail  ...  572 vs 252: fail
     runtime      20.514s           ...  20.660s
     --- tests/block/032.out     2022-04-27 22:02:46.602861565 -0700
     +++ /home/shin/kts/kernel-test-suite/src/blktests/results/nodev/block/032.out.bad2022-04-27 23:18:48.470170788 -0700
     @@ -1,2 +1,2 @@
      Running block/032
     -Test complete
     +Test failed
Quoted 2 lines

I tried v5.18-rcX kernel versions and machines (QEMU or VMware) but the test case failed on all of the trials. Do I miss anything to make the test case pass?

Quoted 6 lines

Hi Shinichiro,

The two tests added by this patch pass when using the legacy block layer (kernel v4.19) but not when using blk-mq. I see this as a (performance) bug in blk-mq. With blk-mq an excessive number of queue runs is triggered for QD=1 if multiple processes try to submit I/O concurrently.

Thanks. So we need the fix in blk-mq before we merge this test case to blktests.

Signature · 2 lines
Best Regards,
Shin'ichiro Kawasaki
#8 lore raw

PATCH 3/3 tests/scsi: Add tests for SCSI devices with gap zones

In reply to Bart Van Assche
---
 tests/scsi/009     | 56 +++++++++++++++++++++++++++++++++++++
 tests/scsi/009.out |  2 ++
 tests/scsi/010     | 70 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/scsi/010.out |  2 ++
 4 files changed, 130 insertions(+)
 create mode 100755 tests/scsi/009
 create mode 100644 tests/scsi/009.out
 create mode 100644 tests/scsi/010
 create mode 100644 tests/scsi/010.out
tests/scsi/009 new file +56 -0
new file mode 100755
index 000000000000..38f771f14e02
@@ -0,0 +1,56 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 Google LLC
+
+. tests/scsi/rc
+. common/scsi_debug
+
+DESCRIPTION="test gap zone support with BTRFS"
+QUICK=1
+
+requires() {
+	_have_fio &&
+	_have_module_param scsi_debug zone_cap_mb &&
+	_have_program mkfs.btrfs
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	local params=(
+		delay=0
+		dev_size_mb=1024
+		sector_size=4096
+		zbc=host-managed
+		zone_cap_mb=3
+		zone_nr_conv=16
+		zone_size_mb=4
+	)
+	_init_scsi_debug "${params[@]}" || return 1
+
+	local dev="/dev/${SCSI_DEBUG_DEVICES[0]}" fail
+
+	mkfs.btrfs "${dev}" >>"${FULL}" 2>&1 &&
+	local mount_dir="$TMPDIR/mnt" &&
+	mkdir -p "${mount_dir}" &&
+	mount -t btrfs "${dev}" "${mount_dir}" &&
+	local fio_args=(
+		--size=1M
+		--directory="${mount_dir}"
+		--time_based
+		--runtime=10
+	) &&
+	_run_fio_verify_io "${fio_args[@]}" >>"${FULL}" 2>&1 ||
+	fail=true
+
+	umount "${mount_dir}" >>"${FULL}" 2>&1
+
+	_exit_scsi_debug
+
+	if [ -z "$fail" ]; then
+		echo "Test complete"
+	else
+		echo "Test failed"
+		return 1
+	fi
+}
tests/scsi/009.out new file +2 -0
new file mode 100644
index 000000000000..f6db2a371d9e
@@ -0,0 +1,2 @@
+Running scsi/009
+Test complete
tests/scsi/010 new file +70 -0

70 lines not on this page · raw message

tests/scsi/010.out new file +2 -0
new file mode 100644
index 000000000000..6581d5eb2c5a
@@ -0,0 +1,2 @@
+Running scsi/010
+Test complete
#9 lore raw

Re: [PATCH blktests 3/3] tests/scsi: Add tests for SCSI devices with gap zones

Quoted 1 line · Bart Van Assche

Hi Bart, the desciption is missing, so I expect you will add it later.

I think the added test cases fit better with zbd group rather than scsi group since the gap between zone capacity and zone size exists for non-scsi ZNS devices also. What do you think?

Signature · 2 lines
Best Regards,
Shin'ichiro Kawasaki
#10 lore raw

Re: [PATCH blktests 3/3] tests/scsi: Add tests for SCSI devices with gap zones

Quoted 8 lines · Shinichiro Kawasaki
Quoted 1 line · Bart Van Assche

Hi Bart, the desciption is missing, so I expect you will add it later.

I think the added test cases fit better with zbd group rather than scsi group since the gap between zone capacity and zone size exists for non-scsi ZNS devices also. What do you think?

Hi Shinichiro,

Thanks for having taken a look. I will add a description and move these tests to the zbd group.

Bart.