Skip to contentSkip to conversation

selftests: ublk: test infrastructure improvements

PATCH Applied Ming Lei 2 participants 11 messages Jan 31

Subsystems (3)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover selftests: ublk: test infrastructure improvements

Hi,

This series improves the ublk selftests infrastructure with better organization and parallel test execution support.

Key improvements:

1. Infrastructure enhancements:
- Add helper functions for device cleanup and parallel execution
- Implement group-based test targets for running test subsets
- Track created devices for proper per-test cleanup
- Increase timeouts to accommodate parallel test execution
- Test time can be reduced to 1/3 with parallel execution by passing JOBS=8

2. Test organization:
- Reorganize tests into logical groups (integrity, recover)
- Refactor large tests into modular functions for maintainability

3. I/O ordering verification:
- Remove test_generic_01.sh prone to false positives from block layer reordering
- Improve test_generic_02.sh with bpftrace-based verification
- Use block_io_start/block_rq_complete tracepoints for accurate tracking

These changes make the test suite more maintainable, easier to run in parallel, and more reliable in detecting actual ublk driver issues vs. false positives.

Ming Lei (9):
selftests: ublk: simplify UBLK_TEST_DIR handling
selftests: ublk: refactor test_loop_08 into separate functions
selftests: ublk: add _ublk_del_dev helper function
selftests: ublk: track created devices for per-test cleanup
selftests: ublk: add group-based test targets
selftests: ublk: add _ublk_sleep helper for parallel execution
selftests: ublk: increase timeouts for parallel test execution
selftests: ublk: reorganize tests into integrity and recover groups
selftests: ublk: improve I/O ordering test with bpftrace

 tools/testing/selftests/ublk/Makefile         |  54 ++++++-
 tools/testing/selftests/ublk/test_common.sh   |  51 +++++--
 .../testing/selftests/ublk/test_generic_01.sh |  47 ------
 .../testing/selftests/ublk/test_generic_02.sh |  22 ++-
 .../testing/selftests/ublk/test_generic_16.sh |   4 +-
 .../{test_null_04.sh => test_integrity_01.sh} |   8 +-
 .../selftests/ublk/test_integrity_02.sh       | 141 ++++++++++++++++++
 tools/testing/selftests/ublk/test_loop_08.sh  | 110 --------------
 tools/testing/selftests/ublk/test_part_02.sh  |   6 +-
 ...{test_generic_04.sh => test_recover_01.sh} |   0
 ...{test_generic_05.sh => test_recover_02.sh} |   0
 ...{test_generic_11.sh => test_recover_03.sh} |   0
 ...{test_generic_14.sh => test_recover_04.sh} |   0
 tools/testing/selftests/ublk/trace/seq_io.bt  |  47 ++++--
 14 files changed, 284 insertions(+), 206 deletions(-)
 delete mode 100755 tools/testing/selftests/ublk/test_generic_01.sh
 rename tools/testing/selftests/ublk/{test_null_04.sh => test_integrity_01.sh} (96%)
 create mode 100755 tools/testing/selftests/ublk/test_integrity_02.sh
 delete mode 100755 tools/testing/selftests/ublk/test_loop_08.sh
 rename tools/testing/selftests/ublk/{test_generic_04.sh => test_recover_01.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_05.sh => test_recover_02.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_11.sh => test_recover_03.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_14.sh => test_recover_04.sh} (100%)
Signature · 1 line
2.47.0
author
#1 lore raw

PATCH 1/9 selftests: ublk: simplify UBLK_TEST_DIR handling

Remove intermediate TDIR variable and set UBLK_TEST_DIR directly in _prep_test(). Remove default initialization since the directory is created dynamically when tests run.

---
 tools/testing/selftests/ublk/test_common.sh | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
tools/testing/selftests/ublk/test_common.sh +1 -4
index 21ba51fcc7d7..8d298a7ee7b1 100755
@@ -124,8 +124,7 @@ _prep_test() {
 	local type=$1
 	shift 1
 	modprobe ublk_drv > /dev/null 2>&1
-	TDIR=$(mktemp -d ${TMPDIR:-.}/ublktest-dir.XXXXXX)
-	export UBLK_TEST_DIR=${TDIR}
+	UBLK_TEST_DIR=$(mktemp -d ${TMPDIR:-.}/ublktest-dir.XXXXXX)
 	UBLK_TMP=$(mktemp ${UBLK_TEST_DIR}/ublk_test_XXXXX)
 	[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "ublk $type: $*"
 	echo "ublk selftest: $TID starting at $(date '+%F %T')" | tee /dev/kmsg
@@ -408,8 +407,6 @@ UBLK_PROG=$(_ublk_test_top_dir)/kublk
 UBLK_TEST_QUIET=1
 UBLK_TEST_SHOW_RESULT=1
 UBLK_BACKFILES=()
-UBLK_TEST_DIR=${TMPDIR:-.}
 export UBLK_PROG
 export UBLK_TEST_QUIET
 export UBLK_TEST_SHOW_RESULT
-export UBLK_TEST_DIR
Signature · 1 line
2.47.0
author
#2 lore raw

PATCH 2/9 selftests: ublk: refactor test_loop_08 into separate functions

In reply to Ming Lei

Encapsulate each test case in its own function for better organization and maintainability:

- _setup_device(): device and backfile initialization
- _test_fill_and_verify(): initial data population
- _test_corrupted_reftag(): reftag corruption detection test
- _test_corrupted_data(): data corruption detection test
- _test_bad_apptag(): apptag mismatch detection test

Also fix temp file creation to use ${UBLK_TEST_DIR}/fio_err_XXXXX instead of creating in current directory.

---
 tools/testing/selftests/ublk/test_loop_08.sh | 199 +++++++++++--------
 1 file changed, 115 insertions(+), 84 deletions(-)
tools/testing/selftests/ublk/test_loop_08.sh +115 -84
index 2caa7ba748fb..aaf1f52da559 100755
@@ -13,98 +13,129 @@ if [[ "$fio_version" =~ fio-[0-9]+\.[0-9]+$ ]]; then
 	exit $UBLK_SKIP_CODE
 fi
 
+ERR_CODE=0
 
-_prep_test "loop" "end-to-end integrity"
+# Global variables set during device setup
+dev_id=""
+fio_args=""
+fio_err=""
 
-_create_backfile 0 256M
-_create_backfile 1 32M # 256M * (64 integrity bytes / 512 data bytes)
-integrity_params="--integrity_capable --integrity_reftag
-                  --metadata_size 64 --pi_offset 56 --csum_type t10dif"
-dev_id=$(_add_ublk_dev -t loop -u $integrity_params "${UBLK_BACKFILES[@]}")
-_check_add_dev $TID $?
-
-# 1M * (64 integrity bytes / 512 data bytes) = 128K
-fio_args="--ioengine io_uring --direct 1 --bsrange 512-1M --iodepth 32
-          --md_per_io_size 128K --pi_act 0 --pi_chk GUARD,REFTAG,APPTAG
-          --filename /dev/ublkb$dev_id"
-fio --name fill --rw randwrite $fio_args > /dev/null
-err=$?
-if [ $err != 0 ]; then
-	echo "fio fill failed"
-	_show_result $TID $err
-fi
+_setup_device() {
+	_create_backfile 0 256M
+	_create_backfile 1 32M # 256M * (64 integrity bytes / 512 data bytes)
 
-fio --name verify --rw randread $fio_args > /dev/null
-err=$?
-if [ $err != 0 ]; then
-	echo "fio verify failed"
-	_show_result $TID $err
-fi
+	local integrity_params="--integrity_capable --integrity_reftag
+		--metadata_size 64 --pi_offset 56 --csum_type t10dif"
+	dev_id=$(_add_ublk_dev -t loop -u $integrity_params "${UBLK_BACKFILES[@]}")
+	_check_add_dev "$TID" $?
 
-fio_err=$(mktemp fio_err_XXXXX)
+	# 1M * (64 integrity bytes / 512 data bytes) = 128K
+	fio_args="--ioengine io_uring --direct 1 --bsrange 512-1M --iodepth 32
+		--md_per_io_size 128K --pi_act 0 --pi_chk GUARD,REFTAG,APPTAG
+		--filename /dev/ublkb$dev_id"
 
-# Overwrite 4-byte reftag at offset 56 + 4 = 60
-dd_reftag_args="bs=1 seek=60 count=4 oflag=dsync conv=notrunc status=none"
-dd if=/dev/urandom "of=${UBLK_BACKFILES[1]}" $dd_reftag_args
-err=$?
-if [ $err != 0 ]; then
-	echo "dd corrupted_reftag failed"
-	rm -f "$fio_err"
-	_show_result $TID $err
-fi
-if fio --name corrupted_reftag --rw randread $fio_args > /dev/null 2> "$fio_err"; then
-	echo "fio corrupted_reftag unexpectedly succeeded"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
-expected_err="REFTAG compare error: LBA: 0 Expected=0, Actual="
-if ! grep -q "$expected_err" "$fio_err"; then
-	echo "fio corrupted_reftag message not found: $expected_err"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
-# Reset to 0
-dd if=/dev/zero "of=${UBLK_BACKFILES[1]}" $dd_reftag_args
-err=$?
-if [ $err != 0 ]; then
-	echo "dd restore corrupted_reftag failed"
-	rm -f "$fio_err"
-	_show_result $TID $err
-fi
+	fio_err=$(mktemp "${UBLK_TEST_DIR}"/fio_err_XXXXX)
+}
 
-dd_data_args="bs=512 count=1 oflag=direct,dsync conv=notrunc status=none"
-dd if=/dev/zero "of=${UBLK_BACKFILES[0]}" $dd_data_args
-err=$?
-if [ $err != 0 ]; then
-	echo "dd corrupted_data failed"
-	rm -f "$fio_err"
-	_show_result $TID $err
-fi
-if fio --name corrupted_data --rw randread $fio_args > /dev/null 2> "$fio_err"; then
-	echo "fio corrupted_data unexpectedly succeeded"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
-expected_err="Guard compare error: LBA: 0 Expected=0, Actual="
-if ! grep -q "$expected_err" "$fio_err"; then
-	echo "fio corrupted_data message not found: $expected_err"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
+_test_fill_and_verify() {
+	fio --name fill --rw randwrite $fio_args > /dev/null
+	if [ $? != 0 ]; then
+		echo "fio fill failed"
+		ERR_CODE=255
+		return 1
+	fi
 
-if fio --name bad_apptag --rw randread $fio_args --apptag 0x4321 > /dev/null 2> "$fio_err"; then
-	echo "fio bad_apptag unexpectedly succeeded"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
-expected_err="APPTAG compare error: LBA: [0-9]* Expected=4321, Actual=1234"
-if ! grep -q "$expected_err" "$fio_err"; then
-	echo "fio bad_apptag message not found: $expected_err"
-	rm -f "$fio_err"
-	_show_result $TID 255
-fi
+	fio --name verify --rw randread $fio_args > /dev/null
+	if [ $? != 0 ]; then
+		echo "fio verify failed"
+		ERR_CODE=255
+		return 1
+	fi
+}
+
+_test_corrupted_reftag() {
+	local dd_reftag_args="bs=1 seek=60 count=4 oflag=dsync conv=notrunc status=none"
+	local expected_err="REFTAG compare error: LBA: 0 Expected=0, Actual="
+
+	# Overwrite 4-byte reftag at offset 56 + 4 = 60
+	dd if=/dev/urandom "of=${UBLK_BACKFILES[1]}" $dd_reftag_args
+	if [ $? != 0 ]; then
+		echo "dd corrupted_reftag failed"
+		ERR_CODE=255
+		return 1
+	fi
+
+	if fio --name corrupted_reftag --rw randread $fio_args > /dev/null 2> "$fio_err"; then
+		echo "fio corrupted_reftag unexpectedly succeeded"
+		ERR_CODE=255
+		return 1
+	fi
+
+	if ! grep -q "$expected_err" "$fio_err"; then
+		echo "fio corrupted_reftag message not found: $expected_err"
+		ERR_CODE=255
+		return 1
+	fi
+
+	# Reset to 0
+	dd if=/dev/zero "of=${UBLK_BACKFILES[1]}" $dd_reftag_args
+	if [ $? != 0 ]; then
+		echo "dd restore corrupted_reftag failed"
+		ERR_CODE=255
+		return 1
+	fi
+}
+
+_test_corrupted_data() {
+	local dd_data_args="bs=512 count=1 oflag=direct,dsync conv=notrunc status=none"
+	local expected_err="Guard compare error: LBA: 0 Expected=0, Actual="
+
+	dd if=/dev/zero "of=${UBLK_BACKFILES[0]}" $dd_data_args
+	if [ $? != 0 ]; then
+		echo "dd corrupted_data failed"
+		ERR_CODE=255
+		return 1
+	fi
+
+	if fio --name corrupted_data --rw randread $fio_args > /dev/null 2> "$fio_err"; then
+		echo "fio corrupted_data unexpectedly succeeded"
+		ERR_CODE=255
+		return 1
+	fi
+
+	if ! grep -q "$expected_err" "$fio_err"; then
+		echo "fio corrupted_data message not found: $expected_err"
+		ERR_CODE=255
+		return 1
+	fi
+}
+
+_test_bad_apptag() {
+	local expected_err="APPTAG compare error: LBA: [0-9]* Expected=4321, Actual=1234"
+
+	if fio --name bad_apptag --rw randread $fio_args --apptag 0x4321 > /dev/null 2> "$fio_err"; then
+		echo "fio bad_apptag unexpectedly succeeded"
+		ERR_CODE=255
+		return 1
+	fi
+
+	if ! grep -q "$expected_err" "$fio_err"; then
+		echo "fio bad_apptag message not found: $expected_err"
+		ERR_CODE=255
+		return 1
+	fi
+}
+
+_prep_test "loop" "end-to-end integrity"
+
+_setup_device
+
+_test_fill_and_verify && \
+_test_corrupted_reftag && \
+_test_corrupted_data && \
+_test_bad_apptag
 
 rm -f "$fio_err"
 
 _cleanup_test
-_show_result $TID 0
+_show_result "$TID" $ERR_CODE
Signature · 1 line
2.47.0
author
#3 lore raw

PATCH 3/9 selftests: ublk: add _ublk_del_dev helper function

In reply to Ming Lei

Add _ublk_del_dev() to delete a specific ublk device by ID and use it in all test scripts instead of calling UBLK_PROG directly.

Also remove unused _remove_ublk_devices() function.

---
 tools/testing/selftests/ublk/test_common.sh     | 13 +++++++------
 tools/testing/selftests/ublk/test_generic_16.sh |  4 ++--
 tools/testing/selftests/ublk/test_null_04.sh    |  8 ++++----
 tools/testing/selftests/ublk/test_part_02.sh    |  4 ++--
 4 files changed, 15 insertions(+), 14 deletions(-)
tools/testing/selftests/ublk/test_common.sh +7 -6
index 8d298a7ee7b1..0f1fdb0892b4 100755
@@ -106,11 +106,6 @@ _check_root() {
 	fi
 }
 
-_remove_ublk_devices() {
-	${UBLK_PROG} del -a
-	modprobe -r ublk_drv > /dev/null 2>&1
-}
-
 _get_ublk_dev_state() {
 	${UBLK_PROG} list -n "$1" | grep "state" | awk '{print $11}'
 }
@@ -277,10 +272,16 @@ __ublk_kill_daemon()
 	echo "$state"
 }
 
-__remove_ublk_dev_return() {
+_ublk_del_dev() {
 	local dev_id=$1
 
 	${UBLK_PROG} del -n "${dev_id}"
+}
+
+__remove_ublk_dev_return() {
+	local dev_id=$1
+
+	_ublk_del_dev "${dev_id}"
 	local res=$?
 	udevadm settle
 	return ${res}
tools/testing/selftests/ublk/test_generic_16.sh +2 -2
index 42e8d2e16ec9..3ef367836ac5 100755
@@ -24,7 +24,7 @@ if ! ${UBLK_PROG} stop -n "${dev_id}" --safe; then
 fi
 
 # Clean up device
-${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1
+_ublk_del_dev "${dev_id}" > /dev/null 2>&1
 udevadm settle
 
 # Test 2: stop --safe on device with active opener should fail
@@ -49,7 +49,7 @@ kill $dd_pid 2>/dev/null
 wait $dd_pid 2>/dev/null
 
 # Now device should be idle, regular delete should work
-${UBLK_PROG} del -n "${dev_id}"
+_ublk_del_dev "${dev_id}"
 udevadm settle
 
 _cleanup_test "null"
tools/testing/selftests/ublk/test_null_04.sh +4 -4
index a5599d38583a..6713b280a6ff 100755
@@ -34,7 +34,7 @@ _test_metadata_only() {
 		"$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 &&
 	_check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0
 
-	${UBLK_PROG} del -n "${dev_id}"
+	_ublk_del_dev "${dev_id}"
 }
 
 _test_integrity_capable_ip() {
@@ -53,7 +53,7 @@ _test_integrity_capable_ip() {
 		"$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 &&
 	_check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0
 
-	${UBLK_PROG} del -n "${dev_id}"
+	_ublk_del_dev "${dev_id}"
 }
 
 _test_integrity_reftag_t10dif() {
@@ -72,7 +72,7 @@ _test_integrity_reftag_t10dif() {
 		"$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 &&
 	_check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0
 
-	${UBLK_PROG} del -n "${dev_id}"
+	_ublk_del_dev "${dev_id}"
 }
 
 _test_nvme_csum() {
@@ -91,7 +91,7 @@ _test_nvme_csum() {
 		"$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 &&
 	_check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 8
 
-	${UBLK_PROG} del -n "${dev_id}"
+	_ublk_del_dev "${dev_id}"
 }
 
 _prep_test "null" "integrity params"
tools/testing/selftests/ublk/test_part_02.sh +2 -2
index 727d0f4610d6..acd098deda3a 100755
@@ -46,13 +46,13 @@ _test_partition_scan_no_hang()
 	if [ "$state" != "${expected_state}" ]; then
 		echo "FAIL: Device state is $state, expected ${expected_state}"
 		ERR_CODE=255
-		${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1
+		_ublk_del_dev "${dev_id}" > /dev/null 2>&1
 		return
 	fi
 	echo "PASS: Device transitioned to ${expected_state} in ${elapsed}s without hanging"
 
 	# Clean up the device
-	${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1
+	_ublk_del_dev "${dev_id}" > /dev/null 2>&1
 }
 
 _prep_test "partition_scan" "verify async partition scan prevents IO hang"
Signature · 1 line
2.47.0
author
#4 lore raw

PATCH 4/9 selftests: ublk: track created devices for per-test cleanup

In reply to Ming Lei

Track device IDs in UBLK_DEVS array when created. Update _cleanup_test() to only delete devices created by this test instead of using 'del -a' which removes all devices.

This prepares for running tests concurrently where each test should only clean up its own devices.

---
 tools/testing/selftests/ublk/test_common.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
tools/testing/selftests/ublk/test_common.sh +12 -1
index 0f1fdb0892b4..422882c32490 100755
@@ -164,7 +164,12 @@ _check_add_dev()
 }
 
 _cleanup_test() {
-	"${UBLK_PROG}" del -a
+	if [ -f "${UBLK_TEST_DIR}/.ublk_devs" ]; then
+		while read -r dev_id; do
+			${UBLK_PROG} del -n "${dev_id}"
+		done < "${UBLK_TEST_DIR}/.ublk_devs"
+		rm -f "${UBLK_TEST_DIR}/.ublk_devs"
+	fi
 
 	_remove_files
 	rmdir ${UBLK_TEST_DIR}
@@ -205,6 +210,7 @@ _create_ublk_dev() {
 	fi
 
 	if [[ "$dev_id" =~ ^[0-9]+$ ]]; then
+		echo "$dev_id" >> "${UBLK_TEST_DIR}/.ublk_devs"
 		echo "${dev_id}"
 	else
 		return 255
@@ -276,6 +282,11 @@ _ublk_del_dev() {
 	local dev_id=$1
 
 	${UBLK_PROG} del -n "${dev_id}"
+
+	# Remove from tracking file
+	if [ -f "${UBLK_TEST_DIR}/.ublk_devs" ]; then
+		sed -i "/^${dev_id}$/d" "${UBLK_TEST_DIR}/.ublk_devs"
+	fi
 }
 
 __remove_ublk_dev_return() {
Signature · 1 line
2.47.0
author
#5 lore raw

PATCH 5/9 selftests: ublk: add group-based test targets

In reply to Ming Lei

Add convenient Makefile targets for running specific test groups:
- run_generic, run_batch, run_null, run_loop, run_stripe, run_stress, etc.
- run_all for running all tests

Test groups are auto-detected from TEST_PROGS using pattern matching (test_<group>_<num>.sh -> group), and targets are generated dynamically using define/eval templates.

Supports parallel execution via JOBS variable:
- JOBS=1 (default): sequential with kselftest TAP output
- JOBS>1: parallel execution with xargs -P

Usage examples:

  make run_null           # Sequential execution
  make run_stress JOBS=4  # Parallel with 4 jobs
  make run_all JOBS=8     # Run all tests with 8 parallel jobs

With JOBS=8, running time of make run_all is reduced to 2m2s from 6m5s in my test VM.

---
 tools/testing/selftests/ublk/Makefile | 36 +++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
tools/testing/selftests/ublk/Makefile +36 -0
index ca8588ed962c..37e012d3a8a7 100644
@@ -72,3 +72,39 @@ $(OUTPUT)/kublk: $(filter-out $(STANDALONE_UTILS),$(wildcard *.c))
 
 check:
 	shellcheck -x -f gcc *.sh
+
+# Test groups for running subsets of tests
+# JOBS=1 (default): sequential with kselftest TAP output
+# JOBS>1: parallel execution with xargs -P
+# Usage: make run_null JOBS=4
+JOBS ?= 1
+
+# Auto-detect test groups from TEST_PROGS (test_<group>_<num>.sh -> group)
+TEST_GROUPS := $(shell echo "$(TEST_PROGS)" | tr ' ' '\n' | \
+	sed 's/test_\([^_]*\)_.*/\1/' | sort -u)
+
+# Template for group test targets
+# $(1) = group name (e.g., null, generic, stress)
+define RUN_GROUP
+run_$(1): all
+	@if [ $$(JOBS) -gt 1 ]; then \
+		echo $$(filter test_$(1)_%.sh,$$(TEST_PROGS)) | tr ' ' '\n' | \
+			xargs -P $$(JOBS) -n1 sh -c './"$$$$0"' || true; \
+	else \
+		$$(call RUN_TESTS, $$(filter test_$(1)_%.sh,$$(TEST_PROGS))); \
+	fi
+.PHONY: run_$(1)
+endef
+
+# Generate targets for each discovered test group
+$(foreach group,$(TEST_GROUPS),$(eval $(call RUN_GROUP,$(group))))
+
+# Run all tests (parallel when JOBS>1)
+run_all: all
+	@if [ $(JOBS) -gt 1 ]; then \
+		echo $(TEST_PROGS) | tr ' ' '\n' | \
+			xargs -P $(JOBS) -n1 sh -c './"$$0"' || true; \
+	else \
+		$(call RUN_TESTS, $(TEST_PROGS)); \
+	fi
+.PHONY: run_all
Signature · 1 line
2.47.0
author
#6 lore raw

PATCH 6/9 selftests: ublk: add _ublk_sleep helper for parallel execution

In reply to Ming Lei

Add _ublk_sleep() helper function that uses different sleep times depending on whether tests run in parallel or sequential mode.

Usage: _ublk_sleep <normal_secs> <parallel_secs>

Export JOBS variable from Makefile so test scripts can detect parallel execution, and use _ublk_sleep in test_part_02.sh to handle the partition scan delay (1s normal, 5s parallel).

---
 tools/testing/selftests/ublk/Makefile        |  1 +
 tools/testing/selftests/ublk/test_common.sh  | 10 ++++++++++
 tools/testing/selftests/ublk/test_part_02.sh |  2 +-
 3 files changed, 12 insertions(+), 1 deletion(-)
tools/testing/selftests/ublk/Makefile +1 -0
index 37e012d3a8a7..1ceae611acb7 100644
@@ -78,6 +78,7 @@ check:
 # JOBS>1: parallel execution with xargs -P
 # Usage: make run_null JOBS=4
 JOBS ?= 1
+export JOBS
 
 # Auto-detect test groups from TEST_PROGS (test_<group>_<num>.sh -> group)
 TEST_GROUPS := $(shell echo "$(TEST_PROGS)" | tr ' ' '\n' | \
tools/testing/selftests/ublk/test_common.sh +10 -0
index 422882c32490..bd27a6875c1a 100755
@@ -15,6 +15,16 @@ _have_program() {
 	return 1
 }
 
+# Sleep with awareness of parallel execution.
+# Usage: _ublk_sleep <normal_secs> <parallel_secs>
+_ublk_sleep() {
+	if [ "${JOBS:-1}" -gt 1 ]; then
+		sleep "$2"
+	else
+		sleep "$1"
+	fi
+}
+
 _get_disk_dev_t() {
 	local dev_id=$1
 	local dev
tools/testing/selftests/ublk/test_part_02.sh +1 -1
index acd098deda3a..7d42ab4d6e83 100755
@@ -33,7 +33,7 @@ _test_partition_scan_no_hang()
 	# The add command should return quickly because partition scan is async.
 	# Now sleep briefly to let the async partition scan work start and hit
 	# the delay in the fault_inject handler.
-	sleep 1
+	_ublk_sleep 1 5
 
 	# Kill the ublk daemon while partition scan is potentially blocked
 	# And check state transitions properly
Signature · 1 line
2.47.0
author
#7 lore raw

PATCH 7/9 selftests: ublk: increase timeouts for parallel test execution

In reply to Ming Lei

When running tests in parallel with high JOBS count (e.g., JOBS=64), the existing timeouts can be insufficient due to system load:

- Increase state wait loops from 20/50 to 100 iterations in _recover_ublk_dev(), __ublk_quiesce_dev(), and __ublk_kill_daemon() to handle slower state transitions under heavy load

- Add --timeout=20 to udevadm settle calls to prevent indefinite hangs when udev event queue is overwhelmed by rapid device creation/deletion

---
 tools/testing/selftests/ublk/test_common.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tools/testing/selftests/ublk/test_common.sh +5 -5
index bd27a6875c1a..c3afd00783a2 100755
@@ -216,7 +216,7 @@ _create_ublk_dev() {
 	fi
 
 	if [ "$settle" = "yes" ]; then
-		udevadm settle
+		udevadm settle --timeout=20
 	fi
 
 	if [[ "$dev_id" =~ ^[0-9]+$ ]]; then
@@ -240,7 +240,7 @@ _recover_ublk_dev() {
 	local state
 
 	dev_id=$(_create_ublk_dev "recover" "yes" "$@")
-	for ((j=0;j<20;j++)); do
+	for ((j=0;j<100;j++)); do
 		state=$(_get_ublk_dev_state "${dev_id}")
 		[ "$state" == "LIVE" ] && break
 		sleep 1
@@ -260,7 +260,7 @@ __ublk_quiesce_dev()
 		return "$state"
 	fi
 
-	for ((j=0;j<50;j++)); do
+	for ((j=0;j<100;j++)); do
 		state=$(_get_ublk_dev_state "${dev_id}")
 		[ "$state" == "$exp_state" ] && break
 		sleep 1
@@ -279,7 +279,7 @@ __ublk_kill_daemon()
 	daemon_pid=$(_get_ublk_daemon_pid "${dev_id}")
 	state=$(_get_ublk_dev_state "${dev_id}")
 
-	for ((j=0;j<50;j++)); do
+	for ((j=0;j<100;j++)); do
 		[ "$state" == "$exp_state" ] && break
 		kill -9 "$daemon_pid" > /dev/null 2>&1
 		sleep 1
@@ -304,7 +304,7 @@ __remove_ublk_dev_return() {
 
 	_ublk_del_dev "${dev_id}"
 	local res=$?
-	udevadm settle
+	udevadm settle --timeout=20
 	return ${res}
 }
 
Signature · 1 line
2.47.0
author
#8 lore raw

PATCH 8/9 selftests: ublk: reorganize tests into integrity and recover groups

In reply to Ming Lei

Move integrity-focused tests into new 'integrity' group:
- test_null_04.sh -> test_integrity_01.sh
- test_loop_08.sh -> test_integrity_02.sh

Move recovery-focused tests into new 'recover' group:
- test_generic_04.sh -> test_recover_01.sh
- test_generic_05.sh -> test_recover_02.sh
- test_generic_11.sh -> test_recover_03.sh
- test_generic_14.sh -> test_recover_04.sh

Update Makefile to reflect the reorganization.

---
 tools/testing/selftests/ublk/Makefile              | 14 ++++++++------
 .../ublk/{test_null_04.sh => test_integrity_01.sh} |  0
 .../ublk/{test_loop_08.sh => test_integrity_02.sh} |  0
 .../{test_generic_04.sh => test_recover_01.sh}     |  0
 .../{test_generic_05.sh => test_recover_02.sh}     |  0
 .../{test_generic_11.sh => test_recover_03.sh}     |  0
 .../{test_generic_14.sh => test_recover_04.sh}     |  0
 7 files changed, 8 insertions(+), 6 deletions(-)
 rename tools/testing/selftests/ublk/{test_null_04.sh => test_integrity_01.sh} (100%)
 rename tools/testing/selftests/ublk/{test_loop_08.sh => test_integrity_02.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_04.sh => test_recover_01.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_05.sh => test_recover_02.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_11.sh => test_recover_03.sh} (100%)
 rename tools/testing/selftests/ublk/{test_generic_14.sh => test_recover_04.sh} (100%)
tools/testing/selftests/ublk/Makefile +8 -6
index 1ceae611acb7..a62a06e13006 100644
@@ -10,18 +10,14 @@ LDLIBS += -lpthread -lm -luring
 TEST_PROGS := test_generic_01.sh
 TEST_PROGS += test_generic_02.sh
 TEST_PROGS += test_generic_03.sh
-TEST_PROGS += test_generic_04.sh
-TEST_PROGS += test_generic_05.sh
 TEST_PROGS += test_generic_06.sh
 TEST_PROGS += test_generic_07.sh
 
 TEST_PROGS += test_generic_08.sh
 TEST_PROGS += test_generic_09.sh
 TEST_PROGS += test_generic_10.sh
-TEST_PROGS += test_generic_11.sh
 TEST_PROGS += test_generic_12.sh
 TEST_PROGS += test_generic_13.sh
-TEST_PROGS += test_generic_14.sh
 TEST_PROGS += test_generic_16.sh
 
 TEST_PROGS += test_batch_01.sh
@@ -31,7 +27,6 @@ TEST_PROGS += test_batch_03.sh
 TEST_PROGS += test_null_01.sh
 TEST_PROGS += test_null_02.sh
 TEST_PROGS += test_null_03.sh
-TEST_PROGS += test_null_04.sh
 TEST_PROGS += test_loop_01.sh
 TEST_PROGS += test_loop_02.sh
 TEST_PROGS += test_loop_03.sh
@@ -39,7 +34,14 @@ TEST_PROGS += test_loop_04.sh
 TEST_PROGS += test_loop_05.sh
 TEST_PROGS += test_loop_06.sh
 TEST_PROGS += test_loop_07.sh
-TEST_PROGS += test_loop_08.sh
+
+TEST_PROGS += test_integrity_01.sh
+TEST_PROGS += test_integrity_02.sh
+
+TEST_PROGS += test_recover_01.sh
+TEST_PROGS += test_recover_02.sh
+TEST_PROGS += test_recover_03.sh
+TEST_PROGS += test_recover_04.sh
 TEST_PROGS += test_stripe_01.sh
 TEST_PROGS += test_stripe_02.sh
 TEST_PROGS += test_stripe_03.sh
tools/testing/selftests/ublk/test_integrity_01.sh renamed from tools/testing/selftests/ublk/test_null_04.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_null_04.sh
rename to tools/testing/selftests/ublk/test_integrity_01.sh
tools/testing/selftests/ublk/test_integrity_02.sh renamed from tools/testing/selftests/ublk/test_loop_08.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_loop_08.sh
rename to tools/testing/selftests/ublk/test_integrity_02.sh
tools/testing/selftests/ublk/test_recover_01.sh renamed from tools/testing/selftests/ublk/test_generic_04.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_generic_04.sh
rename to tools/testing/selftests/ublk/test_recover_01.sh
tools/testing/selftests/ublk/test_recover_02.sh renamed from tools/testing/selftests/ublk/test_generic_05.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_generic_05.sh
rename to tools/testing/selftests/ublk/test_recover_02.sh
tools/testing/selftests/ublk/test_recover_03.sh renamed from tools/testing/selftests/ublk/test_generic_11.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_generic_11.sh
rename to tools/testing/selftests/ublk/test_recover_03.sh
tools/testing/selftests/ublk/test_recover_04.sh renamed from tools/testing/selftests/ublk/test_generic_14.sh +0 -0
similarity index 100%
rename from tools/testing/selftests/ublk/test_generic_14.sh
rename to tools/testing/selftests/ublk/test_recover_04.sh
Signature · 1 line
2.47.0
author
#9 lore raw

PATCH 9/9 selftests: ublk: improve I/O ordering test with bpftrace

In reply to Ming Lei

Remove test_generic_01.sh since block layer may reorder I/O, making the test prone to false positives. Apply the improvements to test_generic_02.sh instead, which supposes for covering ublk dispatch io order.

Rework test_generic_02 to verify that ublk dispatch doesn't reorder I/O by comparing request start order with completion order using bpftrace.

The bpftrace script now:
- Tracks each request's start sequence number in a map keyed by sector
- On completion, verifies the request's start order matches expected completion order
- Reports any out-of-order completions detected

The test script:
- Wait bpftrace BEGIN code block is run
- Pins fio to CPU 0 for deterministic behavior
- Uses block_io_start and block_rq_complete tracepoints
- Checks bpftrace output for reordering errors

---
 tools/testing/selftests/ublk/Makefile         |  3 +-
 .../testing/selftests/ublk/test_generic_01.sh | 47 -------------------
 .../testing/selftests/ublk/test_generic_02.sh | 22 ++++++---
 tools/testing/selftests/ublk/trace/seq_io.bt  | 47 +++++++++++++++----
 4 files changed, 54 insertions(+), 65 deletions(-)
 delete mode 100755 tools/testing/selftests/ublk/test_generic_01.sh
tools/testing/selftests/ublk/Makefile +1 -2
index a62a06e13006..8ac2d4a682a1 100644
@@ -7,8 +7,7 @@ endif
 
 LDLIBS += -lpthread -lm -luring
 
-TEST_PROGS := test_generic_01.sh
-TEST_PROGS += test_generic_02.sh
+TEST_PROGS := test_generic_02.sh
 TEST_PROGS += test_generic_03.sh
 TEST_PROGS += test_generic_06.sh
 TEST_PROGS += test_generic_07.sh
tools/testing/selftests/ublk/test_generic_01.sh deleted +0 -47

47 lines not on this page · raw message

tools/testing/selftests/ublk/test_generic_02.sh +15 -7
index 1d4b1d6e059c..46b657143fd6 100755
@@ -13,7 +13,7 @@ if ! _have_program fio; then
 	exit "$UBLK_SKIP_CODE"
 fi
 
-_prep_test "null" "sequential io order for MQ"
+_prep_test "null" "ublk dispatch won't reorder IO for MQ"
 
 dev_id=$(_add_ublk_dev -t null -q 2)
 _check_add_dev $TID $?
@@ -21,15 +21,20 @@ _check_add_dev $TID $?
 dev_t=$(_get_disk_dev_t "$dev_id")
 bpftrace trace/seq_io.bt "$dev_t" "W" 1 > "$UBLK_TMP" 2>&1 &
 btrace_pid=$!
-sleep 2
 
-if ! kill -0 "$btrace_pid" > /dev/null 2>&1; then
+# Wait for bpftrace probes to be attached (BEGIN block prints BPFTRACE_READY)
+for _ in $(seq 100); do
+	grep -q "BPFTRACE_READY" "$UBLK_TMP" 2>/dev/null && break
+	sleep 0.1
+done
+
+if ! kill -0 "$btrace_pid" 2>/dev/null; then
 	_cleanup_test "null"
 	exit "$UBLK_SKIP_CODE"
 fi
 
-# run fio over this ublk disk
-fio --name=write_seq \
+# run fio over this ublk disk (pinned to CPU 0)
+taskset -c 0 fio --name=write_seq \
     --filename=/dev/ublkb"${dev_id}" \
     --ioengine=libaio --iodepth=16 \
     --rw=write \
@@ -39,8 +44,11 @@ fio --name=write_seq \
 ERR_CODE=$?
 kill "$btrace_pid"
 wait
-if grep -q "io_out_of_order" "$UBLK_TMP"; then
-	cat "$UBLK_TMP"
+
+# Check for out-of-order completions detected by bpftrace
+if grep -q "^out_of_order:" "$UBLK_TMP"; then
+	echo "I/O reordering detected:"
+	grep "^out_of_order:" "$UBLK_TMP"
 	ERR_CODE=255
 fi
 _cleanup_test "null"
tools/testing/selftests/ublk/trace/seq_io.bt +38 -9

61 lines not on this page · raw message

Signature · 1 line
2.47.0
maintainer Applied
#10 lore raw
In reply to Ming Lei
Quoted 5 lines · Ming Lei

This series improves the ublk selftests infrastructure with better organization and parallel test execution support.

Key improvements:

Quoted 8 lines

1. Infrastructure enhancements:
- Add helper functions for device cleanup and parallel execution
- Implement group-based test targets for running test subsets
- Track created devices for proper per-test cleanup
- Increase timeouts to accommodate parallel test execution
- Test time can be reduced to 1/3 with parallel execution by passing JOBS=8

[...]

Applied, thanks!

[1/9] selftests: ublk: simplify UBLK_TEST_DIR handling

      commit: 5af302a15a1d628a025a78892001fe8afea90c60
[2/9] selftests: ublk: refactor test_loop_08 into separate functions
      commit: 842b6520e579b8bd7d6ea09937e1fb7729cce1c5
[3/9] selftests: ublk: add _ublk_del_dev helper function
      commit: 92734a4f3a7a5449b0c7d0160ba658a2b665c31b
[4/9] selftests: ublk: track created devices for per-test cleanup
      commit: 2021e6109de3e97adfce262c40a657ff206ef495
[5/9] selftests: ublk: add group-based test targets
      commit: b6bbc3bec19efd557f888d78865b627b80b37a32
[6/9] selftests: ublk: add _ublk_sleep helper for parallel execution
      commit: 64406dd2f69fe27921c7bf06088871c002cf6186
[7/9] selftests: ublk: increase timeouts for parallel test execution
      commit: 56a08b87f9f2a763cb5546f83b78ebe1e96260af
[8/9] selftests: ublk: reorganize tests into integrity and recover groups
      commit: d9a36ab302b1c90d8f03a3b13538b8676eb6ed3b
[9/9] selftests: ublk: improve I/O ordering test with bpftrace
      commit: 5314d25afbc44d0449fa2519d2c9d7f3c319f74c

Best regards,

Signature · 1 line
Jens Axboe