selftests: ublk: test infrastructure improvements
Subsystems (3)
cover selftests: ublk: test infrastructure improvements
Hi,
This series improves the ublk selftests infrastructure with better organization and parallel test execution support.
Key improvements:
1. Infrastructure enhancements:
- Add helper functions for device cleanup and parallel execution
- Implement group-based test targets for running test subsets
- Track created devices for proper per-test cleanup
- Increase timeouts to accommodate parallel test execution
- Test time can be reduced to 1/3 with parallel execution by passing JOBS=8
2. Test organization:
- Reorganize tests into logical groups (integrity, recover)
- Refactor large tests into modular functions for maintainability
3. I/O ordering verification:
- Remove test_generic_01.sh prone to false positives from block layer reordering
- Improve test_generic_02.sh with bpftrace-based verification
- Use block_io_start/block_rq_complete tracepoints for accurate tracking
These changes make the test suite more maintainable, easier to run in parallel, and more reliable in detecting actual ublk driver issues vs. false positives.
Ming Lei (9):
selftests: ublk: simplify UBLK_TEST_DIR handling
selftests: ublk: refactor test_loop_08 into separate functions
selftests: ublk: add _ublk_del_dev helper function
selftests: ublk: track created devices for per-test cleanup
selftests: ublk: add group-based test targets
selftests: ublk: add _ublk_sleep helper for parallel execution
selftests: ublk: increase timeouts for parallel test execution
selftests: ublk: reorganize tests into integrity and recover groups
selftests: ublk: improve I/O ordering test with bpftrace
tools/testing/selftests/ublk/Makefile | 54 ++++++-
tools/testing/selftests/ublk/test_common.sh | 51 +++++--
.../testing/selftests/ublk/test_generic_01.sh | 47 ------
.../testing/selftests/ublk/test_generic_02.sh | 22 ++-
.../testing/selftests/ublk/test_generic_16.sh | 4 +-
.../{test_null_04.sh => test_integrity_01.sh} | 8 +-
.../selftests/ublk/test_integrity_02.sh | 141 ++++++++++++++++++
tools/testing/selftests/ublk/test_loop_08.sh | 110 --------------
tools/testing/selftests/ublk/test_part_02.sh | 6 +-
...{test_generic_04.sh => test_recover_01.sh} | 0
...{test_generic_05.sh => test_recover_02.sh} | 0
...{test_generic_11.sh => test_recover_03.sh} | 0
...{test_generic_14.sh => test_recover_04.sh} | 0
tools/testing/selftests/ublk/trace/seq_io.bt | 47 ++++--
14 files changed, 284 insertions(+), 206 deletions(-)
delete mode 100755 tools/testing/selftests/ublk/test_generic_01.sh
rename tools/testing/selftests/ublk/{test_null_04.sh => test_integrity_01.sh} (96%)
create mode 100755 tools/testing/selftests/ublk/test_integrity_02.sh
delete mode 100755 tools/testing/selftests/ublk/test_loop_08.sh
rename tools/testing/selftests/ublk/{test_generic_04.sh => test_recover_01.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_05.sh => test_recover_02.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_11.sh => test_recover_03.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_14.sh => test_recover_04.sh} (100%)Signature · 1 line
2.47.0
PATCH 1/9 selftests: ublk: simplify UBLK_TEST_DIR handling
Remove intermediate TDIR variable and set UBLK_TEST_DIR directly in _prep_test(). Remove default initialization since the directory is created dynamically when tests run.
--- tools/testing/selftests/ublk/test_common.sh | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
tools/testing/selftests/ublk/test_common.sh +1 -4
@@ -124,8 +124,7 @@ _prep_test() { local type=$1 shift 1 modprobe ublk_drv > /dev/null 2>&1 - TDIR=$(mktemp -d ${TMPDIR:-.}/ublktest-dir.XXXXXX) - export UBLK_TEST_DIR=${TDIR} + UBLK_TEST_DIR=$(mktemp -d ${TMPDIR:-.}/ublktest-dir.XXXXXX) UBLK_TMP=$(mktemp ${UBLK_TEST_DIR}/ublk_test_XXXXX) [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "ublk $type: $*" echo "ublk selftest: $TID starting at $(date '+%F %T')" | tee /dev/kmsg
@@ -408,8 +407,6 @@ UBLK_PROG=$(_ublk_test_top_dir)/kublk UBLK_TEST_QUIET=1 UBLK_TEST_SHOW_RESULT=1 UBLK_BACKFILES=() -UBLK_TEST_DIR=${TMPDIR:-.} export UBLK_PROG export UBLK_TEST_QUIET export UBLK_TEST_SHOW_RESULT -export UBLK_TEST_DIR
Signature · 1 line
2.47.0
PATCH 2/9 selftests: ublk: refactor test_loop_08 into separate functions
In reply to Ming LeiEncapsulate each test case in its own function for better organization and maintainability:
- _setup_device(): device and backfile initialization
- _test_fill_and_verify(): initial data population
- _test_corrupted_reftag(): reftag corruption detection test
- _test_corrupted_data(): data corruption detection test
- _test_bad_apptag(): apptag mismatch detection test
Also fix temp file creation to use ${UBLK_TEST_DIR}/fio_err_XXXXX instead of creating in current directory.
--- tools/testing/selftests/ublk/test_loop_08.sh | 199 +++++++++++-------- 1 file changed, 115 insertions(+), 84 deletions(-)
tools/testing/selftests/ublk/test_loop_08.sh +115 -84
@@ -13,98 +13,129 @@ if [[ "$fio_version" =~ fio-[0-9]+\.[0-9]+$ ]]; then exit $UBLK_SKIP_CODE fi +ERR_CODE=0 -_prep_test "loop" "end-to-end integrity" +# Global variables set during device setup +dev_id="" +fio_args="" +fio_err="" -_create_backfile 0 256M -_create_backfile 1 32M # 256M * (64 integrity bytes / 512 data bytes) -integrity_params="--integrity_capable --integrity_reftag - --metadata_size 64 --pi_offset 56 --csum_type t10dif" -dev_id=$(_add_ublk_dev -t loop -u $integrity_params "${UBLK_BACKFILES[@]}") -_check_add_dev $TID $? - -# 1M * (64 integrity bytes / 512 data bytes) = 128K -fio_args="--ioengine io_uring --direct 1 --bsrange 512-1M --iodepth 32 - --md_per_io_size 128K --pi_act 0 --pi_chk GUARD,REFTAG,APPTAG - --filename /dev/ublkb$dev_id" -fio --name fill --rw randwrite $fio_args > /dev/null -err=$? -if [ $err != 0 ]; then - echo "fio fill failed" - _show_result $TID $err -fi +_setup_device() { + _create_backfile 0 256M + _create_backfile 1 32M # 256M * (64 integrity bytes / 512 data bytes) -fio --name verify --rw randread $fio_args > /dev/null -err=$? -if [ $err != 0 ]; then - echo "fio verify failed" - _show_result $TID $err -fi + local integrity_params="--integrity_capable --integrity_reftag + --metadata_size 64 --pi_offset 56 --csum_type t10dif" + dev_id=$(_add_ublk_dev -t loop -u $integrity_params "${UBLK_BACKFILES[@]}") + _check_add_dev "$TID" $? -fio_err=$(mktemp fio_err_XXXXX) + # 1M * (64 integrity bytes / 512 data bytes) = 128K + fio_args="--ioengine io_uring --direct 1 --bsrange 512-1M --iodepth 32 + --md_per_io_size 128K --pi_act 0 --pi_chk GUARD,REFTAG,APPTAG + --filename /dev/ublkb$dev_id" -# Overwrite 4-byte reftag at offset 56 + 4 = 60 -dd_reftag_args="bs=1 seek=60 count=4 oflag=dsync conv=notrunc status=none" -dd if=/dev/urandom "of=${UBLK_BACKFILES[1]}" $dd_reftag_args -err=$? -if [ $err != 0 ]; then - echo "dd corrupted_reftag failed" - rm -f "$fio_err" - _show_result $TID $err -fi -if fio --name corrupted_reftag --rw randread $fio_args > /dev/null 2> "$fio_err"; then - echo "fio corrupted_reftag unexpectedly succeeded" - rm -f "$fio_err" - _show_result $TID 255 -fi -expected_err="REFTAG compare error: LBA: 0 Expected=0, Actual=" -if ! grep -q "$expected_err" "$fio_err"; then - echo "fio corrupted_reftag message not found: $expected_err" - rm -f "$fio_err" - _show_result $TID 255 -fi -# Reset to 0 -dd if=/dev/zero "of=${UBLK_BACKFILES[1]}" $dd_reftag_args -err=$? -if [ $err != 0 ]; then - echo "dd restore corrupted_reftag failed" - rm -f "$fio_err" - _show_result $TID $err -fi + fio_err=$(mktemp "${UBLK_TEST_DIR}"/fio_err_XXXXX) +} -dd_data_args="bs=512 count=1 oflag=direct,dsync conv=notrunc status=none" -dd if=/dev/zero "of=${UBLK_BACKFILES[0]}" $dd_data_args -err=$? -if [ $err != 0 ]; then - echo "dd corrupted_data failed" - rm -f "$fio_err" - _show_result $TID $err -fi -if fio --name corrupted_data --rw randread $fio_args > /dev/null 2> "$fio_err"; then - echo "fio corrupted_data unexpectedly succeeded" - rm -f "$fio_err" - _show_result $TID 255 -fi -expected_err="Guard compare error: LBA: 0 Expected=0, Actual=" -if ! grep -q "$expected_err" "$fio_err"; then - echo "fio corrupted_data message not found: $expected_err" - rm -f "$fio_err" - _show_result $TID 255 -fi +_test_fill_and_verify() { + fio --name fill --rw randwrite $fio_args > /dev/null + if [ $? != 0 ]; then + echo "fio fill failed" + ERR_CODE=255 + return 1 + fi -if fio --name bad_apptag --rw randread $fio_args --apptag 0x4321 > /dev/null 2> "$fio_err"; then - echo "fio bad_apptag unexpectedly succeeded" - rm -f "$fio_err" - _show_result $TID 255 -fi -expected_err="APPTAG compare error: LBA: [0-9]* Expected=4321, Actual=1234" -if ! grep -q "$expected_err" "$fio_err"; then - echo "fio bad_apptag message not found: $expected_err" - rm -f "$fio_err" - _show_result $TID 255 -fi + fio --name verify --rw randread $fio_args > /dev/null + if [ $? != 0 ]; then + echo "fio verify failed" + ERR_CODE=255 + return 1 + fi +} + +_test_corrupted_reftag() { + local dd_reftag_args="bs=1 seek=60 count=4 oflag=dsync conv=notrunc status=none" + local expected_err="REFTAG compare error: LBA: 0 Expected=0, Actual=" + + # Overwrite 4-byte reftag at offset 56 + 4 = 60 + dd if=/dev/urandom "of=${UBLK_BACKFILES[1]}" $dd_reftag_args + if [ $? != 0 ]; then + echo "dd corrupted_reftag failed" + ERR_CODE=255 + return 1 + fi + + if fio --name corrupted_reftag --rw randread $fio_args > /dev/null 2> "$fio_err"; then + echo "fio corrupted_reftag unexpectedly succeeded" + ERR_CODE=255 + return 1 + fi + + if ! grep -q "$expected_err" "$fio_err"; then + echo "fio corrupted_reftag message not found: $expected_err" + ERR_CODE=255 + return 1 + fi + + # Reset to 0 + dd if=/dev/zero "of=${UBLK_BACKFILES[1]}" $dd_reftag_args + if [ $? != 0 ]; then + echo "dd restore corrupted_reftag failed" + ERR_CODE=255 + return 1 + fi +} + +_test_corrupted_data() { + local dd_data_args="bs=512 count=1 oflag=direct,dsync conv=notrunc status=none" + local expected_err="Guard compare error: LBA: 0 Expected=0, Actual=" + + dd if=/dev/zero "of=${UBLK_BACKFILES[0]}" $dd_data_args + if [ $? != 0 ]; then + echo "dd corrupted_data failed" + ERR_CODE=255 + return 1 + fi + + if fio --name corrupted_data --rw randread $fio_args > /dev/null 2> "$fio_err"; then + echo "fio corrupted_data unexpectedly succeeded" + ERR_CODE=255 + return 1 + fi + + if ! grep -q "$expected_err" "$fio_err"; then + echo "fio corrupted_data message not found: $expected_err" + ERR_CODE=255 + return 1 + fi +} + +_test_bad_apptag() { + local expected_err="APPTAG compare error: LBA: [0-9]* Expected=4321, Actual=1234" + + if fio --name bad_apptag --rw randread $fio_args --apptag 0x4321 > /dev/null 2> "$fio_err"; then + echo "fio bad_apptag unexpectedly succeeded" + ERR_CODE=255 + return 1 + fi + + if ! grep -q "$expected_err" "$fio_err"; then + echo "fio bad_apptag message not found: $expected_err" + ERR_CODE=255 + return 1 + fi +} + +_prep_test "loop" "end-to-end integrity" + +_setup_device + +_test_fill_and_verify && \ +_test_corrupted_reftag && \ +_test_corrupted_data && \ +_test_bad_apptag rm -f "$fio_err" _cleanup_test -_show_result $TID 0 +_show_result "$TID" $ERR_CODE
Signature · 1 line
2.47.0
PATCH 3/9 selftests: ublk: add _ublk_del_dev helper function
In reply to Ming LeiAdd _ublk_del_dev() to delete a specific ublk device by ID and use it in all test scripts instead of calling UBLK_PROG directly.
Also remove unused _remove_ublk_devices() function.
--- tools/testing/selftests/ublk/test_common.sh | 13 +++++++------ tools/testing/selftests/ublk/test_generic_16.sh | 4 ++-- tools/testing/selftests/ublk/test_null_04.sh | 8 ++++---- tools/testing/selftests/ublk/test_part_02.sh | 4 ++-- 4 files changed, 15 insertions(+), 14 deletions(-)
tools/testing/selftests/ublk/test_common.sh +7 -6
@@ -106,11 +106,6 @@ _check_root() { fi } -_remove_ublk_devices() { - ${UBLK_PROG} del -a - modprobe -r ublk_drv > /dev/null 2>&1 -} - _get_ublk_dev_state() { ${UBLK_PROG} list -n "$1" | grep "state" | awk '{print $11}' }
@@ -277,10 +272,16 @@ __ublk_kill_daemon() echo "$state" } -__remove_ublk_dev_return() { +_ublk_del_dev() { local dev_id=$1 ${UBLK_PROG} del -n "${dev_id}" +} + +__remove_ublk_dev_return() { + local dev_id=$1 + + _ublk_del_dev "${dev_id}" local res=$? udevadm settle return ${res}
tools/testing/selftests/ublk/test_generic_16.sh +2 -2
@@ -24,7 +24,7 @@ if ! ${UBLK_PROG} stop -n "${dev_id}" --safe; then fi # Clean up device -${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1 +_ublk_del_dev "${dev_id}" > /dev/null 2>&1 udevadm settle # Test 2: stop --safe on device with active opener should fail
@@ -49,7 +49,7 @@ kill $dd_pid 2>/dev/null wait $dd_pid 2>/dev/null # Now device should be idle, regular delete should work -${UBLK_PROG} del -n "${dev_id}" +_ublk_del_dev "${dev_id}" udevadm settle _cleanup_test "null"
tools/testing/selftests/ublk/test_null_04.sh +4 -4
@@ -34,7 +34,7 @@ _test_metadata_only() { "$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 && _check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0 - ${UBLK_PROG} del -n "${dev_id}" + _ublk_del_dev "${dev_id}" } _test_integrity_capable_ip() {
@@ -53,7 +53,7 @@ _test_integrity_capable_ip() { "$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 && _check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0 - ${UBLK_PROG} del -n "${dev_id}" + _ublk_del_dev "${dev_id}" } _test_integrity_reftag_t10dif() {
@@ -72,7 +72,7 @@ _test_integrity_reftag_t10dif() { "$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 && _check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 0 - ${UBLK_PROG} del -n "${dev_id}" + _ublk_del_dev "${dev_id}" } _test_nvme_csum() {
@@ -91,7 +91,7 @@ _test_nvme_csum() { "$(cat "/sys/block/ublkb$dev_id/integrity/protection_interval_bytes")" 512 && _check_value "tag_size" "$(cat "/sys/block/ublkb$dev_id/integrity/tag_size")" 8 - ${UBLK_PROG} del -n "${dev_id}" + _ublk_del_dev "${dev_id}" } _prep_test "null" "integrity params"
tools/testing/selftests/ublk/test_part_02.sh +2 -2
@@ -46,13 +46,13 @@ _test_partition_scan_no_hang() if [ "$state" != "${expected_state}" ]; then echo "FAIL: Device state is $state, expected ${expected_state}" ERR_CODE=255 - ${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1 + _ublk_del_dev "${dev_id}" > /dev/null 2>&1 return fi echo "PASS: Device transitioned to ${expected_state} in ${elapsed}s without hanging" # Clean up the device - ${UBLK_PROG} del -n "${dev_id}" > /dev/null 2>&1 + _ublk_del_dev "${dev_id}" > /dev/null 2>&1 } _prep_test "partition_scan" "verify async partition scan prevents IO hang"
Signature · 1 line
2.47.0
PATCH 4/9 selftests: ublk: track created devices for per-test cleanup
In reply to Ming LeiTrack device IDs in UBLK_DEVS array when created. Update _cleanup_test() to only delete devices created by this test instead of using 'del -a' which removes all devices.
This prepares for running tests concurrently where each test should only clean up its own devices.
--- tools/testing/selftests/ublk/test_common.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
tools/testing/selftests/ublk/test_common.sh +12 -1
@@ -164,7 +164,12 @@ _check_add_dev() } _cleanup_test() { - "${UBLK_PROG}" del -a + if [ -f "${UBLK_TEST_DIR}/.ublk_devs" ]; then + while read -r dev_id; do + ${UBLK_PROG} del -n "${dev_id}" + done < "${UBLK_TEST_DIR}/.ublk_devs" + rm -f "${UBLK_TEST_DIR}/.ublk_devs" + fi _remove_files rmdir ${UBLK_TEST_DIR}
@@ -205,6 +210,7 @@ _create_ublk_dev() { fi if [[ "$dev_id" =~ ^[0-9]+$ ]]; then + echo "$dev_id" >> "${UBLK_TEST_DIR}/.ublk_devs" echo "${dev_id}" else return 255
@@ -276,6 +282,11 @@ _ublk_del_dev() { local dev_id=$1 ${UBLK_PROG} del -n "${dev_id}" + + # Remove from tracking file + if [ -f "${UBLK_TEST_DIR}/.ublk_devs" ]; then + sed -i "/^${dev_id}$/d" "${UBLK_TEST_DIR}/.ublk_devs" + fi } __remove_ublk_dev_return() {
Signature · 1 line
2.47.0
PATCH 5/9 selftests: ublk: add group-based test targets
In reply to Ming LeiAdd convenient Makefile targets for running specific test groups:
- run_generic, run_batch, run_null, run_loop, run_stripe, run_stress, etc.
- run_all for running all tests
Test groups are auto-detected from TEST_PROGS using pattern matching (test_<group>_<num>.sh -> group), and targets are generated dynamically using define/eval templates.
Supports parallel execution via JOBS variable:
- JOBS=1 (default): sequential with kselftest TAP output
- JOBS>1: parallel execution with xargs -P
Usage examples:
make run_null # Sequential execution make run_stress JOBS=4 # Parallel with 4 jobs make run_all JOBS=8 # Run all tests with 8 parallel jobs
With JOBS=8, running time of make run_all is reduced to 2m2s from 6m5s in my test VM.
--- tools/testing/selftests/ublk/Makefile | 36 +++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
tools/testing/selftests/ublk/Makefile +36 -0
@@ -72,3 +72,39 @@ $(OUTPUT)/kublk: $(filter-out $(STANDALONE_UTILS),$(wildcard *.c)) check: shellcheck -x -f gcc *.sh + +# Test groups for running subsets of tests +# JOBS=1 (default): sequential with kselftest TAP output +# JOBS>1: parallel execution with xargs -P +# Usage: make run_null JOBS=4 +JOBS ?= 1 + +# Auto-detect test groups from TEST_PROGS (test_<group>_<num>.sh -> group) +TEST_GROUPS := $(shell echo "$(TEST_PROGS)" | tr ' ' '\n' | \ + sed 's/test_\([^_]*\)_.*/\1/' | sort -u) + +# Template for group test targets +# $(1) = group name (e.g., null, generic, stress) +define RUN_GROUP +run_$(1): all + @if [ $$(JOBS) -gt 1 ]; then \ + echo $$(filter test_$(1)_%.sh,$$(TEST_PROGS)) | tr ' ' '\n' | \ + xargs -P $$(JOBS) -n1 sh -c './"$$$$0"' || true; \ + else \ + $$(call RUN_TESTS, $$(filter test_$(1)_%.sh,$$(TEST_PROGS))); \ + fi +.PHONY: run_$(1) +endef + +# Generate targets for each discovered test group +$(foreach group,$(TEST_GROUPS),$(eval $(call RUN_GROUP,$(group)))) + +# Run all tests (parallel when JOBS>1) +run_all: all + @if [ $(JOBS) -gt 1 ]; then \ + echo $(TEST_PROGS) | tr ' ' '\n' | \ + xargs -P $(JOBS) -n1 sh -c './"$$0"' || true; \ + else \ + $(call RUN_TESTS, $(TEST_PROGS)); \ + fi +.PHONY: run_all
Signature · 1 line
2.47.0
PATCH 6/9 selftests: ublk: add _ublk_sleep helper for parallel execution
In reply to Ming LeiAdd _ublk_sleep() helper function that uses different sleep times depending on whether tests run in parallel or sequential mode.
Usage: _ublk_sleep <normal_secs> <parallel_secs>
Export JOBS variable from Makefile so test scripts can detect parallel execution, and use _ublk_sleep in test_part_02.sh to handle the partition scan delay (1s normal, 5s parallel).
--- tools/testing/selftests/ublk/Makefile | 1 + tools/testing/selftests/ublk/test_common.sh | 10 ++++++++++ tools/testing/selftests/ublk/test_part_02.sh | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-)
tools/testing/selftests/ublk/Makefile +1 -0
@@ -78,6 +78,7 @@ check: # JOBS>1: parallel execution with xargs -P # Usage: make run_null JOBS=4 JOBS ?= 1 +export JOBS # Auto-detect test groups from TEST_PROGS (test_<group>_<num>.sh -> group) TEST_GROUPS := $(shell echo "$(TEST_PROGS)" | tr ' ' '\n' | \
tools/testing/selftests/ublk/test_common.sh +10 -0
@@ -15,6 +15,16 @@ _have_program() { return 1 } +# Sleep with awareness of parallel execution. +# Usage: _ublk_sleep <normal_secs> <parallel_secs> +_ublk_sleep() { + if [ "${JOBS:-1}" -gt 1 ]; then + sleep "$2" + else + sleep "$1" + fi +} + _get_disk_dev_t() { local dev_id=$1 local dev
tools/testing/selftests/ublk/test_part_02.sh +1 -1
@@ -33,7 +33,7 @@ _test_partition_scan_no_hang() # The add command should return quickly because partition scan is async. # Now sleep briefly to let the async partition scan work start and hit # the delay in the fault_inject handler. - sleep 1 + _ublk_sleep 1 5 # Kill the ublk daemon while partition scan is potentially blocked # And check state transitions properly
Signature · 1 line
2.47.0
PATCH 7/9 selftests: ublk: increase timeouts for parallel test execution
In reply to Ming LeiWhen running tests in parallel with high JOBS count (e.g., JOBS=64), the existing timeouts can be insufficient due to system load:
- Increase state wait loops from 20/50 to 100 iterations in _recover_ublk_dev(), __ublk_quiesce_dev(), and __ublk_kill_daemon() to handle slower state transitions under heavy load
- Add --timeout=20 to udevadm settle calls to prevent indefinite hangs when udev event queue is overwhelmed by rapid device creation/deletion
--- tools/testing/selftests/ublk/test_common.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
tools/testing/selftests/ublk/test_common.sh +5 -5
@@ -216,7 +216,7 @@ _create_ublk_dev() { fi if [ "$settle" = "yes" ]; then - udevadm settle + udevadm settle --timeout=20 fi if [[ "$dev_id" =~ ^[0-9]+$ ]]; then
@@ -240,7 +240,7 @@ _recover_ublk_dev() { local state dev_id=$(_create_ublk_dev "recover" "yes" "$@") - for ((j=0;j<20;j++)); do + for ((j=0;j<100;j++)); do state=$(_get_ublk_dev_state "${dev_id}") [ "$state" == "LIVE" ] && break sleep 1
@@ -260,7 +260,7 @@ __ublk_quiesce_dev() return "$state" fi - for ((j=0;j<50;j++)); do + for ((j=0;j<100;j++)); do state=$(_get_ublk_dev_state "${dev_id}") [ "$state" == "$exp_state" ] && break sleep 1
@@ -279,7 +279,7 @@ __ublk_kill_daemon() daemon_pid=$(_get_ublk_daemon_pid "${dev_id}") state=$(_get_ublk_dev_state "${dev_id}") - for ((j=0;j<50;j++)); do + for ((j=0;j<100;j++)); do [ "$state" == "$exp_state" ] && break kill -9 "$daemon_pid" > /dev/null 2>&1 sleep 1
@@ -304,7 +304,7 @@ __remove_ublk_dev_return() { _ublk_del_dev "${dev_id}" local res=$? - udevadm settle + udevadm settle --timeout=20 return ${res} }
Signature · 1 line
2.47.0
PATCH 8/9 selftests: ublk: reorganize tests into integrity and recover groups
In reply to Ming LeiMove integrity-focused tests into new 'integrity' group:
- test_null_04.sh -> test_integrity_01.sh
- test_loop_08.sh -> test_integrity_02.sh
Move recovery-focused tests into new 'recover' group:
- test_generic_04.sh -> test_recover_01.sh
- test_generic_05.sh -> test_recover_02.sh
- test_generic_11.sh -> test_recover_03.sh
- test_generic_14.sh -> test_recover_04.sh
Update Makefile to reflect the reorganization.
---
tools/testing/selftests/ublk/Makefile | 14 ++++++++------
.../ublk/{test_null_04.sh => test_integrity_01.sh} | 0
.../ublk/{test_loop_08.sh => test_integrity_02.sh} | 0
.../{test_generic_04.sh => test_recover_01.sh} | 0
.../{test_generic_05.sh => test_recover_02.sh} | 0
.../{test_generic_11.sh => test_recover_03.sh} | 0
.../{test_generic_14.sh => test_recover_04.sh} | 0
7 files changed, 8 insertions(+), 6 deletions(-)
rename tools/testing/selftests/ublk/{test_null_04.sh => test_integrity_01.sh} (100%)
rename tools/testing/selftests/ublk/{test_loop_08.sh => test_integrity_02.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_04.sh => test_recover_01.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_05.sh => test_recover_02.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_11.sh => test_recover_03.sh} (100%)
rename tools/testing/selftests/ublk/{test_generic_14.sh => test_recover_04.sh} (100%)tools/testing/selftests/ublk/Makefile +8 -6
@@ -10,18 +10,14 @@ LDLIBS += -lpthread -lm -luring TEST_PROGS := test_generic_01.sh TEST_PROGS += test_generic_02.sh TEST_PROGS += test_generic_03.sh -TEST_PROGS += test_generic_04.sh -TEST_PROGS += test_generic_05.sh TEST_PROGS += test_generic_06.sh TEST_PROGS += test_generic_07.sh TEST_PROGS += test_generic_08.sh TEST_PROGS += test_generic_09.sh TEST_PROGS += test_generic_10.sh -TEST_PROGS += test_generic_11.sh TEST_PROGS += test_generic_12.sh TEST_PROGS += test_generic_13.sh -TEST_PROGS += test_generic_14.sh TEST_PROGS += test_generic_16.sh TEST_PROGS += test_batch_01.sh
@@ -31,7 +27,6 @@ TEST_PROGS += test_batch_03.sh TEST_PROGS += test_null_01.sh TEST_PROGS += test_null_02.sh TEST_PROGS += test_null_03.sh -TEST_PROGS += test_null_04.sh TEST_PROGS += test_loop_01.sh TEST_PROGS += test_loop_02.sh TEST_PROGS += test_loop_03.sh
@@ -39,7 +34,14 @@ TEST_PROGS += test_loop_04.sh TEST_PROGS += test_loop_05.sh TEST_PROGS += test_loop_06.sh TEST_PROGS += test_loop_07.sh -TEST_PROGS += test_loop_08.sh + +TEST_PROGS += test_integrity_01.sh +TEST_PROGS += test_integrity_02.sh + +TEST_PROGS += test_recover_01.sh +TEST_PROGS += test_recover_02.sh +TEST_PROGS += test_recover_03.sh +TEST_PROGS += test_recover_04.sh TEST_PROGS += test_stripe_01.sh TEST_PROGS += test_stripe_02.sh TEST_PROGS += test_stripe_03.sh
tools/testing/selftests/ublk/test_integrity_01.sh renamed from tools/testing/selftests/ublk/test_null_04.sh +0 -0
tools/testing/selftests/ublk/test_integrity_02.sh renamed from tools/testing/selftests/ublk/test_loop_08.sh +0 -0
tools/testing/selftests/ublk/test_recover_01.sh renamed from tools/testing/selftests/ublk/test_generic_04.sh +0 -0
tools/testing/selftests/ublk/test_recover_02.sh renamed from tools/testing/selftests/ublk/test_generic_05.sh +0 -0
tools/testing/selftests/ublk/test_recover_03.sh renamed from tools/testing/selftests/ublk/test_generic_11.sh +0 -0
tools/testing/selftests/ublk/test_recover_04.sh renamed from tools/testing/selftests/ublk/test_generic_14.sh +0 -0
Signature · 1 line
2.47.0
PATCH 9/9 selftests: ublk: improve I/O ordering test with bpftrace
In reply to Ming LeiRemove test_generic_01.sh since block layer may reorder I/O, making the test prone to false positives. Apply the improvements to test_generic_02.sh instead, which supposes for covering ublk dispatch io order.
Rework test_generic_02 to verify that ublk dispatch doesn't reorder I/O by comparing request start order with completion order using bpftrace.
The bpftrace script now:
- Tracks each request's start sequence number in a map keyed by sector
- On completion, verifies the request's start order matches expected completion order
- Reports any out-of-order completions detected
The test script:
- Wait bpftrace BEGIN code block is run
- Pins fio to CPU 0 for deterministic behavior
- Uses block_io_start and block_rq_complete tracepoints
- Checks bpftrace output for reordering errors
--- tools/testing/selftests/ublk/Makefile | 3 +- .../testing/selftests/ublk/test_generic_01.sh | 47 ------------------- .../testing/selftests/ublk/test_generic_02.sh | 22 ++++++--- tools/testing/selftests/ublk/trace/seq_io.bt | 47 +++++++++++++++---- 4 files changed, 54 insertions(+), 65 deletions(-) delete mode 100755 tools/testing/selftests/ublk/test_generic_01.sh
tools/testing/selftests/ublk/Makefile +1 -2
@@ -7,8 +7,7 @@ endif LDLIBS += -lpthread -lm -luring -TEST_PROGS := test_generic_01.sh -TEST_PROGS += test_generic_02.sh +TEST_PROGS := test_generic_02.sh TEST_PROGS += test_generic_03.sh TEST_PROGS += test_generic_06.sh TEST_PROGS += test_generic_07.sh
tools/testing/selftests/ublk/test_generic_01.sh deleted +0 -47
47 lines not on this page · raw message
tools/testing/selftests/ublk/test_generic_02.sh +15 -7
@@ -13,7 +13,7 @@ if ! _have_program fio; then exit "$UBLK_SKIP_CODE" fi -_prep_test "null" "sequential io order for MQ" +_prep_test "null" "ublk dispatch won't reorder IO for MQ" dev_id=$(_add_ublk_dev -t null -q 2) _check_add_dev $TID $?
@@ -21,15 +21,20 @@ _check_add_dev $TID $? dev_t=$(_get_disk_dev_t "$dev_id") bpftrace trace/seq_io.bt "$dev_t" "W" 1 > "$UBLK_TMP" 2>&1 & btrace_pid=$! -sleep 2 -if ! kill -0 "$btrace_pid" > /dev/null 2>&1; then +# Wait for bpftrace probes to be attached (BEGIN block prints BPFTRACE_READY) +for _ in $(seq 100); do + grep -q "BPFTRACE_READY" "$UBLK_TMP" 2>/dev/null && break + sleep 0.1 +done + +if ! kill -0 "$btrace_pid" 2>/dev/null; then _cleanup_test "null" exit "$UBLK_SKIP_CODE" fi -# run fio over this ublk disk -fio --name=write_seq \ +# run fio over this ublk disk (pinned to CPU 0) +taskset -c 0 fio --name=write_seq \ --filename=/dev/ublkb"${dev_id}" \ --ioengine=libaio --iodepth=16 \ --rw=write \
@@ -39,8 +44,11 @@ fio --name=write_seq \ ERR_CODE=$? kill "$btrace_pid" wait -if grep -q "io_out_of_order" "$UBLK_TMP"; then - cat "$UBLK_TMP" + +# Check for out-of-order completions detected by bpftrace +if grep -q "^out_of_order:" "$UBLK_TMP"; then + echo "I/O reordering detected:" + grep "^out_of_order:" "$UBLK_TMP" ERR_CODE=255 fi _cleanup_test "null"
tools/testing/selftests/ublk/trace/seq_io.bt +38 -9
61 lines not on this page · raw message
Signature · 1 line
2.47.0
Quoted 5 lines · Ming Lei
This series improves the ublk selftests infrastructure with better organization and parallel test execution support.
Key improvements:
Quoted 8 lines
1. Infrastructure enhancements:
- Add helper functions for device cleanup and parallel execution
- Implement group-based test targets for running test subsets
- Track created devices for proper per-test cleanup
- Increase timeouts to accommodate parallel test execution
- Test time can be reduced to 1/3 with parallel execution by passing JOBS=8[...]
Applied, thanks!
[1/9] selftests: ublk: simplify UBLK_TEST_DIR handling
commit: 5af302a15a1d628a025a78892001fe8afea90c60
[2/9] selftests: ublk: refactor test_loop_08 into separate functions
commit: 842b6520e579b8bd7d6ea09937e1fb7729cce1c5
[3/9] selftests: ublk: add _ublk_del_dev helper function
commit: 92734a4f3a7a5449b0c7d0160ba658a2b665c31b
[4/9] selftests: ublk: track created devices for per-test cleanup
commit: 2021e6109de3e97adfce262c40a657ff206ef495
[5/9] selftests: ublk: add group-based test targets
commit: b6bbc3bec19efd557f888d78865b627b80b37a32
[6/9] selftests: ublk: add _ublk_sleep helper for parallel execution
commit: 64406dd2f69fe27921c7bf06088871c002cf6186
[7/9] selftests: ublk: increase timeouts for parallel test execution
commit: 56a08b87f9f2a763cb5546f83b78ebe1e96260af
[8/9] selftests: ublk: reorganize tests into integrity and recover groups
commit: d9a36ab302b1c90d8f03a3b13538b8676eb6ed3b
[9/9] selftests: ublk: improve I/O ordering test with bpftrace
commit: 5314d25afbc44d0449fa2519d2c9d7f3c319f74cBest regards,
Signature · 1 line
Jens Axboe