selftests: ublk: test cleanup & add more tests
Subsystems (3)
v2 cover selftests: ublk: test cleanup & add more tests
Hello Jens,
This patchset cleans up ublk selftests and add more tests:
- two bug fixes(1, 2)
- cleanup (3, 4)
- allow to run tests in parallel(5), also big simplification on test script
- add two stress tests for zero copy(6)
- kublk misc change(7, 8, 9), helps for evaluating performance
- support target specific command line, so help to add new target(Uday is working on fault-inject target) (10)
- add two tests for covering recovery features(11)
- add one heavy io & remove test over recovery enabled device(12), which can catch io hang triggered by several recent patches.
- the last patch is for making sure ublk temp file is cleaned up if test is skipped
With this change, kernel built-in ublk selftests can :
- cover almost all tests done by ublksrv 'make test T=generic', which has been effective to capture driver issue early, so it will make ublk driver development more efficiently
- add more stress tests for covering ublk zc feature, which has found one kernel panic issue introduced recently, fix merged already
- help to add new tests, such as per-target command line, which will help to write fault-inject target
Thanks,
V2:
- use ARRAY_SIZE() (Johannes Thumshirn)
- drop one driver bug fix
- fix ublk temp file cleanup
- improve document
Ming Lei (13):
selftests: ublk: fix ublk_find_tgt()
selftests: ublk: add io_uring uapi header
selftests: ublk: cleanup backfile automatically
selftests: ublk: make sure _add_ublk_dev can return in sub-shell
selftests: ublk: run stress tests in parallel
selftests: ublk: add two stress tests for zero copy feature
selftests: ublk: setup ring with
IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN
selftests: ublk: set queue pthread's cpu affinity
selftests: ublk: increase max nr_queues and queue depth
selftests: ublk: support target specific command line
selftests: ublk: support user recovery
selftests: ublk: add test_stress_05.sh
selftests: ublk: move creating UBLK_TMP into _prep_test()
tools/testing/selftests/ublk/Makefile | 5 + tools/testing/selftests/ublk/kublk.c | 342 ++++++++++++++++-- tools/testing/selftests/ublk/kublk.h | 39 +- tools/testing/selftests/ublk/stripe.c | 28 +- tools/testing/selftests/ublk/test_common.sh | 142 ++++++-- .../testing/selftests/ublk/test_generic_04.sh | 40 ++ .../testing/selftests/ublk/test_generic_05.sh | 44 +++ tools/testing/selftests/ublk/test_loop_01.sh | 8 +- tools/testing/selftests/ublk/test_loop_02.sh | 8 +- tools/testing/selftests/ublk/test_loop_03.sh | 8 +- tools/testing/selftests/ublk/test_loop_04.sh | 9 +- tools/testing/selftests/ublk/test_loop_05.sh | 8 +- .../testing/selftests/ublk/test_stress_01.sh | 45 +-- .../testing/selftests/ublk/test_stress_02.sh | 45 +-- .../testing/selftests/ublk/test_stress_03.sh | 38 ++ .../testing/selftests/ublk/test_stress_04.sh | 37 ++ .../testing/selftests/ublk/test_stress_05.sh | 64 ++++ .../testing/selftests/ublk/test_stripe_01.sh | 12 +- .../testing/selftests/ublk/test_stripe_02.sh | 13 +- .../testing/selftests/ublk/test_stripe_03.sh | 12 +- .../testing/selftests/ublk/test_stripe_04.sh | 13 +- 21 files changed, 786 insertions(+), 174 deletions(-) create mode 100755 tools/testing/selftests/ublk/test_generic_04.sh create mode 100755 tools/testing/selftests/ublk/test_generic_05.sh create mode 100755 tools/testing/selftests/ublk/test_stress_03.sh create mode 100755 tools/testing/selftests/ublk/test_stress_04.sh create mode 100755 tools/testing/selftests/ublk/test_stress_05.sh
Signature · 1 line
2.47.0
PATCH v2 1/13 selftests: ublk: fix ublk_find_tgt()
Bounds check for iterator variable i is missed, so add it and fix ublk_find_tgt().
--- tools/testing/selftests/ublk/kublk.c | 3 +-- tools/testing/selftests/ublk/kublk.h | 2 ++ 2 files changed, 3 insertions(+), 2 deletions(-)
tools/testing/selftests/ublk/kublk.c +1 -2
@@ -14,13 +14,12 @@ static const struct ublk_tgt_ops *tgt_ops_list[] = {static const struct ublk_tgt_ops *ublk_find_tgt (const char *name) { -const struct ublk_tgt_ops *ops;int i;if (name ==NULL )return NULL ; -for (i =0 ;sizeof (tgt_ops_list) /sizeof (ops); i++) +for (i =0 ; i <ARRAY_SIZE (tgt_ops_list); i++)if (strcmp (tgt_ops_list[i]->name, name) ==0 )return tgt_ops_list[i];return NULL ;
tools/testing/selftests/ublk/kublk.h +2 -0
@@ -30,6 +30,8 @@#define min (a, b) ((a) < (b) ? (a) : (b))#endif +#define ARRAY_SIZE (x) (sizeof(x) / sizeof(x[0])) +/****************** part 1: libublk ********************/ #define CTRL_DEV "/dev/ublk-control"
Signature · 1 line
2.47.0
PATCH v2 2/13 selftests: ublk: add io_uring uapi header
In reply to Ming LeiAdd io_uring UAPI header so that ublk can work with latest uapi definition.
Fix the following build failure:
stripe.c: In function ‘stripe_to_uring_op’:
stripe.c:120:29: error: ‘IORING_OP_READV_FIXED’ undeclared (first use in this function); did you mean ‘IORING_OP_READ_FIXED’?
120 | return zc ? IORING_OP_READV_FIXED : IORING_OP_READV;
| ^~~~~~~~~~~~~~~~~~~~~
| IORING_OP_READ_FIXED--- tools/testing/selftests/ublk/kublk.h | 1 + 1 file changed, 1 insertion(+)
tools/testing/selftests/ublk/kublk.h +1 -0
@@ -20,6 +20,7 @@#include <sys/wait.h> #include <sys/eventfd.h> #include <sys/uio.h> +#include <linux/io_uring.h> #include <liburing.h> #include <linux/ublk_cmd.h> #include "ublk_dep.h"
Signature · 1 line
2.47.0
PATCH v2 3/13 selftests: ublk: cleanup backfile automatically
In reply to Ming LeiUse global array of $UBLK_BACKFILES for storing all backfile name, then clean them automatically.
--- tools/testing/selftests/ublk/test_common.sh | 36 ++++++++++++------- tools/testing/selftests/ublk/test_loop_01.sh | 8 ++--- tools/testing/selftests/ublk/test_loop_02.sh | 8 ++--- tools/testing/selftests/ublk/test_loop_03.sh | 8 ++--- tools/testing/selftests/ublk/test_loop_04.sh | 9 +++-- tools/testing/selftests/ublk/test_loop_05.sh | 8 ++--- .../testing/selftests/ublk/test_stress_01.sh | 16 ++++----- .../testing/selftests/ublk/test_stress_02.sh | 16 ++++----- .../testing/selftests/ublk/test_stripe_01.sh | 12 +++---- .../testing/selftests/ublk/test_stripe_02.sh | 13 +++---- .../testing/selftests/ublk/test_stripe_03.sh | 12 +++---- .../testing/selftests/ublk/test_stripe_04.sh | 13 +++---- 12 files changed, 70 insertions(+), 89 deletions(-)
tools/testing/selftests/ublk/test_common.sh +24 -12
@@ -30,18 +30,26 @@ _run_fio_verify_io() { } _create_backfile() { - local my_size=$1 - local my_file + local index=$1 + local new_size=$2 + local old_file + local new_file - my_file=$(mktemp ublk_file_"${my_size}"_XXXXX) - truncate -s "${my_size}" "${my_file}" - echo "$my_file" + old_file="${UBLK_BACKFILES[$index]}" + [ -f "$old_file" ] && rm -f "$old_file" + + new_file=$(mktemp ublk_file_"${new_size}"_XXXXX) + truncate -s "${new_size}" "${new_file}" + UBLK_BACKFILES["$index"]="$new_file" } -_remove_backfile() { - local file=$1 +_remove_files() { + local file - [ -f "$file" ] && rm -f "$file" + for file in "${UBLK_BACKFILES[@]}"; do + [ -f "$file" ] && rm -f "$file" + done + [ -f "$UBLK_TMP" ] && rm -f "$UBLK_TMP" } _create_tmp_dir() {
@@ -129,7 +137,10 @@ _show_result() echo "$1 : [FAIL]" fi fi - [ "$2" -ne 0 ] && exit "$2" + if [ "$2" -ne 0 ]; then + _remove_files + exit "$2" + fi return 0 }
@@ -138,16 +149,16 @@ _check_add_dev() { local tid=$1 local code=$2 - shift 2 + if [ "${code}" -ne 0 ]; then - _remove_test_files "$@" _show_result "${tid}" "${code}" fi } _cleanup_test() { "${UBLK_PROG}" del -a - rm -f "$UBLK_TMP" + + _remove_files } _have_feature()
@@ -247,6 +258,7 @@ UBLK_TMP=$(mktemp ublk_test_XXXXX) UBLK_PROG=$(_ublk_test_top_dir)/kublk UBLK_TEST_QUIET=1 UBLK_TEST_SHOW_RESULT=1 +UBLK_BACKFILES=() export UBLK_PROG export UBLK_TEST_QUIET export UBLK_TEST_SHOW_RESULT
tools/testing/selftests/ublk/test_loop_01.sh +3 -5
@@ -12,10 +12,10 @@ fi _prep_test "loop" "write and verify test" -backfile_0=$(_create_backfile 256M) +_create_backfile 0 256M -dev_id=$(_add_ublk_dev -t loop "$backfile_0") -_check_add_dev $TID $? "${backfile_0}" +dev_id=$(_add_ublk_dev -t loop "${UBLK_BACKFILES[0]}") +_check_add_dev $TID $? # run fio over the ublk disk _run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -23,6 +23,4 @@ ERR_CODE=$? _cleanup_test "loop" -_remove_backfile "$backfile_0" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_loop_02.sh +3 -5
@@ -8,15 +8,13 @@ ERR_CODE=0 _prep_test "loop" "mkfs & mount & umount" -backfile_0=$(_create_backfile 256M) -dev_id=$(_add_ublk_dev -t loop "$backfile_0") -_check_add_dev $TID $? "$backfile_0" +_create_backfile 0 256M +dev_id=$(_add_ublk_dev -t loop "${UBLK_BACKFILES[0]}") +_check_add_dev $TID $? _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? _cleanup_test "loop" -_remove_backfile "$backfile_0" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_loop_03.sh +3 -5
@@ -12,9 +12,9 @@ fi _prep_test "loop" "write and verify over zero copy" -backfile_0=$(_create_backfile 256M) -dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") -_check_add_dev $TID $? "$backfile_0" +_create_backfile 0 256M +dev_id=$(_add_ublk_dev -t loop -z "${UBLK_BACKFILES[0]}") +_check_add_dev $TID $? # run fio over the ublk disk _run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -22,6 +22,4 @@ ERR_CODE=$? _cleanup_test "loop" -_remove_backfile "$backfile_0" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_loop_04.sh +4 -5
@@ -8,15 +8,14 @@ ERR_CODE=0 _prep_test "loop" "mkfs & mount & umount with zero copy" -backfile_0=$(_create_backfile 256M) -dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") -_check_add_dev $TID $? "$backfile_0" +_create_backfile 0 256M + +dev_id=$(_add_ublk_dev -t loop -z "${UBLK_BACKFILES[0]}") +_check_add_dev $TID $? _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? _cleanup_test "loop" -_remove_backfile "$backfile_0" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_loop_05.sh +3 -5
@@ -12,10 +12,10 @@ fi _prep_test "loop" "write and verify test" -backfile_0=$(_create_backfile 256M) +_create_backfile 0 256M -dev_id=$(_add_ublk_dev -q 2 -t loop "$backfile_0") -_check_add_dev $TID $? "${backfile_0}" +dev_id=$(_add_ublk_dev -q 2 -t loop "${UBLK_BACKFILES[0]}") +_check_add_dev $TID $? # run fio over the ublk disk _run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -23,6 +23,4 @@ ERR_CODE=$? _cleanup_test "loop" -_remove_backfile "$backfile_0" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stress_01.sh +6 -10
@@ -10,17 +10,13 @@ ublk_io_and_remove() { local size=$1 shift 1 - local backfile="" - if echo "$@" | grep -q "loop"; then - backfile=${*: -1} - fi + DEV_ID=$(_add_ublk_dev "$@") - _check_add_dev $TID $? "${backfile}" + _check_add_dev $TID $? [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)" if ! __run_io_and_remove "${DEV_ID}" "${size}" "no"; then echo "/dev/ublkc${DEV_ID} isn't removed" - _remove_backfile "${backfile}" exit 255 fi }
@@ -33,15 +29,15 @@ if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -BACK_FILE=$(_create_backfile 256M) -ublk_io_and_remove 256M -t loop -q 4 "${BACK_FILE}" +_create_backfile 0 256M + +ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -ublk_io_and_remove 256M -t loop -q 4 -z "${BACK_FILE}" +ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" ERR_CODE=$? _cleanup_test "stress" -_remove_backfile "${BACK_FILE}" _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stress_02.sh +6 -10
@@ -10,17 +10,13 @@ ublk_io_and_kill_daemon() { local size=$1 shift 1 - local backfile="" - if echo "$@" | grep -q "loop"; then - backfile=${*: -1} - fi + DEV_ID=$(_add_ublk_dev "$@") - _check_add_dev $TID $? "${backfile}" + _check_add_dev $TID $? [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)" if ! __run_io_and_remove "${DEV_ID}" "${size}" "yes"; then echo "/dev/ublkc${DEV_ID} isn't removed res ${res}" - _remove_backfile "${backfile}" exit 255 fi }
@@ -33,15 +29,15 @@ if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -BACK_FILE=$(_create_backfile 256M) -ublk_io_and_kill_daemon 256M -t loop -q 4 "${BACK_FILE}" +_create_backfile 0 256M + +ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${BACK_FILE}" +ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" ERR_CODE=$? _cleanup_test "stress" -_remove_backfile "${BACK_FILE}" _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stripe_01.sh +4 -8
@@ -12,19 +12,15 @@ fi _prep_test "stripe" "write and verify test" -backfile_0=$(_create_backfile 256M) -backfile_1=$(_create_backfile 256M) +_create_backfile 0 256M +_create_backfile 1 256M -dev_id=$(_add_ublk_dev -t stripe "$backfile_0" "$backfile_1") -_check_add_dev $TID $? "${backfile_0}" +dev_id=$(_add_ublk_dev -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}") +_check_add_dev $TID $? # run fio over the ublk disk _run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M ERR_CODE=$? _cleanup_test "stripe" - -_remove_backfile "$backfile_0" -_remove_backfile "$backfile_1" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stripe_02.sh +5 -8
@@ -8,17 +8,14 @@ ERR_CODE=0 _prep_test "stripe" "mkfs & mount & umount" -backfile_0=$(_create_backfile 256M) -backfile_1=$(_create_backfile 256M) -dev_id=$(_add_ublk_dev -t stripe "$backfile_0" "$backfile_1") -_check_add_dev $TID $? "$backfile_0" "$backfile_1" +_create_backfile 0 256M +_create_backfile 1 256M + +dev_id=$(_add_ublk_dev -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}") +_check_add_dev $TID $? _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? _cleanup_test "stripe" - -_remove_backfile "$backfile_0" -_remove_backfile "$backfile_1" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stripe_03.sh +4 -8
@@ -12,19 +12,15 @@ fi _prep_test "stripe" "write and verify test" -backfile_0=$(_create_backfile 256M) -backfile_1=$(_create_backfile 256M) +_create_backfile 0 256M +_create_backfile 1 256M -dev_id=$(_add_ublk_dev -q 2 -t stripe "$backfile_0" "$backfile_1") -_check_add_dev $TID $? "${backfile_0}" +dev_id=$(_add_ublk_dev -q 2 -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}") +_check_add_dev $TID $? # run fio over the ublk disk _run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M ERR_CODE=$? _cleanup_test "stripe" - -_remove_backfile "$backfile_0" -_remove_backfile "$backfile_1" - _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stripe_04.sh +5 -8
@@ -8,17 +8,14 @@ ERR_CODE=0 _prep_test "stripe" "mkfs & mount & umount on zero copy" -backfile_0=$(_create_backfile 256M) -backfile_1=$(_create_backfile 256M) -dev_id=$(_add_ublk_dev -t stripe -z -q 2 "$backfile_0" "$backfile_1") -_check_add_dev $TID $? "$backfile_0" "$backfile_1" +_create_backfile 0 256M +_create_backfile 1 256M + +dev_id=$(_add_ublk_dev -t stripe -z -q 2 "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}") +_check_add_dev $TID $? _mkfs_mount_test /dev/ublkb"${dev_id}" ERR_CODE=$? _cleanup_test "stripe" - -_remove_backfile "$backfile_0" -_remove_backfile "$backfile_1" - _show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
PATCH v2 4/13 selftests: ublk: make sure _add_ublk_dev can return in sub-shell
In reply to Ming LeiDetach ublk daemon from the starting process completely by double-fork and clearing its process group, so that _add_ublk_dev can return from sub-shell.
Then it is more friendly for writing shell test script for adding/recovering ublk device.
Prepare for running ublk test in parallel.
--- tools/testing/selftests/ublk/kublk.c | 30 +++++++++++++++---- tools/testing/selftests/ublk/test_common.sh | 15 +++++----- .../testing/selftests/ublk/test_stress_01.sh | 8 ++--- .../testing/selftests/ublk/test_stress_02.sh | 8 ++--- 4 files changed, 39 insertions(+), 22 deletions(-)
tools/testing/selftests/ublk/kublk.c +24 -6
@@ -654,6 +654,8 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)if (write (evtfd, &id,sizeof (id)) !=sizeof (id))return -EINVAL ; +close (evtfd); +return 0 ; }
@@ -889,24 +891,40 @@ static int cmd_dev_add(struct dev_ctx *ctx)exit (-1 ); } -setsid (); res =fork ();if (res ==0 ) { +int res2; + +setsid (); + res2 =fork (); +if (res2 ==0 ) { +/* prepare for detaching */ +close (STDIN_FILENO ); +close (STDOUT_FILENO ); +close (STDERR_FILENO ); run: - res =__cmd_dev_add (ctx); -return res; + res =__cmd_dev_add (ctx); +return res; + }else { +/* detached from the foreground task */ +exit (EXIT_SUCCESS ); + } }else if (res >0 ) {uint64_t id; +int exit_code =EXIT_FAILURE ; res =read (ctx->_evtfd, &id,sizeof (id));close (ctx->_evtfd);if (res ==sizeof (id) && id !=ERROR_EVTFD_DEVID ) { ctx->dev_id = id -1 ; -return __cmd_dev_list (ctx); +if (__cmd_dev_list (ctx) >=0 ) + exit_code =EXIT_SUCCESS ; } -exit (EXIT_FAILURE ); +/* wait for child and detach from it */ +wait (NULL ); +exit (exit_code); }else { -return res; +exit (EXIT_FAILURE ); } }
tools/testing/selftests/ublk/test_common.sh +7 -8
@@ -170,7 +170,6 @@ _have_feature() } _add_ublk_dev() { - local kublk_temp; local dev_id; if [ ! -c /dev/ublk-control ]; then
@@ -182,17 +181,17 @@ _add_ublk_dev() { fi fi - kublk_temp=$(mktemp /tmp/kublk-XXXXXX) - if ! "${UBLK_PROG}" add "$@" > "${kublk_temp}" 2>&1; then + if ! dev_id=$("${UBLK_PROG}" add "$@" | grep "dev id" | awk -F '[ :]' '{print $3}'); then echo "fail to add ublk dev $*" - rm -f "${kublk_temp}" return 255 fi - - dev_id=$(grep "dev id" "${kublk_temp}" | awk -F '[ :]' '{print $3}') udevadm settle - rm -f "${kublk_temp}" - echo "${dev_id}" + + if [[ "$dev_id" =~ ^[0-9]+$ ]]; then + echo "${dev_id}" + else + return 255 + fi } # kill the ublk daemon and return ublk device state
tools/testing/selftests/ublk/test_stress_01.sh +4 -4
@@ -4,19 +4,19 @@ . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh TID="stress_01" ERR_CODE=0 -DEV_ID=-1 ublk_io_and_remove() { local size=$1 + local dev_id shift 1 - DEV_ID=$(_add_ublk_dev "$@") + dev_id=$(_add_ublk_dev "$@") _check_add_dev $TID $? [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)" - if ! __run_io_and_remove "${DEV_ID}" "${size}" "no"; then - echo "/dev/ublkc${DEV_ID} isn't removed" + if ! __run_io_and_remove "$dev_id" "${size}" "no"; then + echo "/dev/ublkc$dev_id isn't removed" exit 255 fi }
tools/testing/selftests/ublk/test_stress_02.sh +4 -4
@@ -4,19 +4,19 @@ . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh TID="stress_02" ERR_CODE=0 -DEV_ID=-1 ublk_io_and_kill_daemon() { local size=$1 + local dev_id shift 1 - DEV_ID=$(_add_ublk_dev "$@") + dev_id=$(_add_ublk_dev "$@") _check_add_dev $TID $? [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)" - if ! __run_io_and_remove "${DEV_ID}" "${size}" "yes"; then - echo "/dev/ublkc${DEV_ID} isn't removed res ${res}" + if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then + echo "/dev/ublkc$dev_id isn't removed res ${res}" exit 255 fi }
Signature · 1 line
2.47.0
PATCH v2 5/13 selftests: ublk: run stress tests in parallel
In reply to Ming LeiRun stress tests in parallel, meantime add shell local function to simplify the two stress tests.
--- tools/testing/selftests/ublk/test_common.sh | 34 +++++++++++++++- .../testing/selftests/ublk/test_stress_01.sh | 39 +++++++------------ .../testing/selftests/ublk/test_stress_02.sh | 39 +++++++------------ 3 files changed, 63 insertions(+), 49 deletions(-)
tools/testing/selftests/ublk/test_common.sh +33 -1
@@ -230,7 +230,7 @@ __run_io_and_remove() local kill_server=$3 fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \ - --rw=readwrite --iodepth=64 --size="${size}" --numjobs=4 \ + --rw=readwrite --iodepth=256 --size="${size}" --numjobs=4 \ --runtime=20 --time_based > /dev/null 2>&1 & sleep 2 if [ "${kill_server}" = "yes" ]; then
@@ -248,6 +248,38 @@ __run_io_and_remove() wait } +run_io_and_remove() +{ + local size=$1 + local dev_id + shift 1 + + dev_id=$(_add_ublk_dev "$@") + _check_add_dev "$TID" $? + + [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)" + if ! __run_io_and_remove "$dev_id" "${size}" "no"; then + echo "/dev/ublkc$dev_id isn't removed" + exit 255 + fi +} + +run_io_and_kill_daemon() +{ + local size=$1 + local dev_id + shift 1 + + dev_id=$(_add_ublk_dev "$@") + _check_add_dev "$TID" $? + + [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)" + if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then + echo "/dev/ublkc$dev_id isn't removed res ${res}" + exit 255 + fi +} + _ublk_test_top_dir() { cd "$(dirname "$0")" && pwd
tools/testing/selftests/ublk/test_stress_01.sh +15 -24
@@ -7,37 +7,28 @@ ERR_CODE=0 ublk_io_and_remove() { - local size=$1 - local dev_id - shift 1 - - dev_id=$(_add_ublk_dev "$@") - _check_add_dev $TID $? - - [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)" - if ! __run_io_and_remove "$dev_id" "${size}" "no"; then - echo "/dev/ublkc$dev_id isn't removed" - exit 255 + run_io_and_remove "$@" + ERR_CODE=$? + if [ ${ERR_CODE} -ne 0 ]; then + echo "$TID failure: $*" + _show_result $TID $ERR_CODE fi } -_prep_test "stress" "run IO and remove device" - -ublk_io_and_remove 8G -t null -q 4 -ERR_CODE=$? -if [ ${ERR_CODE} -ne 0 ]; then - _show_result $TID $ERR_CODE +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" fi +_prep_test "stress" "run IO and remove device" + _create_backfile 0 256M +_create_backfile 1 128M +_create_backfile 2 128M -ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" -ERR_CODE=$? -if [ ${ERR_CODE} -ne 0 ]; then - _show_result $TID $ERR_CODE -fi +ublk_io_and_remove 8G -t null -q 4 & +ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" & +ublk_io_and_remove 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait -ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" -ERR_CODE=$? _cleanup_test "stress" _show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stress_02.sh +15 -24
@@ -5,39 +5,30 @@ TID="stress_02" ERR_CODE=0 +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + ublk_io_and_kill_daemon() { - local size=$1 - local dev_id - shift 1 - - dev_id=$(_add_ublk_dev "$@") - _check_add_dev $TID $? - - [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)" - if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then - echo "/dev/ublkc$dev_id isn't removed res ${res}" - exit 255 + run_io_and_kill_daemon "$@" + ERR_CODE=$? + if [ ${ERR_CODE} -ne 0 ]; then + echo "$TID failure: $*" + _show_result $TID $ERR_CODE fi } _prep_test "stress" "run IO and kill ublk server" -ublk_io_and_kill_daemon 8G -t null -q 4 -ERR_CODE=$? -if [ ${ERR_CODE} -ne 0 ]; then - _show_result $TID $ERR_CODE -fi - _create_backfile 0 256M +_create_backfile 1 128M +_create_backfile 2 128M -ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" -ERR_CODE=$? -if [ ${ERR_CODE} -ne 0 ]; then - _show_result $TID $ERR_CODE -fi +ublk_io_and_kill_daemon 8G -t null -q 4 & +ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" & +ublk_io_and_kill_daemon 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait -ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" -ERR_CODE=$? _cleanup_test "stress" _show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
PATCH v2 6/13 selftests: ublk: add two stress tests for zero copy feature
In reply to Ming LeiAdd stress_03 & stress_04 for covering zero copy feature.
--- tools/testing/selftests/ublk/Makefile | 2 + .../testing/selftests/ublk/test_stress_03.sh | 38 +++++++++++++++++++ .../testing/selftests/ublk/test_stress_04.sh | 37 ++++++++++++++++++ 3 files changed, 77 insertions(+) create mode 100755 tools/testing/selftests/ublk/test_stress_03.sh create mode 100755 tools/testing/selftests/ublk/test_stress_04.sh
tools/testing/selftests/ublk/Makefile +2 -0
@@ -21,6 +21,8 @@ TEST_PROGS += test_stripe_04.sh TEST_PROGS += test_stress_01.sh TEST_PROGS += test_stress_02.sh +TEST_PROGS += test_stress_03.sh +TEST_PROGS += test_stress_04.sh TEST_GEN_PROGS_EXTENDED = kublk
tools/testing/selftests/ublk/test_stress_03.sh new file +38 -0
@@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh +TID="stress_03" +ERR_CODE=0 + +ublk_io_and_remove() +{ + run_io_and_remove "$@" + ERR_CODE=$? + if [ ${ERR_CODE} -ne 0 ]; then + echo "$TID failure: $*" + _show_result $TID $ERR_CODE + fi +} + +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + +if ! _have_feature "ZERO_COPY"; then + exit "$UBLK_SKIP_CODE" +fi + +_prep_test "stress" "run IO and remove device(zero copy)" + +_create_backfile 0 256M +_create_backfile 1 128M +_create_backfile 2 128M + +ublk_io_and_remove 8G -t null -q 4 -z & +ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" & +ublk_io_and_remove 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait + +_cleanup_test "stress" +_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stress_04.sh new file +37 -0
@@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh +TID="stress_04" +ERR_CODE=0 + +ublk_io_and_kill_daemon() +{ + run_io_and_kill_daemon "$@" + ERR_CODE=$? + if [ ${ERR_CODE} -ne 0 ]; then + echo "$TID failure: $*" + _show_result $TID $ERR_CODE + fi +} + +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi +if ! _have_feature "ZERO_COPY"; then + exit "$UBLK_SKIP_CODE" +fi + +_prep_test "stress" "run IO and kill ublk server(zero copy)" + +_create_backfile 0 256M +_create_backfile 1 128M +_create_backfile 2 128M + +ublk_io_and_kill_daemon 8G -t null -q 4 -z & +ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" & +ublk_io_and_kill_daemon 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait + +_cleanup_test "stress" +_show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
PATCH v2 7/13 selftests: ublk: setup ring with IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN
In reply to Ming LeiIt is observed that this way is more efficient for fast nvme backing file.
--- tools/testing/selftests/ublk/kublk.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/ublk/kublk.c +3 -1
@@ -346,7 +346,9 @@ static int ublk_queue_init(struct ublk_queue *q) } ret =ublk_setup_ring (&q->ring, ring_depth, cq_depth, -IORING_SETUP_COOP_TASKRUN ); +IORING_SETUP_COOP_TASKRUN | +IORING_SETUP_SINGLE_ISSUER | +IORING_SETUP_DEFER_TASKRUN );if (ret <0 ) {ublk_err ("ublk dev %d queue %d setup io_uring failed %d\n" , q->dev->dev_info.dev_id, q->q_id, ret);
Signature · 1 line
2.47.0
PATCH v2 8/13 selftests: ublk: set queue pthread's cpu affinity
In reply to Ming LeiIn NUMA machine, ublk IO performance is very sensitive with queue pthread's affinity setting.
Retrieve queue's affinity and select the 1st cpu as queue thread's sched affinity, and it is observed that single cpu task affinity can get stable & good performance if client application is put on proper cpu.
Dump this info when adding one ublk device. Use shmem to communicate queue's tid between parent and daemon.
--- tools/testing/selftests/ublk/kublk.c | 156 +++++++++++++++++++++++++-- tools/testing/selftests/ublk/kublk.h | 11 +- 2 files changed, 159 insertions(+), 8 deletions(-)
tools/testing/selftests/ublk/kublk.c +149 -7
@@ -206,10 +206,73 @@ static const char *ublk_dev_state_desc(struct ublk_dev *dev) }; } +static void ublk_print_cpu_set (const cpu_set_t *set,char *buf,unsigned len) +{ +unsigned done =0 ; +int i; + +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (CPU_ISSET (i, set)) + done +=snprintf (&buf[done], len - done,"%d " , i); + } +} + +static void ublk_adjust_affinity (cpu_set_t *set) +{ +int j, updated =0 ; + +/* +* Just keep the 1st CPU now. +* +* In future, auto affinity selection can be tried. +*/ +for (j =0 ; j <CPU_SETSIZE ; j++) { +if (CPU_ISSET (j, set)) { +if (!updated) { + updated =1 ; +continue ; + } +CPU_CLR (j, set); + } + } +} + +/* Caller must free the allocated buffer */ +static int ublk_ctrl_get_affinity (struct ublk_dev *ctrl_dev,cpu_set_t **ptr_buf) +{ +struct ublk_ctrl_cmd_data data = { + .cmd_op =UBLK_U_CMD_GET_QUEUE_AFFINITY , + .flags =CTRL_CMD_HAS_DATA |CTRL_CMD_HAS_BUF , + }; +cpu_set_t *buf; +int i, ret; + + buf =malloc (sizeof (cpu_set_t) * ctrl_dev->dev_info.nr_hw_queues); +if (!buf) +return -ENOMEM ; + +for (i =0 ; i < ctrl_dev->dev_info.nr_hw_queues; i++) { + data.data[0 ] = i; + data.len =sizeof (cpu_set_t); + data.addr = (__u64)&buf[i]; + + ret =__ublk_ctrl_cmd (ctrl_dev, &data); +if (ret <0 ) { +free (buf); +return ret; + } +ublk_adjust_affinity (&buf[i]); + } + + *ptr_buf = buf; +return 0 ; +} +static void ublk_ctrl_dump (struct ublk_dev *dev) {struct ublksrv_ctrl_dev_info *info = &dev->dev_info;struct ublk_params p; +cpu_set_t *affinity;int ret; ret =ublk_ctrl_get_params (dev, &p);
@@ -218,12 +281,31 @@ static void ublk_ctrl_dump(struct ublk_dev *dev)return ; } + ret =ublk_ctrl_get_affinity (dev, &affinity); +if (ret <0 ) { +ublk_err ("failed to get affinity %m\n" ); +return ; + } +ublk_log ("dev id %d: nr_hw_queues %d queue_depth %d block size %d dev_capacity %lld\n" , info->dev_id, info->nr_hw_queues, info->queue_depth,1 << p.basic.logical_bs_shift, p.basic.dev_sectors);ublk_log ("\tmax rq size %d daemon pid %d flags 0x%llx state %s\n" , info->max_io_buf_bytes, info->ublksrv_pid, info->flags,ublk_dev_state_desc (dev)); + +if (affinity) { +char buf[512 ]; +int i; + +for (i =0 ; i < info->nr_hw_queues; i++) { +ublk_print_cpu_set (&affinity[i], buf,sizeof (buf)); +printf ("\tqueue %u: tid %d affinity(%s)\n" , + i, dev->q[i].tid, buf); + } +free (affinity); + } +fflush (stdout); }
@@ -603,9 +685,24 @@ static int ublk_process_io(struct ublk_queue *q)return reapped; } +static void ublk_queue_set_sched_affinity (const struct ublk_queue *q, +cpu_set_t *cpuset) +{ +if (sched_setaffinity (0 ,sizeof (*cpuset), cpuset) <0 ) +ublk_err ("ublk dev %u queue %u set affinity failed" , + q->dev->dev_info.dev_id, q->q_id); +} + +struct ublk_queue_info { +struct ublk_queue *q; +sem_t *queue_sem; +cpu_set_t *affinity; +}; +static void *ublk_io_handler_fn (void *data) { -struct ublk_queue *q = data; +struct ublk_queue_info *info = data; +struct ublk_queue *q = info->q;int dev_id = q->dev->dev_info.dev_id;int ret;
@@ -615,6 +712,10 @@ static void *ublk_io_handler_fn(void *data) dev_id, q->q_id);return NULL ; } +/* IO perf is sensitive with queue pthread affinity on NUMA machine*/ +ublk_queue_set_sched_affinity (q, info->affinity); +sem_post (info->queue_sem); +ublk_dbg (UBLK_DBG_QUEUE ,"tid %d: ublk dev %d queue %d started\n" , q->tid, dev_id, q->q_id);
@@ -640,7 +741,7 @@ static void ublk_set_parameters(struct ublk_dev *dev) dev->dev_info.dev_id, ret); } -static int ublk_send_dev_event (const struct dev_ctx *ctx,int dev_id) +static int ublk_send_dev_event (const struct dev_ctx *ctx,struct ublk_dev *dev,int dev_id) {uint64_t id;int evtfd = ctx->_evtfd;
@@ -653,10 +754,14 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)else id =ERROR_EVTFD_DEVID ; +if (dev && ctx->shadow_dev) +memcpy (&ctx->shadow_dev->q, &dev->q,sizeof (dev->q)); +if (write (evtfd, &id,sizeof (id)) !=sizeof (id))return -EINVAL ;close (evtfd); +shmdt (ctx->shadow_dev);return 0 ; }
@@ -664,24 +769,46 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)static int ublk_start_daemon (const struct dev_ctx *ctx,struct ublk_dev *dev) { -int ret, i; -void *thread_ret;const struct ublksrv_ctrl_dev_info *dinfo = &dev->dev_info; +struct ublk_queue_info *qinfo; +cpu_set_t *affinity_buf; +void *thread_ret; +sem_t queue_sem; +int ret, i;ublk_dbg (UBLK_DBG_DEV ,"%s enter\n" , __func__); + qinfo = (struct ublk_queue_info *)calloc (sizeof (struct ublk_queue_info ), + dinfo->nr_hw_queues); +if (!qinfo) +return -ENOMEM ; + +sem_init (&queue_sem,0 ,0 ); ret =ublk_dev_prep (ctx, dev);if (ret)return ret; + ret =ublk_ctrl_get_affinity (dev, &affinity_buf); +if (ret) +return ret; +for (i =0 ; i < dinfo->nr_hw_queues; i++) { dev->q[i].dev = dev; dev->q[i].q_id = i; + + qinfo[i].q = &dev->q[i]; + qinfo[i].queue_sem = &queue_sem; + qinfo[i].affinity = &affinity_buf[i];pthread_create (&dev->q[i].thread,NULL , ublk_io_handler_fn, - &dev->q[i]); + &qinfo[i]); } +for (i =0 ; i < dinfo->nr_hw_queues; i++) +sem_wait (&queue_sem); +free (qinfo); +free (affinity_buf); +/* everything is fine now, start us */ ublk_set_parameters (dev); ret =ublk_ctrl_start_dev (dev,getpid ());
@@ -694,7 +821,7 @@ static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev)if (ctx->fg)ublk_ctrl_dump (dev);else -ublk_send_dev_event (ctx, dev->dev_info.dev_id); +ublk_send_dev_event (ctx, dev, dev->dev_info.dev_id);/* wait until we are terminated */ for (i =0 ; i < dinfo->nr_hw_queues; i++)
@@ -873,7 +1000,7 @@ static int __cmd_dev_add(const struct dev_ctx *ctx) fail:if (ret <0 ) -ublk_send_dev_event (ctx,-1 ); +ublk_send_dev_event (ctx, dev,-1 );ublk_ctrl_deinit (dev);return ret; }
@@ -887,6 +1014,16 @@ static int cmd_dev_add(struct dev_ctx *ctx)if (ctx->fg) goto run; + ctx->_shmid =shmget (IPC_PRIVATE ,sizeof (struct ublk_dev ),IPC_CREAT |0666 ); +if (ctx->_shmid <0 ) { +ublk_err ("%s: failed to shmget %s\n" , __func__,strerror (errno)); +exit (-1 ); + } + ctx->shadow_dev = (struct ublk_dev *)shmat (ctx->_shmid,NULL ,0 ); +if (ctx->shadow_dev == (struct ublk_dev *)-1 ) { +ublk_err ("%s: failed to shmat %s\n" , __func__,strerror (errno)); +exit (-1 ); + } ctx->_evtfd =eventfd (0 ,0 );if (ctx->_evtfd <0 ) {ublk_err ("%s: failed to create eventfd %s\n" , __func__,strerror (errno));
@@ -922,6 +1059,8 @@ static int cmd_dev_add(struct dev_ctx *ctx)if (__cmd_dev_list (ctx) >=0 ) exit_code =EXIT_SUCCESS ; } +shmdt (ctx->shadow_dev); +shmctl (ctx->_shmid,IPC_RMID ,NULL );/* wait for child and detach from it */ wait (NULL );exit (exit_code);
@@ -988,6 +1127,9 @@ static int __cmd_dev_list(struct dev_ctx *ctx)ublk_err ("%s: can't get dev info from %d: %d\n" , __func__, ctx->dev_id, ret); }else { +if (ctx->shadow_dev) +memcpy (&dev->q, ctx->shadow_dev->q,sizeof (dev->q)); +ublk_ctrl_dump (dev); }
tools/testing/selftests/ublk/kublk.h +10 -1
@@ -20,10 +20,15 @@#include <sys/wait.h> #include <sys/eventfd.h> #include <sys/uio.h> +#include <sys/ipc.h> +#include <sys/shm.h> #include <linux/io_uring.h> #include <liburing.h> -#include <linux/ublk_cmd.h> +#include <semaphore.h> + +/* allow ublk_dep.h to override ublk_cmd.h */ #include "ublk_dep.h" +#include <linux/ublk_cmd.h> #define __maybe_unused __attribute__((unused))#define MAX_BACK_FILES 4
@@ -74,6 +79,10 @@ struct dev_ctx {unsigned int chunk_size;int _evtfd; +int _shmid; + +/* built from shmem, only for ublk_dump_dev() */ +struct ublk_dev *shadow_dev; };struct ublk_ctrl_cmd_data {
Signature · 1 line
2.47.0
PATCH v2 9/13 selftests: ublk: increase max nr_queues and queue depth
In reply to Ming LeiIncrease max nr_queues to 32, and queue depth to 1024.
--- tools/testing/selftests/ublk/kublk.c | 2 +- tools/testing/selftests/ublk/kublk.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/ublk/kublk.c +1 -1
@@ -1203,7 +1203,7 @@ static int cmd_dev_get_features(void)static int cmd_dev_help (char *exe) {printf ("%s add -t [null|loop] [-q nr_queues] [-d depth] [-n dev_id] [backfile1] [backfile2] ...\n" , exe); -printf ("\t default: nr_queues=2(max 4), depth=128(max 128), dev_id=-1(auto allocation)\n" ); +printf ("\t default: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n" );printf ("%s del [-n dev_id] -a \n" , exe);printf ("\t -a delete all devices -n delete specified device\n" );printf ("%s list [-n dev_id] -a \n" , exe);
tools/testing/selftests/ublk/kublk.h +2 -2
@@ -50,8 +50,8 @@#define UBLKSRV_IO_IDLE_SECS 20#define UBLK_IO_MAX_BYTES (1 << 20) -#define UBLK_MAX_QUEUES 4 -#define UBLK_QUEUE_DEPTH 128 +#define UBLK_MAX_QUEUES 32 +#define UBLK_QUEUE_DEPTH 1024#define UBLK_DBG_DEV (1U << 0)#define UBLK_DBG_QUEUE (1U << 1)
Signature · 1 line
2.47.0
PATCH v2 10/13 selftests: ublk: support target specific command line
In reply to Ming LeiSupport target specific command line for making related command line code handling more readable & clean.
Also helps for adding new features.
--- tools/testing/selftests/ublk/kublk.c | 59 +++++++++++++++++++++++---- tools/testing/selftests/ublk/kublk.h | 20 +++++++-- tools/testing/selftests/ublk/stripe.c | 28 ++++++++++++- 3 files changed, 95 insertions(+), 12 deletions(-)
tools/testing/selftests/ublk/kublk.c +51 -8
@@ -5,6 +5,8 @@#include "kublk.h" +#define MAX_NR_TGT_ARG 64 +unsigned int ublk_dbg_mask =UBLK_LOG ;static const struct ublk_tgt_ops *tgt_ops_list[] = { &null_tgt_ops,
@@ -1202,12 +1204,25 @@ static int cmd_dev_get_features(void)static int cmd_dev_help (char *exe) { -printf ("%s add -t [null|loop] [-q nr_queues] [-d depth] [-n dev_id] [backfile1] [backfile2] ...\n" , exe); -printf ("\t default: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n" ); +int i; + +printf ("%s add -t [null|loop|stripe] [-q nr_queues] [-d depth] [-n dev_id]\n" , exe); +printf ("\t[--foreground] [--quiet] [-z] [--debug_mask mask]\n" ); +printf ("\t[target options] [backfile1] [backfile2] ...\n" ); +printf ("\tdefault: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n" ); + +for (i =0 ; i <sizeof (tgt_ops_list) /sizeof (tgt_ops_list[0 ]); i++) { +const struct ublk_tgt_ops *ops = tgt_ops_list[i]; + +if (ops->usage) + ops->usage (ops); + } +printf ("\n" ); +printf ("%s del [-n dev_id] -a \n" , exe); -printf ("\t -a delete all devices -n delete specified device\n" ); +printf ("\t -a delete all devices -n delete specified device\n\n" );printf ("%s list [-n dev_id] -a \n" , exe); -printf ("\t -a list all devices, -n list specified device, default -a \n" ); +printf ("\t -a list all devices, -n list specified device, default -a \n\n" );printf ("%s features\n" , exe);return 0 ; }
@@ -1224,9 +1239,9 @@ int main(int argc, char *argv[]) { "quiet", 0, NULL, 0 }, { "zero_copy", 0, NULL, 'z' }, { "foreground", 0, NULL, 0 }, - { "chunk_size", 1, NULL, 0 }, { 0, 0, 0, 0 } }; +const struct ublk_tgt_ops *ops =NULL ;int option_idx, opt;const char *cmd = argv[1 ]; struct dev_ctx ctx = {
@@ -1234,13 +1249,15 @@ int main(int argc, char *argv[]) .nr_hw_queues = 2, .dev_id = -1, .tgt_type = "unknown", - .chunk_size = 65536, /* def chunk size is 64K */ };int ret = -EINVAL , i; +int tgt_argc =1 ; +char *tgt_argv[MAX_NR_TGT_ARG ] = {NULL };if (argc ==1 )return ret; + opterr =0 ; optind =2 ;while ((opt =getopt_long (argc, argv,"t:n:d:q:az" , longopts, &option_idx)) !=-1 ) {
@@ -1271,8 +1288,26 @@ int main(int argc, char *argv[]) ublk_dbg_mask =0 ;if (!strcmp (longopts[option_idx].name,"foreground" )) ctx.fg =1 ; -if (!strcmp (longopts[option_idx].name,"chunk_size" )) - ctx.chunk_size =strtol (optarg,NULL ,10 ); +break ; +case '?' : +/* +* target requires every option must have argument +*/ +if (argv[optind][0 ] =='-' || argv[optind -1 ][0 ] !='-' ) { +fprintf (stderr,"every target option requires argument: %s %s\n" , + argv[optind -1 ], argv[optind]); +exit (EXIT_FAILURE ); + } + +if (tgt_argc < (MAX_NR_TGT_ARG -1 ) /2 ) { + tgt_argv[tgt_argc++] = argv[optind -1 ]; + tgt_argv[tgt_argc++] = argv[optind]; + }else { +fprintf (stderr,"too many target options\n" ); +exit (EXIT_FAILURE ); + } + optind +=1 ; +break ; } }
@@ -1281,6 +1316,14 @@ int main(int argc, char *argv[]) ctx.files[ctx.nr_files++] = argv[i++]; } + ops =ublk_find_tgt (ctx.tgt_type); +if (ops && ops->parse_cmd_line) { + optind =0 ; + + tgt_argv[0 ] = ctx.tgt_type; + ops->parse_cmd_line (&ctx, tgt_argc, tgt_argv); + } +if (!strcmp (cmd,"add" )) ret =cmd_dev_add (&ctx);else if (!strcmp (cmd,"del" ))
tools/testing/selftests/ublk/kublk.h +17 -3
@@ -63,6 +63,11 @@struct ublk_dev ;struct ublk_queue ; +struct stripe_ctx { +/* stripe */ +unsigned int chunk_size; +}; +struct dev_ctx {char tgt_type[16 ];unsigned long flags;
@@ -75,14 +80,15 @@ struct dev_ctx { unsigned int all:1; unsigned int fg:1; -/* stripe */ -unsigned int chunk_size; -int _evtfd;int _shmid;/* built from shmem, only for ublk_dump_dev() */ struct ublk_dev *shadow_dev; + +union { +struct stripe_ctx stripe; + }; };struct ublk_ctrl_cmd_data {
@@ -119,6 +125,14 @@ struct ublk_tgt_ops {int (*queue_io)(struct ublk_queue *,int tag);void (*tgt_io_done)(struct ublk_queue *,int tag,const struct io_uring_cqe *); + +/* +* Target specific command line handling +* +* each option requires argument for target command line +*/ +void (*parse_cmd_line)(struct dev_ctx *ctx,int argc,char *argv[]); +void (*usage)(const struct ublk_tgt_ops *ops); };struct ublk_tgt {
tools/testing/selftests/ublk/stripe.c +27 -1
@@ -281,7 +281,7 @@ static int ublk_stripe_tgt_init(const struct dev_ctx *ctx, struct ublk_dev *dev) .max_sectors = dev->dev_info.max_io_buf_bytes >> 9, }, }; -unsigned chunk_size = ctx->chunk_size; +unsigned chunk_size = ctx->stripe.chunk_size;struct stripe_conf *conf;unsigned chunk_shift;loff_t bytes =0 ;
@@ -344,10 +344,36 @@ static void ublk_stripe_tgt_deinit(struct ublk_dev *dev)backing_file_tgt_deinit (dev); } +static void ublk_stripe_cmd_line (struct dev_ctx *ctx,int argc,char *argv[]) +{ +static const struct option longopts[] = { + {"chunk_size" ,1 ,NULL ,0 }, + {0 ,0 ,0 ,0 } + }; +int option_idx, opt; + + ctx->stripe.chunk_size =65536 ; +while ((opt =getopt_long (argc, argv,"" , + longopts, &option_idx)) !=-1 ) { +switch (opt) { +case 0 : +if (!strcmp (longopts[option_idx].name,"chunk_size" )) + ctx->stripe.chunk_size =strtol (optarg,NULL ,10 ); + } + } +} + +static void ublk_stripe_usage (const struct ublk_tgt_ops *ops) +{ +printf ("\tstripe: [--chunk_size chunk_size (default 65536)]\n" ); +} +const struct ublk_tgt_ops stripe_tgt_ops = { .name ="stripe" , .init_tgt = ublk_stripe_tgt_init, .deinit_tgt = ublk_stripe_tgt_deinit, .queue_io = ublk_stripe_queue_io, .tgt_io_done = ublk_stripe_io_done, + .parse_cmd_line = ublk_stripe_cmd_line, + .usage = ublk_stripe_usage, };
Signature · 1 line
2.47.0
PATCH v2 11/13 selftests: ublk: support user recovery
In reply to Ming LeiAdd user recovery feature.
Meantime add user recovery test: generic_04 and generic_05(zero copy)
--- tools/testing/selftests/ublk/Makefile | 2 + tools/testing/selftests/ublk/kublk.c | 96 +++++++++++++++++-- tools/testing/selftests/ublk/kublk.h | 1 + tools/testing/selftests/ublk/test_common.sh | 57 ++++++++++- .../testing/selftests/ublk/test_generic_04.sh | 40 ++++++++ .../testing/selftests/ublk/test_generic_05.sh | 44 +++++++++ 6 files changed, 230 insertions(+), 10 deletions(-) create mode 100755 tools/testing/selftests/ublk/test_generic_04.sh create mode 100755 tools/testing/selftests/ublk/test_generic_05.sh
tools/testing/selftests/ublk/Makefile +2 -0
@@ -6,6 +6,8 @@ LDLIBS += -lpthread -lm -luring TEST_PROGS := test_generic_01.sh TEST_PROGS += test_generic_02.sh TEST_PROGS += test_generic_03.sh +TEST_PROGS += test_generic_04.sh +TEST_PROGS += test_generic_05.sh TEST_PROGS += test_null_01.sh TEST_PROGS += test_null_02.sh
tools/testing/selftests/ublk/kublk.c +88 -8
160 lines not on this page · raw message
tools/testing/selftests/ublk/kublk.h +1 -0
@@ -79,6 +79,7 @@ struct dev_ctx { unsigned int logging:1; unsigned int all:1; unsigned int fg:1; + unsigned int recovery:1;int _evtfd;int _shmid;
tools/testing/selftests/ublk/test_common.sh +55 -2
82 lines not on this page · raw message
tools/testing/selftests/ublk/test_generic_04.sh new file +40 -0
@@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh + +TID="generic_04" +ERR_CODE=0 + +ublk_run_recover_test() +{ + run_io_and_recover "$@" + ERR_CODE=$? + if [ ${ERR_CODE} -ne 0 ]; then + echo "$TID failure: $*" + _show_result $TID $ERR_CODE + fi +} + +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + +_prep_test "recover" "basic recover function verification" + +_create_backfile 0 256M +_create_backfile 1 128M +_create_backfile 2 128M + +ublk_run_recover_test -t null -q 2 -r 1 & +ublk_run_recover_test -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" & +ublk_run_recover_test -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait + +ublk_run_recover_test -t null -q 2 -r 1 -i 1 & +ublk_run_recover_test -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" & +ublk_run_recover_test -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" & +wait + +_cleanup_test "recover" +_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_generic_05.sh new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v2 12/13 selftests: ublk: add test_stress_05.sh
In reply to Ming LeiAdd test_stress_05.sh for covering removing device with recovery enabled.
io-hang has been observed with the following patch:
https://lore.kernel.org/linux-block/[email protected]/
--- tools/testing/selftests/ublk/Makefile | 1 + .../testing/selftests/ublk/test_stress_05.sh | 64 +++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100755 tools/testing/selftests/ublk/test_stress_05.sh
tools/testing/selftests/ublk/Makefile +1 -0
@@ -25,6 +25,7 @@ TEST_PROGS += test_stress_01.sh TEST_PROGS += test_stress_02.sh TEST_PROGS += test_stress_03.sh TEST_PROGS += test_stress_04.sh +TEST_PROGS += test_stress_05.sh TEST_GEN_PROGS_EXTENDED = kublk
tools/testing/selftests/ublk/test_stress_05.sh new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v2 13/13 selftests: ublk: move creating UBLK_TMP into _prep_test()
In reply to Ming Leitest may exit early because of missing program or not having required feature before calling _prep_test(), then $UBLK_TMP isn't cleaned.
Fix it by moving creating $UBLK_TMP into _prep_test(), any resources created since _prep_test() will be cleaned by _cleanup_test().
--- tools/testing/selftests/ublk/test_common.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/ublk/test_common.sh +1 -1
@@ -114,6 +114,7 @@ _prep_test() { local type=$1 shift 1 modprobe ublk_drv > /dev/null 2>&1 + UBLK_TMP=$(mktemp ublk_test_XXXXX) [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "ublk $type: $*" }
@@ -338,7 +339,6 @@ _ublk_test_top_dir() cd "$(dirname "$0")" && pwd } -UBLK_TMP=$(mktemp ublk_test_XXXXX) UBLK_PROG=$(_ublk_test_top_dir)/kublk UBLK_TEST_QUIET=1 UBLK_TEST_SHOW_RESULT=1
Signature · 1 line
2.47.0
Quoted 39 lines · Ming Lei
Hello Jens,
This patchset cleans up ublk selftests and add more tests:
- two bug
fixes(1, 2)- cleanup (3, 4)
- allow to run tests in parallel(5), also big simplification on test script
- add two stress tests for zero copy(6)
- kublk misc
change(7, 8, 9), helps for evaluating performance- support target specific command line, so help to add new target(Uday is working on fault-inject target) (10)
- add two tests for covering recovery features(11)
- add one heavy io & remove test over recovery enabled device(12), which can catch io hang triggered by several recent patches.
- the last patch is for making sure ublk temp file is cleaned up if test is skipped
With this change, kernel built-in ublk selftests can :
- cover almost all tests done by ublksrv 'make test T=generic', which has been effective to capture driver issue early, so it will make ublk driver development more efficiently
- add more stress tests for covering ublk zc feature, which has found one kernel panic issue introduced recently, fix merged already
- help to add new tests, such as per-target command line, which will help to write fault-inject target
Quoted 7 lines
Thanks,
V2:
- useARRAY_SIZE()(Johannes Thumshirn)
- drop one driver bug fix
- fix ublk temp file cleanup
- improve document
Hello Jens,
Can you consider to merge this patchset to v6.15?
- ublk selftest is just added to 6.15
- it includes bug fixes and nice cleanup/simplification
- test code is always tested fully
Then we can speedup to make the test code mature/stable in this cycle, and later it can backported to liburing/blktest project.
thanks,
Ming
Quoted 10 lines · Ming Lei
This patchset cleans up ublk selftests and add more tests:
- two bug
fixes(1, 2)- cleanup (3, 4)
- allow to run tests in parallel(5), also big simplification on test script
[...]
Applied, thanks!
[01/13] selftests: ublk: fix ublk_find_tgt()
commit: ec120093180b9d92b0c84cb89a205876f9a4cb40
[02/13] selftests: ublk: add io_uring uapi header
commit: 9cad26d66b7a6306fa1e3cf64e30941afdadf6c8
[03/13] selftests: ublk: cleanup backfile automatically
commit: 8d31a7e505340a69528cbccb0894ef530f123cbb
[04/13] selftests: ublk: make sure _add_ublk_dev can return in sub-shell
commit: 573840ab90ad5bfc8711f0252cf88db028ad473e
[05/13] selftests: ublk: run stress tests in parallel
commit: bb2cabf23568d74407a3881e81f43777f490299b
[06/13] selftests: ublk: add two stress tests for zero copy feature
commit: d836590d9a9e1d822667e2720ef0d5e69a566aef
[07/13] selftests: ublk: setup ring with IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN
commit: 62867a046a223e6eb771e23d2048e839c1d949d7
[08/13] selftests: ublk: set queue pthread's cpu affinity
commit: 2f0a692a93a585ead9ccffd0642694946d74411f
[09/13] selftests: ublk: increase max nr_queues and queue depth
commit: 6c62fd04e8bfc06f37ccda0d12fd367591445954
[10/13] selftests: ublk: support target specific command line
commit: 810b88f3dcb6d04e274b37d05f421330e20a3714
[11/13] selftests: ublk: support user recovery
commit: 57e13a2e8cd208db254968631820fc1353da9db0
[12/13] selftests: ublk: add test_stress_05.sh
commit: 2f9a30bd16643d842da0921dc37bf00c750b0a8b
[13/13] selftests: ublk: move creating UBLK_TMP into _prep_test()
commit: 3bf540609cab0402a7c3e40c1425532f3376318aBest regards,
Signature · 1 line
Jens Axboe