add blkdev type environment variable
Subsystems (3)
cover add blkdev type environment variable
During the last batch of refactoring I noticed that we could reduce the number of tests a bit. There are tests which are almost identically except how the target is configured, file vs block device backend.
By introducing a configure knob, we can drop the duplicates and even make some of the tests a bit more versatile. Not all tests exists in file and block device backend version. Thus we increase the test coverage with this series. And not really surprising, there is a fallout. The nvme/028 test with file backend is failing in my setup but I was not able to figure out where things go wrong yet. I'll provide some logs later.
Daniel Wagner (3):
nvme/rc: add blkdev type environment variable nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
tests
nvme/{021,022,025,026,027,028}: do not hard code target blkdev type
Documentation/running-tests.md | 2 ++ tests/nvme/006 | 5 ++-- tests/nvme/007 | 28 -------------------- tests/nvme/007.out | 2 -- tests/nvme/008 | 4 +-- tests/nvme/009 | 36 ------------------------- tests/nvme/009.out | 3 --- tests/nvme/010 | 4 +-- tests/nvme/011 | 39 --------------------------- tests/nvme/011.out | 3 --- tests/nvme/012 | 4 +-- tests/nvme/013 | 43 ------------------------------ tests/nvme/013.out | 3 --- tests/nvme/014 | 4 +-- tests/nvme/015 | 48 ---------------------------------- tests/nvme/015.out | 4 --- tests/nvme/019 | 4 +-- tests/nvme/020 | 40 ---------------------------- tests/nvme/020.out | 4 --- tests/nvme/021 | 6 ++--- tests/nvme/022 | 6 ++--- tests/nvme/023 | 4 +-- tests/nvme/024 | 40 ---------------------------- tests/nvme/024.out | 2 -- tests/nvme/025 | 6 ++--- tests/nvme/026 | 6 ++--- tests/nvme/027 | 6 ++--- tests/nvme/028 | 6 ++--- tests/nvme/rc | 3 ++- 29 files changed, 36 insertions(+), 329 deletions(-) delete mode 100755 tests/nvme/007 delete mode 100644 tests/nvme/007.out delete mode 100755 tests/nvme/009 delete mode 100644 tests/nvme/009.out delete mode 100755 tests/nvme/011 delete mode 100644 tests/nvme/011.out delete mode 100755 tests/nvme/013 delete mode 100644 tests/nvme/013.out delete mode 100755 tests/nvme/015 delete mode 100644 tests/nvme/015.out delete mode 100755 tests/nvme/020 delete mode 100644 tests/nvme/020.out delete mode 100755 tests/nvme/024 delete mode 100644 tests/nvme/024.out
Signature · 1 line
2.44.0
PATCH 1/3 nvme/rc: add blkdev type environment variable
Introduce nvmet_blkdev_type environment variable which allows to control the target setup. This allows us to drop duplicate tests which just differ how the target is setup.
--- Documentation/running-tests.md | 2 ++ tests/nvme/rc | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)
Documentation/running-tests.md +2 -0
@@ -109,6 +109,8 @@ The NVMe tests can be additionally parameterized via environment variables. and 'G' postfix are supported. - nvme_num_iter: 1000 (default) The number of iterations a test should do. +- nvmet_blkdev_type: 'device' (default), 'file' + Select different target block device type backends. ### Running nvme-rdma and SRP tests
tests/nvme/rc +2 -1
@@ -21,6 +21,7 @@ export def_subsys_uuid="91fdba0d-f87b-4c25-b80f-db7be1418b9e" nvme_trtype=${nvme_trtype:-"loop"} nvme_img_size=${nvme_img_size:-"1G"} nvme_num_iter=${nvme_num_iter:-"1000"} +nvmet_blkdev_type=${nvmet_blkdev_type:-"device"} # TMPDIR can not be referred out of test() or test_device() context. Instead of # global variable def_flie_path, use this getter function.
@@ -826,7 +827,7 @@ _find_nvme_passthru_loop_dev() { } _nvmet_target_setup() { - local blkdev_type="device" + local blkdev_type="${nvmet_blkdev_type}" local blkdev local ctrlkey="" local hostkey=""
Signature · 1 line
2.44.0
PATCH 2/3 nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type tests
In reply to Daniel WagnerThere are various tests which only differ on the blkdev type of the target. With the newly added feature which allows to control the target blkdev type via the environment, these duplicate tests are not necessary anymore and reduces the maintenance overhead.
--- tests/nvme/006 | 5 ++--- tests/nvme/007 | 28 --------------------------- tests/nvme/007.out | 2 -- tests/nvme/008 | 4 ++-- tests/nvme/009 | 36 ---------------------------------- tests/nvme/009.out | 3 --- tests/nvme/010 | 4 ++-- tests/nvme/011 | 39 ------------------------------------- tests/nvme/011.out | 3 --- tests/nvme/012 | 4 ++-- tests/nvme/013 | 43 ----------------------------------------- tests/nvme/013.out | 3 --- tests/nvme/014 | 4 ++-- tests/nvme/015 | 48 ---------------------------------------------- tests/nvme/015.out | 4 ---- tests/nvme/019 | 4 ++-- tests/nvme/020 | 40 -------------------------------------- tests/nvme/020.out | 4 ---- tests/nvme/023 | 4 ++-- tests/nvme/024 | 40 -------------------------------------- tests/nvme/024.out | 2 -- 21 files changed, 14 insertions(+), 310 deletions(-) delete mode 100755 tests/nvme/007 delete mode 100644 tests/nvme/007.out delete mode 100755 tests/nvme/009 delete mode 100644 tests/nvme/009.out delete mode 100755 tests/nvme/011 delete mode 100644 tests/nvme/011.out delete mode 100755 tests/nvme/013 delete mode 100644 tests/nvme/013.out delete mode 100755 tests/nvme/015 delete mode 100644 tests/nvme/015.out delete mode 100755 tests/nvme/020 delete mode 100644 tests/nvme/020.out delete mode 100755 tests/nvme/024 delete mode 100644 tests/nvme/024.out
tests/nvme/006 +2 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMeOF target creation with a block device backed ns. +# Test NVMeOF target creation. . tests/nvme/rc -DESCRIPTION="create an NVMeOF target with a block device-backed ns" +DESCRIPTION="create an NVMeOF target" QUICK=1 requires() {
@@ -20,7 +20,6 @@ test() { _setup_nvmet - _nvmet_target_setup _nvmet_target_cleanup
tests/nvme/007 deleted +0 -28
@@ -1,28 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: GPL-2.0+ -# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. -# -# Test NVMeOF target creation with a file backed ns. - -. tests/nvme/rc - -DESCRIPTION="create an NVMeOF target with a file-backed ns" -QUICK=1 - -requires() { - _nvme_requires - _require_nvme_trtype_is_fabrics -} - -test() { - echo "Running ${TEST_NAME}" - - _setup_nvmet - - - _nvmet_target_setup --blkdev file - - _nvmet_target_cleanup - - echo "Test complete" -}
tests/nvme/007.out deleted +0 -2
@@ -1,2 +0,0 @@ -Running nvme/007 -Test complete
tests/nvme/008 +2 -2
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMeOF host creation with a block device backed ns. +# Test NVMeOF host creation. . tests/nvme/rc -DESCRIPTION="create an NVMeOF host with a block device-backed ns" +DESCRIPTION="create an NVMeOF host" QUICK=1 requires() {
tests/nvme/009 deleted +0 -36
@@ -1,36 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: GPL-2.0+ -# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. -# -# Test NVMeOF host creation with a file backed ns. - -. tests/nvme/rc - -DESCRIPTION="create an NVMeOF host with a file-backed ns" -QUICK=1 - -requires() { - _nvme_requires - _require_nvme_trtype_is_fabrics -} - -test() { - echo "Running ${TEST_NAME}" - - _setup_nvmet - - local nvmedev - - _nvmet_target_setup --blkdev file - - _nvme_connect_subsys - - nvmedev=$(_find_nvme_dev "${def_subsysnqn}") - _check_uuid "${nvmedev}" - - _nvme_disconnect_subsys - - _nvmet_target_cleanup - - echo "Test complete" -}
tests/nvme/009.out deleted +0 -3
@@ -1,3 +0,0 @@ -Running nvme/009 -disconnected 1 controller(s) -Test complete
tests/nvme/010 +2 -2
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# This is a data verification test for block device backed ns. +# This is a data verification test. . tests/nvme/rc -DESCRIPTION="run data verification fio job on NVMeOF block device-backed ns" +DESCRIPTION="run data verification fio job" TIMED=1 requires() {
tests/nvme/011 deleted +0 -39
@@ -1,39 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: GPL-2.0+ -# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. -# -# This is a data verification test for file backed ns. - -. tests/nvme/rc - -DESCRIPTION="run data verification fio job on NVMeOF file-backed ns" -TIMED=1 - -requires() { - _nvme_requires - _have_fio - _require_nvme_trtype_is_fabrics -} - -test() { - echo "Running ${TEST_NAME}" - - _setup_nvmet - - local ns - - _nvmet_target_setup --blkdev file - - _nvme_connect_subsys - - ns=$(_find_nvme_ns "${def_subsys_uuid}") - - _run_fio_verify_io --size="${nvme_img_size}" \ - --filename="$/dev/{ns}" - - _nvme_disconnect_subsys - - _nvmet_target_cleanup - - echo "Test complete" -}
tests/nvme/011.out deleted +0 -3
@@ -1,3 +0,0 @@ -Running nvme/011 -disconnected 1 controller(s) -Test complete
tests/nvme/012 +2 -2
@@ -2,12 +2,12 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test mkfs with data verification for block device backed ns. +# Test mkfs with data verification. . tests/nvme/rc . common/xfs -DESCRIPTION="run mkfs and data verification fio job on NVMeOF block device-backed ns" +DESCRIPTION="run mkfs and data verification fio" TIMED=1 requires() {
tests/nvme/013 deleted +0 -43
43 lines not on this page · raw message
tests/nvme/013.out deleted +0 -3
@@ -1,3 +0,0 @@ -Running nvme/013 -disconnected 1 controller(s) -Test complete
tests/nvme/014 +2 -2
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMeOF flush command from host with a block device backed ns. +# Test NVMeOF flush command from host. . tests/nvme/rc -DESCRIPTION="flush a NVMeOF block device-backed ns" +DESCRIPTION="flush a command from host" QUICK=1 requires() {
tests/nvme/015 deleted +0 -48
48 lines not on this page · raw message
tests/nvme/015.out deleted +0 -4
@@ -1,4 +0,0 @@ -Running nvme/015 -NVMe Flush: success -disconnected 1 controller(s) -Test complete
tests/nvme/019 +2 -2
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe DSM Discard command on NVMeOF with a block-device ns. +# Test NVMe DSM Discard command. . tests/nvme/rc -DESCRIPTION="test NVMe DSM Discard command on NVMeOF block-device ns" +DESCRIPTION="test NVMe DSM Discard command" QUICK=1 requires() {
tests/nvme/020 deleted +0 -40
@@ -1,40 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: GPL-2.0+ -# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. -# -# Test NVMe DSM Discard command on NVMeOF with a file-backed ns. - -. tests/nvme/rc - -DESCRIPTION="test NVMe DSM Discard command on NVMeOF file-backed ns" -QUICK=1 - -requires() { - _nvme_requires - _require_nvme_trtype_is_fabrics -} - -test() { - echo "Running ${TEST_NAME}" - - _setup_nvmet - - local ns - local nblk_range="10,10,10,10,10,10,10,10,10,10" - local sblk_range="100,200,300,400,500,600,700,800,900,1000" - - _nvmet_target_setup --blkdev file - - _nvme_connect_subsys - - ns=$(_find_nvme_ns "${def_subsys_uuid}") - - nvme dsm "/dev/${ns}" --ad \ - --slbs "${sblk_range}" --blocks "${nblk_range}" - - _nvme_disconnect_subsys - - _nvmet_target_cleanup - - echo "Test complete" -}
tests/nvme/020.out deleted +0 -4
@@ -1,4 +0,0 @@ -Running nvme/020 -NVMe DSM: success -disconnected 1 controller(s) -Test complete
tests/nvme/023 +2 -2
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe smart-log command on NVMeOF with a block-device ns. +# Test NVMe smart-log command. . tests/nvme/rc -DESCRIPTION="test NVMe smart-log command on NVMeOF block-device ns" +DESCRIPTION="test NVMe smart-log command" QUICK=1 requires() {
tests/nvme/024 deleted +0 -40
@@ -1,40 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: GPL-2.0+ -# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. -# -# Test NVMe smart-log command on NVMeOF with a file-backed ns. - -. tests/nvme/rc - -DESCRIPTION="test NVMe smart-log command on NVMeOF file-backed ns" -QUICK=1 - -requires() { - _nvme_requires - _have_loop - _require_nvme_trtype_is_fabrics -} - -test() { - echo "Running ${TEST_NAME}" - - _setup_nvmet - - local ns - - _nvmet_target_setup --blkdev file - - _nvme_connect_subsys - - ns=$(_find_nvme_ns ${def_subsys_uuid}) - - if ! nvme smart-log "/dev/${ns}" >> "$FULL" 2>&1; then - echo "ERROR: smart-log file-ns failed" - fi - - _nvme_disconnect_subsys >> "$FULL" 2>&1 - - _nvmet_target_cleanup - - echo "Test complete" -}
tests/nvme/024.out deleted +0 -2
@@ -1,2 +0,0 @@ -Running nvme/024 -Test complete
Signature · 1 line
2.44.0
PATCH 3/3 nvme/{021,022,025,026,027,028}: do not hard code target blkdev type
In reply to Daniel WagnerThere is no need to hardcode the target blkdev type. This allows the user to select different blkdev types via the nvmet_blkdev_type environment variable.
--- tests/nvme/021 | 6 +++--- tests/nvme/022 | 6 +++--- tests/nvme/025 | 6 +++--- tests/nvme/026 | 6 +++--- tests/nvme/027 | 6 +++--- tests/nvme/028 | 6 +++--- 6 files changed, 18 insertions(+), 18 deletions(-)
tests/nvme/021 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe list command on NVMeOF with a file-backed ns. +# Test NVMe list command. . tests/nvme/rc -DESCRIPTION="test NVMe list command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe list command" QUICK=1 requires() {
@@ -22,7 +22,7 @@ test() { local ns - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
tests/nvme/022 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe reset command on NVMeOF with a file-backed ns. +# Test NVMe reset command. . tests/nvme/rc -DESCRIPTION="test NVMe reset command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe reset command" QUICK=1 requires() {
@@ -22,7 +22,7 @@ test() { local nvmedev - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
tests/nvme/025 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe effects-log command on NVMeOF with a file-backed ns. +# Test NVMe effects-log command. . tests/nvme/rc -DESCRIPTION="test NVMe effects-log command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe effects-log" QUICK=1 requires() {
@@ -22,7 +22,7 @@ test() { local ns - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
tests/nvme/026 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe ns-descs command on NVMeOF with a file-backed ns. +# Test NVMe ns-descs command. . tests/nvme/rc -DESCRIPTION="test NVMe ns-descs command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe ns-descs" QUICK=1 requires() {
@@ -22,7 +22,7 @@ test() { local ns - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
tests/nvme/027 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe ns-rescan command on NVMeOF with a file-backed ns. +# Test NVMe ns-rescan command. . tests/nvme/rc -DESCRIPTION="test NVMe ns-rescan command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe ns-rescan command" QUICK=1 requires() {
@@ -22,7 +22,7 @@ test() { local nvmedev - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
tests/nvme/028 +3 -3
@@ -2,11 +2,11 @@ # SPDX-License-Identifier: GPL-2.0+ # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates. # -# Test NVMe list-subsys command on NVMeOF with a file-backed ns. +# Test NVMe list-subsys command. . tests/nvme/rc -DESCRIPTION="test NVMe list-subsys command on NVMeOF file-backed ns" +DESCRIPTION="test NVMe list-subsys" QUICK=1 requires() {
@@ -20,7 +20,7 @@ test() { _setup_nvmet - _nvmet_target_setup --blkdev file + _nvmet_target_setup _nvme_connect_subsys
Signature · 1 line
2.44.0
Quoted 1 line · Daniel Wagner
The nvme/028 test with file backend is failing in my setup
^^^^^^^^^^^^^
block device backend
Quoted 10 lines · Daniel Wagner
During the last batch of refactoring I noticed that we could reduce the number of tests a bit. There are tests which are almost identically except how the target is configured, file vs block device backend.
By introducing a configure knob, we can drop the duplicates and even make some of the tests a bit more versatile. Not all tests exists in file and block device backend version. Thus we increase the test coverage with this series. And not really surprising, there is a fallout. The nvme/028 test with file backend is failing in my setup but I was not able to figure out where things go wrong yet. I'll provide some logs later.
Hi Daniel, I find this series reduces code duplications further, and it expands test coverage with small test script changes. Good.
On the other hand, I see that the series has a couple of drawbacks:
1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.
2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.
To avoid the drawbacks, how about this?
- Do not provide nvmet_blkdev_type as a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)
- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.
I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?
tests/nvme/006 +7 -3
@@ -15,14 +15,18 @@ requires() { _require_nvme_trtype_is_fabrics } -test() { - echo "Running ${TEST_NAME}" - +do_test() { _setup_nvmet _nvmet_target_setup _nvmet_target_cleanup +} + +test() { + echo "Running ${TEST_NAME}" + + _nvmet_run_for_each_blkdev_type do_test echo "Test complete" }
tests/nvme/rc +9 -0
@@ -996,6 +996,15 @@ _nvmet_passthru_target_cleanup() { _remove_nvmet_host "${def_hostnqn}" } +_nvmet_run_for_each_blkdev_type() { + local blkdev_type + + for blkdev_type in device file; do + nvmet_blkdev_type=$blkdev_type + eval "$1" + done +} + _discovery_genctr() { _nvme_discover "${nvme_trtype}" | sed -n -e 's/^.*Generation counter \([0-9]\+\).*$/\1/p'
Hi Shinichiro,
Quoted 11 lines · Shinichiro Kawasaki
On the other hand, I see that the series has a couple of drawbacks:
1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.
2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.
Yes, I agree. These drawbacks should be addressed somehow.
Quoted 6 lines
To avoid the drawbacks, how about this?
- Do not provide
nvmet_blkdev_typeas a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)
Quoted 6 lines
- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.
I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?
Ideally we don't have to introduce additional common setup logic into each test. Also for debugging purpose it might sense to run a test only in one configuration. So it might make sense still to have user visible environment variable
nvmet_blkdev_types="file device"
as default.
Quoted 5 lines
tests/nvme/006 +1
-3-test() { - echo "Running ${TEST_NAME}" - +do_test() { _setup_nvmet
tests/nvme/006 requires() { · line 29
Quoted 10 lines
tests/nvme/006 +6
-0_nvmet_target_setup _nvmet_target_cleanup +} + +test() { + echo "Running ${TEST_NAME}" + + _nvmet_run_for_each_blkdev_type do_test
I was wondering if the nvmet_run_for_each_blkdev_type logic could be in common code, so we don't have to add a do_test function. Basically having a common code for a bunch of configuration variables (matrix tests). This could also be useful for nvmet_trtype etc.
The generic setup could be requested via the require hook.
requires() = {
_nvmet_setup_target
}
What do you think about this idea?
Thanks,
Daniel
Shinichiro/Daniel,
Quoted 10 lines · Daniel Wagner
Hi Shinichiro,
Quoted 7 lines · Shinichiro Kawasaki
On the other hand, I see that the series has a couple of drawbacks:
1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.
reducing code is always encouraged, but please don't change the test coverage and
make user add additional steps to run the tests, IOW ./check nvme should run all
the tests case by default as it dose today, this will keep your changes backward
compatible ..
I believe that is not too hard to achieve ?
-ck
Quoted 2 lines · Chaitanya Kulkarni
reducing code is always encouraged, but please don't change the test coverage and
This series increases the test coverage.
Quoted 5 lines
make user add additional steps to run the tests, IOW ./check nvme should run all
the tests case by default as it dose today, this will keep your changes backward
compatible ..
Well, that is not really true right now. E.g. nvme_trtype is loop by default. And you need to specify the additional transports manually.
This is also why I suggest we think about some sort of generic way to do this types of 'matrix configuration'(*) out of the box.
(* don't know how to name this better)
Quoted 47 lines · Daniel Wagner
Hi Shinichiro,
Quoted 11 lines · Shinichiro Kawasaki
On the other hand, I see that the series has a couple of drawbacks:
1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.
2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.
Yes, I agree. These drawbacks should be addressed somehow.
Quoted 6 lines
To avoid the drawbacks, how about this?
- Do not provide
nvmet_blkdev_typeas a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)Quoted 6 lines
- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.
I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?
Ideally we don't have to introduce additional common setup logic into each test. Also for debugging purpose it might sense to run a test only in one configuration. So it might make sense still to have user visible environment variable
nvmet_blkdev_types="file device"
as default.
Quoted 8 lines
tests/nvme/006 +1
-3-test() { - echo "Running ${TEST_NAME}" - +do_test() { _setup_nvmet _nvmet_target_setup
Quoted 12 lines
tests/nvme/006 requires() { · line 29
Quoted 7 lines
tests/nvme/006 +6
-0_nvmet_target_cleanup +} + +test() { + echo "Running ${TEST_NAME}" + + _nvmet_run_for_each_blkdev_type do_testI was wondering if the
nvmet_run_for_each_blkdev_typelogic could be in common code, so we don't have to add ado_testfunction. Basically having a common code for a bunch of configuration variables (matrix tests).
Actually, similar feature is implemented in the common code so that some test case can be run twice, once for regular the block device, and one more time for the zoned block device. You can find test cases with CAN_BE_ZONED=1 flag. They are run twice when RUN_ZONED_TESTS is set in the config.
To be precise, this applies to the test cases with test() function. CAN_BE_ZONED has different meaning for test cases with test_device().
Now we want to run some of test cases twice for the two nvmet block device types. This is essentially common feature as the repeated runs for the CAN_BE_ZONED test cases. I think it's time to generalize these two uses cases and support "repeated test case runs with different test conditions".
Quoted 1 line
This could also be useful for
nvmet_trtypeetc.
Yes, when I run for all of nvmet_trtype variations, I see some test cases are repeated with same condition. Waste of time.
Quoted 7 lines
The generic setup could be requested via the require hook.
requires()= {
_nvmet_setup_target
}
Hmm, I think this abuses the hook. IMO, it's the better to introduce a new hook.
Quoted 2 lines
What do you think about this idea?
It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays NVMET_BLKDEV_TYPES and NVMET_TR_TYPES. When these two are set in config file as follows,
NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)
it will run a single test case as follows. 2 x 3 = 6 times repeptitions.
$ sudo ./check nvme/006
nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target) [passed]
runtime 0.090s ... 0.091s
nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target) [passed]
runtime 0.310s ... 0.305s
nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target) [passed]
runtime 0.149s ... 0.153s
nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target) [passed]
runtime 0.138s ... 0.135s
nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target) [passed]
runtime 0.300s ... 0.305s
nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target) [passed]
runtime 0.141s ... 0.147sI hope this meets your needs.
Quoted 5 lines · Shinichiro Kawasaki
Actually, similar feature is implemented in the common code so that some test case can be run twice, once for regular the block device, and one more time for the zoned block device. You can find test cases with CAN_BE_ZONED=1 flag. They are run twice when
RUN_ZONED_TESTSis set in the config.
Quoted 7 lines
To be precise, this applies to the test cases with
test()function.CAN_BE_ZONEDhas different meaning for test cases withtest_device().Now we want to run some of test cases twice for the two nvmet block device types. This is essentially common feature as the repeated runs for the
CAN_BE_ZONEDtest cases. I think it's time to generalize these two uses cases and support "repeated test case runs with different test conditions".
Sounds reasonable.
Quoted 6 lines
Quoted 4 lines
requires()= {
_nvmet_setup_target
}Hmm, I think this abuses the hook. IMO, it's the better to introduce a new hook.
Indeed :)
Quoted 27 lines
Quoted 2 lines
What do you think about this idea?
It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays
NVMET_BLKDEV_TYPESandNVMET_TR_TYPES. When these two are set in config file as follows,NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)it will run a single test case as follows. 2 x 3 = 6 times repeptitions.
$ sudo ./check nvme/006 nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target) [passed] runtime 0.090s ... 0.091s nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target) [passed] runtime 0.310s ... 0.305s nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target) [passed] runtime 0.149s ... 0.153s nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target) [passed] runtime 0.138s ... 0.135s nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target) [passed] runtime 0.300s ... 0.305s nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target) [passed] runtime 0.141s ... 0.147s
Quoted 1 line
I hope this meets your needs.
Yes, this is very useful.
Quoted 1 line
I quickly looked into the changes. The only thing I'd say it looks a bit hard to extend if we have yet another variable. But maybe I'd make it too complex. I don't think we have to be too future proof here, because we can change this part without problems, it is all 'under the hood' and doesn't change the 'user interface'.
Great stuff!
Quoted 1 line · Daniel Wagner
On Thu, Apr 04, 2024 at 07:43:28AM +0000, Shinichiro Kawasaki wrote:
...
Quoted 30 lines
Quoted 24 lines
It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays
NVMET_BLKDEV_TYPESandNVMET_TR_TYPES. When these two are set in config file as follows,NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)it will run a single test case as follows. 2 x 3 = 6 times repeptitions.
$ sudo ./check nvme/006 nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target) [passed] runtime 0.090s ... 0.091s nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target) [passed] runtime 0.310s ... 0.305s nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target) [passed] runtime 0.149s ... 0.153s nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target) [passed] runtime 0.138s ... 0.135s nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target) [passed] runtime 0.300s ... 0.305s nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target) [passed] runtime 0.141s ... 0.147sQuoted 1 line
I hope this meets your needs.
Yes, this is very useful.
Quoted 1 line
Quoted 7 lines
I quickly looked into the changes. The only thing I'd say it looks a bit hard to extend if we have yet another variable. But maybe I'd make it too complex. I don't think we have to be too future proof here, because we can change this part without problems, it is all 'under the hood' and doesn't change the 'user interface'.
Great stuff!
Okay, thanks for the positive comment. I will brush up the patches and post for review. Let me have several days.