Skip to contentSkip to conversation

add blkdev type environment variable

PATCH In review Daniel Wagner 3 participants 12 messages Apr 2, 2024 – Apr 4, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover add blkdev type environment variable

During the last batch of refactoring I noticed that we could reduce the number of tests a bit. There are tests which are almost identically except how the target is configured, file vs block device backend.

By introducing a configure knob, we can drop the duplicates and even make some of the tests a bit more versatile. Not all tests exists in file and block device backend version. Thus we increase the test coverage with this series. And not really surprising, there is a fallout. The nvme/028 test with file backend is failing in my setup but I was not able to figure out where things go wrong yet. I'll provide some logs later.

Daniel Wagner (3):
nvme/rc: add blkdev type environment variable nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
tests
nvme/{021,022,025,026,027,028}: do not hard code target blkdev type

 Documentation/running-tests.md |  2 ++
 tests/nvme/006                 |  5 ++--
 tests/nvme/007                 | 28 --------------------
 tests/nvme/007.out             |  2 --
 tests/nvme/008                 |  4 +--
 tests/nvme/009                 | 36 -------------------------
 tests/nvme/009.out             |  3 ---
 tests/nvme/010                 |  4 +--
 tests/nvme/011                 | 39 ---------------------------
 tests/nvme/011.out             |  3 ---
 tests/nvme/012                 |  4 +--
 tests/nvme/013                 | 43 ------------------------------
 tests/nvme/013.out             |  3 ---
 tests/nvme/014                 |  4 +--
 tests/nvme/015                 | 48 ----------------------------------
 tests/nvme/015.out             |  4 ---
 tests/nvme/019                 |  4 +--
 tests/nvme/020                 | 40 ----------------------------
 tests/nvme/020.out             |  4 ---
 tests/nvme/021                 |  6 ++---
 tests/nvme/022                 |  6 ++---
 tests/nvme/023                 |  4 +--
 tests/nvme/024                 | 40 ----------------------------
 tests/nvme/024.out             |  2 --
 tests/nvme/025                 |  6 ++---
 tests/nvme/026                 |  6 ++---
 tests/nvme/027                 |  6 ++---
 tests/nvme/028                 |  6 ++---
 tests/nvme/rc                  |  3 ++-
 29 files changed, 36 insertions(+), 329 deletions(-)
 delete mode 100755 tests/nvme/007
 delete mode 100644 tests/nvme/007.out
 delete mode 100755 tests/nvme/009
 delete mode 100644 tests/nvme/009.out
 delete mode 100755 tests/nvme/011
 delete mode 100644 tests/nvme/011.out
 delete mode 100755 tests/nvme/013
 delete mode 100644 tests/nvme/013.out
 delete mode 100755 tests/nvme/015
 delete mode 100644 tests/nvme/015.out
 delete mode 100755 tests/nvme/020
 delete mode 100644 tests/nvme/020.out
 delete mode 100755 tests/nvme/024
 delete mode 100644 tests/nvme/024.out
Signature · 1 line
2.44.0
#1 lore raw

PATCH 1/3 nvme/rc: add blkdev type environment variable

Introduce nvmet_blkdev_type environment variable which allows to control the target setup. This allows us to drop duplicate tests which just differ how the target is setup.

---
 Documentation/running-tests.md | 2 ++
 tests/nvme/rc                  | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
Documentation/running-tests.md +2 -0
index ae80860c917a..9cc53cb8d147 100644
@@ -109,6 +109,8 @@ The NVMe tests can be additionally parameterized via environment variables.
 	and 'G' postfix are supported.
 - nvme_num_iter: 1000 (default)
   The number of iterations a test should do.
+- nvmet_blkdev_type: 'device' (default), 'file'
+  Select different target block device type backends.
 
 ### Running nvme-rdma and SRP tests
 
tests/nvme/rc +2 -1
index 203cf0c7903b..5fa1871f7fd8 100644
@@ -21,6 +21,7 @@ export def_subsys_uuid="91fdba0d-f87b-4c25-b80f-db7be1418b9e"
 nvme_trtype=${nvme_trtype:-"loop"}
 nvme_img_size=${nvme_img_size:-"1G"}
 nvme_num_iter=${nvme_num_iter:-"1000"}
+nvmet_blkdev_type=${nvmet_blkdev_type:-"device"}
 
 # TMPDIR can not be referred out of test() or test_device() context. Instead of
 # global variable def_flie_path, use this getter function.
@@ -826,7 +827,7 @@ _find_nvme_passthru_loop_dev() {
 }
 
 _nvmet_target_setup() {
-	local blkdev_type="device"
+	local blkdev_type="${nvmet_blkdev_type}"
 	local blkdev
 	local ctrlkey=""
 	local hostkey=""
Signature · 1 line
2.44.0
#2 lore raw

PATCH 2/3 nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type tests

In reply to Daniel Wagner

There are various tests which only differ on the blkdev type of the target. With the newly added feature which allows to control the target blkdev type via the environment, these duplicate tests are not necessary anymore and reduces the maintenance overhead.

---
 tests/nvme/006     |  5 ++---
 tests/nvme/007     | 28 ---------------------------
 tests/nvme/007.out |  2 --
 tests/nvme/008     |  4 ++--
 tests/nvme/009     | 36 ----------------------------------
 tests/nvme/009.out |  3 ---
 tests/nvme/010     |  4 ++--
 tests/nvme/011     | 39 -------------------------------------
 tests/nvme/011.out |  3 ---
 tests/nvme/012     |  4 ++--
 tests/nvme/013     | 43 -----------------------------------------
 tests/nvme/013.out |  3 ---
 tests/nvme/014     |  4 ++--
 tests/nvme/015     | 48 ----------------------------------------------
 tests/nvme/015.out |  4 ----
 tests/nvme/019     |  4 ++--
 tests/nvme/020     | 40 --------------------------------------
 tests/nvme/020.out |  4 ----
 tests/nvme/023     |  4 ++--
 tests/nvme/024     | 40 --------------------------------------
 tests/nvme/024.out |  2 --
 21 files changed, 14 insertions(+), 310 deletions(-)
 delete mode 100755 tests/nvme/007
 delete mode 100644 tests/nvme/007.out
 delete mode 100755 tests/nvme/009
 delete mode 100644 tests/nvme/009.out
 delete mode 100755 tests/nvme/011
 delete mode 100644 tests/nvme/011.out
 delete mode 100755 tests/nvme/013
 delete mode 100644 tests/nvme/013.out
 delete mode 100755 tests/nvme/015
 delete mode 100644 tests/nvme/015.out
 delete mode 100755 tests/nvme/020
 delete mode 100644 tests/nvme/020.out
 delete mode 100755 tests/nvme/024
 delete mode 100644 tests/nvme/024.out
tests/nvme/006 +2 -3
index d85f64b702eb..65f2a0da0441 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMeOF target creation with a block device backed ns.
+# Test NVMeOF target creation.
 
 . tests/nvme/rc
 
-DESCRIPTION="create an NVMeOF target with a block device-backed ns"
+DESCRIPTION="create an NVMeOF target"
 QUICK=1
 
 requires() {
@@ -20,7 +20,6 @@ test() {
 
 	_setup_nvmet
 
-
 	_nvmet_target_setup
 
 	_nvmet_target_cleanup
tests/nvme/007 deleted +0 -28
deleted file mode 100755
index b14243576d32..000000000000
@@ -1,28 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
-#
-# Test NVMeOF target creation with a file backed ns.
-
-. tests/nvme/rc
-
-DESCRIPTION="create an NVMeOF target with a file-backed ns"
-QUICK=1
-
-requires() {
-	_nvme_requires
-	_require_nvme_trtype_is_fabrics
-}
-
-test() {
-	echo "Running ${TEST_NAME}"
-
-	_setup_nvmet
-
-
-	_nvmet_target_setup --blkdev file
-
-	_nvmet_target_cleanup
-
-	echo "Test complete"
-}
tests/nvme/007.out deleted +0 -2
deleted file mode 100644
index fdb347204ea8..000000000000
@@ -1,2 +0,0 @@
-Running nvme/007
-Test complete
tests/nvme/008 +2 -2
index a5d06818c6e4..420f9c0b1789 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMeOF host creation with a block device backed ns.
+# Test NVMeOF host creation.
 
 . tests/nvme/rc
 
-DESCRIPTION="create an NVMeOF host with a block device-backed ns"
+DESCRIPTION="create an NVMeOF host"
 QUICK=1
 
 requires() {
tests/nvme/009 deleted +0 -36
deleted file mode 100755
index a1655d43e56f..000000000000
@@ -1,36 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
-#
-# Test NVMeOF host creation with a file backed ns.
-
-. tests/nvme/rc
-
-DESCRIPTION="create an NVMeOF host with a file-backed ns"
-QUICK=1
-
-requires() {
-	_nvme_requires
-	_require_nvme_trtype_is_fabrics
-}
-
-test() {
-	echo "Running ${TEST_NAME}"
-
-	_setup_nvmet
-
-	local nvmedev
-
-	_nvmet_target_setup --blkdev file
-
-	_nvme_connect_subsys
-
-	nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
-	_check_uuid "${nvmedev}"
-
-	_nvme_disconnect_subsys
-
-	_nvmet_target_cleanup
-
-	echo "Test complete"
-}
tests/nvme/009.out deleted +0 -3
deleted file mode 100644
index 4d53a8e8ab19..000000000000
@@ -1,3 +0,0 @@
-Running nvme/009
-disconnected 1 controller(s)
-Test complete
tests/nvme/010 +2 -2
index 6feb39153e99..a3b6b2dd1478 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# This is a data verification test for block device backed ns.
+# This is a data verification test.
 
 . tests/nvme/rc
 
-DESCRIPTION="run data verification fio job on NVMeOF block device-backed ns"
+DESCRIPTION="run data verification fio job"
 TIMED=1
 
 requires() {
tests/nvme/011 deleted +0 -39
deleted file mode 100755
index eee044cbb4f8..000000000000
@@ -1,39 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
-#
-# This is a data verification test for file backed ns.
-
-. tests/nvme/rc
-
-DESCRIPTION="run data verification fio job on NVMeOF file-backed ns"
-TIMED=1
-
-requires() {
-	_nvme_requires
-	_have_fio
-	_require_nvme_trtype_is_fabrics
-}
-
-test() {
-	echo "Running ${TEST_NAME}"
-
-	_setup_nvmet
-
-	local ns
-
-	_nvmet_target_setup --blkdev file
-
-	_nvme_connect_subsys
-
-	ns=$(_find_nvme_ns "${def_subsys_uuid}")
-
-	_run_fio_verify_io --size="${nvme_img_size}" \
-		--filename="$/dev/{ns}"
-
-	_nvme_disconnect_subsys
-
-	_nvmet_target_cleanup
-
-	echo "Test complete"
-}
tests/nvme/011.out deleted +0 -3
deleted file mode 100644
index ebbb4f7c65bf..000000000000
@@ -1,3 +0,0 @@
-Running nvme/011
-disconnected 1 controller(s)
-Test complete
tests/nvme/012 +2 -2
index 64cb6ecf0191..b09a55c63e1f 100755
@@ -2,12 +2,12 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test mkfs with data verification for block device backed ns.
+# Test mkfs with data verification.
 
 . tests/nvme/rc
 . common/xfs
 
-DESCRIPTION="run mkfs and data verification fio job on NVMeOF block device-backed ns"
+DESCRIPTION="run mkfs and data verification fio"
 TIMED=1
 
 requires() {
tests/nvme/013 deleted +0 -43

43 lines not on this page · raw message

tests/nvme/013.out deleted +0 -3
deleted file mode 100644
index a7271700adfe..000000000000
@@ -1,3 +0,0 @@
-Running nvme/013
-disconnected 1 controller(s)
-Test complete
tests/nvme/014 +2 -2
index e56e3212cf28..2c855c89f0b7 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMeOF flush command from host with a block device backed ns.
+# Test NVMeOF flush command from host.
 
 . tests/nvme/rc
 
-DESCRIPTION="flush a NVMeOF block device-backed ns"
+DESCRIPTION="flush a command from host"
 QUICK=1
 
 requires() {
tests/nvme/015 deleted +0 -48

48 lines not on this page · raw message

tests/nvme/015.out deleted +0 -4
deleted file mode 100644
index f854f0b52dc0..000000000000
@@ -1,4 +0,0 @@
-Running nvme/015
-NVMe Flush: success
-disconnected 1 controller(s)
-Test complete
tests/nvme/019 +2 -2
index 1cd5378e9dd4..d86ca6daed31 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe DSM Discard command on NVMeOF with a block-device ns.
+# Test NVMe DSM Discard command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe DSM Discard command on NVMeOF block-device ns"
+DESCRIPTION="test NVMe DSM Discard command"
 QUICK=1
 
 requires() {
tests/nvme/020 deleted +0 -40
deleted file mode 100755
index 0364c4e0dd4f..000000000000
@@ -1,40 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
-#
-# Test NVMe DSM Discard command on NVMeOF with a file-backed ns.
-
-. tests/nvme/rc
-
-DESCRIPTION="test NVMe DSM Discard command on NVMeOF file-backed ns"
-QUICK=1
-
-requires() {
-	_nvme_requires
-	_require_nvme_trtype_is_fabrics
-}
-
-test() {
-	echo "Running ${TEST_NAME}"
-
-	_setup_nvmet
-
-	local ns
-	local nblk_range="10,10,10,10,10,10,10,10,10,10"
-	local sblk_range="100,200,300,400,500,600,700,800,900,1000"
-
-	_nvmet_target_setup --blkdev file
-
-	_nvme_connect_subsys
-
-	ns=$(_find_nvme_ns "${def_subsys_uuid}")
-
-	nvme dsm "/dev/${ns}" --ad \
-		--slbs "${sblk_range}" --blocks "${nblk_range}"
-
-	_nvme_disconnect_subsys
-
-	_nvmet_target_cleanup
-
-	echo "Test complete"
-}
tests/nvme/020.out deleted +0 -4
deleted file mode 100644
index 61be280a64cc..000000000000
@@ -1,4 +0,0 @@
-Running nvme/020
-NVMe DSM: success
-disconnected 1 controller(s)
-Test complete
tests/nvme/023 +2 -2
index d8f17ae7a8ea..d6713fb30b0d 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe smart-log command on NVMeOF with a block-device ns.
+# Test NVMe smart-log command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe smart-log command on NVMeOF block-device ns"
+DESCRIPTION="test NVMe smart-log command"
 QUICK=1
 
 requires() {
tests/nvme/024 deleted +0 -40
deleted file mode 100755
index a5121940a543..000000000000
@@ -1,40 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0+
-# Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
-#
-# Test NVMe smart-log command on NVMeOF with a file-backed ns.
-
-. tests/nvme/rc
-
-DESCRIPTION="test NVMe smart-log command on NVMeOF file-backed ns"
-QUICK=1
-
-requires() {
-	_nvme_requires
-	_have_loop
-	_require_nvme_trtype_is_fabrics
-}
-
-test() {
-	echo "Running ${TEST_NAME}"
-
-	_setup_nvmet
-
-	local ns
-
-	_nvmet_target_setup --blkdev file
-
-	_nvme_connect_subsys
-
-	ns=$(_find_nvme_ns ${def_subsys_uuid})
-
-	if ! nvme smart-log "/dev/${ns}" >> "$FULL" 2>&1; then
-		echo "ERROR: smart-log file-ns failed"
-	fi
-
-	_nvme_disconnect_subsys >> "$FULL" 2>&1
-
-	_nvmet_target_cleanup
-
-	echo "Test complete"
-}
tests/nvme/024.out deleted +0 -2
deleted file mode 100644
index 76c3e2931686..000000000000
@@ -1,2 +0,0 @@
-Running nvme/024
-Test complete
Signature · 1 line
2.44.0
#3 lore raw

PATCH 3/3 nvme/{021,022,025,026,027,028}: do not hard code target blkdev type

In reply to Daniel Wagner

There is no need to hardcode the target blkdev type. This allows the user to select different blkdev types via the nvmet_blkdev_type environment variable.

---
 tests/nvme/021 | 6 +++---
 tests/nvme/022 | 6 +++---
 tests/nvme/025 | 6 +++---
 tests/nvme/026 | 6 +++---
 tests/nvme/027 | 6 +++---
 tests/nvme/028 | 6 +++---
 6 files changed, 18 insertions(+), 18 deletions(-)
tests/nvme/021 +3 -3
index 7ee1f078cd60..0cd6b2b4fd67 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe list command on NVMeOF with a file-backed ns.
+# Test NVMe list command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe list command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe list command"
 QUICK=1
 
 requires() {
@@ -22,7 +22,7 @@ test() {
 
 	local ns
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
tests/nvme/022 +3 -3
index 7ce33ddbf006..41dca1497a91 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe reset command on NVMeOF with a file-backed ns.
+# Test NVMe reset command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe reset command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe reset command"
 QUICK=1
 
 requires() {
@@ -22,7 +22,7 @@ test() {
 
 	local nvmedev
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
tests/nvme/025 +3 -3
index 3f9a615e542e..603dc793f6a4 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe effects-log command on NVMeOF with a file-backed ns.
+# Test NVMe effects-log command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe effects-log command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe effects-log"
 QUICK=1
 
 requires() {
@@ -22,7 +22,7 @@ test() {
 
 	local ns
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
tests/nvme/026 +3 -3
index 28fd151d9a77..a96b40ce9cc1 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe ns-descs command on NVMeOF with a file-backed ns.
+# Test NVMe ns-descs command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe ns-descs command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe ns-descs"
 QUICK=1
 
 requires() {
@@ -22,7 +22,7 @@ test() {
 
 	local ns
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
tests/nvme/027 +3 -3
index 053fd58f9369..9eb8f244097b 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe ns-rescan command on NVMeOF with a file-backed ns.
+# Test NVMe ns-rescan command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe ns-rescan command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe ns-rescan command"
 QUICK=1
 
 requires() {
@@ -22,7 +22,7 @@ test() {
 
 	local nvmedev
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
tests/nvme/028 +3 -3
index 9f4a90581984..8754bc7e8805 100755
@@ -2,11 +2,11 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright (c) 2017-2018 Western Digital Corporation or its affiliates.
 #
-# Test NVMe list-subsys command on NVMeOF with a file-backed ns.
+# Test NVMe list-subsys command.
 
 . tests/nvme/rc
 
-DESCRIPTION="test NVMe list-subsys command on NVMeOF file-backed ns"
+DESCRIPTION="test NVMe list-subsys"
 QUICK=1
 
 requires() {
@@ -20,7 +20,7 @@ test() {
 
 	_setup_nvmet
 
-	_nvmet_target_setup --blkdev file
+	_nvmet_target_setup
 
 	_nvme_connect_subsys
 
Signature · 1 line
2.44.0
#5 lore raw
In reply to Daniel Wagner
Quoted 10 lines · Daniel Wagner

During the last batch of refactoring I noticed that we could reduce the number of tests a bit. There are tests which are almost identically except how the target is configured, file vs block device backend.

By introducing a configure knob, we can drop the duplicates and even make some of the tests a bit more versatile. Not all tests exists in file and block device backend version. Thus we increase the test coverage with this series. And not really surprising, there is a fallout. The nvme/028 test with file backend is failing in my setup but I was not able to figure out where things go wrong yet. I'll provide some logs later.

Hi Daniel, I find this series reduces code duplications further, and it expands test coverage with small test script changes. Good.

On the other hand, I see that the series has a couple of drawbacks:

1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.

2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.

To avoid the drawbacks, how about this?

- Do not provide nvmet_blkdev_type as a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)

- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.

I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?

tests/nvme/006 +7 -3
index 65f2a0d..6241961 100755
@@ -15,14 +15,18 @@ requires() {
 	_require_nvme_trtype_is_fabrics
 }
 
-test() {
-	echo "Running ${TEST_NAME}"
-
+do_test() {
 	_setup_nvmet
 
 	_nvmet_target_setup
 
 	_nvmet_target_cleanup
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	_nvmet_run_for_each_blkdev_type do_test
 
 	echo "Test complete"
 }
tests/nvme/rc +9 -0
index 5fa1871..4155411 100644
@@ -996,6 +996,15 @@ _nvmet_passthru_target_cleanup() {
 	_remove_nvmet_host "${def_hostnqn}"
 }
 
+_nvmet_run_for_each_blkdev_type() {
+	local blkdev_type
+
+	for blkdev_type in device file; do
+		nvmet_blkdev_type=$blkdev_type
+		eval "$1"
+	done
+}
+
 _discovery_genctr() {
 	_nvme_discover "${nvme_trtype}" |
 		sed -n -e 's/^.*Generation counter \([0-9]\+\).*$/\1/p'
#6 lore raw

Hi Shinichiro,

Quoted 11 lines · Shinichiro Kawasaki

On the other hand, I see that the series has a couple of drawbacks:

1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.

2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.

Yes, I agree. These drawbacks should be addressed somehow.

Quoted 6 lines

To avoid the drawbacks, how about this?

- Do not provide nvmet_blkdev_type as a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)

Quoted 6 lines

- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.

I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?

Ideally we don't have to introduce additional common setup logic into each test. Also for debugging purpose it might sense to run a test only in one configuration. So it might make sense still to have user visible environment variable

nvmet_blkdev_types="file device"

as default.

Quoted 5 lines
tests/nvme/006 +1 -3
-test() {
-	echo "Running ${TEST_NAME}"
-
+do_test() {
 	_setup_nvmet

tests/nvme/006 requires() { · line 29

Quoted 10 lines
tests/nvme/006 +6 -0
 
 	_nvmet_target_setup
 
 	_nvmet_target_cleanup
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	_nvmet_run_for_each_blkdev_type do_test

I was wondering if the nvmet_run_for_each_blkdev_type logic could be in common code, so we don't have to add a do_test function. Basically having a common code for a bunch of configuration variables (matrix tests). This could also be useful for nvmet_trtype etc.

The generic setup could be requested via the require hook.

requires() = {

_nvmet_setup_target
}

What do you think about this idea?

Thanks,
Daniel

#7 lore raw

Shinichiro/Daniel,

Quoted 10 lines · Daniel Wagner

Hi Shinichiro,

Quoted 7 lines · Shinichiro Kawasaki

On the other hand, I see that the series has a couple of drawbacks:

1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.

reducing code is always encouraged, but please don't change the test coverage and
make user add additional steps to run the tests, IOW ./check nvme should run all
the tests case by default as it dose today, this will keep your changes backward
compatible ..

I believe that is not too hard to achieve ?

-ck

#8 lore raw
Quoted 2 lines · Chaitanya Kulkarni

reducing code is always encouraged, but please don't change the test coverage and

This series increases the test coverage.

Quoted 5 lines

make user add additional steps to run the tests, IOW ./check nvme should run all
the tests case by default as it dose today, this will keep your changes backward
compatible ..

Well, that is not really true right now. E.g. nvme_trtype is loop by default. And you need to specify the additional transports manually.

This is also why I suggest we think about some sort of generic way to do this types of 'matrix configuration'(*) out of the box.

(* don't know how to name this better)

#9 lore raw
In reply to Daniel Wagner
Quoted 47 lines · Daniel Wagner

Hi Shinichiro,

Quoted 11 lines · Shinichiro Kawasaki

On the other hand, I see that the series has a couple of drawbacks:

1) When blktests users run with the default knob only, the test coverage will be smaller. To keep the current test coverage, the users need to run the check script twice: nvmet_blkdev_type=file and nvmet_blkdev_type=device. Some users may not do it and lose the test coverage. And some users, e.g., CKI project, need to adjust their script for this change.

2) When the users run the check script twice to keep the test coverage, some test cases are executed twice under the exact same test conditions. This will waste some of the test run effort.

Yes, I agree. These drawbacks should be addressed somehow.

Quoted 6 lines

To avoid the drawbacks, how about this?

- Do not provide nvmet_blkdev_type as a knob for users. Keep it as just a global variable in tests/nvme/rc. (It should be renamed to clarify that it is not for users.)

Quoted 6 lines

- Introduce a helper function to do the same test twice for nvmet_blkdev_type= file and nvmet_blkdev_type=device. Call this helper function from a single test case to cover both the blkdev types.

I attach a patch at the end of this email to show the ideas above. It applies the idea to nvme/006 as an example. What do you think?

Ideally we don't have to introduce additional common setup logic into each test. Also for debugging purpose it might sense to run a test only in one configuration. So it might make sense still to have user visible environment variable

nvmet_blkdev_types="file device"

as default.

Quoted 8 lines
tests/nvme/006 +1 -3
-test() {
-	echo "Running ${TEST_NAME}"
-
+do_test() {
 	_setup_nvmet
 
 	_nvmet_target_setup
Quoted 12 lines

tests/nvme/006 requires() { · line 29

Quoted 7 lines
tests/nvme/006 +6 -0
 	_nvmet_target_cleanup
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	_nvmet_run_for_each_blkdev_type do_test

I was wondering if the nvmet_run_for_each_blkdev_type logic could be in common code, so we don't have to add a do_test function. Basically having a common code for a bunch of configuration variables (matrix tests).

Actually, similar feature is implemented in the common code so that some test case can be run twice, once for regular the block device, and one more time for the zoned block device. You can find test cases with CAN_BE_ZONED=1 flag. They are run twice when RUN_ZONED_TESTS is set in the config.

To be precise, this applies to the test cases with test() function. CAN_BE_ZONED has different meaning for test cases with test_device().

Now we want to run some of test cases twice for the two nvmet block device types. This is essentially common feature as the repeated runs for the CAN_BE_ZONED test cases. I think it's time to generalize these two uses cases and support "repeated test case runs with different test conditions".

Quoted 1 line

This could also be useful for nvmet_trtype etc.

Yes, when I run for all of nvmet_trtype variations, I see some test cases are repeated with same condition. Waste of time.

Quoted 7 lines

The generic setup could be requested via the require hook.

requires() = {

_nvmet_setup_target
}

Hmm, I think this abuses the hook. IMO, it's the better to introduce a new hook.

Quoted 2 lines

What do you think about this idea?

It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays NVMET_BLKDEV_TYPES and NVMET_TR_TYPES. When these two are set in config file as follows,

NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)

it will run a single test case as follows. 2 x 3 = 6 times repeptitions.

$ sudo ./check nvme/006
nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target)  [passed]
    runtime  0.090s  ...  0.091s
nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target)  [passed]
    runtime  0.310s  ...  0.305s
nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target)   [passed]
    runtime  0.149s  ...  0.153s
nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target)    [passed]
    runtime  0.138s  ...  0.135s
nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target)    [passed]
    runtime  0.300s  ...  0.305s
nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target)     [passed]
    runtime  0.141s  ...  0.147s

I hope this meets your needs.

[*] https://github.com/kawasaki/blktests/tree/conditions

#10 lore raw
Quoted 5 lines · Shinichiro Kawasaki

Actually, similar feature is implemented in the common code so that some test case can be run twice, once for regular the block device, and one more time for the zoned block device. You can find test cases with CAN_BE_ZONED=1 flag. They are run twice when RUN_ZONED_TESTS is set in the config.

Quoted 7 lines

To be precise, this applies to the test cases with test() function. CAN_BE_ZONED has different meaning for test cases with test_device().

Now we want to run some of test cases twice for the two nvmet block device types. This is essentially common feature as the repeated runs for the CAN_BE_ZONED test cases. I think it's time to generalize these two uses cases and support "repeated test case runs with different test conditions".

Sounds reasonable.

Quoted 6 lines
Quoted 4 lines

requires() = {

_nvmet_setup_target
}

Hmm, I think this abuses the hook. IMO, it's the better to introduce a new hook.

Indeed :)

Quoted 27 lines
Quoted 2 lines

What do you think about this idea?

It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays NVMET_BLKDEV_TYPES and NVMET_TR_TYPES. When these two are set in config file as follows,

NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)

it will run a single test case as follows. 2 x 3 = 6 times repeptitions.

$ sudo ./check nvme/006
nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target)  [passed]
    runtime  0.090s  ...  0.091s
nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target)  [passed]
    runtime  0.310s  ...  0.305s
nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target)   [passed]
    runtime  0.149s  ...  0.153s
nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target)    [passed]
    runtime  0.138s  ...  0.135s
nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target)    [passed]
    runtime  0.300s  ...  0.305s
nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target)     [passed]
    runtime  0.141s  ...  0.147s
Quoted 1 line

I hope this meets your needs.

Yes, this is very useful.

Quoted 1 line

[*] https://github.com/kawasaki/blktests/tree/conditions

I quickly looked into the changes. The only thing I'd say it looks a bit hard to extend if we have yet another variable. But maybe I'd make it too complex. I don't think we have to be too future proof here, because we can change this part without problems, it is all 'under the hood' and doesn't change the 'user interface'.

Great stuff!

#11 lore raw
Quoted 1 line · Daniel Wagner

On Thu, Apr 04, 2024 at 07:43:28AM +0000, Shinichiro Kawasaki wrote:

...

Quoted 30 lines
Quoted 24 lines

It sounds an interesting idea :) I prototyped the common code change based on the idea and shared it on GitHub [*]. It introduces two new config arrays NVMET_BLKDEV_TYPES and NVMET_TR_TYPES. When these two are set in config file as follows,

NVMET_BLKDEV_TYPES=(device file)
NVMET_TR_TYPES=(loop rdma tcp)

it will run a single test case as follows. 2 x 3 = 6 times repeptitions.

$ sudo ./check nvme/006
nvme/006(nvmet dev=device tr=loop)(create an NVMeOF target)  [passed]
    runtime  0.090s  ...  0.091s
nvme/006(nvmet dev=device tr=rdma)(create an NVMeOF target)  [passed]
    runtime  0.310s  ...  0.305s
nvme/006(nvmet dev=device tr=tcp)(create an NVMeOF target)   [passed]
    runtime  0.149s  ...  0.153s
nvme/006(nvmet dev=file tr=loop)(create an NVMeOF target)    [passed]
    runtime  0.138s  ...  0.135s
nvme/006(nvmet dev=file tr=rdma)(create an NVMeOF target)    [passed]
    runtime  0.300s  ...  0.305s
nvme/006(nvmet dev=file tr=tcp)(create an NVMeOF target)     [passed]
    runtime  0.141s  ...  0.147s
Quoted 1 line

I hope this meets your needs.

Yes, this is very useful.

Quoted 1 line

[*] https://github.com/kawasaki/blktests/tree/conditions

Quoted 7 lines

I quickly looked into the changes. The only thing I'd say it looks a bit hard to extend if we have yet another variable. But maybe I'd make it too complex. I don't think we have to be too future proof here, because we can change this part without problems, it is all 'under the hood' and doesn't change the 'user interface'.

Great stuff!

Okay, thanks for the positive comment. I will brush up the patches and post for review. Let me have several days.