nvme testsuite runtime optimization
Subsystems (3)
v4 cover nvme testsuite runtime optimization
I've updated the series with the comments from the last round.
changes
v4:
- renamed test
- updated commit messages
- fixed whitespace damage
- fixed ShellCheck reports
- dropped 'limit number of random jobs'
- fio runtime 'invinity' changed to 1d
- fixed typo and updated documentation
- added tags
v3:
- 'nvme/rc' instead of 'nvme-rc' subject prefi x
- fixed spaces vs tabs indention
- nvme/014, nvme/015 dropped -1 in #block calculation
- common/xfs: use 'tail -1' instead 'awk'
- common/xfs: no need for printf for integer division
- nvme/035 limit disk size via nvme_img_size
- set defaults to previous values: nvme_img_size=1G, nvme_num_iter=1000
- nvme/032, nvme/040 let fio figure out image size
set timeout to infinity
cleanup fio
- add documentation for nvme_tr_type, nvme_img_size and nvme_num_iter
- limited nproc to 32 for num jobs for random fio job
- drop job size limit calculation
- drop nvme_num_iter from nvme/002
- limit xfs fio job size inside _xfs_run_fio_verify_io()
- added reviewed tags
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- made image size configurable via nvme_img_size env
- make number of iteration configurable via nvme_num_iter
- do not hard code test values
- calculate job size
- use runtime for fio background jobs
- https://lore.kernel.org/linux-nvme/[email protected]/
v1:
- initial version
- https://lore.kernel.org/linux-nvme/[email protected]/
Daniel Wagner (11):
nvme/rc: Auto convert test device size info nvme{014/015}: Do not hard code device size for dd test common/xfs: Make size argument optional for _xfs_run_fio_verify_io common/xfs: Limit fio size job to fit into xfs fs nvme{032,040}: Use runtime fio background jobs
Documentation: Add info on nvme_trtype
nvme: Make test image size configurable
nvme: Add minimal test image size requirement nvme{045,047}: Calculate IO size for random fio jobs nvme{002,030}: Move discovery generation counter code to rc nvme{016,017}: Make the number iterations configurable
Documentation/running-tests.md | 12 + common/rc | 30 +- common/xfs | 17 +- tests/nvme/002 | 4 +- tests/nvme/002.out | 3004 -------------------------------- tests/nvme/004 | 2 +- tests/nvme/005 | 2 +- tests/nvme/006 | 2 +- tests/nvme/007 | 2 +- tests/nvme/008 | 2 +- tests/nvme/009 | 2 +- tests/nvme/010 | 5 +- tests/nvme/011 | 5 +- tests/nvme/012 | 5 +- tests/nvme/013 | 5 +- tests/nvme/014 | 12 +- tests/nvme/015 | 12 +- tests/nvme/016 | 2 +- tests/nvme/017 | 4 +- tests/nvme/018 | 2 +- tests/nvme/019 | 2 +- tests/nvme/020 | 2 +- tests/nvme/021 | 2 +- tests/nvme/022 | 2 +- tests/nvme/023 | 2 +- tests/nvme/024 | 2 +- tests/nvme/025 | 2 +- tests/nvme/026 | 2 +- tests/nvme/027 | 2 +- tests/nvme/028 | 2 +- tests/nvme/029 | 2 +- tests/nvme/030 | 22 +- tests/nvme/031 | 2 +- tests/nvme/032 | 4 +- tests/nvme/034 | 2 +- tests/nvme/035 | 4 +- tests/nvme/040 | 11 +- tests/nvme/041 | 2 +- tests/nvme/042 | 2 +- tests/nvme/043 | 2 +- tests/nvme/044 | 2 +- tests/nvme/045 | 6 +- tests/nvme/047 | 8 +- tests/nvme/048 | 2 +- tests/nvme/rc | 40 + 45 files changed, 183 insertions(+), 3079 deletions(-)
Signature · 1 line
2.40.0
PATCH v4 1/11 nvme/rc: Auto convert test device size info
Introduce a convert_to_mb() helper which converts the size argument to MBytes and use in test device require function. This makes it possible to use user input strings in future.
--- common/rc | 30 +++++++++++++++++++++++++++--- tests/nvme/035 | 2 +- 2 files changed, 28 insertions(+), 4 deletions(-)
common/rc +27 -3
@@ -324,9 +324,12 @@ _get_pci_parent_from_blkdev() { tail -2 | head -1 } -_require_test_dev_size_mb() { - local require_sz_mb=$1 - local test_dev_sz_mb=$(($(blockdev --getsize64 "$TEST_DEV")/1024/1024)) +_require_test_dev_size() { + local require_sz_mb + local test_dev_sz_mb + + require_sz_mb="$(convert_to_mb "$1")" + test_dev_sz_mb="$(($(blockdev --getsize64 "$TEST_DEV")/1024/1024))" if (( "$test_dev_sz_mb" < "$require_sz_mb" )); then SKIP_REASONS+=("${TEST_DEV} required at least ${require_sz_mb}m")
@@ -422,3 +425,24 @@ _have_writeable_kmsg() { _run_user() { su "$NORMAL_USER" -c "$1" } + +convert_to_mb() +{ + local str=$1 + local res + + res=$(echo "${str}" | sed -n 's/^\([0-9]\+\)$/\1/p') + if [[ -n "${res}" ]]; then + echo "$((res / 1024 / 1024))" + fi + + res=$(echo "${str}" | sed -n 's/^\([0-9]\+\)[mM]$/\1/p') + if [[ -n "${res}" ]]; then + echo "$((res))" + fi + + res=$(echo "${str}" | sed -n 's/^\([0-9]\+\)[gG]$/\1/p') + if [[ -n "${res}" ]]; then + echo "$((res * 1024))" + fi +}
tests/nvme/035 +1 -1
@@ -17,7 +17,7 @@ requires() { } device_requires() { - _require_test_dev_size_mb 1024 + _require_test_dev_size 1024m } test_device() {
Signature · 1 line
2.40.0
PATCH v4 2/11 nvme{014/015}: Do not hard code device size for dd test
In reply to Daniel WagnerRead the block device sizes instead hard coding them, so that the device size can be configurable in future.
--- tests/nvme/014 | 10 +++++++++- tests/nvme/015 | 10 +++++++++- 2 files changed, 18 insertions(+), 2 deletions(-)
tests/nvme/014 +9 -1
@@ -23,6 +23,9 @@ test() { local port local nvmedev local loop_dev + local size + local bs + local count local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1"
@@ -41,7 +44,12 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - dd if=/dev/urandom of="/dev/${nvmedev}n1" count=128000 bs=4k status=none + size="$(blockdev --getsize64 "/dev/${nvmedev}n1")" + bs="$(blockdev --getbsz "/dev/${nvmedev}n1")" + count=$((size / bs)) + + dd if=/dev/urandom of="/dev/${nvmedev}n1" \ + count="${count}" bs="${bs}" status=none nvme flush "/dev/${nvmedev}" -n 1
tests/nvme/015 +9 -1
@@ -22,6 +22,9 @@ test() { local port local nvmedev + local size + local bs + local count local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1"
@@ -38,7 +41,12 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - dd if=/dev/urandom of="/dev/${nvmedev}n1" count=128000 bs=4k status=none + size="$(blockdev --getsize64 "/dev/${nvmedev}n1")" + bs="$(blockdev --getbsz "/dev/${nvmedev}n1")" + count=$((size / bs)) + + dd if=/dev/urandom of="/dev/${nvmedev}n1" \ + count="${count}" bs="${bs}" status=none nvme flush "/dev/${nvmedev}n1" -n 1
Signature · 1 line
2.40.0
PATCH v4 3/11 common/xfs: Make size argument optional for _xfs_run_fio_verify_io
In reply to Daniel WagnerMake the size argument optional by reading the filesystem info. The caller doesn't have to guess (or calculate) how big the max IO size. The log data structure of XFS is reducing the capacity.
--- common/xfs | 14 +++++++++++++- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 3 files changed, 15 insertions(+), 3 deletions(-)
common/xfs +13 -1
@@ -24,10 +24,22 @@ _xfs_run_fio_verify_io() { local mount_dir="/mnt/blktests" local bdev=$1 local sz=$2 + local sz_mb + local avail + local avail_mb _xfs_mkfs_and_mount "${bdev}" "${mount_dir}" >> "${FULL}" 2>&1 - _run_fio_verify_io --size="$sz" --directory="${mount_dir}/" + avail="$(df --output=avail "${mount_dir}" | tail -1)" + avail_mb="$((avail / 1024))" + + if [[ -z "${sz}" ]]; then + sz_mb="${avail_mb}" + else + sz_mb="$(convert_to_mb "${sz}")" + fi + + _run_fio_verify_io --size="${sz_mb}m" --directory="${mount_dir}/" umount "${mount_dir}" >> "${FULL}" 2>&1 rm -fr "${mount_dir}"
tests/nvme/012 +1 -1
@@ -44,7 +44,7 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - _xfs_run_fio_verify_io "/dev/${nvmedev}n1" "900m" + _xfs_run_fio_verify_io "/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
tests/nvme/013 +1 -1
@@ -41,7 +41,7 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - _xfs_run_fio_verify_io "/dev/${nvmedev}n1" "900m" + _xfs_run_fio_verify_io "/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
Signature · 1 line
2.40.0
PATCH v4 4/11 common/xfs: Limit fio size job to fit into xfs fs
In reply to Daniel WagnerThe usable capacity of the filesystem is less than the raw partition/device size due to the additional meta/log data.
Ensure that the job size for fio is not exceeding the limits.
Because we have hard coded the path where we mount the filesystem and don't want to expose this, we just update max size inside _xfs_run_fio_verify_io(). No need to leak this into the caller.
--- common/xfs | 3 +++ 1 file changed, 3 insertions(+)
common/xfs +3 -0
@@ -37,6 +37,9 @@ _xfs_run_fio_verify_io() { sz_mb="${avail_mb}" else sz_mb="$(convert_to_mb "${sz}")" + if [[ "${sz_mb}" -gt "${avail_mb}" ]]; then + sz_mb="${avail_mb}" + fi fi _run_fio_verify_io --size="${sz_mb}m" --directory="${mount_dir}/"
Signature · 1 line
2.40.0
Re: [PATCH blktests v4 04/11] common/xfs: Limit fio size job to fit into xfs fs
Quoted 12 lines · Daniel Wagner
The usable capacity of the filesystem is less than the raw partition/device size due to the additional meta/log data.
Ensure that the job size for fio is not exceeding the limits.
Because we have hard coded the path where we mount the filesystem and don't want to expose this, we just update max size inside
_xfs_run_fio_verify_io(). No need to leak this into the caller.Signed-off-by Daniel Wagner <[email protected]>---
make sense for the xfs fio verify, looks good.
-ck
PATCH v4 5/11 nvme{032,040}: Use runtime fio background jobs
In reply to Daniel WagnerThe fio jobs are supposed to run long in background during the test. Instead relying on a job size use explicit runtime for this.
--- tests/nvme/032 | 4 ++-- tests/nvme/040 | 9 +++++++-- 2 files changed, 9 insertions(+), 4 deletions(-)
tests/nvme/032 +2 -2
@@ -38,8 +38,8 @@ test_device() { sysfs="/sys/bus/pci/devices/${pdev}" # start fio job - _run_fio_rand_io --filename="$TEST_DEV" --size=1g \ - --group_reporting &> /dev/null & + _run_fio_rand_io --filename="$TEST_DEV" \ + --group_reporting --time_based --runtime=1d &> /dev/null & sleep 5
tests/nvme/040 +7 -2
@@ -21,6 +21,7 @@ test() { local port local loop_dev local nvmedev + local fio_pid echo "Running ${TEST_NAME}"
@@ -37,8 +38,10 @@ test() { # start fio job echo "starting background fio" - _run_fio_rand_io --filename="/dev/${nvmedev}n1" --size=1g \ - --group_reporting --ramp_time=5 &> /dev/null & + _run_fio_rand_io --filename="/dev/${nvmedev}n1" \ + --group_reporting --ramp_time=5 \ + --time_based --runtime=1d &> /dev/null & + fio_pid=$! sleep 5 # do reset/remove operation
@@ -48,6 +51,8 @@ test() { echo "deleting controller" _nvme_delete_ctrl "${nvmedev}" + { kill "${fio_pid}"; wait; } &> /dev/null + _remove_nvmet_subsystem_from_port "${port}" "${subsys}" _remove_nvmet_subsystem "${subsys}" _remove_nvmet_port "${port}"
Signature · 1 line
2.40.0
Re: [PATCH blktests v4 05/11] nvme{032,040}: Use runtime fio background jobs
Quoted 4 lines · Daniel Wagner
The fio jobs are supposed to run long in background during the test. Instead relying on a job size use explicit runtime for this.
Signed-off-by Daniel Wagner <[email protected]>
Is there any issue with the exiting approach for this change ?
-ck
Re: [PATCH blktests v4 05/11] nvme{032,040}: Use runtime fio background jobs
Quoted 7 lines · Chaitanya Kulkarni
Quoted 4 lines · Daniel Wagner
The fio jobs are supposed to run long in background during the test. Instead relying on a job size use explicit runtime for this.
Signed-off-by Daniel Wagner <[email protected]>Is there any issue with the exiting approach for this change ?
The expectation of the test here is that there is a background job running. Depending on the job size is an indirect way to express run at least for x seconds. This gives a variable runtime as it depends the how fast fio jobs gets executed. Explicitly telling the runtime is my opinion more robust and documents the indention better.
Chaitanya Kulkarni
Reviewed-byRe: [PATCH blktests v4 05/11] nvme{032,040}: Use runtime fio background jobs
Quoted 5 lines · Daniel Wagner
Quoted 4 lines · Chaitanya Kulkarni
Quoted 3 lines · Daniel Wagner
The fio jobs are supposed to run long in background during the test. Instead relying on a job size use explicit runtime for this.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Signed-off-by Daniel Wagner <[email protected]>Is there any issue with the exiting approach for this change ?
The expectation of the test here is that there is a background job running. Depending on the job size is an indirect way to express run at least for x seconds. This gives a variable runtime as it depends the how fast fio jobs gets executed. Explicitly telling the runtime is my opinion more robust and documents the indention better.
agree, it is better to kill on rely on the variable while test is running ...
-ck
PATCH v4 6/11 Documentation: Add info on nvme_trtype
In reply to Daniel WagnerMention that the nvme tests can be parametrized.
--- Documentation/running-tests.md | 7 +++++++ 1 file changed, 7 insertions(+)
Documentation/running-tests.md +7 -0
@@ -98,6 +98,13 @@ requires that the kernel be compiled with `CONFIG_BLK_DEV_ZONED` enabled. RUN_ZONED_TESTS=1 ``` +### NVMe test parameterizing + +The NVMe tests can be additionally parameterized via environment variables. + +- nvme_trtype: 'loop' (default), 'tcp', 'rdma' and 'fc' + Run the tests with the given transport. + ### Running nvme-rdma nvmeof-mp srp tests Most of these tests will use the rdma_rxe (soft-RoCE) driver by default. The siw (soft-iWARP) driver is also supported.
Signature · 1 line
2.40.0
PATCH v4 7/11 nvme: Make test image size configurable
In reply to Daniel WagnerThe reduce the overall runtime of the testsuite by making the default size of the test image small. For verification jobs, the default can be overwriten via the newly introduced nvme_img_size environment variable.
--- Documentation/running-tests.md | 3 +++ tests/nvme/004 | 2 +- tests/nvme/005 | 2 +- tests/nvme/006 | 2 +- tests/nvme/007 | 2 +- tests/nvme/008 | 2 +- tests/nvme/009 | 2 +- tests/nvme/010 | 5 +++-- tests/nvme/011 | 5 +++-- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- tests/nvme/014 | 2 +- tests/nvme/015 | 2 +- tests/nvme/017 | 2 +- tests/nvme/018 | 2 +- tests/nvme/019 | 2 +- tests/nvme/020 | 2 +- tests/nvme/021 | 2 +- tests/nvme/022 | 2 +- tests/nvme/023 | 2 +- tests/nvme/024 | 2 +- tests/nvme/025 | 2 +- tests/nvme/026 | 2 +- tests/nvme/027 | 2 +- tests/nvme/028 | 2 +- tests/nvme/029 | 2 +- tests/nvme/031 | 2 +- tests/nvme/034 | 2 +- tests/nvme/035 | 4 ++-- tests/nvme/040 | 2 +- tests/nvme/041 | 2 +- tests/nvme/042 | 2 +- tests/nvme/043 | 2 +- tests/nvme/044 | 2 +- tests/nvme/045 | 2 +- tests/nvme/047 | 2 +- tests/nvme/048 | 2 +- tests/nvme/rc | 1 + 38 files changed, 45 insertions(+), 39 deletions(-)
Documentation/running-tests.md +3 -0
@@ -104,6 +104,9 @@ The NVMe tests can be additionally parameterized via environment variables. - nvme_trtype: 'loop' (default), 'tcp', 'rdma' and 'fc' Run the tests with the given transport. +- nvme_img_size: '1G' (default) + Run the tests with given image size in bytes. 'm', 'M', 'g' + and 'G' postfix are supported. ### Running nvme-rdma nvmeof-mp srp tests
tests/nvme/004 +1 -1
@@ -25,7 +25,7 @@ test() { local port port="$(_create_nvmet_port "${nvme_trtype}")" - truncate -s 1G "$TMPDIR/img" + truncate -s "${nvme_img_size}" "$TMPDIR/img" local loop_dev loop_dev="$(losetup -f --show "$TMPDIR/img")"
tests/nvme/005 +1 -1
@@ -24,7 +24,7 @@ test() { local port port="$(_create_nvmet_port "${nvme_trtype}")" - truncate -s 1G "$TMPDIR/img" + truncate -s "${nvme_img_size}" "$TMPDIR/img" local loop_dev loop_dev="$(losetup -f --show "$TMPDIR/img")"
tests/nvme/006 +1 -1
@@ -24,7 +24,7 @@ test() { _setup_nvmet - truncate -s 1G "$TMPDIR/img" + truncate -s "${nvme_img_size}" "$TMPDIR/img" loop_dev="$(losetup -f --show "$TMPDIR/img")"
tests/nvme/007 +1 -1
@@ -25,7 +25,7 @@ test() { file_path="${TMPDIR}/img" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/008 +1 -1
@@ -26,7 +26,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/009 +1 -1
@@ -24,7 +24,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/010 +3 -2
@@ -26,7 +26,7 @@ test() { local file_path="${TMPDIR}/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
@@ -41,7 +41,8 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - _run_fio_verify_io --size=950m --filename="/dev/${nvmedev}n1" + _run_fio_verify_io --size="${nvme_img_size}" \ + --filename="/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
tests/nvme/011 +3 -2
@@ -26,7 +26,7 @@ test() { local file_path="${TMPDIR}/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
@@ -39,7 +39,8 @@ test() { cat "/sys/block/${nvmedev}n1/uuid" cat "/sys/block/${nvmedev}n1/wwid" - _run_fio_verify_io --size=950m --filename="/dev/${nvmedev}n1" + _run_fio_verify_io --size="${nvme_img_size}" \ + --filename="/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
tests/nvme/012 +1 -1
@@ -29,7 +29,7 @@ test() { local file_path="${TMPDIR}/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/013 +1 -1
@@ -28,7 +28,7 @@ test() { local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/014 +1 -1
@@ -29,7 +29,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/015 +1 -1
@@ -28,7 +28,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/017 +1 -1
@@ -25,7 +25,7 @@ test() { file_path="${TMPDIR}/img" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/018 +1 -1
@@ -26,7 +26,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/019 +1 -1
@@ -28,7 +28,7 @@ test() { local nblk_range="10,10,10,10,10,10,10,10,10,10" local sblk_range="100,200,300,400,500,600,700,800,900,1000" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/020 +1 -1
@@ -26,7 +26,7 @@ test() { local nblk_range="10,10,10,10,10,10,10,10,10,10" local sblk_range="100,200,300,400,500,600,700,800,900,1000" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/021 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/022 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/023 +1 -1
@@ -26,7 +26,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/024 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/025 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/026 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/027 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/028 +1 -1
@@ -25,7 +25,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
tests/nvme/029 +1 -1
@@ -59,7 +59,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 1G "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/031 +1 -1
@@ -33,7 +33,7 @@ test() { _setup_nvmet - truncate -s 1G "$TMPDIR/img" + truncate -s "${nvme_img_size}" "$TMPDIR/img" loop_dev="$(losetup -f --show "$TMPDIR/img")"
tests/nvme/034 +1 -1
@@ -26,7 +26,7 @@ test_device() { port=$(_nvmet_passthru_target_setup "${subsys}") nsdev=$(_nvmet_passthru_target_connect "${nvme_trtype}" "${subsys}") - _run_fio_verify_io --size=950m --filename="${nsdev}" + _run_fio_verify_io --size="${nvme_img_size}" --filename="${nsdev}" _nvme_disconnect_subsys "${subsys}" _nvmet_passthru_target_cleanup "${port}" "${subsys}"
tests/nvme/035 +2 -2
@@ -17,7 +17,7 @@ requires() { } device_requires() { - _require_test_dev_size 1024m + _require_test_dev_size "${nvme_img_size}" } test_device() {
@@ -32,7 +32,7 @@ test_device() { port=$(_nvmet_passthru_target_setup "${subsys}") nsdev=$(_nvmet_passthru_target_connect "${nvme_trtype}" "${subsys}") - _xfs_run_fio_verify_io "${nsdev}" "900m" + _xfs_run_fio_verify_io "${nsdev}" "${nvme_img_size}" _nvme_disconnect_subsys "${subsys}" _nvmet_passthru_target_cleanup "${port}" "${subsys}"
tests/nvme/040 +1 -1
@@ -26,7 +26,7 @@ test() { echo "Running ${TEST_NAME}" _setup_nvmet - truncate -s 1G "$TMPDIR/img" + truncate -s "${nvme_img_size}" "$TMPDIR/img" loop_dev="$(losetup -f --show "$TMPDIR/img")" port="$(_create_nvmet_port "${nvme_trtype}")"
tests/nvme/041 +1 -1
@@ -44,7 +44,7 @@ test() { _setup_nvmet - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "b92842df-a394-44b1-84a4-92ae7d112861"
tests/nvme/042 +1 -1
@@ -41,7 +41,7 @@ test() { _setup_nvmet - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" port="$(_create_nvmet_port "${nvme_trtype}")"
tests/nvme/043 +1 -1
@@ -42,7 +42,7 @@ test() { _setup_nvmet - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" port="$(_create_nvmet_port "${nvme_trtype}")"
tests/nvme/044 +1 -1
@@ -53,7 +53,7 @@ test() { _setup_nvmet - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" port="$(_create_nvmet_port "${nvme_trtype}")"
tests/nvme/045 +1 -1
@@ -55,7 +55,7 @@ test() { _setup_nvmet - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" port="$(_create_nvmet_port "${nvme_trtype}")"
tests/nvme/047 +1 -1
@@ -28,7 +28,7 @@ test() { local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1" - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" loop_dev="$(losetup -f --show "${file_path}")"
tests/nvme/048 +1 -1
@@ -84,7 +84,7 @@ test() { fi hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}" - truncate -s 512M "${file_path}" + truncate -s "${nvme_img_size}" "${file_path}" _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ "b92842df-a394-44b1-84a4-92ae7d112861"
tests/nvme/rc +1 -0
@@ -17,6 +17,7 @@ def_local_wwpn="0x20001100aa000002" def_hostnqn="$(cat /etc/nvme/hostnqn 2> /dev/null)" def_hostid="$(cat /etc/nvme/hostid 2> /dev/null)" nvme_trtype=${nvme_trtype:-"loop"} +nvme_img_size=${nvme_img_size:-"1G"} _nvme_requires() { _have_program nvme
Signature · 1 line
2.40.0
PATCH v4 8/11 nvme: Add minimal test image size requirement
In reply to Daniel WagnerSome tests need a minimal test image size to work correctly. Thus add a helper to check the size and update these tests accordingly.
The image minimum is 4M because some of the test have hard coded values. All tests which use the xfs fio verification job have a minimum requirement of 350M impossed by the xfs filesystem.
--- tests/nvme/012 | 1 + tests/nvme/013 | 1 + tests/nvme/045 | 2 +- tests/nvme/rc | 15 +++++++++++++++ 4 files changed, 18 insertions(+), 1 deletion(-)
tests/nvme/012 +1 -0
@@ -16,6 +16,7 @@ requires() { _have_fio _have_loop _require_nvme_trtype_is_fabrics + _require_nvme_test_img_size 350m } test() {
tests/nvme/013 +1 -0
@@ -15,6 +15,7 @@ requires() { _have_xfs _have_fio _require_nvme_trtype_is_fabrics + _require_nvme_test_img_size 350m } test() {
tests/nvme/045 +1 -1
@@ -120,7 +120,7 @@ test() { nvmedev=$(_find_nvme_dev "${subsys_name}") - _run_fio_rand_io --size=8m --filename="/dev/${nvmedev}n1" + _run_fio_rand_io --size=4m --filename="/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
tests/nvme/rc +15 -0
@@ -21,6 +21,7 @@ nvme_img_size=${nvme_img_size:-"1G"} _nvme_requires() { _have_program nvme + _require_nvme_test_img_size 4m case ${nvme_trtype} in loop) _have_driver nvme-loop
@@ -94,6 +95,20 @@ _require_test_dev_is_nvme() { return 0 } +_require_nvme_test_img_size() { + local require_sz_mb + local nvme_img_size_mb + + require_sz_mb="$(convert_to_mb "$1")" + nvme_img_size_mb="$(convert_to_mb "${nvme_img_size}")" + + if ((nvme_img_size_mb < require_sz_mb)); then + SKIP_REASONS+=("nvme_img_size must be at least ${require_sz_mb}m") + return 1 + fi + return 0 +} + _require_nvme_trtype() { local trtype for trtype in "$@"; do
Signature · 1 line
2.40.0
PATCH v4 9/11 nvme{045,047}: Calculate IO size for random fio jobs
In reply to Daniel Wagner_nvme_calc_run_io_size() returns the jobs size for _run_fio_rand_io() function. The jobs size is the size per job, thus we have to divide through the number of CPUs.
_xfs_run_fio_verify_io() is replaced with _run_fio_rand_io() because the former has a minimum nvme_img_size of 350M. Both tests nvme/{045,047} just want some IO to verify that the path is working. Thus reduce the min nmve_img_size requirement switch to _run_fio_rand_io()
--- tests/nvme/045 | 4 +++- tests/nvme/047 | 6 ++++-- tests/nvme/rc | 10 ++++++++++ 3 files changed, 17 insertions(+), 3 deletions(-)
tests/nvme/045 +3 -1
@@ -31,6 +31,7 @@ test() { local ctrlkey local new_ctrlkey local ctrldev + local rand_io_size echo "Running ${TEST_NAME}"
@@ -120,7 +121,8 @@ test() { nvmedev=$(_find_nvme_dev "${subsys_name}") - _run_fio_rand_io --size=4m --filename="/dev/${nvmedev}n1" + rand_io_size="$(_nvme_calc_rand_io_size 4m)" + _run_fio_rand_io --size="${rand_io_size}" --filename="/dev/${nvmedev}n1" _nvme_disconnect_subsys "${subsys_name}"
tests/nvme/047 +4 -2
@@ -25,6 +25,7 @@ test() { local port local nvmedev local loop_dev + local rand_io_size local file_path="$TMPDIR/img" local subsys_name="blktests-subsystem-1"
@@ -42,7 +43,8 @@ test() { nvmedev=$(_find_nvme_dev "${subsys_name}") - _xfs_run_fio_verify_io /dev/"${nvmedev}n1" "1m" || echo FAIL + rand_io_size="$(_nvme_calc_rand_io_size 4M)" + _run_fio_rand_io --filename="/dev/${nvmedev}n1" --size="${rand_io_size}" _nvme_disconnect_subsys "${subsys_name}" >> "$FULL" 2>&1
@@ -50,7 +52,7 @@ test() { --nr-write-queues 1 \ --nr-poll-queues 1 || echo FAIL - _xfs_run_fio_verify_io /dev/"${nvmedev}n1" "1m" || echo FAIL + _run_fio_rand_io --filename="/dev/${nvmedev}n1" --size="${rand_io_size}" _nvme_disconnect_subsys "${subsys_name}" >> "$FULL" 2>&1
tests/nvme/rc +10 -0
@@ -150,6 +150,16 @@ _test_dev_nvme_nsid() { cat "${TEST_DEV_SYSFS}/nsid" } +_nvme_calc_rand_io_size() { + local img_size_mb + local io_size_kb + + img_size_mb="$(convert_to_mb "$1")" + io_size_kb="$(((img_size_mb * 1024) / $(nproc)))" + + echo "${io_size_kb}k" +} + _nvme_fcloop_add_rport() { local local_wwnn="$1" local local_wwpn="$2"
Signature · 1 line
2.40.0
Re: [PATCH blktests v4 09/11] nvme{045,047}: Calculate IO size for random fio jobs
Quoted 3 lines · Daniel Wagner
_nvme_calc_run_io_size()returns the jobs size for_run_fio_rand_io()function. The jobs size is the size per job, thus we have to divide through the number of CPUs.
sorry I didn't understand why we have to divide through number of CPUs ? isn't tht will change the current job size of the test ?
unless we are increasing somewhere which I missed it .
-ck
Re: [PATCH blktests v4 09/11] nvme{045,047}: Calculate IO size for random fio jobs
Quoted 9 lines · Chaitanya Kulkarni
Quoted 3 lines · Daniel Wagner
_nvme_calc_run_io_size()returns the jobs size for_run_fio_rand_io()function. The jobs size is the size per job, thus we have to divide through the number of CPUs.sorry I didn't understand why we have to divide through number of CPUs ? isn't tht will change the current job size of the test ?
unless we are increasing somewhere which I missed it .
This change reduces the I/O size per job, but it keeps the total I/O size regardless of the number of CPUs. This will keep test case runtime reasonable on systems with hundreds of CPUs.
As for the test case nvme/045, it tests re-authentication. I don't think it requires total I/O size proportional to number of CPUs. As for the test case nvme/047, it exercises different queue types (write queue and poll queue). Does it require total I/O size proportional to number of CPUs? Daniel is the test case author, and I guessed he is ok with the change.
Re: [PATCH blktests v4 09/11] nvme{045,047}: Calculate IO size for random fio jobs
Sorry for the late response, had to deal with a lot of high prio stuff...
Quoted 6 lines · Shinichiro Kawasaki
Quoted 5 lines · Chaitanya Kulkarni
Quoted 3 lines · Daniel Wagner
_nvme_calc_run_io_size()returns the jobs size for_run_fio_rand_io()function. The jobs size is the size per job, thus we have to divide through the number of CPUs.
Quoted 8 lines
Quoted 4 lines
sorry I didn't understand why we have to divide through number of CPUs ? isn't tht will change the current job size of the test ?
unless we are increasing somewhere which I missed it .
This change reduces the I/O size per job, but it keeps the total I/O size regardless of the number of CPUs. This will keep test case runtime reasonable on systems with hundreds of CPUs.
Yes, indeed.
Quoted 5 lines
As for the test case nvme/045, it tests re-authentication. I don't think it requires total I/O size proportional to number of CPUs. As for the test case nvme/047, it exercises different queue types (write queue and poll queue). Does it require total I/O size proportional to number of CPUs? Daniel is the test case author, and I guessed he is ok with the change.
Yes :)
Thanks for applying these patches!
PATCH v4 10/11 nvme{002,030}: Move discovery generation counter code to rc
In reply to Daniel WagnerMove the discovery generation counter code to rc so that we can reuse it in 002.
--- tests/nvme/002 | 4 +- tests/nvme/002.out | 3004 -------------------------------------------- tests/nvme/030 | 22 +- tests/nvme/rc | 13 + 4 files changed, 20 insertions(+), 3023 deletions(-)
tests/nvme/002 +3 -1
@@ -27,12 +27,14 @@ test() { local loop_dev loop_dev="$(losetup -f)" + local genctr=1 + for ((i = 0; i < iterations; i++)); do _create_nvmet_subsystem "blktests-subsystem-$i" "${loop_dev}" _add_nvmet_subsys_to_port "${port}" "blktests-subsystem-$i" done - _nvme_discover "${nvme_trtype}" | _filter_discovery + genctr=$(_check_genctr "${genctr}" "adding a subsystem to a port") for ((i = iterations - 1; i >= 0; i--)); do _remove_nvmet_subsystem_from_port "${port}" "blktests-subsystem-$i"
tests/nvme/002.out +0 -3004
2,985 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.40.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH blktests v4 10/11] nvme{002,030}: Move discovery generation counter code to rc
Quoted 7 lines · Daniel Wagner
Move the discovery generation counter code to rc so that we can reuse it in 002.
Reviewed-by Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
Looks good.
-ck
PATCH v4 11/11 nvme{016,017}: Make the number iterations configurable
In reply to Daniel WagnerSome tests hard code high values of iterations. This makes them run relatively long compared to the other tests. Introduce a new environment variable nvme_num_iter to allow tune the runtime.
--- Documentation/running-tests.md | 2 ++ tests/nvme/016 | 2 +- tests/nvme/017 | 2 +- tests/nvme/rc | 1 + 4 files changed, 5 insertions(+), 2 deletions(-)
Documentation/running-tests.md +2 -0
@@ -107,6 +107,8 @@ The NVMe tests can be additionally parameterized via environment variables. - nvme_img_size: '1G' (default) Run the tests with given image size in bytes. 'm', 'M', 'g' and 'G' postfix are supported. +- nvme_num_iter: 1000 (default) + The number of iterations a test should do. ### Running nvme-rdma nvmeof-mp srp tests
tests/nvme/016 +1 -1
@@ -17,7 +17,7 @@ test() { echo "Running ${TEST_NAME}" local port - local iterations=1000 + local iterations="${nvme_num_iter}" local loop_dev local subsys_nqn="blktests-subsystem-1"
tests/nvme/017 +1 -1
@@ -18,7 +18,7 @@ test() { local port local file_path - local iterations=1000 + local iterations="${nvme_num_iter}" local subsys_name="blktests-subsystem-1" _setup_nvmet
tests/nvme/rc +1 -0
@@ -18,6 +18,7 @@ def_hostnqn="$(cat /etc/nvme/hostnqn 2> /dev/null)" def_hostid="$(cat /etc/nvme/hostid 2> /dev/null)" nvme_trtype=${nvme_trtype:-"loop"} nvme_img_size=${nvme_img_size:-"1G"} +nvme_num_iter=${nvme_num_iter:-"1000"} _nvme_requires() { _have_program nvme
Signature · 1 line
2.40.0
Shinichiro Kawasaki
AppliedQuoted 12 lines · Daniel Wagner
I've updated the series with the comments from the last round.
changes
v4:
- renamed test
- updated commit messages
- fixed whitespace damage
- fixed ShellCheck reports
- dropped 'limit number of random jobs'
- fio runtime 'invinity' changed to 1d
- fixed typo and updated documentation
- added tags
I've applied this v4 series. Thanks!