test queue count changes on reconnect
Subsystems (2)
v5 cover test queue count changes on reconnect
The target is allowed to change the number of i/o queues. test if the host is able to reconnect in this scenario.
I was able to test this successfully with tcp and rdma. fc still fails due to the race condition in the target code.
Next attempt to get this series upstream :)
This version is based on my previous posted nvme/047 patches [1]
[1] https://lore.kernel.org/linux-nvme/[email protected]/
v5:
- moved generic parts to nvme/rc
- renamed test to 048
- rebased ontop of nvme/047
v4:
- do not remove ports instead depend on host removing controllers, see
https://lore.kernel.org/linux-nvme/[email protected]/
- https://lore.kernel.org/linux-nvme/[email protected]/
v3:
- Added comment why at least 2 CPUs are needed for the test
- Fixed shell quoting in _set_nvmet_attr_qid_max
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- detect if attr_qid_max is available
- https://lore.kernel.org/linux-block/[email protected]/
v1:
- https://lore.kernel.org/linux-block/[email protected]/
Daniel Wagner (4):
nvme/rc: Add setter for attr_qid_max
nvme/rc: Add nvmet attribute feature detection function nvme/rc: Add helper to wait for nvme ctrl state nvme/048: test queue count changes on reconnect
tests/nvme/048 | 81 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nvme/048.out | 3 ++ tests/nvme/rc | 58 +++++++++++++++++++++++++++++++++ 3 files changed, 142 insertions(+) create mode 100755 tests/nvme/048 create mode 100644 tests/nvme/048.out
Signature · 1 line
2.40.0
PATCH v5 1/4 nvme/rc: Add setter for attr_qid_max
--- tests/nvme/rc | 10 ++++++++++ 1 file changed, 10 insertions(+)
tests/nvme/rc +10 -0
@@ -617,6 +617,16 @@ _set_nvmet_dhgroup() { "${cfs_path}/dhchap_dhgroup" } +_set_nvmet_attr_qid_max() { + local nvmet_subsystem="$1" + local qid_max="$2" + local cfs_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}" + + if [[ -f "${cfs_path}/attr_qid_max" ]]; then + echo $qid_max > "${cfs_path}/attr_qid_max" + fi +} + _find_nvme_dev() { local subsys=$1 local subsysnqn
Signature · 1 line
2.40.0
Re: [PATCH blktests v5 1/4] nvme/rc: Add setter for attr_qid_max
Quoted 14 lines · Daniel Wagner
Signed-off-by Daniel Wagner <[email protected]>--- tests/nvme/rc | 10 ++++++++++ 1 file changed, 10 insertions(+)tests/nvme/rc +1
-0@@ -617,6 +617,16 @@ _set_nvmet_dhgroup() { "${cfs_path}/dhchap_dhgroup" } +_set_nvmet_attr_qid_max() {
tests/nvme/rc _set_nvmet_dhgroup() { · line 630
Quoted 10 lines
tests/nvme/rc +9
-0+ local nvmet_subsystem="$1" + local qid_max="$2" + local cfs_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}" + + if [[ -f "${cfs_path}/attr_qid_max" ]]; then + echo $qid_max > "${cfs_path}/attr_qid_max" + fi +} +
this is only used in the testcase patch #4, until we get a second user move it to patch #4 ?
in case this was already discussed and decision made to keep it in rc, please ignore this comment.
-ck
Re: [PATCH blktests v5 1/4] nvme/rc: Add setter for attr_qid_max
Quoted 5 lines · Chaitanya Kulkarni
this is only used in the testcase patch #4, until we get a second user move it to patch #4 ?
in case this was already discussed and decision made to keep it in rc, please ignore this comment.
There wasn't any decision on this topic. I was not sure if I should put it in rc but I saw there are already _set_nvmet_*() functions. Thus I came to the conclusion it makes maintaining these helper function simpler in future because they are all in one file. If someone touches all _set_nvmet_*() function this one is not forgotten.
The same goes for the other rc helpers (patch 1-3).
That said, I really do not insit in putiting in rc.
Re: [PATCH blktests v5 1/4] nvme/rc: Add setter for attr_qid_max
Quoted 12 lines · Daniel Wagner
Quoted 5 lines · Chaitanya Kulkarni
this is only used in the testcase patch #4, until we get a second user move it to patch #4 ?
in case this was already discussed and decision made to keep it in rc, please ignore this comment.
There wasn't any decision on this topic. I was not sure if I should put it in rc but I saw there are already _set_nvmet_*() functions. Thus I came to the conclusion it makes maintaining these helper function simpler in future because they are all in one file. If someone touches all _set_nvmet_*() function this one is not forgotten.
Quoted 3 lines
The same goes for the other rc helpers (patch 1-3).
That said, I really do not insit in putiting in rc.
if there are functions in rc which are only testcase specific let's keep those there, we can always do cleanup later ..
-ck
Re: [PATCH blktests v5 1/4] nvme/rc: Add setter for attr_qid_max
On Thu, Apr 06, 2023 at 07:05:45AM +0000, Chaitanya Kulkarni wrote:
if there are functions in rc which are only testcase specific let's keep
Quoted 1 line
those there, we can always do cleanup later ..
Sure, works for me too.
PATCH v5 2/4 nvme/rc: Add nvmet attribute feature detection function
In reply to Daniel WagnerIn order to detect if the target supports a attribute, we have to setup a target and check if the attribute file appears.
--- tests/nvme/rc | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
tests/nvme/rc +25 -0
@@ -627,6 +627,31 @@ _set_nvmet_attr_qid_max() { fi } +_detect_nvmet_subsys_attr() { + local attr="$1" + local file_path="${TMPDIR}/img" + local subsys_name="blktests-feature-detect" + local cfs_path="${NVMET_CFS}/subsystems/${subsys_name}" + local port + + truncate -s 1M "${file_path}" + + _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ + "b92842df-a394-44b1-84a4-92ae7d112332" + port="$(_create_nvmet_port "${nvme_trtype}")" + + local val=1 + [[ -f "${cfs_path}/${attr}" ]] && val=0 + + _remove_nvmet_subsystem "${subsys_name}" + + _remove_nvmet_port "${port}" + + rm "${file_path}" + + return "${val}" +} + _find_nvme_dev() { local subsys=$1 local subsysnqn
Signature · 1 line
2.40.0
PATCH v5 3/4 nvme/rc: Add helper to wait for nvme ctrl state
In reply to Daniel WagnerThe function waits until it sees a given state (or timeouts). Unfortunately, we can't use /sys/class/nvme/nvme%d because this symlink will be removed if the controller is going through resetting state. Thus use the real nvme%d directly.
--- tests/nvme/rc | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
tests/nvme/rc +23 -0
@@ -16,6 +16,7 @@ def_local_wwnn="0x10001100aa000002" def_local_wwpn="0x20001100aa000002" def_hostnqn="$(cat /etc/nvme/hostnqn 2> /dev/null)" def_hostid="$(cat /etc/nvme/hostid 2> /dev/null)" +def_state_timeout=20 nvme_trtype=${nvme_trtype:-"loop"} _nvme_requires() {
@@ -198,6 +199,28 @@ _nvme_fcloop_del_tport() { echo "wwnn=${wwnn},wwpn=${wwpn}" > ${loopctl}/del_target_port 2> /dev/null } +_nvmf_wait_for_state() { + local subsys_name="$1" + local state="$2" + local timeout="${3:-$def_state_timeout}" + + local nvmedev=$(_find_nvme_dev "${subsys_name}") + local state_file="/sys/class/nvme-fabrics/ctl/${nvmedev}/state" + + local start_time=$(date +%s) + local end_time + + while ! grep -q "${state}" "${state_file}"; do + sleep 1 + end_time=$(date +%s) + if (( end_time - start_time > timeout )); then + echo "expected state \"${state}\" not " \ + "reached within ${timeout} seconds" + break + fi + done +} + _cleanup_fcloop() { local local_wwnn="${1:-$def_local_wwnn}" local local_wwpn="${2:-$def_local_wwpn}"
Signature · 1 line
2.40.0
Re: [PATCH blktests v5 3/4] nvme/rc: Add helper to wait for nvme ctrl state
Quoted 8 lines · Daniel Wagner
The function waits until it sees a given state (or timeouts). Unfortunately, we can't use /sys/class/nvme/nvme%d because this symlink will be removed if the controller is going through resetting state. Thus use the real nvme%d directly.
Signed-off-by Daniel Wagner <[email protected]>---
Looks good.
-ck
PATCH v5 4/4 nvme/048: test queue count changes on reconnect
In reply to Daniel WagnerThe target is allowed to change the number of I/O queues. Test if the host is able to reconnect in this scenario.
--- tests/nvme/048 | 81 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nvme/048.out | 3 ++ 2 files changed, 84 insertions(+) create mode 100755 tests/nvme/048 create mode 100644 tests/nvme/048.out
tests/nvme/048 new file +81 -0
@@ -0,0 +1,81 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-3.0+ +# Copyright (C) 2023 Daniel Wagner, SUSE Labs +# +# Test queue count changes on reconnect + +. tests/nvme/rc + +DESCRIPTION="Test queue count changes on reconnect" +QUICK=1 + +requires() { + _nvme_requires + _have_loop + _require_nvme_trtype tcp rdma fc + _require_min_cpus 2 +} + +set_max_qid() { + local port="$1" + local subsys_name="$2" + local max_qid="$3" + + _set_nvmet_attr_qid_max "${subsys_name}" "${max_qid}" + + # Setting qid_max forces a disconnect and the reconntect attempt starts + # 10s after connection lost, thus we will see the connecting state. + _nvmf_wait_for_state "${subsys_name}" "connecting" + _nvmf_wait_for_state "${subsys_name}" "live" +} + +test() { + local port + local subsys_name="blktests-subsystem-1" + local hostid + local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}" + local file_path="${TMPDIR}/img" + + echo "Running ${TEST_NAME}" + + hostid="$(uuidgen)" + if [ -z "$hostid" ] ; then + echo "uuidgen failed" + return 1 + fi + + if ! _detect_nvmet_subsys_attr "attr_qid_max"; then + SKIP_REASONS+=("missing attr_qid_max feature") + return 1 + fi + + truncate -s 512M "${file_path}" + + _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ + "b92842df-a394-44b1-84a4-92ae7d112861" + port="$(_create_nvmet_port "${nvme_trtype}")" + _add_nvmet_subsys_to_port "${port}" "${subsys_name}" + _create_nvmet_host "${subsys_name}" "${hostnqn}" + + _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \ + --hostnqn "${hostnqn}" \ + --hostid "${hostid}" + + _nvmf_wait_for_state "${subsys_name}" "live" + + set_max_qid "${port}" "${subsys_name}" 1 + set_max_qid "${port}" "${subsys_name}" 128 + + _nvme_disconnect_subsys "${subsys_name}" + + _remove_nvmet_subsystem_from_port "${port}" "${subsys_name}" + _remove_nvmet_subsystem "${subsys_name}" + + _remove_nvmet_port "${port}" + + _remove_nvmet_host "${hostnqn}" + + rm "${file_path}" + + echo "Test complete" +}
tests/nvme/048.out new file +3 -0
@@ -0,0 +1,3 @@ +Running nvme/048 +NQN:blktests-subsystem-1 disconnected 1 controller(s) +Test complete
Signature · 1 line
2.40.0
Re: [PATCH blktests v5 4/4] nvme/048: test queue count changes on reconnect
Quoted 6 lines · Daniel Wagner
The target is allowed to change the number of I/O queues. Test if the host is able to reconnect in this scenario.
Signed-off-by Daniel Wagner <[email protected]>---
Looks good.
-ck
Re: [PATCH blktests v5 4/4] nvme/048: test queue count changes on reconnect
In reply to Daniel Wagnertests/nvme/048 line 55
Quoted 10 lines
tests/nvme/048 new file +9
-0+ if ! _detect_nvmet_subsys_attr "attr_qid_max"; then + SKIP_REASONS+=("missing attr_qid_max feature") + return 1 + fi + + truncate -s 512M "${file_path}" + + _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ + "b92842df-a394-44b1-84a4-92ae7d112861"
by checking following after create subsystem in testcase itself we avoid whole process of creating and deleting subsystem and additional function in the rc file, because we are already creating subsystem as a part of the testcase :-
local attr="${NVMET_CFS}/subsystems/${subsys_name}/attr_qid_max"
#above tow vars go top of this function
if [ -f "${attr}" ];then
SKIP_REASONS+=("missing attr_qid_max feature") #do appropriate error handling and jump to unwind code
fi
again please ignore this comment if decision has been made to keep it this way for some reason...
tests/nvme/048 line 59
Quoted 4 lines
tests/nvme/048 new file +4
-0+ port="$(_create_nvmet_port "${nvme_trtype}")" + _add_nvmet_subsys_to_port "${port}" "${subsys_name}" + _create_nvmet_host "${subsys_name}" "${hostnqn}" +
-ck
Re: [PATCH blktests v5 4/4] nvme/048: test queue count changes on reconnect
Quoted 20 lines · Chaitanya Kulkarni
tests/nvme/048 line 55
Quoted 10 lines
tests/nvme/048 new file +9
-0+ if ! _detect_nvmet_subsys_attr "attr_qid_max"; then + SKIP_REASONS+=("missing attr_qid_max feature") + return 1 + fi + + truncate -s 512M "${file_path}" + + _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ + "b92842df-a394-44b1-84a4-92ae7d112861"by checking following after create subsystem in testcase itself we avoid whole process of creating and deleting subsystem and additional function in the rc file, because we are already creating subsystem as a part of the testcase :-
local attr="${NVMET_CFS}/subsystems/${subsys_name}/attr_qid_max"
#above tow vars go top of this function
Quoted 7 lines
if [ -f "${attr}" ];then
SKIP_REASONS+=("missingattr_qid_maxfeature") #do appropriate error handling and jump to unwind code
fiagain please ignore this comment if decision has been made to keep it this way for some reason...
Again, no decision here. I think I overengineered this part slightly. Indeed if we are goint to setup a controller anyway we should try to avoid double work. This should also speed up the test slightly.
Talking about execution time, I was thinking on reducing the timeout value to reduce the overall runtime.
Re: [PATCH blktests v5 4/4] nvme/048: test queue count changes on reconnect
Quoted 24 lines · Daniel Wagner
Quoted 23 lines · Chaitanya Kulkarni
tests/nvme/048 line 55
Quoted 9 lines
tests/nvme/048 new file +9
-0+ if ! _detect_nvmet_subsys_attr "attr_qid_max"; then + SKIP_REASONS+=("missing attr_qid_max feature") + return 1 + fi + + truncate -s 512M "${file_path}" + + _create_nvmet_subsystem "${subsys_name}" "${file_path}" \ + "b92842df-a394-44b1-84a4-92ae7d112861"by checking following after create subsystem in testcase itself we avoid whole process of creating and deleting subsystem and additional function in the rc file, because we are already creating subsystem as a part of the testcase :-
local attr="${NVMET_CFS}/subsystems/${subsys_name}/attr_qid_max"
#above tow vars go top of this function
if [ -f "${attr}" ];then
SKIP_REASONS+=("missingattr_qid_maxfeature") #do appropriate error handling and jump to unwind code
fi
Quoted 8 lines
Quoted 2 lines
again please ignore this comment if decision has been made to keep it this way for some reason...
Again, no decision here. I think I overengineered this part slightly. Indeed if we are goint to setup a controller anyway we should try to avoid double work. This should also speed up the test slightly.
Talking about execution time, I was thinking on reducing the timeout value to reduce the overall runtime.
okay let's go with the above suggestion than ? and overall let's only add to rc when there are multiple users to the function with prep patches moving the code, otherwise it gets bloated for no reason..
overall speed of the testcase is also very important since this gets run with bunch of other testcases/testsuite(s) so keeping the testcase speed minimum is always desirable ...
-ck