Skip to contentSkip to conversation

test queue count changes on reconnect

PATCH Applied Daniel Wagner 2 participants 8 messages Apr 6, 2023 – Apr 12, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v6 cover test queue count changes on reconnect

The target is allowed to change the number of i/o queues. Test if the host is able to reconnect in this scenario.

I've incorperated Chaitanya's feedback and also optimized the runtime for the good and the bad case (rdma started to fail today, something to fix). Before it run for rougly 25 seconds when everytying was okay.

# nvme_trtype=rdma ./check nvme/048
nvme/048 (Test queue count changes on reconnect) [failed]
runtime 6.488s ... 6.404s
--- tests/nvme/048.out 2023-04-06 09:38:29.574194562 +0200
+++ /home/wagi/work/blktests/results/nodev/nvme/048.out.bad 2023-04-06 10:09:47.692036702 +0200 @@ -1,3 +1,11 @@
Running nvme/048
-NQN:blktests-subsystem-1 disconnected 1 controller(s)
+grep: /sys/class/nvme-fabrics/ctl//state: No such file or directory
+grep: /sys/class/nvme-fabrics/ctl//state: No such file or directory
+grep: /sys/class/nvme-fabrics/ctl//state: No such file or directory
+grep: /sys/class/nvme-fabrics/ctl//state: No such file or directory
+grep: /sys/class/nvme-fabrics/ctl//state: No such file or directory
...
(Run 'diff -u tests/nvme/048.out /home/wagi/work/blktests/results/nodev/nvme/048.out.bad' to see the entire diff)
# nvme_trtype=tcp ./check nvme/048
nvme/048 (Test queue count changes on reconnect) [passed]

      runtime  6.350s  ...  6.227s

This version is based on my previous posted nvme/047 patches [1]

[1] https://lore.kernel.org/linux-nvme/[email protected]/

v6:
- moved generic rc bits back into test case
- added checks to fail early
- added timeout values parser for connect call
- reduced timeouts (runtime reduction for good and bad case)
- fixed shellcheck warnings
v5:
- moved generic parts to nvme/rc
- renamed test to 048
- rebased ontop of nvme/047
- https://lore.kernel.org/linux-nvme/[email protected]/
v4:
- do not remove ports instead depend on host removing controllers, see
https://lore.kernel.org/linux-nvme/[email protected]/
- https://lore.kernel.org/linux-nvme/[email protected]/
v3:
- Added comment why at least 2 CPUs are needed for the test
- Fixed shell quoting in _set_nvmet_attr_qid_max
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- detect if attr_qid_max is available
- https://lore.kernel.org/linux-block/[email protected]/
v1:
- https://lore.kernel.org/linux-block/[email protected]/

Daniel Wagner (2):
nvme/rc: Add timeout argument parsing to _nvme_connect_subsys() nvme/048: test queue count changes on reconnect

 tests/nvme/048     | 125 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/048.out |   3 ++
 tests/nvme/rc      |  24 +++++++++
 3 files changed, 152 insertions(+)
 create mode 100755 tests/nvme/048
 create mode 100644 tests/nvme/048.out
Signature · 1 line
2.40.0
#1 lore raw

PATCH v6 1/2 nvme/rc: Add timeout argument parsing to _nvme_connect_subsys()

Add the possiblity for tests to specify the timeout values. This makes it possible to reduce the test runtime.

---
 tests/nvme/rc | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tests/nvme/rc +24 -0
index a3c9b42d91e6..b44239446dcf 100644
@@ -338,6 +338,9 @@ _nvme_connect_subsys() {
 	local nr_io_queues=""
 	local nr_write_queues=""
 	local nr_poll_queues=""
+	local keep_alive_tmo=""
+	local reconnect_delay=""
+	local ctrl_loss_tmo=""
 
 	while [[ $# -gt 0 ]]; do
 		case $1 in
@@ -381,6 +384,18 @@ _nvme_connect_subsys() {
 				nr_poll_queues="$2"
 				shift 2
 				;;
+			-k|--keep-alive-tmo)
+				keep_alive_tmo="$2"
+				shift 2
+				;;
+			-c|--reconnect-delay)
+				reconnect_delay="$2"
+				shift 2
+				;;
+			-l|--ctrl-loss-tmo)
+				ctrl_loss_tmo="$2"
+				shift 2
+				;;
 			*)
 				positional_args+=("$1")
 				shift
@@ -420,6 +435,15 @@ _nvme_connect_subsys() {
 	if [[ -n "${nr_poll_queues}" ]]; then
 		ARGS+=(--nr-poll-queues="${nr_poll_queues}")
 	fi
+	if [[ -n "${keep_alive_tmo}" ]]; then
+		ARGS+=(--keep-alive-tmo="${keep_alive_tmo}")
+	fi
+	if [[ -n "${reconnect_delay}" ]]; then
+		ARGS+=(--reconnect-delay="${reconnect_delay}")
+	fi
+	if [[ -n "${ctrl_loss_tmo}" ]]; then
+		ARGS+=(--ctrl-loss-tmo="${ctrl_loss_tmo}")
+	fi
 
 	nvme connect "${ARGS[@]}" 2> /dev/null
 }
Signature · 1 line
2.40.0
#2 lore raw

PATCH v6 2/2 nvme/048: test queue count changes on reconnect

In reply to Daniel Wagner

The target is allowed to change the number of I/O queues. Test if the host is able to reconnect in this scenario.

---
 tests/nvme/048     | 125 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/048.out |   3 ++
 2 files changed, 128 insertions(+)
 create mode 100755 tests/nvme/048
 create mode 100644 tests/nvme/048.out
tests/nvme/048 new file +125 -0
new file mode 100755
index 000000000000..926f9f3de955
@@ -0,0 +1,125 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2023 Daniel Wagner, SUSE Labs
+#
+# Test queue count changes on reconnect
+
+. tests/nvme/rc
+
+DESCRIPTION="Test queue count changes on reconnect"
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype tcp rdma fc
+	_require_min_cpus 2
+}
+
+nvmf_wait_for_state() {
+	local def_state_timeout=5
+	local subsys_name="$1"
+	local state="$2"
+	local timeout="${3:-$def_state_timeout}"
+	local nvmedev
+	local state_file
+	local start_time
+	local end_time
+
+	nvmedev=$(_find_nvme_dev "${subsys_name}")
+	state_file="/sys/class/nvme-fabrics/ctl/${nvmedev}/state"
+
+	start_time=$(date +%s)
+	while ! grep -q "${state}" "${state_file}"; do
+		sleep 1
+		end_time=$(date +%s)
+                if (( end_time - start_time > timeout )); then
+			echo "expected state \"${state}\" not " \
+				"reached within ${timeout} seconds"
+			return 1
+		fi
+	done
+
+	return 0
+}
+
+set_nvmet_attr_qid_max() {
+	local nvmet_subsystem="$1"
+	local qid_max="$2"
+	local cfs_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
+
+	echo "${qid_max}" > "${cfs_path}/attr_qid_max"
+}
+
+set_qid_max() {
+	local port="$1"
+	local subsys_name="$2"
+	local qid_max="$3"
+
+	set_nvmet_attr_qid_max "${subsys_name}" "${qid_max}"
+
+	# Setting qid_max forces a disconnect and the reconntect attempt starts
+	nvmf_wait_for_state "${subsys_name}" "connecting" || return 1
+	nvmf_wait_for_state "${subsys_name}" "live" || return 1
+
+	return 0
+}
+
+test() {
+	local subsys_name="blktests-subsystem-1"
+	local cfs_path="${NVMET_CFS}/subsystems/${subsys_name}"
+	local file_path="${TMPDIR}/img"
+	local skipped=false
+	local hostnqn
+	local hostid
+	local port
+
+	echo "Running ${TEST_NAME}"
+
+	hostid="$(uuidgen)"
+	if [ -z "$hostid" ] ; then
+		echo "uuidgen failed"
+		return 1
+	fi
+	hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}" \
+		"b92842df-a394-44b1-84a4-92ae7d112861"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}"
+
+	if [[ -f "${cfs_path}/attr_qid_max" ]] ; then
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+					--hostnqn "${hostnqn}" \
+					--hostid "${hostid}" \
+					--keep-alive-tmo 1 \
+					--reconnect-delay 2
+
+		if ! nvmf_wait_for_state "${subsys_name}" "live" ; then
+			echo FAIL
+		else
+			set_qid_max "${port}" "${subsys_name}" 1 || echo FAIL
+			set_qid_max "${port}" "${subsys_name}" 128 || echo FAIL
+		fi
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	else
+		SKIP_REASONS+=("missing attr_qid_max feature")
+		skipped=true
+	fi
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+	_remove_nvmet_port "${port}"
+	_remove_nvmet_host "${hostnqn}"
+
+	rm "${file_path}"
+
+	if [[ "${skipped}" = true ]] ; then
+		return 1
+	fi
+
+	echo "Test complete"
+}
tests/nvme/048.out new file +3 -0
new file mode 100644
index 000000000000..7f986ef9637d
@@ -0,0 +1,3 @@
+Running nvme/048
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.40.0
#3 lore raw

Re: [PATCH blktests v6 2/2] nvme/048: test queue count changes on reconnect

tests/nvme/048 line 76

Quoted 10 lines · Daniel Wagner
tests/nvme/048 new file +10 -0
+test() {
+	local subsys_name="blktests-subsystem-1"
+	local cfs_path="${NVMET_CFS}/subsystems/${subsys_name}"
+	local file_path="${TMPDIR}/img"
+	local skipped=false
+	local hostnqn
+	local hostid
+	local port
+
+	echo "Running ${TEST_NAME}"

I think I missing a _nvmet_setup call here.

#4 lore raw

Re: [PATCH blktests v6 2/2] nvme/048: test queue count changes on reconnect

In reply to Daniel Wagner
Quoted 42 lines · Daniel Wagner

The target is allowed to change the number of I/O queues. Test if the host is able to reconnect in this scenario.

---
 tests/nvme/048     | 125 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/048.out |   3 ++
 2 files changed, 128 insertions(+)
 create mode 100755 tests/nvme/048
 create mode 100644 tests/nvme/048.out
tests/nvme/048 new file +25 -0
new file mode 100755
index 000000000000..926f9f3de955
@@ -0,0 +1,125 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2023 Daniel Wagner, SUSE Labs
+#
+# Test queue count changes on reconnect
+
+. tests/nvme/rc
+
+DESCRIPTION="Test queue count changes on reconnect"
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype tcp rdma fc
+	_require_min_cpus 2
+}
+
+nvmf_wait_for_state() {
+	local def_state_timeout=5
+	local subsys_name="$1"
+	local state="$2"
+	local timeout="${3:-$def_state_timeout}"
+	local nvmedev
+	local state_file
+	local start_time

tests/nvme/048 line 35

Quoted 10 lines
tests/nvme/048 new file +10 -0
+	local end_time
+
+	nvmedev=$(_find_nvme_dev "${subsys_name}")
+	state_file="/sys/class/nvme-fabrics/ctl/${nvmedev}/state"
+
+	start_time=$(date +%s)
+	while ! grep -q "${state}" "${state_file}"; do
+		sleep 1
+		end_time=$(date +%s)
+                if (( end_time - start_time > timeout )); then

Nit: the line above has spaces instead of tabs.

Quoted 102 lines
tests/nvme/048 new file +90 -0
+			echo "expected state \"${state}\" not " \
+				"reached within ${timeout} seconds"
+			return 1
+		fi
+	done
+
+	return 0
+}
+
+set_nvmet_attr_qid_max() {
+	local nvmet_subsystem="$1"
+	local qid_max="$2"
+	local cfs_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
+
+	echo "${qid_max}" > "${cfs_path}/attr_qid_max"
+}
+
+set_qid_max() {
+	local port="$1"
+	local subsys_name="$2"
+	local qid_max="$3"
+
+	set_nvmet_attr_qid_max "${subsys_name}" "${qid_max}"
+
+	# Setting qid_max forces a disconnect and the reconntect attempt starts
+	nvmf_wait_for_state "${subsys_name}" "connecting" || return 1
+	nvmf_wait_for_state "${subsys_name}" "live" || return 1
+
+	return 0
+}
+
+test() {
+	local subsys_name="blktests-subsystem-1"
+	local cfs_path="${NVMET_CFS}/subsystems/${subsys_name}"
+	local file_path="${TMPDIR}/img"
+	local skipped=false
+	local hostnqn
+	local hostid
+	local port
+
+	echo "Running ${TEST_NAME}"
+
+	hostid="$(uuidgen)"
+	if [ -z "$hostid" ] ; then
+		echo "uuidgen failed"
+		return 1
+	fi
+	hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}" \
+		"b92842df-a394-44b1-84a4-92ae7d112861"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}"
+
+	if [[ -f "${cfs_path}/attr_qid_max" ]] ; then
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+					--hostnqn "${hostnqn}" \
+					--hostid "${hostid}" \
+					--keep-alive-tmo 1 \
+					--reconnect-delay 2
+
+		if ! nvmf_wait_for_state "${subsys_name}" "live" ; then
+			echo FAIL
+		else
+			set_qid_max "${port}" "${subsys_name}" 1 || echo FAIL
+			set_qid_max "${port}" "${subsys_name}" 128 || echo FAIL
+		fi
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	else
+		SKIP_REASONS+=("missing attr_qid_max feature")
+		skipped=true
+	fi
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+	_remove_nvmet_port "${port}"
+	_remove_nvmet_host "${hostnqn}"
+
+	rm "${file_path}"
+
+	if [[ "${skipped}" = true ]] ; then
+		return 1
+	fi
+
+	echo "Test complete"
+}
tests/nvme/048.out new file +3 -0
new file mode 100644
index 000000000000..7f986ef9637d
@@ -0,0 +1,3 @@
+Running nvme/048
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.40.0
#5 lore raw
In reply to Daniel Wagner
Quoted 1 line · Daniel Wagner

good and the bad case (rdma started to fail today, something to fix).

After adding the missing _nvmet_setup() call, tcp and rdma works fine.

#6 lore raw
Quoted 4 lines · Daniel Wagner
Quoted 1 line · Daniel Wagner

good and the bad case (rdma started to fail today, something to fix).

After adding the missing _nvmet_setup() call, tcp and rdma works fine.

I also confirmed this. I observed test pass with tcp and rdma. Good. Still lockdep warning is observed and fc transport causes system hang, but I think those issues should be fixed after applying the patches to blktests master.

I walked through the patches and they look good enough to apply for me. I will wait a few more days in case anyone has more comments. If there is no more comments, no need to post v7. I will add the missing _setup_nvmet() and replace the spaces in the 2nd patch.

#7 lore raw
Quoted 6 lines · Shin'ichiro Kawasaki
Quoted 4 lines · Daniel Wagner
Quoted 1 line · Daniel Wagner

good and the bad case (rdma started to fail today, something to fix).

After adding the missing _nvmet_setup() call, tcp and rdma works fine.

Quoted 8 lines

I also confirmed this. I observed test pass with tcp and rdma. Good. Still lockdep warning is observed and fc transport causes system hang, but I think those issues should be fixed after applying the patches to blktests master.

I walked through the patches and they look good enough to apply for me. I will wait a few more days in case anyone has more comments. If there is no more comments, no need to post v7. I will add the missing _setup_nvmet() and replace the spaces in the 2nd patch.

I've applied this series with the edits. Thanks!