nvme: add nvme pci timeout testcase

PATCH Superseded Chaitanya Kulkarni 3 participants 3 messages Jan 23, 2024 – Jan 29, 2024

Subsystems (2)

Block layer NVMe

lore

Share & tools

lore

Chaitanya Kulkarni

author

2024-01-23 22:56 UTC #0 lore raw

PATCH v3 1/1 nvme: add nvme pci timeout testcase

Trigger and test nvme-pci timeout with concurrent fio jobs.

Signed-off-by Chaitanya Kulkarni <[email protected]>

---
V3:-

1. Add CAN_BE_ZONED.
2. Add FAULT_INJECTION_DEBUG_FS check in requires.
3. Remove _require_nvme_trtype pci in requires().
4. Remove device_requires().
5. Store fio output in FULL.
6. Revmoe shellcheck and use grep I/O error value to pass/fail testcase.

---
 tests/nvme/050     | 69 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/050.out |  2 ++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/nvme/050
 create mode 100644 tests/nvme/050.out

tests/nvme/050 new file +69 -0

new file mode 100755
index 0000000..cacaba6
@@ -0,0 +1,69 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2024 Chaitanya Kulkarni
+#
+# Test NVMe-PCI timeout with FIO jobs by triggering the nvme_timeout function.
+#
+
+. tests/nvme/rc
+
+DESCRIPTION="test nvme-pci timeout with fio jobs"
+CAN_BE_ZONED=1
+
+sysfs_path="/sys/kernel/debug/fail_io_timeout/"
+#restrict test to nvme-pci only
+nvme_trtype=pci
+
+# fault injection config array
+declare -A fi_array
+
+requires() {
+	_have_fio
+	_nvme_requires
+	_have_kernel_option FAIL_IO_TIMEOUT
+	_have_kernel_option FAULT_INJECTION_DEBUG_FS
+}
+
+save_fi_settings() {
+	for fi_attr in probability interval times space verbose
+	do
+		fi_array["${fi_attr}"]=$(cat "${sysfs_path}/${fi_attr}")
+	done
+}
+
+restore_fi_settings() {
+	for fi_attr in probability interval times space verbose
+	do
+		echo "${fi_array["${fi_attr}"]}" > "${sysfs_path}/${fi_attr}"
+	done
+}
+
+test_device() {
+	local nvme_ns
+	local io_fimeout_fail
+
+	echo "Running ${TEST_NAME}"
+
+	nvme_ns="$(basename "${TEST_DEV}")"
+	io_fimeout_fail="$(cat /sys/block/"${nvme_ns}"/io-timeout-fail)"
+	save_fi_settings
+	echo 1 > /sys/block/"${nvme_ns}"/io-timeout-fail
+
+	echo 100 > /sys/kernel/debug/fail_io_timeout/probability
+	echo   1 > /sys/kernel/debug/fail_io_timeout/interval
+	echo  -1 > /sys/kernel/debug/fail_io_timeout/times
+	echo   0 > /sys/kernel/debug/fail_io_timeout/space
+	echo   1 > /sys/kernel/debug/fail_io_timeout/verbose
+
+	fio --bs=4k --rw=randread --norandommap --numjobs="$(nproc)" \
+	    --name=reads --direct=1 --filename="${TEST_DEV}" --group_reporting \
+	    --time_based --runtime=1m >& "$FULL"
+
+	if grep -q "Input/output error" "$FULL"; then
+		echo "Test complete"
+	else
+		echo "Test failed"
+	fi
+	restore_fi_settings
+	echo "${io_fimeout_fail}" > /sys/block/"${nvme_ns}"/io-timeout-fail
+}

tests/nvme/050.out new file +2 -0

new file mode 100644
index 0000000..b78b05f
@@ -0,0 +1,2 @@
+Running nvme/050
+Test complete

Signature · 1 line

2.40.0

Chaitanya Kulkarni

2024-01-29 11:14 UTC #1 lore raw

Quoted 6 lines · Chaitanya Kulkarni

Trigger and test nvme-pci timeout with concurrent fio jobs.
Signed-off-by Chaitanya Kulkarni <[email protected]>
---
V3:-

Quoted 9 lines

1. Add CAN_BE_ZONED.
2. Add FAULT_INJECTION_DEBUG_FS check in requires.
3. Remove _require_nvme_trtype pci in requires().
4. Remove device_requires().
5. Store fio output in FULL.
6. Revmoe shellcheck and use grep I/O error value to pass/fail testcase.
---

is there any objections on this patch ?

-ck

Shinichiro Kawasaki

2024-01-29 11:57 UTC #2 lore raw

Quoted 14 lines · Chaitanya Kulkarni

Quoted 13 lines · Chaitanya Kulkarni
Trigger and test nvme-pci timeout with concurrent fio jobs.
Signed-off-by Chaitanya Kulkarni <[email protected]>
---
V3:-
1. Add CAN_BE_ZONED.
2. Add FAULT_INJECTION_DEBUG_FS check in requires.
3. Remove _require_nvme_trtype pci in requires().
4. Remove device_requires().
5. Store fio output in FULL.
6. Revmoe shellcheck and use grep I/O error value to pass/fail testcase.

Quoted 4 lines

Quoted 2 lines
---
is there any objections on this patch ?

Thanks for the V3 patch. It looks almost good, except one thing: as I commented on V1 and V2, this test case often leaves the target NVME device with zero size. It will make the following test cases fail (I mean the future test cases to be added).

I suggest to remove and rescan the device to regain the good status of the device. The following change will do it. What do you think?

(If we take this change as it is, it will recover the io_timeout_fail sysfs attribute also, so the code to save and restore io_timeout_fail can be removed.)

tests/nvme/050 +10 -0

index cacaba6..cb1c6f5 100755
@@ -41,9 +41,12 @@ restore_fi_settings() {
 test_device() {
 	local nvme_ns
 	local io_fimeout_fail
+	local pdev
 
 	echo "Running ${TEST_NAME}"
 
+	pdev=$(_get_pci_dev_from_blkdev)
+
 	nvme_ns="$(basename "${TEST_DEV}")"
 	io_fimeout_fail="$(cat /sys/block/"${nvme_ns}"/io-timeout-fail)"
 	save_fi_settings

@@ -66,4 +69,11 @@ test_device() {
 	fi
 	restore_fi_settings
 	echo "${io_fimeout_fail}" > /sys/block/"${nvme_ns}"/io-timeout-fail
+
+	# Remove and rescan the NVME device to ensure that it has come back
+	echo 1 > "/sys/bus/pci/devices/$pdev/remove"
+	echo 1 > /sys/bus/pci/rescan
+	if [[ ! -b $TEST_DEV ]]; then
+		echo "Failed to regain $TEST_DEV"
+	fi
 }