Skip to contentSkip to conversation

Testsuite for nvme in-band authentication

PATCH In review Hannes Reinecke 2 participants 20 messages Jun 10, 2022 – Jun 20, 2022 NVMe

lore
Share & tools
lore
#7 lore raw

PATCH v5 5/10 nvme/rc: add more arguments to _nvme_connect_subsys()

In reply to Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Add optional arguments for setting 'hostnqn', 'hostid', 'hostkey', and 'ctrlkey' to _nvme_connect_subsys() to make it usable for testing nvme in-band authentication.

---
 tests/nvme/rc | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
tests/nvme/rc +16 -0
index 2873037..b7bb3cd 100644
@@ -190,11 +190,27 @@ _nvme_connect_subsys() {
 	local subsysnqn="$2"
 	local traddr="${3:-$def_traddr}"
 	local trsvcid="${4:-$def_trsvcid}"
+	local hostnqn="${5:-$def_hostnqn}"
+	local hostid="${6:-$def_hostid}"
+	local hostkey="${7}"
+	local ctrlkey="${8}"
 
 	ARGS=(-t "${trtype}" -n "${subsysnqn}")
 	if [[ "${trtype}" != "loop" ]]; then
 		ARGS+=(-a "${traddr}" -s "${trsvcid}")
 	fi
+	if [[ "${hostnqn}" != "$def_hostnqn" ]]; then
+		ARGS+=(--hostnqn="${hostnqn}")
+	fi
+	if [[ "${hostid}" != "$def_hostid" ]]; then
+		ARGS+=(--hostid="${hostid}")
+	fi
+	if [[ -n "${hostkey}" ]]; then
+		ARGS+=(--dhchap-secret="${hostkey}")
+	fi
+	if [[ -n "${ctrlkey}" ]]; then
+		ARGS+=(--dhchap-ctrl-secret="${ctrlkey}")
+	fi
 	nvme connect "${ARGS[@]}"
 }
 
Signature · 1 line
2.26.2
#5 lore raw

PATCH v5 3/10 nvme/rc: clear hosts directory in _cleanup_nvmet()

In reply to Hannes Reinecke

From: Hannes Reinecke <[email protected]>

When clearing out the configuration we need to clear the 'hosts' directory, too, as scripts might have added host configurations.

---
 tests/nvme/rc | 6 ++++++
 1 file changed, 6 insertions(+)
tests/nvme/rc +6 -0
index 454609a..cc48802 100644
@@ -124,6 +124,12 @@ _cleanup_nvmet() {
 		rmdir "${subsys}"
 	done
 
+	for host in "${NVMET_CFS}"/hosts/*; do
+		name=$(basename "${host}")
+		echo "WARNING: Test did not clean up host: ${name}"
+		rmdir "${host}"
+	done
+
 	shopt -u nullglob
 	trap SIGINT
 
Signature · 1 line
2.26.2
#17 lore raw

PATCH v5 9/10 nvme/043: test bi-directional authentication

In reply to Hannes Reinecke
---
 tests/nvme/043     | 106 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/043.out |   8 ++++
 2 files changed, 114 insertions(+)
 create mode 100644 tests/nvme/043
 create mode 100644 tests/nvme/043.out
tests/nvme/043 new file +106 -0
new file mode 100644
index 0000000..fdccb10
@@ -0,0 +1,106 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Test bi-directional authentication
+
+. tests/nvme/rc
+
+DESCRIPTION="Test bi-directional authentication"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrlkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"
+		return 1
+	fi
+
+	ctrlkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"
+		return 1
+	fi
+
+	_setup_nvmet
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}" "${hostkey}" "${ctrlkey}"
+
+	_set_nvmet_dhgroup "${hostnqn}" "ffdhe2048"
+
+	# Step 1: Connect with host authentication only
+	echo "Test host authentication"
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trsvcid}" \
+			     "${hostnqn}" "${hostid}" \
+			     "${hostkey}"
+	if [ $? -ne 0 ] ; then
+		echo "nvme connect failed"
+	fi
+
+	udevadm settle
+
+	_nvme_disconnect_subsys "${subsys_name}"
+
+	# Step 2: Connect with host authentication
+	# and invalid ctrl authentication
+	echo "Test host authentication and invalid ctrl authentication"
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trsvcid}" \
+			     "${hostnqn}" "${hostid}" \
+			     "${hostkey}" "${hostkey}"
+	if [ $? -eq 0 ] ; then
+		echo "nvme connect succeeded (should have failed)"
+		_nvme_disconnect_subsys "${subsys_name}"
+	fi
+
+	# Step 3: Connect with host authentication
+	# and valid ctrl authentication
+	echo "Test host authentication and valid ctrl authentication"
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trsvcid}" \
+			     "${hostnqn}" "${hostid}" \
+			     "${hostkey}" "${ctrlkey}"
+	if [ $? -ne 0 ] ; then
+		echo "nvme connect failed"
+	fi
+
+	udevadm settle
+
+	_nvme_disconnect_subsys "${subsys_name}"
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+
+	_remove_nvmet_port "${port}"
+
+	_remove_nvmet_host "${hostnqn}"
+
+	rm ${file_path}
+
+	echo "Test complete"
+}
tests/nvme/043.out new file +8 -0
new file mode 100644
index 0000000..a49408d
@@ -0,0 +1,8 @@
+Running nvme/043
+Test host authentication
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test host authentication and invalid ctrl authentication
+no controller found: failed to write to nvme-fabrics device
+Test host authentication and valid ctrl authentication
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.26.2
#10 lore raw

PATCH v5 6/10 nvme/040: create authenticated connections

In reply to Hannes Reinecke
---
 tests/nvme/040     | 81 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/040.out |  7 ++++
 2 files changed, 88 insertions(+)
 create mode 100644 tests/nvme/040
 create mode 100644 tests/nvme/040.out
tests/nvme/040 new file +81 -0
new file mode 100644
index 0000000..ce6157f
@@ -0,0 +1,81 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Create authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Create authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"
+		return 1
+	fi
+
+	_setup_nvmet
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}" \
+		"b92842df-a394-44b1-84a4-92ae7d112861"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}" "${hostkey}"
+
+	# Test unauthenticated connection (should fail)
+	echo "Test unauthenticated connection"
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trscvid}" \
+			     "${hostnqn}" "${hostid}"
+	if [ $? -eq 0 ] ; then
+		echo "nvme connect succeeded"
+	fi
+
+	_nvme_disconnect_subsys "${subsys_name}"
+
+	# Test authenticated connection
+	echo "Test authenticated connection"
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trscvid}" \
+			     "${hostnqn}" "${hostid}" "${hostkey}"
+	if [ $? -ne 0 ] ; then
+		echo "nvme connect failed"
+	fi
+
+	udevadm settle
+
+	_nvme_disconnect_subsys "${subsys_name}"
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+
+	_remove_nvmet_port "${port}"
+
+	_remove_nvmet_host "${hostnqn}"
+
+	rm "${file_path}"
+
+	echo "Test complete"
+}
tests/nvme/040.out new file +7 -0
new file mode 100644
index 0000000..b5da85f
@@ -0,0 +1,7 @@
+Running nvme/040
+Test unauthenticated connection
+no controller found: failed to write to nvme-fabrics device
+NQN:blktests-subsystem-1 disconnected 0 controller(s)
+Test authenticated connection
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.26.2
#2 lore raw

PATCH v5 2/10 nvme/rc: clear allowed_hosts subdirectory

In reply to Hannes Reinecke

When removing a subsystem we need to clear out the allowed_hosts subdirectory, otherwise removal will fail.

---
 tests/nvme/rc | 1 +
 1 file changed, 1 insertion(+)
tests/nvme/rc +1 -0
index ad4324f..454609a 100644
@@ -264,6 +264,7 @@ _remove_nvmet_subsystem() {
 	local subsys_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
 
 	_remove_nvmet_ns "${nvmet_subsystem}" "1"
+	rm "${subsys_path}"/allowed_hosts/*
 	rmdir "${subsys_path}"
 }
 
Signature · 1 line
2.26.2
#1 lore raw

PATCH v5 1/10 nvme/rc: do not print error message when no nvme device is found

In reply to Hannes Reinecke

_find_nvme_dev() will only print an nvme device node if one is found, so no point in printing an error message.

---
 tests/nvme/rc | 1 +
 1 file changed, 1 insertion(+)
tests/nvme/rc +1 -0
index 998b181..ad4324f 100644
@@ -308,6 +308,7 @@ _find_nvme_dev() {
 	local subsysnqn
 	local dev
 	for dev in /sys/class/nvme/nvme*; do
+		[ -e "$dev" ] || continue
 		dev="$(basename "$dev")"
 		subsysnqn="$(cat "/sys/class/nvme/${dev}/subsysnqn")"
 		if [[ "$subsysnqn" == "$subsys" ]]; then
Signature · 1 line
2.26.2
#6 lore raw

PATCH v5 4/10 nvme/rc: add functions for in-band authentication

In reply to Hannes Reinecke

Add functions to enable in-band authentication.

---
 tests/nvme/rc | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
tests/nvme/rc +73 -0
index cc48802..2873037 100644
@@ -75,6 +75,17 @@ _require_nvme_trtype_is_fabrics() {
 	return 0
 }
 
+_require_nvme_cli_auth() {
+	local hostkey
+
+	hostkey="$(nvme gen-dhchap-key -n nvmf-test-subsys 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		SKIP_REASON="nvme gen-dhchap-key command missing"
+		return 1
+	fi
+	return 0
+}
+
 _test_dev_nvme_ctrl() {
 	echo "/dev/char/$(cat "${TEST_DEV_SYSFS}/device/dev")"
 }
@@ -255,6 +266,25 @@ _create_nvmet_subsystem() {
 	_create_nvmet_ns "${nvmet_subsystem}" "1" "${blkdev}" "${uuid}"
 }
 
+_create_nvmet_host() {
+	local nvmet_subsystem="$1"
+	local nvmet_hostnqn="$2"
+	local nvmet_hostkey="$3"
+	local nvmet_ctrlkey="$4"
+	local cfs_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
+	local host_path="${NVMET_CFS}/hosts/${nvmet_hostnqn}"
+
+	mkdir "${host_path}"
+	echo 0 > "${cfs_path}/attr_allow_any_host"
+	ln -s "${host_path}" "${cfs_path}/allowed_hosts/${nvmet_hostnqn}"
+	if [[ "${nvmet_hostkey}" ]] ; then
+		echo "${nvmet_hostkey}" > "${host_path}/dhchap_key"
+	fi
+	if [[ "${nvmet_ctrlkey}" ]] ; then
+		echo "${nvmet_ctrlkey}" > "${host_path}/dhchap_ctrl_key"
+	fi
+}
+
 _remove_nvmet_ns() {
 	local nvmet_subsystem="$1"
 	local nsid=$2
@@ -274,6 +304,13 @@ _remove_nvmet_subsystem() {
 	rmdir "${subsys_path}"
 }
 
+_remove_nvmet_host() {
+	local nvmet_host="$1"
+	local host_path="${NVMET_CFS}/hosts/${nvmet_host}"
+
+	rmdir "${host_path}"
+}
+
 _create_nvmet_passthru() {
 	local nvmet_subsystem="$1"
 	local subsys_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
@@ -310,6 +347,42 @@ _remove_nvmet_subsystem_from_port() {
 	rm "${NVMET_CFS}/ports/${port}/subsystems/${nvmet_subsystem}"
 }
 
+_set_nvmet_hostkey() {
+	local nvmet_hostnqn="$1"
+	local nvmet_hostkey="$2"
+	local cfs_path="${NVMET_CFS}/hosts/${nvmet_hostnqn}"
+
+	echo "${nvmet_hostkey}" > \
+	     "${cfs_path}/dhchap_key"
+}
+
+_set_nvmet_ctrlkey() {
+	local nvmet_hostnqn="$1"
+	local nvmet_ctrlkey="$2"
+	local cfs_path="${NVMET_CFS}/hosts/${nvmet_hostnqn}"
+
+	echo "${nvmet_ctrlkey}" > \
+	     "${cfs_path}/dhchap_ctrl_key"
+}
+
+_set_nvmet_hash() {
+	local nvmet_hostnqn="$1"
+	local nvmet_hash="$2"
+	local cfs_path="${NVMET_CFS}/hosts/${nvmet_hostnqn}"
+
+	echo "${nvmet_hash}" > \
+	     "${cfs_path}/dhchap_hash"
+}
+
+_set_nvmet_dhgroup() {
+	local nvmet_hostnqn="$1"
+	local nvmet_dhgroup="$2"
+	local cfs_path="${NVMET_CFS}/hosts/${nvmet_hostnqn}"
+
+	echo "${nvmet_dhgroup}" > \
+	     "${cfs_path}/dhchap_dhgroup"
+}
+
 _find_nvme_dev() {
 	local subsys=$1
 	local subsysnqn
Signature · 1 line
2.26.2
#0 lore raw

v5 cover Testsuite for nvme in-band authentication

Hi all,

some people (Hi Sagi!) have complained that nvme in-band authentication provide far too many configuration options to test it with some easy commands. So here's a test suite for testing (most of) the various configuration options and features.

As usual, comments and reviews are welcome.

Changes to v4:
- Rebase to current HEAD
- Use return code from 'nvme connect' instead of looking for the controller node

Changes to v3:
- Rebase to current HEAD

Changes to v2:
- Merge first two tests
- Add optional arguments to _nvme_connect_subsys()
- Convert to use _nvme_connect_subsys()

Changes to v1:
- Move tests to the 'nvme' directory
- Check for authentication failure on invalid keys

Hannes Reinecke (10):
nvme/rc: do not print error message when no nvme device is found nvme/rc: clear allowed_hosts subdirectory
nvme/rc: clear hosts directory in _cleanup_nvmet() nvme/rc: add functions for in-band authentication nvme/rc: add more arguments to _nvme_connect_subsys() nvme/040: create authenticated connections
nvme/041: test dhchap key types for authenticated connections nvme/042: test hash and dh group variations for authenticated
connections
nvme/043: test bi-directional authentication nvme/044: test re-authentication

 tests/nvme/040     |  81 +++++++++++++++++++++++++++++
 tests/nvme/040.out |   7 +++
 tests/nvme/041     |  95 ++++++++++++++++++++++++++++++++++
 tests/nvme/041.out |  16 ++++++
 tests/nvme/042     |  89 ++++++++++++++++++++++++++++++++
 tests/nvme/042.out |  18 +++++++
 tests/nvme/043     | 106 ++++++++++++++++++++++++++++++++++++++
 tests/nvme/043.out |   8 +++
 tests/nvme/044     | 125 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/044.out |  12 +++++
 tests/nvme/rc      |  97 +++++++++++++++++++++++++++++++++++
 11 files changed, 654 insertions(+)
 create mode 100644 tests/nvme/040
 create mode 100644 tests/nvme/040.out
 create mode 100644 tests/nvme/041
 create mode 100644 tests/nvme/041.out
 create mode 100644 tests/nvme/042
 create mode 100644 tests/nvme/042.out
 create mode 100644 tests/nvme/043
 create mode 100644 tests/nvme/043.out
 create mode 100644 tests/nvme/044
 create mode 100644 tests/nvme/044.out
Signature · 1 line
2.26.2
#16 lore raw

PATCH v5 8/10 nvme/042: test hash and dh group variations for authenticated connections

---
 tests/nvme/042     | 89 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/042.out | 18 ++++++++++
 2 files changed, 107 insertions(+)
 create mode 100644 tests/nvme/042
 create mode 100644 tests/nvme/042.out
tests/nvme/042 new file +89 -0
new file mode 100644
index 0000000..0b9762a
@@ -0,0 +1,89 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Test hash and dh group variations for authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Test hash and DH group variations for authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	_setup_nvmet
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}" "${hostkey}"
+
+	for hash in "hmac(sha256)" "hmac(sha384)" "hmac(sha512)" ; do
+
+		echo "Testing hash ${hash}"
+
+		_set_nvmet_hash "${hostnqn}" "${hash}"
+
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+				     "${def_traddr}" "${def_trsvcid}" \
+				     "${hostnqn}" "${hostid}" \
+				     "${hostkey}"
+		if [ $? -ne 0 ] ; then
+			echo "nvme connect failed"
+		fi
+
+		udevadm settle
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	done
+
+	for dhgroup in "ffdhe2048" "ffdhe3072" "ffdhe4096" "ffdhe6144" "ffdhe8192" ; do
+
+		echo "Testing DH group ${dhgroup}"
+
+		_set_nvmet_dhgroup "${hostnqn}" "${dhgroup}"
+
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+				     "${def_traddr}" "${def_trsvcid}" \
+				     "${hostnqn}" "${hostid}" \
+				     "${hostkey}"
+
+		if [ $? -ne 0 ] ; then
+			echo "nvme connect failed"
+		fi
+
+		udevadm settle
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	done
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+
+	_remove_nvmet_port "${port}"
+
+	_remove_nvmet_host "${hostnqn}"
+
+	rm ${file_path}
+
+	echo "Test complete"
+}
tests/nvme/042.out new file +18 -0
new file mode 100644
index 0000000..c05aa2f
@@ -0,0 +1,18 @@
+Running nvme/042
+Testing hash hmac(sha256)
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing hash hmac(sha384)
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing hash hmac(sha512)
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing DH group ffdhe2048
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing DH group ffdhe3072
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing DH group ffdhe4096
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing DH group ffdhe6144
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing DH group ffdhe8192
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.26.2
#13 lore raw

PATCH v5 7/10 nvme/041: test dhchap key types for authenticated connections

In reply to Hannes Reinecke
---
 tests/nvme/041     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/041.out | 16 ++++++++
 2 files changed, 111 insertions(+)
 create mode 100644 tests/nvme/041
 create mode 100644 tests/nvme/041.out
tests/nvme/041 new file +95 -0
new file mode 100644
index 0000000..2ded9c9
@@ -0,0 +1,95 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Test dhchap key types for authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Test dhchap key types for authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	_setup_nvmet
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}"
+
+	for hmac in 0 1 2 3; do
+		echo "Testing hmac ${hmac}"
+		hostkey="$(nvme gen-dhchap-key --hmac=${hmac} -n ${subsys_name} 2> /dev/null)"
+		if [ $? -ne 0 ] ; then
+			echo "couldn't generate host key for hmac ${hmac}"
+			return 1
+		fi
+		_set_nvmet_hostkey "${hostnqn}" "${hostkey}"
+
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+				     "${def_traddr}" "${def_trsvcid}" \
+				     "${hostnqn}" "${hostid}" \
+				     "${hostkey}"
+		if [ $? -ne 0 ] ; then
+			echo "nvme connect failed"
+		fi
+
+		udevadm settle
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	done
+
+	for key_len in 32 48 64; do
+		echo "Testing key length ${key_len}"
+		hostkey="$(nvme gen-dhchap-key --key-length=${key_len} -n ${subsys_name} 2> /dev/null)"
+		if [ $? -ne 0 ] ; then
+			echo "couldn't generate host key for length ${key_len}"
+			return 1
+		fi
+		_set_nvmet_hostkey "${hostnqn}" "${hostkey}"
+
+		_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+				     "${def_traddr}" "${def_trsvcid}" \
+				     "${hostnqn}" "${hostid}" \
+				     "${hostkey}"
+
+		if [ $? -ne 0 ] ; then
+			echo "nvme connect failed"
+		fi
+
+		udevadm settle
+
+		_nvme_disconnect_subsys "${subsys_name}"
+	done
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+
+	_remove_nvmet_port "${port}"
+
+	_remove_nvmet_host "${hostnqn}"
+
+	rm ${file_path}
+
+	echo "Test complete"
+}
tests/nvme/041.out new file +16 -0
new file mode 100644
index 0000000..49a56ea
@@ -0,0 +1,16 @@
+Running nvme/041
+Testing hmac 0
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing hmac 1
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing hmac 2
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing hmac 3
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing key length 32
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing key length 48
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Testing key length 64
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.26.2
#18 lore raw

PATCH v5 10/10 nvme/044: test re-authentication

In reply to Hannes Reinecke
---
 tests/nvme/044     | 125 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/044.out |  12 +++++
 2 files changed, 137 insertions(+)
 create mode 100644 tests/nvme/044
 create mode 100644 tests/nvme/044.out
tests/nvme/044 new file +125 -0
new file mode 100644
index 0000000..060194e
@@ -0,0 +1,125 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Test re-authentication
+
+. tests/nvme/rc
+
+DESCRIPTION="Test re-authentication"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrlkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"
+		return 1
+	fi
+
+	ctrlkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"
+		return 1
+	fi
+
+	_setup_nvmet
+
+	truncate -s 512M "${file_path}"
+
+	_create_nvmet_subsystem "${subsys_name}" "${file_path}"
+	port="$(_create_nvmet_port "${nvme_trtype}")"
+	_add_nvmet_subsys_to_port "${port}" "${subsys_name}"
+	_create_nvmet_host "${subsys_name}" "${hostnqn}" "${hostkey}" "${ctrlkey}"
+
+	_set_nvmet_dhgroup "${hostnqn}" "ffdhe2048"
+
+	_nvme_connect_subsys "${nvme_trtype}" "${subsys_name}" \
+			     "${def_traddr}" "${def_trsvcid}" \
+			     "${hostnqn}" "${hostid}" \
+			     "${hostkey}" "${ctrlkey}"
+	if [ $? -ne 0 ] ; then
+		echo "nvme connect failed"
+	fi
+
+	udevadm settle
+
+	echo "Re-authenticate with original host key"
+
+	ctrldev=$(_find_nvme_dev "${subsys_name}")
+	if [ -z "$ctrldev" ] ; then
+		echo "nvme controller not found"
+	fi
+	echo "${hostkey}" > /sys/class/nvme/${ctrldev}/dhchap_secret
+
+	echo "Renew host key on the controller"
+
+	new_hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+
+	_set_nvmet_hostkey "${hostnqn}" "${new_hostkey}"
+
+	echo "Re-authenticate with new host key"
+
+	echo "${new_hostkey}" > /sys/class/nvme/${ctrldev}/dhchap_secret
+
+	echo "Renew ctrl key on the controller"
+
+	new_ctrlkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+
+	_set_nvmet_ctrlkey "${hostnqn}" "${new_ctrlkey}"
+
+	echo "Re-authenticate with new ctrl key"
+
+	echo "${new_ctrlkey}" > /sys/class/nvme/${ctrldev}/dhchap_ctrl_secret
+
+	echo "Change DH group to ffdhe8192"
+
+	_set_nvmet_dhgroup "${hostnqn}" "ffdhe8192"
+
+	echo "Re-authenticate with changed DH group"
+
+	echo "${new_hostkey}" > /sys/class/nvme/${ctrldev}/dhchap_secret
+
+	echo "Change hash to hmac(sha512)"
+
+	_set_nvmet_hash "${hostnqn}" "hmac(sha512)"
+
+	echo "Re-authenticate with changed hash"
+
+	echo "${new_hostkey}" > /sys/class/nvme/${ctrldev}/dhchap_secret
+
+	nvmedev=$(_find_nvme_dev "${subsys_name}")
+
+	_run_fio_rand_io --size=8m --filename="/dev/${nvmedev}n1"
+
+	_nvme_disconnect_subsys "${subsys_name}"
+
+	_remove_nvmet_subsystem_from_port "${port}" "${subsys_name}"
+	_remove_nvmet_subsystem "${subsys_name}"
+
+	_remove_nvmet_port "${port}"
+
+	_remove_nvmet_host "${hostnqn}"
+
+	rm ${file_path}
+
+	echo "Test complete"
+}
tests/nvme/044.out new file +12 -0
new file mode 100644
index 0000000..51a7092
@@ -0,0 +1,12 @@
+Running nvme/044
+Re-authenticate with original host key
+Renew host key on the controller
+Re-authenticate with new host key
+Renew ctrl key on the controller
+Re-authenticate with new ctrl key
+Change DH group to ffdhe8192
+Re-authenticate with changed DH group
+Change hash to hmac(sha512)
+Re-authenticate with changed hash
+NQN:blktests-subsystem-1 disconnected 1 controller(s)
+Test complete
Signature · 1 line
2.26.2
#3 lore raw

Re: [PATCH 02/10] nvme/rc: clear allowed_hosts subdirectory

In reply to Hannes Reinecke
Quoted 14 lines · Hannes Reinecke

When removing a subsystem we need to clear out the allowed_hosts subdirectory, otherwise removal will fail.

---
 tests/nvme/rc | 1 +
 1 file changed, 1 insertion(+)
tests/nvme/rc +0 -0
index ad4324f..454609a 100644

tests/nvme/rc _remove_nvmet_subsystem() { · line 267

Quoted 4 lines
tests/nvme/rc +1 -0
@@ -264,6 +264,7 @@ _remove_nvmet_subsystem() {
 	local subsys_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
 
 	_remove_nvmet_ns "${nvmet_subsystem}" "1"
+	rm "${subsys_path}"/allowed_hosts/*

When the allowed_hosts directory is empty, the rm command reports an error, and it makes nvme/003 fail:

nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]

    runtime  10.481s  ...  10.481s
    --- tests/nvme/003.out      2022-06-02 10:18:53.539739780 +0900
    +++ /home/shin/kts/kernel-test-suite/src/blktests/results/nodev/nvme/003.out.bad    2022-06-14 13:10:22.804354624 +0900
    @@ -1,3 +1,4 @@
     Running nvme/003
     NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s)
    +rm: cannot remove '/sys/kernel/config/nvmet//subsystems/blktests-subsystem-1/allowed_hosts/*': No such file or directory
     Test complete

How about to add -f option to the rm?

Signature · 1 line
Shin'ichiro Kawasaki
#8 lore raw

Re: [PATCH 05/10] nvme/rc: add more arguments to _nvme_connect_subsys()

In reply to Hannes Reinecke
Quoted 16 lines · Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Add optional arguments for setting 'hostnqn', 'hostid', 'hostkey', and 'ctrlkey' to _nvme_connect_subsys() to make it usable for testing nvme in-band authentication.

---
 tests/nvme/rc | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
tests/nvme/rc +0 -0
index 2873037..b7bb3cd 100644

tests/nvme/rc _nvme_connect_subsys() { · line 194

Quoted 5 lines
tests/nvme/rc +2 -0
@@ -190,11 +190,27 @@ _nvme_connect_subsys() {
 	local subsysnqn="$2"
 	local traddr="${3:-$def_traddr}"
 	local trsvcid="${4:-$def_trsvcid}"
+	local hostnqn="${5:-$def_hostnqn}"
+	local hostid="${6:-$def_hostid}"

I can not find def_hostnqn and def_hostid defined. Did they slip out from this series?

Quoted 26 lines
tests/nvme/rc +14 -0
+	local hostkey="${7}"
+	local ctrlkey="${8}"
 
 	ARGS=(-t "${trtype}" -n "${subsysnqn}")
 	if [[ "${trtype}" != "loop" ]]; then
 		ARGS+=(-a "${traddr}" -s "${trsvcid}")
 	fi
+	if [[ "${hostnqn}" != "$def_hostnqn" ]]; then
+		ARGS+=(--hostnqn="${hostnqn}")
+	fi
+	if [[ "${hostid}" != "$def_hostid" ]]; then
+		ARGS+=(--hostid="${hostid}")
+	fi
+	if [[ -n "${hostkey}" ]]; then
+		ARGS+=(--dhchap-secret="${hostkey}")
+	fi
+	if [[ -n "${ctrlkey}" ]]; then
+		ARGS+=(--dhchap-ctrl-secret="${ctrlkey}")
+	fi
 	nvme connect "${ARGS[@]}"
 }
 
Signature · 1 line
2.26.2
Signature · 1 line
Shin'ichiro Kawasaki
#11 lore raw

Re: [PATCH 06/10] nvme/040: create authenticated connections

In reply to Hannes Reinecke
Quoted 37 lines · Hannes Reinecke
---
 tests/nvme/040     | 81 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/040.out |  7 ++++
 2 files changed, 88 insertions(+)
 create mode 100644 tests/nvme/040
 create mode 100644 tests/nvme/040.out
tests/nvme/040 new file +23 -0
new file mode 100644
index 0000000..ce6157f
@@ -0,0 +1,81 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Create authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Create authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"

tests/nvme/040 line 33

Quoted 10 lines
tests/nvme/040 new file +10 -0
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
+
+	hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"

This message is a bit confusing. _require_nvme_cli_auth in requries() already confirmed that nvme gen-dhchap-key command is not missing. I think "nvme gen-dhchap-key command failed" is a bit better message here.

Signature · 1 line
Shin'ichiro Kawasaki
#14 lore raw

Re: [PATCH 07/10] nvme/041: test dhchap key types for authenticated connections

In reply to Hannes Reinecke
Quoted 18 lines · Hannes Reinecke
---
 tests/nvme/041     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/041.out | 16 ++++++++
 2 files changed, 111 insertions(+)
 create mode 100644 tests/nvme/041
 create mode 100644 tests/nvme/041.out
tests/nvme/041 new file +4 -0
new file mode 100644
index 0000000..2ded9c9
@@ -0,0 +1,95 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#

tests/nvme/041 line 14

Quoted 10 lines
tests/nvme/041 new file +10 -0
+# Test dhchap key types for authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Test dhchap key types for authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have loop

_have_loop? '_' is missing before loop.

Signature · 1 line
Shin'ichiro Kawasaki
#19 lore raw
In reply to Hannes Reinecke

Hi Hannes, thanks for sharing the series. Here are general minor comments on it.

- Newly added test case scripts have file mode 644. I recommend 755 to indicate that they are executable and to keep consistency with other test cases.
- Could you please run "make check" and address shellcheck warnings? Shellcheck is good to reduce human-error, and I wish to keep zero warning.
- In the series, I observe bash condition expressions with single brackets [] and double brackets [[ ]] are mixed. Originally double brackets were recommended for blktests scripts then I rather prefer double brackets. Having said that many single brackets are already used in blktests, so single bracket is still ok.

Signature · 1 line
Shin'ichiro Kawasaki
#4 lore raw

Re: [PATCH 02/10] nvme/rc: clear allowed_hosts subdirectory

In reply to Shinichiro Kawasaki
Quoted 33 lines · Shinichiro Kawasaki
Quoted 14 lines · Hannes Reinecke

When removing a subsystem we need to clear out the allowed_hosts subdirectory, otherwise removal will fail.

---
  tests/nvme/rc | 1 +
  1 file changed, 1 insertion(+)
tests/nvme/rc +0 -0
index ad4324f..454609a 100644

tests/nvme/rc _remove_nvmet_subsystem() { · line 267

Quoted 4 lines
tests/nvme/rc +1 -0
@@ -264,6 +264,7 @@ _remove_nvmet_subsystem() {
  	local subsys_path="${NVMET_CFS}/subsystems/${nvmet_subsystem}"
 
  	_remove_nvmet_ns "${nvmet_subsystem}" "1"
+	rm "${subsys_path}"/allowed_hosts/*

When the allowed_hosts directory is empty, the rm command reports an error, and it makes nvme/003 fail:

nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]

     runtime  10.481s  ...  10.481s
     --- tests/nvme/003.out      2022-06-02 10:18:53.539739780 +0900
     +++ /home/shin/kts/kernel-test-suite/src/blktests/results/nodev/nvme/003.out.bad    2022-06-14 13:10:22.804354624 +0900
     @@ -1,3 +1,4 @@
      Running nvme/003
      NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s)
     +rm: cannot remove '/sys/kernel/config/nvmet//subsystems/blktests-subsystem-1/allowed_hosts/*': No such file or directory
      Test complete
Quoted 2 lines

How about to add -f option to the rm?

Yes, good point.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#9 lore raw

Re: [PATCH 05/10] nvme/rc: add more arguments to _nvme_connect_subsys()

In reply to Shinichiro Kawasaki
Quoted 12 lines · Shinichiro Kawasaki
Quoted 11 lines · Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Add optional arguments for setting 'hostnqn', 'hostid', 'hostkey', and 'ctrlkey' to _nvme_connect_subsys() to make it usable for testing nvme in-band authentication.

---
  tests/nvme/rc | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
Quoted 14 lines

tests/nvme/rc _nvme_connect_subsys() { · line 194

Quoted 10 lines
tests/nvme/rc +2 -0
index 2873037..b7bb3cd 100644
@@ -190,11 +190,27 @@ _nvme_connect_subsys() {
  	local subsysnqn="$2"
  	local traddr="${3:-$def_traddr}"
  	local trsvcid="${4:-$def_trsvcid}"
+	local hostnqn="${5:-$def_hostnqn}"
+	local hostid="${6:-$def_hostid}"

I can not find def_hostnqn and def_hostid defined. Did they slip out from this series?

No, not really; these are optional args anyway.
But I probably should add code to read them in from /etc/nvme/

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#12 lore raw

Re: [PATCH 06/10] nvme/040: create authenticated connections

In reply to Shinichiro Kawasaki
Quoted 44 lines · Shinichiro Kawasaki
Quoted 43 lines · Hannes Reinecke
---
  tests/nvme/040     | 81 ++++++++++++++++++++++++++++++++++++++++++++++
  tests/nvme/040.out |  7 ++++
  2 files changed, 88 insertions(+)

create mode 100644 tests/nvme/040
create mode 100644 tests/nvme/040.out

tests/nvme/040 new file +29 -0
new file mode 100644
index 0000000..ce6157f
@@ -0,0 +1,81 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Create authenticated connections
+
+. tests/nvme/rc
+
+DESCRIPTION="Create authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_fabrics
+	_require_nvme_cli_auth
+}
+
+
+test() {
+	local port
+	local subsys_name="blktests-subsystem-1"
+	local hostid="$(uuidgen)"
+	local hostnqn="nqn.2014-08.org.nvmexpress:uuid:${hostid}"
+	local file_path="${TMPDIR}/img"
+	local hostkey
+	local ctrldev
+
+	echo "Running ${TEST_NAME}"
Quoted 9 lines

tests/nvme/040 line 33

Quoted 4 lines
tests/nvme/040 new file +4 -0
+
+	hostkey="$(nvme gen-dhchap-key -n ${subsys_name} 2> /dev/null)"
+	if [ $? -ne 0 ] ; then
+		echo "nvme gen-dhchap-key command missing"

This message is a bit confusing. _require_nvme_cli_auth in requries() already confirmed that nvme gen-dhchap-key command is not missing. I think "nvme gen-dhchap-key command failed" is a bit better message here.

Okay, will be fixing it up.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#15 lore raw

Re: [PATCH 07/10] nvme/041: test dhchap key types for authenticated connections

In reply to Shinichiro Kawasaki
Quoted 23 lines · Shinichiro Kawasaki
Quoted 22 lines · Hannes Reinecke
---
  tests/nvme/041     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
  tests/nvme/041.out | 16 ++++++++
  2 files changed, 111 insertions(+)

create mode 100644 tests/nvme/041
create mode 100644 tests/nvme/041.out

tests/nvme/041 new file +8 -0
new file mode 100644
index 0000000..2ded9c9
@@ -0,0 +1,95 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2021 Hannes Reinecke, SUSE Labs
+#
+# Test dhchap key types for authenticated connections
+
+. tests/nvme/rc
+
Quoted 9 lines

tests/nvme/041 line 14

Quoted 6 lines
tests/nvme/041 new file +6 -0
+DESCRIPTION="Test dhchap key types for authenticated connections"
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have loop

_have_loop? '_' is missing before loop.

Correct, will be fixing it up.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman