Skip to contentSkip to conversation

vfio: selftests: Automate finding/setting up devices for testing

PATCH In review Josh Hilke 3 participants 9 messages Apr 9 – Apr 14

Subsystems (4)
lore
Share & tools
lore

Josh Hilke

author
#0 lore raw

cover vfio: selftests: Automate finding/setting up devices for testing

This series introduces two improvements to make setting up devices for VFIO selftests more user-friendly.

1. Add functionality to the VFIO selftest setup script to find devices on a user's system that are compatible with the VFIO selftest framework (i.e. devices that have a VFIO selftest driver).

Example:

  $ ./tools/testing/selftests/vfio/scripts/setup.sh -l
    Supported devices:
    0000:6a:01.0
    0000:6f:01.0
    0000:74:01.0
    0000:79:01.0
    0000:e7:01.0
    0000:ec:01.0
    0000:f1:01.0
    0000:f6:01.0

2. Modify the VFIO selftest setup script to try to find and set up a compatible device automatically. "Set up the device for testing" means unbind the device from its driver, then bind it to VFIO. The user is no longer required to find a segment:bus:device.function number to pass to the script (though this functionality is still available through a -d option).

Example of automated device setup:

    $ ./tools/testing/selftests/vfio/scripts/setup.sh
    + echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
    + echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
    + echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
    Successfully set up 0000:6a:01.0

Example of manual device setup:

    $ ./tools/testing/selftests/vfio/scripts/setup.sh -d 0000:6a:01.0
    + echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
    + echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
    + echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
    Successfully set up 0000:6a:01.0

Josh Hilke (2):
vfio: selftests: Find devices that are compatible with VFIO selftests
vfio: selftests: Automate finding/setting up devices for testing

 tools/testing/selftests/vfio/scripts/setup.sh | 63 +++++++++++++++++--
 1 file changed, 59 insertions(+), 4 deletions(-)
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
author
#1 lore raw

PATCH 1/2 vfio: selftests: Find devices that are compatible with VFIO selftests

Update the VFIO selftest setup script to print the segment:bus:device.function numbers of devices on the user's system that have VFIO selftest drivers (i.e. are compatible with the VFIO selftest infrastructure). This makes it easy for users to quickly find devices that are compatible with VFIO selftests.

Example of how to list compatible devices on the system:

$ ./tools/testing/selftests/vfio/scripts/setup.sh -l
Supported devices:
0000:6a:01.0
0000:6f:01.0
0000:74:01.0
0000:79:01.0
0000:e7:01.0
0000:ec:01.0
0000:f1:01.0
0000:f6:01.0

Example of setting up a device:

$ ./tools/testing/selftests/vfio/scripts/setup.sh -d 0000:6a:01.0
+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
Successfully set up 0000:6a:01.0

This makes the script much more user friendly so that no one has to dig around in the VFIO selftest libraries to figure out which devices can run the selftests.

---
 tools/testing/selftests/vfio/scripts/setup.sh | 43 +++++++++++++++++--
 1 file changed, 40 insertions(+), 3 deletions(-)
tools/testing/selftests/vfio/scripts/setup.sh +40 -3
index 49a499e51cbe..2cde07eb9c8b 100755
@@ -3,18 +3,54 @@ set -e
 
 source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
 
+# List of devices which have a VFIO selftest driver
+DEVICES=(
+	"8086:0b25" # Intel Data Streaming Accelerator
+	"8086:0cf8" # Intel CBDMA
+)
+
+function print_supported_devices() {
+	local vendor_device_id
+	local id
+
+	for vendor_device_id in "${DEVICES[@]}"; do
+		read -r id <<< "${vendor_device_id}"
+		lspci -D -d "${id}" | awk '{ print $1 }'
+	done
+}
+
+function usage() {
+	echo "usage: $0 [-l] [-d <segment:bus:device.function>]" >&2
+	echo "" >&2
+	echo "  -l  List segment:bus:device.function numbers of supported devices." >&2
+	echo "  -d  segment:bus:device.function to set up." >&2
+}
+
 function main() {
 	local device_bdf
 	local device_dir
 	local numvfs
 	local driver
+	local bdf_list=()
+
+	while getopts "ld:" opt; do
+		case ${opt} in
+			l)
+			 	echo "Supported devices: "
+				print_supported_devices
+				exit 0
+				;;
+			d) bdf_list+=("${OPTARG}") ;;
+			*) usage; exit 1 ;;
+		esac
+	done
 
-	if [ $# = 0 ]; then
-		echo "usage: $0 segment:bus:device.function ..." >&2
+	if [ ${#bdf_list[@]} -eq 0 ]; then
+		usage
 		exit 1
 	fi
 
-	for device_bdf in "$@"; do
+	for device_bdf in "${bdf_list[@]}"; do
 		test -d /sys/bus/pci/devices/${device_bdf}
 
 		device_dir=${DEVICES_DIR}/${device_bdf}
@@ -42,6 +78,7 @@ function main() {
 
 		bind ${device_bdf} vfio-pci
 		touch ${device_dir}/vfio-pci
+		echo "Successfully set up ${device_bdf}"
 	done
 }
 
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer
#2 lore raw

Re: [PATCH 1/2] vfio: selftests: Find devices that are compatible with VFIO selftests

Quoted 40 lines · Josh Hilke

Update the VFIO selftest setup script to print the segment:bus:device.function numbers of devices on the user's system that have VFIO selftest drivers (i.e. are compatible with the VFIO selftest infrastructure). This makes it easy for users to quickly find devices that are compatible with VFIO selftests.

Example of how to list compatible devices on the system:

$ ./tools/testing/selftests/vfio/scripts/setup.sh -l
Supported devices:
0000:6a:01.0
0000:6f:01.0
0000:74:01.0
0000:79:01.0
0000:e7:01.0
0000:ec:01.0
0000:f1:01.0
0000:f6:01.0

Example of setting up a device:

$ ./tools/testing/selftests/vfio/scripts/setup.sh -d 0000:6a:01.0
+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
Successfully set up 0000:6a:01.0

This makes the script much more user friendly so that no one has to dig around in the VFIO selftest libraries to figure out which devices can run the selftests.

---
 tools/testing/selftests/vfio/scripts/setup.sh | 43 +++++++++++++++++--
 1 file changed, 40 insertions(+), 3 deletions(-)
tools/testing/selftests/vfio/scripts/setup.sh +0 -0
index 49a499e51cbe..2cde07eb9c8b 100755

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 8

Quoted 6 lines
tools/testing/selftests/vfio/scripts/setup.sh +3 -0
@@ -3,18 +3,54 @@ set -e
 
 source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
 
+# List of devices which have a VFIO selftest driver
+DEVICES=(
+	"8086:0b25" # Intel Data Streaming Accelerator

Please also add the new DMR and GNR-D DSA devices that were added recently:

https://github.com/awilliam/linux-vfio/commit/c82cfe15916d33e89c2d2efeeb624e8c9c2c4ca8

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 17

Quoted 9 lines
tools/testing/selftests/vfio/scripts/setup.sh +9 -0
+	"8086:0cf8" # Intel CBDMA
+)
+
+function print_supported_devices() {
+	local vendor_device_id
+	local id
+
+	for vendor_device_id in "${DEVICES[@]}"; do
+		read -r id <<< "${vendor_device_id}"

Sashiko seems to think the read here is unecessary.

https://sashiko.dev/#/patchset/20260409015139.2800185-1-jrhilke%40google.com?part=1

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 18

Quoted 1 line
tools/testing/selftests/vfio/scripts/setup.sh +1 -0
+		lspci -D -d "${id}" | awk '{ print $1 }'

Let's print the whole line and let the user parse out the BDF with awk. That will let the user see which type of device it its rather than just raw BDFs.

Quoted 14 lines
tools/testing/selftests/vfio/scripts/setup.sh +10 -0
+	done
+}
+
+function usage() {
+	echo "usage: $0 [-l] [-d <segment:bus:device.function>]" >&2
+	echo "" >&2
+	echo "  -l  List segment:bus:device.function numbers of supported devices." >&2
+	echo "  -d  segment:bus:device.function to set up." >&2
+}
+
 function main() {
 	local device_bdf
 	local device_dir
 	local numvfs

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 42

Quoted 10 lines
tools/testing/selftests/vfio/scripts/setup.sh +9 -0
 	local driver
+	local bdf_list=()
+
+	while getopts "ld:" opt; do
+		case ${opt} in
+			l)
+			 	echo "Supported devices: "
+				print_supported_devices
+				exit 0
+				;;

nit: Put the echo in print_supported_devices and this can be a one-liner like the rest of the options.

	while getopts "ld:" opt; do
		case ${opt} in
			l) print_supported_devices; exit 0 ;;
			d) bdf_list+=("${OPTARG}") ;;
			*) usage; exit 1 ;;
		esac
	done

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 43

Quoted 1 line
tools/testing/selftests/vfio/scripts/setup.sh +1 -0
+			d) bdf_list+=("${OPTARG}") ;;

Let's keep the current semantics where the list of BDFs is passed as positional arguments. The next commit would just pick a device if there are no positional arguments.

This will keep the script backwards compatible with current usage and also keep it simple (no need to pass extra -d flag for every BDF):

./setup.sh BDF1 BDF2 BDF3
vs.
./setup.sh -d BDF1 -d BDF2 -d BDF3

Quoted 27 lines
tools/testing/selftests/vfio/scripts/setup.sh +7 -3
+			*) usage; exit 1 ;;
+		esac
+	done
 
-	if [ $# = 0 ]; then
-		echo "usage: $0 segment:bus:device.function ..." >&2
+	if [ ${#bdf_list[@]} -eq 0 ]; then
+		usage
 		exit 1
 	fi
 
-	for device_bdf in "$@"; do
+	for device_bdf in "${bdf_list[@]}"; do
 		test -d /sys/bus/pci/devices/${device_bdf}
 
 		device_dir=${DEVICES_DIR}/${device_bdf}
@@ -42,6 +78,7 @@ function main() {
 
 		bind ${device_bdf} vfio-pci
 		touch ${device_dir}/vfio-pci
+		echo "Successfully set up ${device_bdf}"
 	done
 }
 
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
author
#3 lore raw

Re: [PATCH 1/2] vfio: selftests: Find devices that are compatible with VFIO selftests

Quoted 8 lines · David Matlack

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 8

Quoted 3 lines
tools/testing/selftests/vfio/scripts/setup.sh +3 -0
+# List of devices which have a VFIO selftest driver
+DEVICES=(
+     "8086:0b25" # Intel Data Streaming Accelerator

Please also add the new DMR and GNR-D DSA devices that were added recently:

https://github.com/awilliam/linux-vfio/commit/c82cfe15916d33e89c2d2efeeb624e8c9c2c4ca8

Ack. Will do in v3.

Quoted 10 lines

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 17

Quoted 6 lines
tools/testing/selftests/vfio/scripts/setup.sh +6 -0
+function print_supported_devices() {
+     local vendor_device_id
+     local id
+
+     for vendor_device_id in "${DEVICES[@]}"; do
+             read -r id <<< "${vendor_device_id}"

Sashiko seems to think the read here is unecessary.

https://sashiko.dev/#/patchset/20260409015139.2800185-1-jrhilke%40google.com?part=1

Ah yea, the read is a vestige of a different iteration I had of this script. I'll remove it in v3.

Quoted 5 lines
Quoted 1 line

+ lspci -D -d "${id}" | awk '{ print $1 }'

Let's print the whole line and let the user parse out the BDF with awk. That will let the user see which type of device it its rather than just raw BDFs.

Will do in v3.

Quoted 11 lines

tools/testing/selftests/vfio/scripts/setup.sh set -e · line 42

Quoted 7 lines
tools/testing/selftests/vfio/scripts/setup.sh +7 -0
+     while getopts "ld:" opt; do
+             case ${opt} in
+                     l)
+                             echo "Supported devices: "
+                             print_supported_devices
+                             exit 0
+                             ;;

nit: Put the echo in print_supported_devices and this can be a one-liner like the rest of the options.

Quoted 7 lines
        while getopts "ld:" opt; do
                case ${opt} in
                        l) print_supported_devices; exit 0 ;;
                        d) bdf_list+=("${OPTARG}") ;;
                        *) usage; exit 1 ;;
                esac
        done

Will do in v3.

Quoted 6 lines
Quoted 1 line

+ d) bdf_list+=("${OPTARG}") ;;

Let's keep the current semantics where the list of BDFs is passed as positional arguments. The next commit would just pick a device if there are no positional arguments.

Quoted 6 lines

This will keep the script backwards compatible with current usage and also keep it simple (no need to pass extra -d flag for every BDF):

./setup.sh BDF1 BDF2 BDF3
vs.
./setup.sh -d BDF1 -d BDF2 -d BDF3

Works for me. Will do in v3.

author
#4 lore raw

PATCH 2/2 vfio: selftests: Automate setting up devices for testing

In reply to Josh Hilke

Update the VFIO selftest setup script to find a device that has a VFIO selftest driver on the system, and set it up (i.e unbind that device from its driver, and then bind that device to VFIO).

This makes the process of setting up a device to VFIO selftests much easier, as users don't have to manually figure out which device to use. Though, users can still manually setup a device by passing the -d argument with a segment:bus:device.function number.

Example of automated device setup:

	$  ./tools/testing/selftests/vfio/scripts/setup.sh./setup.sh
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
	+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
	Successfully set up 0000:6a:01.0

Example of automated device setup where no compatible devices exist:

	$ . ./tools/testing/selftests/vfio/scripts/setup.sh/setup.sh
	No available supported devices found on the system.

Example of manual device setup:

	$  ./tools/testing/selftests/vfio/scripts/setup.sh -d 0000:6a:01.0
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
	+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
	Successfully set up 0000:6a:01.0
---
 tools/testing/selftests/vfio/scripts/setup.sh | 22 +++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
tools/testing/selftests/vfio/scripts/setup.sh +20 -2
index 2cde07eb9c8b..6d4492b211ea 100755
@@ -19,11 +19,30 @@ function print_supported_devices() {
 	done
 }
 
+function pick_device() {
+	local bdf
+
+	while read -r bdf; do
+		if [ -n "${bdf}" ]; then
+			if [ -d "${DEVICES_DIR}/${bdf}" ]; then
+				echo "${bdf} has already been set up, exiting." >&2
+				exit 0
+			fi
+			echo "${bdf}"
+			return 0
+		fi
+	done <<< "$(print_supported_devices)"
+
+	echo "No available supported devices found on the system." >&2
+	exit 1
+}
+
 function usage() {
 	echo "usage: $0 [-l] [-d <segment:bus:device.function>]" >&2
 	echo "" >&2
 	echo "  -l  List segment:bus:device.function numbers of supported devices." >&2
 	echo "  -d  segment:bus:device.function to set up." >&2
+	echo "      If -d is not specified, a device will be automatically picked." >&2
 }
 
 function main() {
@@ -46,8 +65,7 @@ function main() {
 	done
 
 	if [ ${#bdf_list[@]} -eq 0 ]; then
-		usage
-		exit 1
+		bdf_list=($(pick_device))
 	fi
 
 	for device_bdf in "${bdf_list[@]}"; do
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer
#5 lore raw

Re: [PATCH 2/2] vfio: selftests: Automate setting up devices for testing

Quoted 4 lines · Josh Hilke

Update the VFIO selftest setup script to find a device that has a VFIO selftest driver on the system, and set it up (i.e unbind that device from its driver, and then bind that device to VFIO).

Quoted 8 lines

This makes the process of setting up a device to VFIO selftests much easier, as users don't have to manually figure out which device to use. Though, users can still manually setup a device by passing the -d argument with a segment:bus:device.function number.

Example of automated device setup:

	$  ./tools/testing/selftests/vfio/scripts/setup.sh./setup.sh

Script path here got garbled

Quoted 8 lines

+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind Successfully set up 0000:6a:01.0

Example of automated device setup where no compatible devices exist:

	$ . ./tools/testing/selftests/vfio/scripts/setup.sh/setup.sh

Here too, also the extra '.' before the path looks like a mistake.

Quoted 11 lines

No available supported devices found on the system.

Example of manual device setup:

	$  ./tools/testing/selftests/vfio/scripts/setup.sh -d 0000:6a:01.0
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/idxd/unbind
	+ echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override
	+ echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
	Successfully set up 0000:6a:01.0

Please add Suggested-by tag for Sean and also Cc Sean on next version of the series :)

Quoted 17 lines
---
 tools/testing/selftests/vfio/scripts/setup.sh | 22 +++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
tools/testing/selftests/vfio/scripts/setup.sh +5 -0
index 2cde07eb9c8b..6d4492b211ea 100755
@@ -19,11 +19,30 @@ function print_supported_devices() {
 	done
 }
 
+function pick_device() {
+	local bdf
+
+	while read -r bdf; do
+		if [ -n "${bdf}" ]; then

tools/testing/selftests/vfio/scripts/setup.sh function print_supported_devices() { · line 36

Quoted 10 lines
tools/testing/selftests/vfio/scripts/setup.sh +10 -0
+			if [ -d "${DEVICES_DIR}/${bdf}" ]; then
+				echo "${bdf} has already been set up, exiting." >&2
+				exit 0
+			fi
+			echo "${bdf}"
+			return 0
+		fi
+	done <<< "$(print_supported_devices)"
+
+	echo "No available supported devices found on the system." >&2

I think we should make it clear what "supported" means in this log. This is about finding a device that has a VFIO selftests driver, which is only required by some tests.

Same goes for the -l option in the previous commit actually and the usage message below.

Quoted 13 lines
tools/testing/selftests/vfio/scripts/setup.sh +4 -0
+	exit 1
+}
+
 function usage() {
 	echo "usage: $0 [-l] [-d <segment:bus:device.function>]" >&2
 	echo "" >&2
 	echo "  -l  List segment:bus:device.function numbers of supported devices." >&2
 	echo "  -d  segment:bus:device.function to set up." >&2
+	echo "      If -d is not specified, a device will be automatically picked." >&2
 }
 
 function main() {

tools/testing/selftests/vfio/scripts/setup.sh function main() { · line 68

Quoted 6 lines
tools/testing/selftests/vfio/scripts/setup.sh +1 -2
@@ -46,8 +65,7 @@ function main() {
 	done
 
 	if [ ${#bdf_list[@]} -eq 0 ]; then
-		usage
-		exit 1
+		bdf_list=($(pick_device))

Sashiko thinks this will ignore any errors from pick_device():

https://sashiko.dev/#/patchset/20260409015139.2800185-1-jrhilke%40google.com?part=2

Quoted 6 lines
tools/testing/selftests/vfio/scripts/setup.sh +0 -0
 	fi
 
 	for device_bdf in "${bdf_list[@]}"; do
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer
#6 lore raw

Re: [PATCH 2/2] vfio: selftests: Automate setting up devices for testing

In reply to Josh Hilke
Quoted 3 lines · Josh Hilke

Update the VFIO selftest setup script to find a device that has a VFIO selftest driver on the system, and set it up (i.e unbind that device from its driver, and then bind that device to VFIO).

I am leaning toward putting this behind an explicit flag (maybe -p for pick_device or -a for automatic device setup?) that way we are sure the user is ok with letting the script start messing with devices on their system.

#7 lore raw

Re: [PATCH 2/2] vfio: selftests: Automate setting up devices for testing

Quoted 9 lines · David Matlack
Quoted 3 lines · Josh Hilke

Update the VFIO selftest setup script to find a device that has a VFIO selftest driver on the system, and set it up (i.e unbind that device from its driver, and then bind that device to VFIO).

I am leaning toward putting this behind an explicit flag (maybe -p for pick_device or -a for automatic device setup?) that way we are sure the user is ok with letting the script start messing with devices on their system.

In the spirit of "do one thing and do it well", I would rather put "discovery" in a separate script entirely, and then tweak setup.sh as necessary to allow piping in a BDF from the discovery script.

The mechanics of configuring a device for VFIO (setup.sh) are largely generic, i.e. not unique to the selftests, whereas print_supported_devices() is very much tightly coupled to selftests.

And with respect to making sure the user actually wants the script to mess with their system, the permissions required for each script can and should be different. lspci, and discovery in general, typically doesn't require root, whereas most systems will be configured to have things like "sriov_numvfs" writable only from root.

author
#8 lore raw

Re: [PATCH 2/2] vfio: selftests: Automate setting up devices for testing

Quoted 20 lines · Sean Christopherson
Quoted 9 lines · David Matlack
Quoted 3 lines · Josh Hilke

Update the VFIO selftest setup script to find a device that has a VFIO selftest driver on the system, and set it up (i.e unbind that device from its driver, and then bind that device to VFIO).

I am leaning toward putting this behind an explicit flag (maybe -p for pick_device or -a for automatic device setup?) that way we are sure the user is ok with letting the script start messing with devices on their system.

In the spirit of "do one thing and do it well", I would rather put "discovery" in a separate script entirely, and then tweak setup.sh as necessary to allow piping in a BDF from the discovery script.

The mechanics of configuring a device for VFIO (setup.sh) are largely generic, i.e. not unique to the selftests, whereas print_supported_devices() is very much tightly coupled to selftests.

Quoted 5 lines

And with respect to making sure the user actually wants the script to mess with their system, the permissions required for each script can and should be different. lspci, and discovery in general, typically doesn't require root, whereas most systems will be configured to have things like "sriov_numvfs" writable only from root.

I'll put device discovery in a separate script for v3.

Regarding David's feedback, I'll also use -p for automatic device setup, and make the phrasing for "supported devices" more precise.