Skip to contentSkip to conversation

vfio: selftest: Add SR-IOV UAPI test

PATCH Superseded Raghavendra Rao Ananta 4 participants 20 messages Apr 2 – May 5

Subsystems (4)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v7 cover vfio: selftest: Add SR-IOV UAPI test

Hello,

This series adds a vfio selftest, vfio_pci_sriov_uapi_test.c, to get some coverage on SR-IOV UAPI handling. Specifically, it includes the following cases that iterates over all the iommu modes:
- Setting correct/incorrect/NULL tokens during device init.
- Close the PF device immediately after setting the token.
- Change/override the PF's token after device init.

The test takes care of creating/setting up the VF device, and hence, it can be executed like any other test, simply by passing the PF's BDF to run.sh. For example,

$ ./scripts/setup.sh 0000:16:00.1
$ ./scripts/run.sh ./vfio_pci_sriov_uapi_test

Created 1 VF (0000:1a:00.0) under the PF: 0000:16:00.1
TAP version 13
1..45
# Starting 45 tests from 15 test cases.
# RUN vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.init_token_match ...
# OK vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.init_token_match
ok 1 vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.init_token_match
# RUN vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.pf_early_close ...
# OK vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.pf_early_close
ok 2 vfio_pci_sriov_uapi_test.vfio_type1_iommu_same_uuid.pf_early_close [...]
# RUN vfio_pci_sriov_uapi_test.iommufd_null_uuid.override_token ...
# OK vfio_pci_sriov_uapi_test.iommufd_null_uuid.override_token
ok 45 vfio_pci_sriov_uapi_test.iommufd_null_uuid.override_token # PASSED: 45 / 45 tests passed.
# Totals: pass:45 fail:0 xfail:0 xpass:0 skip:0 error:0

Thank you.
Raghavendra

v7: Suggestions by Vipin and David
- Introduce and use readlink_safe() to use readlink() functionality without worrying about buffer overruns and non '\0'terminations. (David)
- Define 'group'with 'int', instead of 'unsigned int' in sysfs_iommu_group_get() (Vipin)
- Add comments and define macros, such as DRIVER_NAME_SZ,to make the sysfs lib more readable (Vipin).
- Replace strcpy() with strncpy() in sysfs_driver_get(). (Vipin)
- Replace the FIXTURE_SETUP() and FIXTURE_TEARDOWN() with setup_vf() and teardown_vf() (implemented as an exit handler) that are called outside the test harness. This is to leave the test with the VF destroyed if any of the setup fails. (Vipin and David)
- Add asserts after every PF's device_init(). (Vipin)

v6:
- Terminate the output buffers of readlink() with '\0' in sysfs.c.

v5:
- Rebase to v7.0-rc1 (Alex).

v4: Suggestions by David and Alex
- Assert that the value computed in sysfs_val_get() in an int. Rename the function to sysfs_val_get_int() to better reflect what the function is doing. (Alex)
- Add the missing Signed-off-by tag in patch-7 (David).

v3: Suggestions by David Matlack (thanks!)
- Introduce a patch to add -Wall and -Werror to the vfio Makefile.
- Use snprintf_assert() where they were missed.
- Rename the functions as suggested in the sysfs lib and the test file.
- Alloc the output char * buffer in the functions sysfs_driver_get() and sysfs_sriov_vf_bdf_get() instead of relying on the caller to pass one. The caller is now responsible for freeing these buffers.
- Remove unnecessary initializations of local variables in sysfs and the vfio_pci_device libraries.
- Move the inclusion of -luuid to the top level Makefile.
- Introduce vfio_pci_device_{alloc|free}() and let the test and the functions in vfio_pci_device.c use this.
- Return -errno for the ioctl failure in __vfio_device_bind_iommufd() instead of directly calling ioctl_assert().
- Since the vfio-pci driver sets the 'driver_override' to the driver of PF, instead of clearing sriov_drivers_autoprobe and binding the VF explicitly to the 'vfio-pci' driver, only assert that it's already bound.
- By extension to the above point, remove the unnecessary functions from the sysfs lib.

v2: Suggestions by David Matlack (thank you)
- Introduce snprintf_assert() to check against content trucation.
- Introduce a new sysfs library to handle all the common vfio/pci sysfs operations.
- Rename vfio_pci_container_get_device_fd() to vfio_pci_group_get_device_fd().
- Use a fixed size 'arg' array instead of dynamic allocation in __vfio_pci_group_get_device_fd().
- Exclude vfio_pci_device_init() to accept the 'vf_token' arg.
- Move the vfio_pci_sriov_uapi_test.c global variable to the FIXTURE() struct or as TEST_F() local variables.
- test_vfio_pci_container_setup() returns 'int' to indicate status.
- Skip the test if nr_vfs != 0.
- Explicitly set "sriov_drivers_autoprobe" for the PF.
- Make sure to bind the VF device to the "vfio-pci" driver.
- Cleanup the things done by FIXTURE_SETUP() in FIXTURE_TEARDOWN().

v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Raghavendra Rao Ananta (8):
vfio: selftests: Add -Wall and -Werror to the Makefile
vfio: selftests: Introduce snprintf_assert()
vfio: selftests: Introduce a sysfs lib
vfio: selftests: Extend container/iommufd setup for passing vf_token
vfio: selftests: Expose more vfio_pci_device functions
vfio: selftests: Add helper to set/override a vf_token
vfio: selftests: Add helpers to alloc/free vfio_pci_device
vfio: selftests: Add tests to validate SR-IOV UAPI

 tools/testing/selftests/vfio/Makefile         |   4 +
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../vfio/lib/include/libvfio/assert.h         |   5 +
 .../vfio/lib/include/libvfio/sysfs.h          |  12 +
 .../lib/include/libvfio/vfio_pci_device.h     |  11 +
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 150 ++++++++++++
 .../selftests/vfio/lib/vfio_pci_device.c      | 157 +++++++++---
 .../selftests/vfio/vfio_dma_mapping_test.c    |   6 +-
 .../selftests/vfio/vfio_pci_device_test.c     |  21 +-
 .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 230 ++++++++++++++++++
 11 files changed, 547 insertions(+), 51 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
 create mode 100644 tools/testing/selftests/vfio/lib/sysfs.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c

base-commit: 6de23f81a5e08be8fbf5e8d7e9febc72a5b5f27f --
2.53.0.1213.gd9a14994de-goog

#1 lore raw

PATCH v7 1/8 vfio: selftests: Add -Wall and -Werror to the Makefile

Add the compiler flags, -Wall and -Werror, to catch all the build warnings and flag them as a build error, respectively. This is to ensure that no obvious programmer errors are introduced. We can add -Wno-* flags in the future to ignore specific warnings as necesasry.

---
 tools/testing/selftests/vfio/Makefile | 1 +
 1 file changed, 1 insertion(+)
tools/testing/selftests/vfio/Makefile +1 -0
index 8e90e409e91d8..6a9ac6dd32cb6 100644
@@ -23,6 +23,7 @@ include lib/libvfio.mk
 
 CFLAGS += -I$(top_srcdir)/tools/include
 CFLAGS += -MD
+CFLAGS += -Wall -Werror
 CFLAGS += $(EXTRA_CFLAGS)
 
 LDFLAGS += -pthread
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#2 lore raw

PATCH v7 2/8 vfio: selftests: Introduce snprintf_assert()

In reply to Raghavendra Rao Ananta

Introduce snprintf_assert() to protect the users of snprintf() to fail if the requested operation was truncated due to buffer limits. VFIO tests and libraries, including a new sysfs library that will be introduced by an upcoming patch, rely quite heavily on snprintf()s to build PCI sysfs paths. Having a protection against this will be helpful to prevent false test failures.

---
 .../vfio/lib/include/libvfio/assert.h         |  5 +++++
 .../selftests/vfio/lib/vfio_pci_device.c      |  8 +++----
 .../selftests/vfio/vfio_dma_mapping_test.c    |  6 +++---
 .../selftests/vfio/vfio_pci_device_test.c     | 21 ++++++++++---------
 4 files changed, 23 insertions(+), 17 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/assert.h +5 -0
index f4ebd122d9b6f..77b68c7129a64 100644
@@ -51,4 +51,9 @@
 	VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #_arg, __ret); \
 } while (0)
 
+#define snprintf_assert(_s, _size, _fmt, ...) do {                      \
+	int __ret = snprintf(_s, _size, _fmt, ##__VA_ARGS__);           \
+	VFIO_ASSERT_LT(__ret, _size);                                   \
+} while (0)
+
 #endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H */
tools/testing/selftests/vfio/lib/vfio_pci_device.c +4 -4
index 4e5871f1ebc3b..1538d2b30ae59 100644
@@ -209,7 +209,7 @@ static unsigned int vfio_pci_get_group_from_dev(const char *bdf)
 	unsigned int group;
 	int ret;
 
-	snprintf(sysfs_path, PATH_MAX, "%s/%s/iommu_group", PCI_SYSFS_PATH, bdf);
+	snprintf_assert(sysfs_path, PATH_MAX, "%s/%s/iommu_group", PCI_SYSFS_PATH, bdf);
 
 	ret = readlink(sysfs_path, dev_iommu_group_path, sizeof(dev_iommu_group_path));
 	VFIO_ASSERT_NE(ret, -1, "Failed to get the IOMMU group for device: %s\n", bdf);
@@ -229,7 +229,7 @@ static void vfio_pci_group_setup(struct vfio_pci_device *device, const char *bdf
 	int group;
 
 	group = vfio_pci_get_group_from_dev(bdf);
-	snprintf(group_path, sizeof(group_path), "/dev/vfio/%d", group);
+	snprintf_assert(group_path, sizeof(group_path), "/dev/vfio/%d", group);
 
 	device->group_fd = open(group_path, O_RDWR);
 	VFIO_ASSERT_GE(device->group_fd, 0, "open(%s) failed\n", group_path);
@@ -300,7 +300,7 @@ const char *vfio_pci_get_cdev_path(const char *bdf)
 	cdev_path = calloc(PATH_MAX, 1);
 	VFIO_ASSERT_NOT_NULL(cdev_path);
 
-	snprintf(dir_path, sizeof(dir_path), "/sys/bus/pci/devices/%s/vfio-dev/", bdf);
+	snprintf_assert(dir_path, sizeof(dir_path), "/sys/bus/pci/devices/%s/vfio-dev/", bdf);
 
 	dir = opendir(dir_path);
 	VFIO_ASSERT_NOT_NULL(dir, "Failed to open directory %s\n", dir_path);
@@ -310,7 +310,7 @@ const char *vfio_pci_get_cdev_path(const char *bdf)
 		if (strncmp("vfio", entry->d_name, 4))
 			continue;
 
-		snprintf(cdev_path, PATH_MAX, "/dev/vfio/devices/%s", entry->d_name);
+		snprintf_assert(cdev_path, PATH_MAX, "/dev/vfio/devices/%s", entry->d_name);
 		break;
 	}
 
tools/testing/selftests/vfio/vfio_dma_mapping_test.c +3 -3
index abb170bdcef7e..7d0de8c79de13 100644
@@ -44,9 +44,9 @@ static int intel_iommu_mapping_get(const char *bdf, u64 iova,
 	FILE *file;
 	char *rest;
 
-	snprintf(iommu_mapping_path, sizeof(iommu_mapping_path),
-		 "/sys/kernel/debug/iommu/intel/%s/domain_translation_struct",
-		 bdf);
+	snprintf_assert(iommu_mapping_path, sizeof(iommu_mapping_path),
+			"/sys/kernel/debug/iommu/intel/%s/domain_translation_struct",
+			bdf);
 
 	printf("Searching for IOVA 0x%lx in %s\n", iova, iommu_mapping_path);
 
tools/testing/selftests/vfio/vfio_pci_device_test.c +11 -10
index 7c0fe8ce3a61f..93c11fd5e0818 100644
@@ -39,16 +39,17 @@ FIXTURE_TEARDOWN(vfio_pci_device_test)
 	iommu_cleanup(self->iommu);
 }
 
-#define read_pci_id_from_sysfs(_file) ({							\
-	char __sysfs_path[PATH_MAX];								\
-	char __buf[32];										\
-	int __fd;										\
-												\
-	snprintf(__sysfs_path, PATH_MAX, "/sys/bus/pci/devices/%s/%s", device_bdf, _file);	\
-	ASSERT_GT((__fd = open(__sysfs_path, O_RDONLY)), 0);					\
-	ASSERT_GT(read(__fd, __buf, ARRAY_SIZE(__buf)), 0);					\
-	ASSERT_EQ(0, close(__fd));								\
-	(u16)strtoul(__buf, NULL, 0);								\
+#define read_pci_id_from_sysfs(_file) ({					\
+	char __sysfs_path[PATH_MAX];						\
+	char __buf[32];								\
+	int __fd;								\
+										\
+	snprintf_assert(__sysfs_path, PATH_MAX, "/sys/bus/pci/devices/%s/%s",	\
+			device_bdf, _file);					\
+	ASSERT_GT((__fd = open(__sysfs_path, O_RDONLY)), 0);			\
+	ASSERT_GT(read(__fd, __buf, ARRAY_SIZE(__buf)), 0);			\
+	ASSERT_EQ(0, close(__fd));						\
+	(u16)strtoul(__buf, NULL, 0);						\
 })
 
 TEST_F(vfio_pci_device_test, config_space_read_write)
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#3 lore raw

PATCH v7 3/8 vfio: selftests: Introduce a sysfs lib

In reply to Raghavendra Rao Ananta

Introduce a sysfs library to handle the common reads/writes to the PCI sysfs files, for example, getting the total number of VFs supported by the device via /sys/bus/pci/devices/$BDF/sriov_totalvfs. The library will be used in the upcoming test patch to configure the VFs for a given PF device.

Since readlink() is quite commonly used in the lib, introduce and use readlink_safe() to take care of potential buffer overrun errors and to safely terminate the buffer with '\0'.

Opportunistically, move vfio_pci_get_group_from_dev() to this library as it falls under the same bucket. Rename it to sysfs_iommu_group_get() to align with other function names.

---
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../vfio/lib/include/libvfio/sysfs.h          |  12 ++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 150 ++++++++++++++++++
 .../selftests/vfio/lib/vfio_pci_device.c      |  22 +--
 5 files changed, 165 insertions(+), 21 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h
 create mode 100644 tools/testing/selftests/vfio/lib/sysfs.c
tools/testing/selftests/vfio/lib/include/libvfio.h +1 -0
index 1b6da54cc2cb7..07862b470777b 100644
@@ -5,6 +5,7 @@
 #include <libvfio/assert.h>
 #include <libvfio/iommu.h>
 #include <libvfio/iova_allocator.h>
+#include <libvfio/sysfs.h>
 #include <libvfio/vfio_pci_device.h>
 #include <libvfio/vfio_pci_driver.h>
 
tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h new file +12 -0
new file mode 100644
index 0000000000000..c1d945ea62a10
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H
+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H
+
+int sysfs_sriov_totalvfs_get(const char *bdf);
+int sysfs_sriov_numvfs_get(const char *bdf);
+void sysfs_sriov_numvfs_set(const char *bdfs, int numvfs);
+char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
+int sysfs_iommu_group_get(const char *bdf);
+char *sysfs_driver_get(const char *bdf);
+
+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H */
tools/testing/selftests/vfio/lib/libvfio.mk +1 -0
index 9f47bceed16f4..b7857319c3f1f 100644
@@ -6,6 +6,7 @@ LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
 LIBVFIO_C := iommu.c
 LIBVFIO_C += iova_allocator.c
 LIBVFIO_C += libvfio.c
+LIBVFIO_C += sysfs.c
 LIBVFIO_C += vfio_pci_device.c
 LIBVFIO_C += vfio_pci_driver.c
 
tools/testing/selftests/vfio/lib/sysfs.c new file +150 -0

150 lines not on this page · raw message

tools/testing/selftests/vfio/lib/vfio_pci_device.c +1 -21
index 1538d2b30ae59..82f255f0486dc 100644
@@ -25,8 +25,6 @@
 #include "kselftest.h"
 #include <libvfio.h>
 
-#define PCI_SYSFS_PATH	"/sys/bus/pci/devices"
-
 static void vfio_pci_irq_set(struct vfio_pci_device *device,
 			     u32 index, u32 vector, u32 count, int *fds)
 {
@@ -202,24 +200,6 @@ void vfio_pci_device_reset(struct vfio_pci_device *device)
 	ioctl_assert(device->fd, VFIO_DEVICE_RESET, NULL);
 }
 
-static unsigned int vfio_pci_get_group_from_dev(const char *bdf)
-{
-	char dev_iommu_group_path[PATH_MAX] = {0};
-	char sysfs_path[PATH_MAX] = {0};
-	unsigned int group;
-	int ret;
-
-	snprintf_assert(sysfs_path, PATH_MAX, "%s/%s/iommu_group", PCI_SYSFS_PATH, bdf);
-
-	ret = readlink(sysfs_path, dev_iommu_group_path, sizeof(dev_iommu_group_path));
-	VFIO_ASSERT_NE(ret, -1, "Failed to get the IOMMU group for device: %s\n", bdf);
-
-	ret = sscanf(basename(dev_iommu_group_path), "%u", &group);
-	VFIO_ASSERT_EQ(ret, 1, "Failed to get the IOMMU group for device: %s\n", bdf);
-
-	return group;
-}
-
 static void vfio_pci_group_setup(struct vfio_pci_device *device, const char *bdf)
 {
 	struct vfio_group_status group_status = {
@@ -228,7 +208,7 @@ static void vfio_pci_group_setup(struct vfio_pci_device *device, const char *bdf
 	char group_path[32];
 	int group;
 
-	group = vfio_pci_get_group_from_dev(bdf);
+	group = sysfs_iommu_group_get(bdf);
 	snprintf_assert(group_path, sizeof(group_path), "/dev/vfio/%d", group);
 
 	device->group_fd = open(group_path, O_RDWR);
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer
#4 lore raw

Re: [PATCH v7 3/8] vfio: selftests: Introduce a sysfs lib

Quoted 14 lines · On Thu, 2 Apr 2026 17:30:54 +0000 Raghavendra Rao Ananta <[email protected]>…
tools/testing/selftests/vfio/lib/sysfs.c new file +8 -0
new file mode 100644
index 0000000000000..f1e6889556527
@@ -0,0 +1,150 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <fcntl.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+#include <linux/limits.h>
+
+#include <libvfio.h>

tools/testing/selftests/vfio/lib/sysfs.c line 18

Quoted 10 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +10 -0
+
+/* Enough to hold MODULE_NAME_LEN */
+#define DRIVER_NAME_SZ 64
+
+#define readlink_safe(_path, _buf) ({				\
+	int __ret = readlink(_path, _buf, sizeof(_buf) - 1);	\
+	if (__ret != -1)					\
+		_buf[__ret] = 0;				\
+	__ret;							\
+})

Not a current issue, but the use of sizeof() here is a danger for future users that might pass something other than a stack array. We could avoid this by adding a build time test (untested):

    _Static_assert(!__builtin_types_compatible_p(                       \
                       __typeof__(_buf), char *),                       \
                   "readlink_safe: _buf must be an array, not a pointer"); \

Maybe overkill for an internal helper, but we could avoid the hazard.

...

Quoted 12 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +12 -0
+char *sysfs_driver_get(const char *bdf)
+{
+	char driver_path[PATH_MAX];
+	char path[PATH_MAX];
+	char *out_driver;
+	int ret;
+
+	out_driver = calloc(DRIVER_NAME_SZ, sizeof(char));
+	VFIO_ASSERT_NOT_NULL(out_driver);
+
+	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/devices/%s/driver", bdf);
+	ret = readlink_safe(path, driver_path);

tools/testing/selftests/vfio/lib/sysfs.c line 148

Quoted 10 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +10 -0
+	if (ret == -1) {
+		free(out_driver);
+
+		if (errno == ENOENT)
+			return NULL;
+
+		VFIO_FAIL("Failed to read %s\n", path);
+	}
+
+	strncpy(out_driver, basename(driver_path), DRIVER_NAME_SZ - 1);

Nit, I'm not really sure what DRIVER_NAME_SZ buys us other than an artificial limit. We're already bound by PATH_MAX and could just do a strdup() for the return. Neither are blockers. Thanks,

Alex

Quoted 2 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +2 -0
+	return out_driver;
+}
#5 lore raw

Re: [PATCH v7 3/8] vfio: selftests: Introduce a sysfs lib

Quoted 32 lines · Alex Williamson
Quoted 14 lines · On Thu, 2 Apr 2026 17:30:54 +0000 Raghavendra Rao Ananta <[email protected]>…
tools/testing/selftests/vfio/lib/sysfs.c new file +8 -0
new file mode 100644
index 0000000000000..f1e6889556527
@@ -0,0 +1,150 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <fcntl.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+#include <linux/limits.h>
+
+#include <libvfio.h>

tools/testing/selftests/vfio/lib/sysfs.c line 18

Quoted 10 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +10 -0
+
+/* Enough to hold MODULE_NAME_LEN */
+#define DRIVER_NAME_SZ 64
+
+#define readlink_safe(_path, _buf) ({                                \
+     int __ret = readlink(_path, _buf, sizeof(_buf) - 1);    \
+     if (__ret != -1)                                        \
+             _buf[__ret] = 0;                                \
+     __ret;                                                  \
+})

Not a current issue, but the use of sizeof() here is a danger for future users that might pass something other than a stack array. We could avoid this by adding a build time test (untested):

Quoted 6 lines
    _Static_assert(!__builtin_types_compatible_p(                       \
                       __typeof__(_buf), char *),                       \
                   "readlink_safe: _buf must be an array, not a pointer"); \

Maybe overkill for an internal helper, but we could avoid the hazard.

Good point! It makes sense to have a build-time check for this. I will incorporate the _Static_assert to avoid the hazard.

Quoted 19 lines

...

Quoted 18 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +18 -0
+char *sysfs_driver_get(const char *bdf)
+{
+     char driver_path[PATH_MAX];
+     char path[PATH_MAX];
+     char *out_driver;
+     int ret;
+
+     out_driver = calloc(DRIVER_NAME_SZ, sizeof(char));
+     VFIO_ASSERT_NOT_NULL(out_driver);
+
+     snprintf_assert(path, PATH_MAX, "/sys/bus/pci/devices/%s/driver", bdf);
+     ret = readlink_safe(path, driver_path);
+     if (ret == -1) {
+             free(out_driver);
+
+             if (errno == ENOENT)
+                     return NULL;
+
Quoted 9 lines

tools/testing/selftests/vfio/lib/sysfs.c line 148

Quoted 4 lines
tools/testing/selftests/vfio/lib/sysfs.c new file +4 -0
+             VFIO_FAIL("Failed to read %s\n", path);
+     }
+
+     strncpy(out_driver, basename(driver_path), DRIVER_NAME_SZ - 1);

Nit, I'm not really sure what DRIVER_NAME_SZ buys us other than an artificial limit. We're already bound by PATH_MAX and could just do a strdup() for the return. Neither are blockers. Thanks,

Ah, why didn't I think of strdup(). It's a better approach. I'll update this in v8.

Thank you.
Raghavendra

#6 lore raw

PATCH v7 4/8 vfio: selftests: Extend container/iommufd setup for passing vf_token

In reply to Raghavendra Rao Ananta

A UUID is normally set as a vf_token to correspond the VFs with the PFs, if they are both bound by the vfio-pci driver. This is true for iommufd-based approach and container-based approach. The token can be set either during device creation (VFIO_GROUP_GET_DEVICE_FD) in container-based approach or during iommu bind (VFIO_DEVICE_BIND_IOMMUFD) in the iommu-fd case. Hence extend the functions, vfio_pci_iommufd_setup() and vfio_pci_container_setup(), to accept vf_token as an (optional) argument and handle the necessary setup.

No functional changes are expected.

---
 tools/testing/selftests/vfio/Makefile         |  2 +
 .../selftests/vfio/lib/vfio_pci_device.c      | 46 +++++++++++++++----
 2 files changed, 40 insertions(+), 8 deletions(-)
tools/testing/selftests/vfio/Makefile +2 -0
index 6a9ac6dd32cb6..f27ed18070f14 100644
@@ -28,6 +28,8 @@ CFLAGS += $(EXTRA_CFLAGS)
 
 LDFLAGS += -pthread
 
+LDLIBS += -luuid
+
 $(TEST_GEN_PROGS): %: %.o $(LIBVFIO_O)
 	$(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $< $(LIBVFIO_O) $(LDLIBS) -o $@
 
tools/testing/selftests/vfio/lib/vfio_pci_device.c +38 -8

94 lines not on this page · raw message

Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#7 lore raw

PATCH v7 5/8 vfio: selftests: Expose more vfio_pci_device functions

In reply to Raghavendra Rao Ananta

Refactor and make the functions called under device initialization public. A later patch adds a test that calls these functions to validate the UAPI of SR-IOV devices. Opportunistically, to test the success and failure cases of the UAPI, split the functions dealing with VFIO_GROUP_GET_DEVICE_FD and VFIO_DEVICE_BIND_IOMMUFD into a core function and another one that asserts the ioctl. The former will be used for testing the SR-IOV UAPI, hence only export these.

---
 .../lib/include/libvfio/vfio_pci_device.h     |  7 +++
 .../selftests/vfio/lib/vfio_pci_device.c      | 47 ++++++++++++++-----
 2 files changed, 42 insertions(+), 12 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +7 -0
index 2858885a89bbb..898de032fed5a 100644
@@ -122,4 +122,11 @@ static inline bool vfio_pci_device_match(struct vfio_pci_device *device,
 
 const char *vfio_pci_get_cdev_path(const char *bdf);
 
+void vfio_pci_group_setup(struct vfio_pci_device *device, const char *bdf);
+void __vfio_pci_group_get_device_fd(struct vfio_pci_device *device,
+				    const char *bdf, const char *vf_token);
+void vfio_container_set_iommu(struct vfio_pci_device *device);
+void vfio_pci_cdev_open(struct vfio_pci_device *device, const char *bdf);
+int __vfio_device_bind_iommufd(int device_fd, int iommufd, const char *vf_token);
+
 #endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H */
tools/testing/selftests/vfio/lib/vfio_pci_device.c +35 -12

105 lines not on this page · raw message

Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#8 lore raw

PATCH v7 6/8 vfio: selftests: Add helper to set/override a vf_token

In reply to Raghavendra Rao Ananta

Add a helper function, vfio_device_set_vf_token(), to set or override a vf_token. Not only at init, but a vf_token can also be set via the VFIO_DEVICE_FEATURE ioctl, by setting the VFIO_DEVICE_FEATURE_PCI_VF_TOKEN flag. Hence, add an API to utilize this functionality from the test code. The subsequent commit will use this to test the functionality of this method to set the vf_token.

---
 .../lib/include/libvfio/vfio_pci_device.h     |  2 ++
 .../selftests/vfio/lib/vfio_pci_device.c      | 34 +++++++++++++++++++
 2 files changed, 36 insertions(+)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +2 -0
index 898de032fed5a..4ebdc00e20fca 100644
@@ -129,4 +129,6 @@ void vfio_container_set_iommu(struct vfio_pci_device *device);
 void vfio_pci_cdev_open(struct vfio_pci_device *device, const char *bdf);
 int __vfio_device_bind_iommufd(int device_fd, int iommufd, const char *vf_token);
 
+void vfio_device_set_vf_token(int fd, const char *vf_token);
+
 #endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H */
tools/testing/selftests/vfio/lib/vfio_pci_device.c +34 -0
index 3123ba591f088..4673b148f8c44 100644
@@ -113,6 +113,40 @@ static void vfio_pci_irq_get(struct vfio_pci_device *device, u32 index,
 	ioctl_assert(device->fd, VFIO_DEVICE_GET_IRQ_INFO, irq_info);
 }
 
+static int vfio_device_feature_ioctl(int fd, u32 flags, void *data,
+				     size_t data_size)
+{
+	u8 buffer[sizeof(struct vfio_device_feature) + data_size] = {};
+	struct vfio_device_feature *feature = (void *)buffer;
+
+	memcpy(feature->data, data, data_size);
+
+	feature->argsz = sizeof(buffer);
+	feature->flags = flags;
+
+	return ioctl(fd, VFIO_DEVICE_FEATURE, feature);
+}
+
+static void vfio_device_feature_set(int fd, u16 feature, void *data, size_t data_size)
+{
+	u32 flags = VFIO_DEVICE_FEATURE_SET | feature;
+	int ret;
+
+	ret = vfio_device_feature_ioctl(fd, flags, data, data_size);
+	VFIO_ASSERT_EQ(ret, 0, "Failed to set feature %u\n", feature);
+}
+
+void vfio_device_set_vf_token(int fd, const char *vf_token)
+{
+	uuid_t token_uuid = {0};
+
+	VFIO_ASSERT_NOT_NULL(vf_token, "vf_token is NULL");
+	VFIO_ASSERT_EQ(uuid_parse(vf_token, token_uuid), 0);
+
+	vfio_device_feature_set(fd, VFIO_DEVICE_FEATURE_PCI_VF_TOKEN,
+				token_uuid, sizeof(uuid_t));
+}
+
 static void vfio_pci_region_get(struct vfio_pci_device *device, int index,
 				struct vfio_region_info *info)
 {
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#9 lore raw

PATCH v7 7/8 vfio: selftests: Add helpers to alloc/free vfio_pci_device

In reply to Raghavendra Rao Ananta

Add a helper, vfio_pci_device_alloc(), to allocate 'struct vfio_pci_device'. The subsequent test patch will utilize this to get the struct with very minimal initialization done. Internally, let vfio_pci_device_init() also make use of this function and later do the full initialization.

Symmetrically, add a free variant, vfio_pci_device_free(), to be used in a similar fashion.

No functional change intended.

---
 .../vfio/lib/include/libvfio/vfio_pci_device.h |  2 ++
 .../selftests/vfio/lib/vfio_pci_device.c       | 18 ++++++++++++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +2 -0
index 4ebdc00e20fca..3eabead717bbd 100644
@@ -38,6 +38,8 @@ struct vfio_pci_device {
 #define dev_info(_dev, _fmt, ...) printf("%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
 #define dev_err(_dev, _fmt, ...) fprintf(stderr, "%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
 
+struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu);
+void vfio_pci_device_free(struct vfio_pci_device *device);
 struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu);
 void vfio_pci_device_cleanup(struct vfio_pci_device *device);
 
tools/testing/selftests/vfio/lib/vfio_pci_device.c +16 -2
index 4673b148f8c44..4ff76970e3791 100644
@@ -419,7 +419,7 @@ static void vfio_pci_iommufd_setup(struct vfio_pci_device *device,
 	vfio_device_attach_iommufd_pt(device->fd, device->iommu->ioas_id);
 }
 
-struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu)
+struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu)
 {
 	struct vfio_pci_device *device;
 
@@ -430,6 +430,20 @@ struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iomm
 	device->iommu = iommu;
 	device->bdf = bdf;
 
+	return device;
+}
+
+void vfio_pci_device_free(struct vfio_pci_device *device)
+{
+	free(device);
+}
+
+struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu)
+{
+	struct vfio_pci_device *device;
+
+	device = vfio_pci_device_alloc(bdf, iommu);
+
 	if (iommu->mode->container_path)
 		vfio_pci_container_setup(device, bdf, NULL);
 	else
@@ -462,5 +476,5 @@ void vfio_pci_device_cleanup(struct vfio_pci_device *device)
 	if (device->group_fd)
 		VFIO_ASSERT_EQ(close(device->group_fd), 0);
 
-	free(device);
+	vfio_pci_device_free(device);
 }
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#10 lore raw

PATCH v7 8/8 vfio: selftests: Add tests to validate SR-IOV UAPI

In reply to Raghavendra Rao Ananta

Add a selftest, vfio_pci_sriov_uapi_test.c, to validate the SR-IOV UAPI, including the following cases, iterating over all the IOMMU modes currently supported:
- Setting correct/incorrect/NULL tokens during device init.
- Close the PF device immediately after setting the token.
- Change/override the PF's token after device init.

---
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 230 ++++++++++++++++++
 2 files changed, 231 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c
tools/testing/selftests/vfio/Makefile +1 -0
index f27ed18070f14..d0c8cea53eb54 100644
@@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test
 TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
+TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +230 -0

230 lines not on this page · raw message

Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer
#11 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 74

Quoted 11 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +11 -0
+/* clang-format off */
+FIXTURE(vfio_pci_sriov_uapi_test) {};
+/* clang-format on */
+
+FIXTURE_SETUP(vfio_pci_sriov_uapi_test)
+{
+}
+
+FIXTURE_TEARDOWN(vfio_pci_sriov_uapi_test)
+{
+}

Please do iommu_init() iommu_cleanup() here to reduce code duplication and as an added bonus you can drop the clang-format comments above.

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 181

Quoted 5 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +5 -0
+static void vf_teardown(void)
+{
+	/* Destroy the VF only when the main/parent process exits. */
+	if (getpid() != main_pid)
+		return;

Is this because the child processes created by test_harness_run() to run the TEST_F()s inherit the atexit() call? If so please clarify that in the comment in more detail.

Quoted 30 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +30 -0
+
+	free(vf_bdf);
+	sysfs_sriov_numvfs_set(pf_bdf, 0);
+}
+
+static void vf_setup(void)
+{
+	char *vf_driver;
+	int nr_vfs;
+
+	nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+	if (nr_vfs <= 0)
+		ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
+
+	nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+	if (nr_vfs != 0)
+		ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
+
+	/* Create only one VF for testing */
+	sysfs_sriov_numvfs_set(pf_bdf, 1);
+
+	/*
+	 * Setup an exit handler to destroy the VF in case of failures
+	 * during further setup at the end of the test run.
+	 */
+	main_pid = getpid();
+	VFIO_ASSERT_EQ(atexit(vf_teardown), 0);
+
+	vf_bdf = sysfs_sriov_vf_bdf_get(pf_bdf, 0);
+

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 221

Quoted 10 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +10 -0
+	/*
+	 * The VF inherits the driver from the PF.
+	 * Ensure this is 'vfio-pci' before proceeding.
+	 */
+	vf_driver = sysfs_driver_get(vf_bdf);
+	VFIO_ASSERT_NE(vf_driver, NULL);
+	VFIO_ASSERT_EQ(strcmp(vf_driver, "vfio-pci"), 0);
+	free(vf_driver);
+
+	printf("Created 1 VF (%s) under the PF: %s\n", vf_bdf, pf_bdf);

Do we actually need atexit()? This code can go into main and we can do the VF cleanup before returning from main after test_harness_run() returns. I don't think you even need to check the VF driver. If the VF is not bound to vfio-pci then that will be caught by the test cases when they call device_init(vf_bdf, ...).

That would make the logic more straightforward and get rid of the main_pid stuff too.

#12 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 18 lines · David Matlack

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 74

Quoted 11 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +11 -0
+/* clang-format off */
+FIXTURE(vfio_pci_sriov_uapi_test) {};
+/* clang-format on */
+
+FIXTURE_SETUP(vfio_pci_sriov_uapi_test)
+{
+}
+
+FIXTURE_TEARDOWN(vfio_pci_sriov_uapi_test)
+{
+}

Please do iommu_init() iommu_cleanup() here to reduce code duplication and as an added bonus you can drop the clang-format comments above.

I think even Vipin suggested this. Last time I tried this, IIRC got a build error that 'variant' was inaccessible. But I guess I just had to re-org my code to make it work.
I'll do this in v8.

Quoted 10 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 181

Quoted 5 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +5 -0
+static void vf_teardown(void)
+{
+     /* Destroy the VF only when the main/parent process exits. */
+     if (getpid() != main_pid)
+             return;

Is this because the child processes created by test_harness_run() to run the TEST_F()s inherit the atexit() call? If so please clarify that in the comment in more detail.

Yes, that's the reason, and I assumed the comment made that clear. I'll re-write it if required.

Quoted 38 lines
Quoted 38 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +38 -0
+
+     free(vf_bdf);
+     sysfs_sriov_numvfs_set(pf_bdf, 0);
+}
+
+static void vf_setup(void)
+{
+     char *vf_driver;
+     int nr_vfs;
+
+     nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+     if (nr_vfs <= 0)
+             ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
+
+     nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+     if (nr_vfs != 0)
+             ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
+
+     /* Create only one VF for testing */
+     sysfs_sriov_numvfs_set(pf_bdf, 1);
+
+     /*
+      * Setup an exit handler to destroy the VF in case of failures
+      * during further setup at the end of the test run.
+      */
+     main_pid = getpid();
+     VFIO_ASSERT_EQ(atexit(vf_teardown), 0);
+
+     vf_bdf = sysfs_sriov_vf_bdf_get(pf_bdf, 0);
+
+     /*
+      * The VF inherits the driver from the PF.
+      * Ensure this is 'vfio-pci' before proceeding.
+      */
+     vf_driver = sysfs_driver_get(vf_bdf);
+     VFIO_ASSERT_NE(vf_driver, NULL);
+     VFIO_ASSERT_EQ(strcmp(vf_driver, "vfio-pci"), 0);
+     free(vf_driver);
Quoted 9 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 221

Quoted 2 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +2 -0
+
+     printf("Created 1 VF (%s) under the PF: %s\n", vf_bdf, pf_bdf);

Do we actually need atexit()? This code can go into main and we can do the VF cleanup before returning from main after test_harness_run() returns. I don't think you even need to check the VF driver. If the VF is not bound to vfio-pci then that will be caught by the test cases when they call device_init(vf_bdf, ...).

Other than the asserts on 'vf_driver', the calls to sysfs_sriov_vf_bdf_get() and sysfs_driver_get() themselves have a potential chance of triggering asserts, although this is unlikely. But the bigger reason is that test_harness_run() unconditionally calls ksft_exit() at the end, forcing it to be the last function called from main(), regardless of the tests' outcome. Hence, the exit handler.

I see atexit() used in some selftests, so I thought it was normal and acceptable.

Thank you.
Raghavendra

maintainer
#13 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 92 lines · Raghavendra Rao Ananta
Quoted 18 lines · David Matlack

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 74

Quoted 11 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +11 -0
+/* clang-format off */
+FIXTURE(vfio_pci_sriov_uapi_test) {};
+/* clang-format on */
+
+FIXTURE_SETUP(vfio_pci_sriov_uapi_test)
+{
+}
+
+FIXTURE_TEARDOWN(vfio_pci_sriov_uapi_test)
+{
+}

Please do iommu_init() iommu_cleanup() here to reduce code duplication and as an added bonus you can drop the clang-format comments above.

I think even Vipin suggested this. Last time I tried this, IIRC got a build error that 'variant' was inaccessible. But I guess I just had to re-org my code to make it work.
I'll do this in v8.

Quoted 10 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 181

Quoted 5 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +5 -0
+static void vf_teardown(void)
+{
+     /* Destroy the VF only when the main/parent process exits. */
+     if (getpid() != main_pid)
+             return;

Is this because the child processes created by test_harness_run() to run the TEST_F()s inherit the atexit() call? If so please clarify that in the comment in more detail.

Yes, that's the reason, and I assumed the comment made that clear. I'll re-write it if required.

Quoted 38 lines
Quoted 38 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +38 -0
+
+     free(vf_bdf);
+     sysfs_sriov_numvfs_set(pf_bdf, 0);
+}
+
+static void vf_setup(void)
+{
+     char *vf_driver;
+     int nr_vfs;
+
+     nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+     if (nr_vfs <= 0)
+             ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
+
+     nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+     if (nr_vfs != 0)
+             ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
+
+     /* Create only one VF for testing */
+     sysfs_sriov_numvfs_set(pf_bdf, 1);
+
+     /*
+      * Setup an exit handler to destroy the VF in case of failures
+      * during further setup at the end of the test run.
+      */
+     main_pid = getpid();
+     VFIO_ASSERT_EQ(atexit(vf_teardown), 0);
+
+     vf_bdf = sysfs_sriov_vf_bdf_get(pf_bdf, 0);
+
+     /*
+      * The VF inherits the driver from the PF.
+      * Ensure this is 'vfio-pci' before proceeding.
+      */
+     vf_driver = sysfs_driver_get(vf_bdf);
+     VFIO_ASSERT_NE(vf_driver, NULL);
+     VFIO_ASSERT_EQ(strcmp(vf_driver, "vfio-pci"), 0);
+     free(vf_driver);
Quoted 9 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 221

Quoted 2 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +2 -0
+
+     printf("Created 1 VF (%s) under the PF: %s\n", vf_bdf, pf_bdf);

Do we actually need atexit()? This code can go into main and we can do the VF cleanup before returning from main after test_harness_run() returns. I don't think you even need to check the VF driver. If the VF is not bound to vfio-pci then that will be caught by the test cases when they call device_init(vf_bdf, ...).

Other than the asserts on 'vf_driver', the calls to sysfs_sriov_vf_bdf_get() and sysfs_driver_get() themselves have a potential chance of triggering asserts, although this is unlikely. But the bigger reason is that test_harness_run() unconditionally calls ksft_exit() at the end, forcing it to be the last function called from main(), regardless of the tests' outcome. Hence, the exit handler.

Quoted 2 lines

I see atexit() used in some selftests, so I thought it was normal and acceptable.

Ah ok if test_harness_run() exits instead of returning then I agree with using atexit(). Thanks for the explanation!

#14 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

In reply to Raghavendra Rao Ananta
Quoted 19 lines · Raghavendra Rao Ananta
Quoted 18 lines · David Matlack

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 74

Quoted 11 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +11 -0
+/* clang-format off */
+FIXTURE(vfio_pci_sriov_uapi_test) {};
+/* clang-format on */
+
+FIXTURE_SETUP(vfio_pci_sriov_uapi_test)
+{
+}
+
+FIXTURE_TEARDOWN(vfio_pci_sriov_uapi_test)
+{
+}

Please do iommu_init() iommu_cleanup() here to reduce code duplication and as an added bonus you can drop the clang-format comments above.

Quoted 4 lines

I think even Vipin suggested this. Last time I tried this, IIRC got a build error that 'variant' was inaccessible. But I guess I just had to re-org my code to make it work.
I'll do this in v8.

Build error is because FIXTURE_VARIANT is coming after FIXTURE_SETUP, if you move it before it should work.

#15 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

In reply to Raghavendra Rao Ananta

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 50

Quoted 12 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +11 -0
+static struct vfio_pci_device *device_init(const char *bdf, struct iommu *iommu,
+					   const char *vf_token, int *out_ret)
+{
+	struct vfio_pci_device *device = vfio_pci_device_alloc(bdf, iommu);
+
+	if (iommu->mode->container_path)
+		*out_ret = container_setup(device, bdf, vf_token);
+	else
+		*out_ret = iommufd_setup(device, bdf, vf_token);
+
+	return device;

I will recommend to return the error code and pass struct vfio_pci_device **out_dev in the arguments. This seems more natural compared to having a last argument as an ret value which is checked in the caller.

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 93

Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +4 -0
+
+/*
+ * PF's token is always set with UUID_1 and VF's token is rotated with
+ * various tokens (including UUID_1 and NULL).

Nit: s/UUID_1/UUID_2

Quoted 18 lines

+ * This asserts if the VF device is successfully created for a match
+ * in the token or actually fails during a mismatch.
+ */

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +15 -0
+#define ASSERT_VF_CREATION(_ret) do {					\
+	if (!variant->vf_token || strcmp(UUID_1, variant->vf_token)) {	\
+		ASSERT_NE((_ret), 0);					\
+	} else {							\
+		ASSERT_EQ((_ret), 0);					\
+	}								\
+} while (0)
+
+/*
+ * Validate if the UAPI handles correctly and incorrectly set token on the VF.
+ */
+TEST_F(vfio_pci_sriov_uapi_test, init_token_match)
+{
+	struct vfio_pci_device *pf;
+	struct vfio_pci_device *vf;

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 121

Quoted 10 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +10 -0
+	struct iommu *iommu;
+	int ret;
+
+	iommu = iommu_init(variant->iommu_mode);
+
+	pf = device_init(pf_bdf, iommu, UUID_1, &ret);
+	ASSERT_EQ(ret, 0);
+
+	vf = device_init(vf_bdf, iommu, variant->vf_token, &ret);
+	ASSERT_VF_CREATION(ret);

ASSERT_VF_CREATION() name is confusing, as it is asserting both success and failure ret value based on the variant passed.

I will recommend to rename it to ASSERT_COND_VF_CREATION(), or, may be create a wrapper function to check if current test is a UUID_1 variant or not, and then directly the assert needed.

Quoted 4 lines

+/*
+ * After setting a token on the PF, validate if the VF can still set the
+ * expected token.
+ */

This comment seems incorrect. VF doesn't set the token, it just provides the token which is set on a PF.

May be a comment can be "After closing the PF, validate VF access still needs the right token.

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 165

Quoted 10 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +10 -0
+TEST_F(vfio_pci_sriov_uapi_test, override_token)
+{
+	struct vfio_pci_device *pf;
+	struct vfio_pci_device *vf;
+	struct iommu *iommu;
+	int ret;
+
+	iommu = iommu_init(variant->iommu_mode);
+
+	pf = device_init(pf_bdf, iommu, UUID_2, &ret);

I am assuming because of this, you cannot move device_init and device_cleanup calls to FIXTURE_SETUP and FIXTURE_TEARDOWN respectively.

Can we just start this test with device_cleanup(), then do init with UUID_2? This will allow to reduce the code in all of the tests by moving things to corresponding setup and teardown functions. WDYT?

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 198

Quoted 13 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +13 -0
+
+static void vf_setup(void)
+{
+	char *vf_driver;
+	int nr_vfs;
+
+	nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+	if (nr_vfs <= 0)
+		ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
+
+	nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+	if (nr_vfs != 0)
+		ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);

Why would we want to skip if VFs are already enabled. Just set it to 0 if it is already there and set it to 1 unconditionally after that.

#16 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 11 lines · Vipin Sharma
Quoted 9 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +8 -0
+static struct vfio_pci_device *device_init(const char *bdf, struct iommu *iommu,
+                                        const char *vf_token, int *out_ret)
+{
+     struct vfio_pci_device *device = vfio_pci_device_alloc(bdf, iommu);
+
+     if (iommu->mode->container_path)
+             *out_ret = container_setup(device, bdf, vf_token);
+     else
Quoted 9 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 50

Quoted 3 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +3 -0
+             *out_ret = iommufd_setup(device, bdf, vf_token);
+
+     return device;

I will recommend to return the error code and pass struct vfio_pci_device **out_dev in the arguments. This seems more natural compared to having a last argument as an ret value which is checked in the caller.

The idea was to keep it in line with vfio_pci_device_init(), so I returned 'struct vfio_pci_device *'. But, it's not a big deal. I've changed it to return an 'int' if that's what we prefer.

Quoted 7 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 93

Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +4 -0
+
+/*
+ * PF's token is always set with UUID_1 and VF's token is rotated with
+ * various tokens (including UUID_1 and NULL).

Nit: s/UUID_1/UUID_2

We rotate with UUID_1, UUID_2, and NULL. I've adjusted the comment to incude all three.

Quoted 61 lines
Quoted 18 lines

+ * This asserts if the VF device is successfully created for a match
+ * in the token or actually fails during a mismatch.
+ */

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +15 -0
+#define ASSERT_VF_CREATION(_ret) do {                                        \
+     if (!variant->vf_token || strcmp(UUID_1, variant->vf_token)) {  \
+             ASSERT_NE((_ret), 0);                                   \
+     } else {                                                        \
+             ASSERT_EQ((_ret), 0);                                   \
+     }                                                               \
+} while (0)
+
+/*
+ * Validate if the UAPI handles correctly and incorrectly set token on the VF.
+ */
+TEST_F(vfio_pci_sriov_uapi_test, init_token_match)
+{
+     struct vfio_pci_device *pf;
+     struct vfio_pci_device *vf;

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 121

Quoted 10 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +10 -0
+     struct iommu *iommu;
+     int ret;
+
+     iommu = iommu_init(variant->iommu_mode);
+
+     pf = device_init(pf_bdf, iommu, UUID_1, &ret);
+     ASSERT_EQ(ret, 0);
+
+     vf = device_init(vf_bdf, iommu, variant->vf_token, &ret);
+     ASSERT_VF_CREATION(ret);

ASSERT_VF_CREATION() name is confusing, as it is asserting both success and failure ret value based on the variant passed.

I will recommend to rename it to ASSERT_COND_VF_CREATION(), or, may be create a wrapper function to check if current test is a UUID_1 variant or not, and then directly the assert needed.

Quoted 4 lines

+/*
+ * After setting a token on the PF, validate if the VF can still set the
+ * expected token.
+ */

This comment seems incorrect. VF doesn't set the token, it just provides the token which is set on a PF.

May be a comment can be "After closing the PF, validate VF access still needs the right token.

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 165

Quoted 10 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +10 -0
+TEST_F(vfio_pci_sriov_uapi_test, override_token)
+{
+     struct vfio_pci_device *pf;
+     struct vfio_pci_device *vf;
+     struct iommu *iommu;
+     int ret;
+
+     iommu = iommu_init(variant->iommu_mode);
+
+     pf = device_init(pf_bdf, iommu, UUID_2, &ret);

I am assuming because of this, you cannot move device_init and device_cleanup calls to FIXTURE_SETUP and FIXTURE_TEARDOWN respectively.

Quoted 4 lines

Can we just start this test with device_cleanup(), then do init with UUID_2? This will allow to reduce the code in all of the tests by moving things to corresponding setup and teardown functions. WDYT?

Yes it was intentionally kept this way for the 'override_token' test. Also see the previous 'pf_early_close' test that performs a premature cleanup of the PF. To accommodate these (and any future TEST_F()s we may want to add) based on your suggestion, we'd have to create special/conditional statements across the tests and I'd like to avoid that if possible. The current setup clearly shows what each test does/requires.

Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +9 -0
+
+static void vf_setup(void)
+{
+     char *vf_driver;
+     int nr_vfs;
+
+     nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+     if (nr_vfs <= 0)
+             ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
Quoted 9 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 198

Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +4 -0
+
+     nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+     if (nr_vfs != 0)
+             ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);

Why would we want to skip if VFs are already enabled. Just set it to 0 if it is already there and set it to 1 unconditionally after that.

This actually goes back to a previous discussion with David where we agreed to avoid such situations. For instance, what if the device is already in use elsewhere.

Other than these, I've addressed other comments.

Thank you.
Raghavendra

#17 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 21 lines · Raghavendra Rao Ananta
Quoted 16 lines · Vipin Sharma

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 165

Quoted 10 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +10 -0
+TEST_F(vfio_pci_sriov_uapi_test, override_token)
+{
+     struct vfio_pci_device *pf;
+     struct vfio_pci_device *vf;
+     struct iommu *iommu;
+     int ret;
+
+     iommu = iommu_init(variant->iommu_mode);
+
+     pf = device_init(pf_bdf, iommu, UUID_2, &ret);

I am assuming because of this, you cannot move device_init and device_cleanup calls to FIXTURE_SETUP and FIXTURE_TEARDOWN respectively.

Quoted 4 lines

Can we just start this test with device_cleanup(), then do init with UUID_2? This will allow to reduce the code in all of the tests by moving things to corresponding setup and teardown functions. WDYT?

Quoted 8 lines

Yes it was intentionally kept this way for the 'override_token' test. Also see the previous 'pf_early_close' test that performs a premature cleanup of the PF. To accommodate these (and any future TEST_F()s we may want to add) based on your suggestion, we'd have to create special/conditional statements across the tests and I'd like to avoid that if possible. The current setup clearly shows what each test does/requires.

I think if you make device_cleanup() handle already cleaned up device as no-op this should avoid any special handling.

iommu_init() and device_init() with UUID_1 will be used by all three tests in FIXTURE_SETUP(). Their corresponding cleanup will be in FIXTURE_TEARDOWN().

This way only pf_early_close() will have extra call of device_cleanup(pf).

Can you tell more about special/conditional statements which each test will have to write? Currently, we don't have any tests like that, we can revisit it when we see that kind of issue.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +9 -0
+
+static void vf_setup(void)
+{
+     char *vf_driver;
+     int nr_vfs;
+
+     nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
+     if (nr_vfs <= 0)
+             ksft_exit_skip("SR-IOV may not be supported by the PF: %s\n", pf_bdf);
Quoted 9 lines

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 198

Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +4 -0
+
+     nr_vfs = sysfs_sriov_numvfs_get(pf_bdf);
+     if (nr_vfs != 0)
+             ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);

Why would we want to skip if VFs are already enabled. Just set it to 0 if it is already there and set it to 1 unconditionally after that.

Quoted 4 lines

This actually goes back to a previous discussion with David where we agreed to avoid such situations. For instance, what if the device is already in use elsewhere.

I think this is what you are referring to:
https://lore.kernel.org/all/[email protected]
--------

	> > > +     snprintf(path, PATH_MAX, "%s/%s/sriov_numvfs", PCI_SYSFS_PATH, pf_dev_bdf);
	> > > +     ASSERT_GT(fd = open(path, O_RDWR), 0);
	> > > +     ASSERT_GT(read(fd, buf, ARRAY_SIZE(buf)), 0);
	> > > +     nr_vfs = strtoul(buf, NULL, 0);
	> > > +     if (nr_vfs == 0)

> >
> > If VFs are already enabled, shouldn't the test fail or skip? > >
> My idea was to simply "steal" the device that was already created and > use it. Do we want to skip it, as you suggested?

If a VF already exists it might be bound to a different driver, and may be in use by something else. I think the only safe thing to do is to bail if a VF already exists. If the test creates the VF, then it knows that it owns it.
--------

If we are running a test harness, and one test failed without clearing up VFs then all the following tests will pay penalty this way. As this is a selftest code, device assigned to it is for testing not for some production work at the same time, I am not able to see why it will be a unsafe.

My recommendation will be to reset and not skip, I don't think there are any practical risks here. I will leave it to you to make a final decision on this.

#18 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 32 lines · Vipin Sharma
Quoted 21 lines · Raghavendra Rao Ananta
Quoted 16 lines · Vipin Sharma

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 165

Quoted 10 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +10 -0
+TEST_F(vfio_pci_sriov_uapi_test, override_token)
+{
+     struct vfio_pci_device *pf;
+     struct vfio_pci_device *vf;
+     struct iommu *iommu;
+     int ret;
+
+     iommu = iommu_init(variant->iommu_mode);
+
+     pf = device_init(pf_bdf, iommu, UUID_2, &ret);

I am assuming because of this, you cannot move device_init and device_cleanup calls to FIXTURE_SETUP and FIXTURE_TEARDOWN respectively.

Quoted 4 lines

Can we just start this test with device_cleanup(), then do init with UUID_2? This will allow to reduce the code in all of the tests by moving things to corresponding setup and teardown functions. WDYT?

Quoted 8 lines

Yes it was intentionally kept this way for the 'override_token' test. Also see the previous 'pf_early_close' test that performs a premature cleanup of the PF. To accommodate these (and any future TEST_F()s we may want to add) based on your suggestion, we'd have to create special/conditional statements across the tests and I'd like to avoid that if possible. The current setup clearly shows what each test does/requires.

Quoted 3 lines

I think if you make device_cleanup() handle already cleaned up device as no-op this should avoid any special handling.

Having a conditional check to avoid doing the cleanup only to satisfy the 'pf_early_close' test was the special handling that I was referring to.
No other test requires this.

Quoted 4 lines

iommu_init() and device_init() with UUID_1 will be used by all three tests in FIXTURE_SETUP(). Their corresponding cleanup will be in FIXTURE_TEARDOWN().

'override_token' first initializes the PF with UUID_2 as the token and then overrides to UUID_1.
To easily satisfy all the tests, ASSERT_COND_VF_CREATION() was built on the assumption that when the VF is verified, the pf token will always be UUID_1.

Anyway, I tried moving the PF setup into FIXTURE_SETUP() and the PF/VF cleanups into FIXTURE_CLEANUP(), and I adjusted ASSERT_COND_VF_CREATION() to accommodate any PF token for comparison. I'll push these changes to v8 soon. Let me know what you think.

Quoted 9 lines

If we are running a test harness, and one test failed without clearing up VFs then all the following tests will pay penalty this way. As this is a selftest code, device assigned to it is for testing not for some production work at the same time, I am not able to see why it will be a unsafe.

My recommendation will be to reset and not skip, I don't think there are any practical risks here. I will leave it to you to make a final decision on this.

Not just selftests, it's possible that the VF devices were created for use by actual vfio users. Hence, I'd like to avoid that assumption and skip touching the devices altogether.

Thank you.
Raghavendra

#19 lore raw

Re: [PATCH v7 8/8] vfio: selftests: Add tests to validate SR-IOV UAPI

Quoted 53 lines · Raghavendra Rao Ananta
Quoted 32 lines · Vipin Sharma
Quoted 21 lines · Raghavendra Rao Ananta
Quoted 16 lines · Vipin Sharma

tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c line 165

Quoted 10 lines · Raghavendra Rao Ananta
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c new file +10 -0
+TEST_F(vfio_pci_sriov_uapi_test, override_token)
+{
+     struct vfio_pci_device *pf;
+     struct vfio_pci_device *vf;
+     struct iommu *iommu;
+     int ret;
+
+     iommu = iommu_init(variant->iommu_mode);
+
+     pf = device_init(pf_bdf, iommu, UUID_2, &ret);

I am assuming because of this, you cannot move device_init and device_cleanup calls to FIXTURE_SETUP and FIXTURE_TEARDOWN respectively.

Quoted 4 lines

Can we just start this test with device_cleanup(), then do init with UUID_2? This will allow to reduce the code in all of the tests by moving things to corresponding setup and teardown functions. WDYT?

Quoted 8 lines

Yes it was intentionally kept this way for the 'override_token' test. Also see the previous 'pf_early_close' test that performs a premature cleanup of the PF. To accommodate these (and any future TEST_F()s we may want to add) based on your suggestion, we'd have to create special/conditional statements across the tests and I'd like to avoid that if possible. The current setup clearly shows what each test does/requires.

Quoted 3 lines

I think if you make device_cleanup() handle already cleaned up device as no-op this should avoid any special handling.

Having a conditional check to avoid doing the cleanup only to satisfy the 'pf_early_close' test was the special handling that I was referring to.
No other test requires this.

Quoted 4 lines

iommu_init() and device_init() with UUID_1 will be used by all three tests in FIXTURE_SETUP(). Their corresponding cleanup will be in FIXTURE_TEARDOWN().

'override_token' first initializes the PF with UUID_2 as the token and then overrides to UUID_1.
To easily satisfy all the tests, ASSERT_COND_VF_CREATION() was built on the assumption that when the VF is verified, the pf token will always be UUID_1.

Quoted 5 lines

Anyway, I tried moving the PF setup into FIXTURE_SETUP() and the PF/VF cleanups into FIXTURE_CLEANUP(), and I adjusted ASSERT_COND_VF_CREATION() to accommodate any PF token for comparison. I'll push these changes to v8 soon. Let me know what you think.

I am assuming you meant for both iommu and PF setup/cleanup in the FIXTURE macro. Sounds good, looking forward to v8.

Thanks
Vipin