Skip to contentSkip to conversation

Issues with Multi-GPU Device Isolation for Fault Injection Tests

PATCH Peter Senna Tschudin 2 participants 2 messages Nov 22, 2025 – Nov 25, 2025

lore
Share & tools
lore
#0 lore raw

Hello,

I've been working on the multi-GPU device isolation feature for fault injection tests that was requested, but I've run into some blockers...

As discussed, fault injection tests operate at the driver level and affect all GPUs bound to the xe driver. The goal was to automatically unbind non-selected devices when running tests with --device on multi-GPU systems, then rebind them after the tests complete.

The approach:
- Scans /sys/bus/pci/drivers/xe/ to detect all bound devices
- Detects when --device flag is used via igt_device_filter_count()
- Unbinds non-selected devices before running tests
- Attempts to restore the system state after tests complete

The problem is that rebinding devices does not work reliably. I've encountered two major issues:
1. Device Rebinding Issues:
- Rebinding either works but returns an error code from sysfs, or
- It completely locks the machine, usually on the second rebind attempt
- This makes the cleanup unreliable and potentially dangerous

2. Module Reload Alternative:
- I tried replacing rebind with a full module reload (using
igt_xe_driver_unload() and load_and_check_xe(NULL))
- This approach also failed and did not provide a reliable solution as
the driver fails to unload.

I am under the impression that unbinding a GPU from the driver leaves the driver in an unclean state that is preventing successful module reload and successful GPU bind. As I used the standard igt library functions that are used elsewhere in the code, I am uncertain if there is a way out here.

Given these issues, I don't believe the current approach of runtime device management is viable for production use. Please see the attached code for details on what I've tried.

Ideas?

Thank you,

Peter

From 8f90be5f622cebc294dab14111eea4ac9e4ee3d7 Mon Sep 17 00:00:00 2001
From: Peter Senna Tschudin <[email protected]>
Date: Wed, 19 Nov 2025 22:24:48 +0100
Subject: [PATCH i-g-t] devc

---
 tests/intel/xe_fault_injection.c | 276 +++++++++++++++++++++++++++++++
 1 file changed, 276 insertions(+)
(quoted diff) +0 -0

b/tests/intel/xe_fault_injection.c
index 4b4881634..38676bbbb 100644

tests/intel/xe_fault_injection.c +98 -0

110 lines not on this page · raw message

xe_device_context *ctx, int fd)
+{
+ char selected_pci_slot[NAME_MAX];
+ int i;
+
+ igt_device_get_pci_slot_name(fd, selected_pci_slot);
+ igt_debug("Device selected via --device filter: %s\n", selected_pci_slot);
+
+ for (i = 0; i < ctx->device_count; i++) {
+ if (strcmp(ctx->devices[i].pci_slot, selected_pci_slot) == 0) {
+ ctx->devices[i].is_selected = true;
+ ctx->selected_device = &ctx->devices[i];
+ return &ctx->devices[i];
+ }
+ }
+
+ igt_warn("Selected device %s not found in scanned devices\n",
+ selected_pci_slot);
+ return NULL;
+}
+
+/**
+ * unbind_other_devices - Unbind all non-selected Xe devices
+ * @ctx: Device context
+ *
+ * Unbinds all Xe devices except the selected one from the xe driver.
+ * Marks devices as needing rebind for cleanup.
+ *
+ * Returns: 0 on success, -1 on error
+ */
+static int unbind_other_devices(struct xe_device_context *ctx) +{
+ int i;
+ int ret = 0;
+
+ if (!ctx->multi_gpu_system)
+ return 0;
+
+ if (!ctx->selected_device) {
+ igt_warn("No selected device, cannot unbind others\n");
+ return -1;
+ }
+
+ igt_info("Unbinding non-selected Xe devices...\n");
+
+ for (i = 0; i < ctx->device_count; i++) {
+ if (ctx->devices[i].is_selected)
+ continue;
+
+ igt_info("Unbinding device: %s\n", ctx->devices[i].pci_slot);
+
+ if (igt_kmod_unbind("xe", ctx->devices[i].pci_slot) != 0) {
+ igt_warn("Failed to unbind device %s\n",
+ ctx->devices[i].pci_slot);
+ ret = -1;
+ continue;
+ }
+
+ ctx->devices[i].needs_rebind = true;
+ }
+
+ return ret;
+}
+
+/**
+ * cleanup_device_context - Cleanup device context after tests
+ * @ctx: Device context
+ *
+ * Reloads the xe module if any devices were unbound during test setup.
+ * This is safer than rebinding individual devices as it avoids issues
+ * with module dependencies (e.g., audio modules).
+ */
+static void cleanup_device_context(struct xe_device_context *ctx) +{
+ bool any_unbound = false;
+ int i;
+
+ if (!ctx->multi_gpu_system)
+ return;
+
+ /* Check if any devices were unbound */
+ for (i = 0; i < ctx->device_count; i++) {
+ if (ctx->devices[i].needs_rebind) {
+ if (igt_kmod_bind("xe", ctx->devices[i].pci_slot) != 0) {
+ igt_warn("Failed to rebind device %s\n",
+ ctx->devices[i].pci_slot);
+ }
+ any_unbound = true;
+ }
+ }
+
+ if (!any_unbound)
+ return;
+
+ igt_info("Reloading xe module to restore all devices...\n");
+
+ /* Unload and reload the xe module */
+ if (igt_xe_driver_unload() != 0) {
+ igt_warn("Failed to unload xe driver\n");
+ return;
+ }
+
+ if (igt_xe_driver_load(NULL) != 0) {
+ igt_warn("Failed to reload xe driver\n");
+ return;
+ }
+
+ igt_info("Xe module reloaded successfully\n");
+}
+
+/**
+ * validate_device_context - Validate device setup and apply policy
+ * @ctx: Device context
+ * @fd: Open file descriptor to device
+ *
+ * Validates the device setup according to the test's requirements:
+ * - Single GPU: Tests run normally
+ * - Multiple GPUs with --device: Unbind non-selected devices
+ * - Multiple GPUs without --device: Skip all tests with warning
+ *
+ * Returns: 0 if tests should proceed, -1 if tests should be skipped
+ */
+static int validate_device_context(struct xe_device_context *ctx, int fd) +{
+ bool device_filter_used;
+
+ if (scan_xe_devices(ctx) != 0) {
+ igt_warn("Failed to scan Xe devices\n");
+ return -1;
+ }
+
+ /* Check if --device filter was explicitly used */
+ device_filter_used = (igt_device_filter_count() > 0);
+
+ igt_debug("Xe devices found: %d, --device filter used: %s\n",
+ ctx->device_count, device_filter_used ? "yes" : "no");
+
+ if (!ctx->multi_gpu_system) {
+ igt_info("Single Xe device detected, proceeding normally\n");
+ return 0;
+ }
+
+ /* Multiple GPUs detected */
+ igt_info("Multiple Xe devices detected (%d devices)\n",
ctx->device_count);
+
+ if (!device_filter_used) {
+ igt_warn("Multiple Xe devices bound to driver, but no device selected
with --device\n");
+ igt_warn("Fault injection affects all devices bound to the driver.\n");
+ igt_warn("Please use --device to select exactly one GPU.\n");
+ return -1;
+ }
+
+ /* Device filter was used, find which device was selected */
+ if (!get_selected_device(ctx, fd)) {
+ igt_warn("Could not determine selected device from --device filter\n");
+ return -1;
+ }
+
+ igt_info("Device %s selected, unbinding other devices\n",
+ ctx->selected_device->pci_slot);
+
+ if (unbind_other_devices(ctx) != 0) {
+ igt_warn("Failed to unbind other devices\n");
+ return -1;
+ }
+
+ return 0;
+}
+
static int fail_function_open(void)
{
int debugfs_fail_function_dir_fd;
@@ -561,6 +827,7 @@ igt_main_args("I:", NULL, help_str, opt_handler, NULL)
static uint32_t devid;
char pci_slot[NAME_MAX];
bool is_vf_device;
+ struct xe_device_context device_ctx;
const struct section {
const char *name;
unsigned int flags;
@@ -635,6 +902,12 @@ igt_main_args("I:", NULL, help_str, opt_handler, NULL)
setup_injection_fault(&default_fault_params); igt_install_exit_handler(cleanup_injection_fault); is_vf_device = intel_is_vf_device(fd);
+
+ /* Validate device setup for multi-GPU systems */
+ if (!validate_device_context(&device_ctx, fd) == 0) {
+ igt_skip("Multiple Xe devices bound to driver, but no device
selected with --device\n"
+ "Please use --device to select exactly one GPU.\n");
+ }
}

for (const struct section *s = vm_create_fail_functions; s->name; s++)
@@ -686,6 +959,9 @@ igt_main_args("I:", NULL, help_str, opt_handler, NULL)
}

igt_fixture {
+ /* Cleanup: reload xe module if any devices were unbound */
+ cleanup_device_context(&device_ctx);
+
close(sysfs);
drm_close_driver(fd);
injection_list_clear();

Signature · 1 line
2.43.0
#1 lore raw

✗ Fi.CI.BUILD: failure for Issues with Multi-GPU Device Isolation for Fault Injection Tests

== Series Details ==

== Series Details ==

Series: Issues with Multi-GPU Device Isolation for Fault Injection Tests
URL : https://patchwork.freedesktop.org/series/157939/
State : failure

== Summary ==

Patch is empty.
When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To record the empty patch as an empty commit, run "git am --allow-empty". To restore the original branch and stop patching, run "git am --abort".