Skip to contentSkip to conversation

Xe driver asynchronous notification mechanism

RFC PATCH In review Thomas Hellström 1 participant 4 messages Mar 10 – Mar 18

Subsystems (3)
lore
Share & tools
lore

Thomas Hellström

author maintainer
#0 lore raw

cover Xe driver asynchronous notification mechanism

There is a need to inform user-space clients when a rebind worker has ran out of memory so that it can react, adjust its working-set and restart the job. This patch series aims to start a discussion about the bet way to accomplish this.

The series builds on the core "general notification mechanism" or "watch_queue", and attaches a watch queue to each xe drm file.

The watch_queue is extremely flexible and allows filtering out events of interest at the kernel level. There can be multiple listeners.

Another approach would be to use drm events, but then there could only be one listener per open file and no filtering. Otoh drm events would probably have the shortest delivery latency.

Finally there is eventfd (man 2 eventfd) but doesn't appear to allow carrying metadata.

Any feedback appreciated, also on method preference.

Patch 1 extende the watch_queue interface slightly, Patch 2 implements delivery of a VM rebind worker error.

Note this is to be regarded as a POC at this time. No need for a detailed review.

A user-space igt user is posted as an RFC here:

https://patchwork.freedesktop.org/series/162576/

RESEND:
- Widening the audience to solicit watch_queue feedback.

Thomas Hellström (2):
watch_queue: Add a DRM_XE_NOTIFY watch type and export init_watch() drm/xe: Add watch_queue-based device event notification

 drivers/gpu/drm/xe/Kconfig           |   1 +
 drivers/gpu/drm/xe/Makefile          |   1 +
 drivers/gpu/drm/xe/xe_device.c       |   7 ++
 drivers/gpu/drm/xe/xe_device_types.h |   6 ++
 drivers/gpu/drm/xe/xe_vm.c           |   7 +-
 drivers/gpu/drm/xe/xe_vm_types.h     |   2 +
 drivers/gpu/drm/xe/xe_watch_queue.c  | 107 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_watch_queue.h  |  20 +++++
 include/uapi/drm/xe_drm.h            |  46 ++++++++++++
 include/uapi/drm/xe_drm_events.h     |  56 ++++++++++++++
 include/uapi/linux/watch_queue.h     |   3 +-
 kernel/watch_queue.c                 |  13 +++-
 12 files changed, 263 insertions(+), 6 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_watch_queue.c
 create mode 100644 drivers/gpu/drm/xe/xe_watch_queue.h
 create mode 100644 include/uapi/drm/xe_drm_events.h
Signature · 1 line
2.53.0
author maintainer
#1 lore raw

RFC PATCH 1/2 watch_queue: Add a DRM_XE_NOTIFY watch type and export init_watch()

Add a DRM_XE_NOTIFY watch type for asynchronous error notifications from the DRM_XE kernel module.

Also export the init_watch() function for use from kernel drivers. Use EXPORT_SYMBOL() to align with other exports from the same file.

Assisted-by GitHub Copilot:claude-sonnet-4.6
---
 include/uapi/linux/watch_queue.h |  3 ++-
 kernel/watch_queue.c             | 13 ++++++++++---
 2 files changed, 12 insertions(+), 4 deletions(-)
include/uapi/linux/watch_queue.h +2 -1
index c3d8320b5d3a..c800c153989d 100644
@@ -14,7 +14,8 @@
 enum watch_notification_type {
 	WATCH_TYPE_META		= 0,	/* Special record */
 	WATCH_TYPE_KEY_NOTIFY	= 1,	/* Key change event notification */
-	WATCH_TYPE__NR		= 2
+	WATCH_TYPE_DRM_XE_NOTIFY	= 2,	/* DRM device event notification */
+	WATCH_TYPE__NR		= 3
 };
 
 enum watch_meta_notification_subtype {
kernel/watch_queue.c +10 -3
index 538520861e8b..701b5c388808 100644
@@ -445,11 +445,17 @@ static void put_watch(struct watch *watch)
 }
 
 /**
- * init_watch - Initialise a watch
+ * init_watch() - Initialise a watch subscription
  * @watch: The watch to initialise.
- * @wqueue: The queue to assign.
+ * @wqueue: The watch queue (notification pipe) to associate with the watch.
  *
- * Initialise a watch and set the watch queue.
+ * Initialise a newly allocated watch object and associate it with @wqueue.
+ * The caller must subsequently set @watch->id and @watch->info_id before
+ * calling add_watch_to_object() to subscribe the watch to a notification
+ * source.
+ *
+ * The watch queue reference is held internally; call put_watch_queue() if
+ * the watch is not successfully passed to add_watch_to_object().
  */
 void init_watch(struct watch *watch, struct watch_queue *wqueue)
 {
@@ -458,6 +464,7 @@ void init_watch(struct watch *watch, struct watch_queue *wqueue)
 	INIT_HLIST_NODE(&watch->queue_node);
 	rcu_assign_pointer(watch->queue, wqueue);
 }
+EXPORT_SYMBOL(init_watch);
 
 static int add_one_watch(struct watch *watch, struct watch_list *wlist, struct watch_queue *wqueue)
 {
Signature · 1 line
2.53.0
author maintainer
#2 lore raw

Watch queue maintainer? WAS:[RFC PATCH RESEND 1/2] watch_queue: Add a DRM_XE_NOTIFY watch type and export init_watch()

Hi Christian,

I saw you relatively recently commited some patches to the watch_queue implementation. We're drafting the below change that probably is going to use it but at this time it's only RFC.

Do you think the below would be acceptable? If so could you help acking for merge once the series as a whole is ready to go?

Any feedback appreciated,

Thanks,
Thomas

Quoted 70 lines · Thomas Hellström

Add a DRM_XE_NOTIFY watch type for asynchronous error notifications from the DRM_XE kernel module.

Also export the init_watch() function for use from kernel drivers. Use EXPORT_SYMBOL() to align with other exports from the same file.

Assisted-by GitHub Copilot:claude-sonnet-4.6

---
 include/uapi/linux/watch_queue.h |  3 ++- kernel/watch_queue.c             | 13 ++++++++++---

 2 files changed, 12 insertions(+), 4 deletions(-)
include/uapi/linux/watch_queue.h +0 -0
index c3d8320b5d3a..c800c153989d 100644
@@ -14,7 +14,8 @@
  enum watch_notification_type {

  WATCH_TYPE_META = 0, /* Special record */
  WATCH_TYPE_KEY_NOTIFY = 1, /* Key change event
notification */

include/uapi/linux/watch_queue.h +1 -1
-	WATCH_TYPE__NR		= 2
+	WATCH_TYPE_DRM_XE_NOTIFY	= 2,	/* DRM device event

notification */

include/uapi/linux/watch_queue.h +1 -0
+	WATCH_TYPE__NR		= 3

 };

 enum watch_meta_notification_subtype {

kernel/watch_queue.c +0 -0
index 538520861e8b..701b5c388808 100644
@@ -445,11 +445,17 @@ static void put_watch(struct watch *watch)
  }


 /**
- * init_watch - Initialise a watch
+ * init_watch() - Initialise a watch subscription
  * @watch: The watch to initialise.
- * @wqueue: The queue to assign.
+ * @wqueue: The watch queue (notification pipe) to associate with the watch.
  *
- * Initialise a watch and set the watch queue.
+ * Initialise a newly allocated watch object and associate it with @wqueue.
+ * The caller must subsequently set @watch->id and @watch->info_id before
+ * calling add_watch_to_object() to subscribe the watch to a notification
+ * source.
+ *
+ * The watch queue reference is held internally; call put_watch_queue() if
+ * the watch is not successfully passed to add_watch_to_object(). */
 void init_watch(struct watch *watch, struct watch_queue *wqueue) {

kernel/watch_queue.c +1 -0
@@ -458,6 +464,7 @@ void init_watch(struct watch *watch, struct
 watch_queue *wqueue)
  	INIT_HLIST_NODE(&watch->queue_node);
  	rcu_assign_pointer(watch->queue, wqueue);
  }
+EXPORT_SYMBOL(init_watch);


 static int add_one_watch(struct watch *watch, struct watch_list *wlist, struct watch_queue *wqueue)
 {

author maintainer
#3 lore raw

RFC PATCH 2/2 drm/xe: Add watch_queue-based device event notification

In reply to Thomas Hellström

Add a watch_queue notification channel tied to struct xe_vm so that userspace can subscribe to asynchronous GPU device events via the general kernel notification mechanism.

Introduce DRM_IOCTL_XE_WATCH_QUEUE to let userspace subscribe a notification pipe (opened with pipe2(O_NOTIFICATION_PIPE)) to the device event stream. Embed the watch_id field (0-255) in the WATCH_INFO_ID field of every notification, allowing multiple watches to share a single pipe and be told apart by the reader.

Deliver notifications as struct drm_xe_watch_notification records, with type always set to WATCH_TYPE_DRM_XE_NOTIFY and subtype drawn from enum drm_xe_watch_event. Define DRM_XE_WATCH_EVENT_DEVICE_RESET as the first event, to be posted by the GPU reset path to inform userspace that in-flight work has been lost. Expose xe_watch_queue_post_event() as the in-kernel posting API.

Add event definitions in a separate uapi header, <drm/xe_drm_events.h>. The main reason is that the header needs to include <linux/watch_queue.h> which in turn includes <linux/fcntl.h> which may conflict with the system <fcntl.h>. Hence user-space must pay special attention when including this file.

Assisted-by N/A:claude-sonnet-4.6 Tool1 Tool2
---
 drivers/gpu/drm/xe/Kconfig           |   1 +
 drivers/gpu/drm/xe/Makefile          |   1 +
 drivers/gpu/drm/xe/xe_device.c       |   7 ++
 drivers/gpu/drm/xe/xe_device_types.h |   6 ++
 drivers/gpu/drm/xe/xe_vm.c           |   7 +-
 drivers/gpu/drm/xe/xe_vm_types.h     |   2 +
 drivers/gpu/drm/xe/xe_watch_queue.c  | 107 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_watch_queue.h  |  20 +++++
 include/uapi/drm/xe_drm.h            |  46 ++++++++++++
 include/uapi/drm/xe_drm_events.h     |  56 ++++++++++++++
 10 files changed, 251 insertions(+), 2 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_watch_queue.c
 create mode 100644 drivers/gpu/drm/xe/xe_watch_queue.h
 create mode 100644 include/uapi/drm/xe_drm_events.h
drivers/gpu/drm/xe/Kconfig +1 -0
index 4d7dcaff2b91..dbdc2fb49c53 100644
@@ -25,6 +25,7 @@ config DRM_XE
 	select DRM_MIPI_DSI
 	select RELAY
 	select IRQ_WORK
+	select WATCH_QUEUE
 	# xe depends on ACPI_VIDEO when ACPI is enabled
 	# but for select to work, need to select ACPI_VIDEO's dependencies, ick
 	select BACKLIGHT_CLASS_DEVICE if ACPI
drivers/gpu/drm/xe/Makefile +1 -0
index ff778fb2d4ff..1129583865ad 100644
@@ -144,6 +144,7 @@ xe-y += xe_bb.o \
 	xe_vsec.o \
 	xe_wa.o \
 	xe_wait_user_fence.o \
+	xe_watch_queue.o \
 	xe_wopcm.o
 
 xe-$(CONFIG_I2C)	+= xe_i2c.o
drivers/gpu/drm/xe/xe_device.c +7 -0
index 3462645ca13c..89bc221546ce 100644
@@ -9,6 +9,7 @@
 #include <linux/delay.h>
 #include <linux/fault-inject.h>
 #include <linux/units.h>
+#include <linux/watch_queue.h>
 
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_client.h>
@@ -75,6 +76,7 @@
 #include "xe_vsec.h"
 #include "xe_wait_user_fence.h"
 #include "xe_wa.h"
+#include "xe_watch_queue.h"
 
 #include <generated/xe_device_wa_oob.h>
 #include <generated/xe_wa_oob.h>
@@ -110,6 +112,8 @@ static int xe_file_open(struct drm_device *dev, struct drm_file *file)
 	file->driver_priv = xef;
 	kref_init(&xef->refcount);
 
+	init_watch_list(&xef->watch_list, NULL);
+
 	task = get_pid_task(rcu_access_pointer(file->pid), PIDTYPE_PID);
 	if (task) {
 		xef->process_name = kstrdup(task->comm, GFP_KERNEL);
@@ -124,6 +128,8 @@ static void xe_file_destroy(struct kref *ref)
 {
 	struct xe_file *xef = container_of(ref, struct xe_file, refcount);
 
+	remove_watch_from_object(&xef->watch_list, NULL, 0, true);
+
 	xa_destroy(&xef->exec_queue.xa);
 	mutex_destroy(&xef->exec_queue.lock);
 	xa_destroy(&xef->vm.xa);
@@ -211,6 +217,7 @@ static const struct drm_ioctl_desc xe_ioctls[] = {
 			  DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(XE_EXEC_QUEUE_SET_PROPERTY, xe_exec_queue_set_property_ioctl,
 			  DRM_RENDER_ALLOW),
+	DRM_IOCTL_DEF_DRV(XE_WATCH_QUEUE, xe_watch_queue_ioctl, DRM_RENDER_ALLOW),
 };
 
 static long xe_drm_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
drivers/gpu/drm/xe/xe_device_types.h +6 -0
index caa8f34a6744..a42e6125c069 100644
@@ -11,6 +11,7 @@
 #include <drm/drm_device.h>
 #include <drm/drm_file.h>
 #include <drm/ttm/ttm_device.h>
+#include <linux/watch_queue.h>
 
 #include "xe_devcoredump_types.h"
 #include "xe_heci_gsc.h"
@@ -629,6 +630,11 @@ struct xe_file {
 
 	/** @refcount: ref count of this xe file */
 	struct kref refcount;
+
+#ifdef CONFIG_WATCH_QUEUE
+	/** @watch_list: per-file notification source for device events */
+	struct watch_list watch_list;
+#endif
 };
 
 #endif
drivers/gpu/drm/xe/xe_vm.c +5 -2
index 548b0769b3ef..1f331a2b2ecc 100644
@@ -13,6 +13,7 @@
 #include <drm/drm_print.h>
 #include <drm/ttm/ttm_tt.h>
 #include <uapi/drm/xe_drm.h>
+#include <uapi/drm/xe_drm_events.h>
 #include <linux/ascii85.h>
 #include <linux/delay.h>
 #include <linux/kthread.h>
@@ -40,6 +41,7 @@
 #include "xe_tlb_inval.h"
 #include "xe_trace_bo.h"
 #include "xe_wa.h"
+#include "xe_watch_queue.h"
 
 static struct drm_gem_object *xe_vm_obj(struct xe_vm *vm)
 {
@@ -567,13 +569,13 @@ static void preempt_rebind_work_func(struct work_struct *w)
 	}
 
 	if (err) {
-		drm_warn(&vm->xe->drm, "VM worker error: %d\n", err);
+		xe_watch_queue_post_vm_err_event(vm->xef, vm->id, err);
+		drm_dbg(&vm->xe->drm, "VM worker error: %d\n", err);
 		xe_vm_kill(vm, true);
 	}
 	up_write(&vm->lock);
 
 	free_preempt_fences(&preempt_fences);
-
 	trace_xe_vm_rebind_worker_exit(vm);
 }
 
@@ -2008,6 +2010,7 @@ int xe_vm_create_ioctl(struct drm_device *dev, void *data,
 	if (err)
 		goto err_close_and_put;
 
+	vm->id = id;
 	args->vm_id = id;
 
 	return 0;
drivers/gpu/drm/xe/xe_vm_types.h +2 -0
index 1f6f7e30e751..df559cf87b4c 100644
@@ -365,6 +365,8 @@ struct xe_vm {
 	bool batch_invalidate_tlb;
 	/** @xef: Xe file handle for tracking this VM's drm client */
 	struct xe_file *xef;
+	/** @id: The id of the VM in the VM table of @xef. */
+	u32 id;
 };
 
 /** struct xe_vma_op_map - VMA map operation */
drivers/gpu/drm/xe/xe_watch_queue.c new file +107 -0

107 lines not on this page · raw message

drivers/gpu/drm/xe/xe_watch_queue.h new file +20 -0
new file mode 100644
index 000000000000..ad199ee68205
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _XE_WATCH_QUEUE_H_
+#define _XE_WATCH_QUEUE_H_
+
+#include <linux/types.h>
+
+struct drm_device;
+struct drm_file;
+struct xe_file;
+
+int xe_watch_queue_ioctl(struct drm_device *dev, void *data,
+			 struct drm_file *file);
+void xe_watch_queue_post_vm_err_event(struct xe_file *xef, u32 vm_id,
+				      int error_code);
+
+#endif /* _XE_WATCH_QUEUE_H_ */
include/uapi/drm/xe_drm.h +46 -0

70 lines not on this page · raw message

include/uapi/drm/xe_drm_events.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.53.0