Fine grained fault locking, threaded prefetch, storm cache
Subsystems (2)
Matthew Brost
maintainerv8 cover Fine grained fault locking, threaded prefetch, storm cache
Fine-grained fault locking provides immediate benefits: it allows page faults from the same VM to be processed in parallel (unless they target the same range) and enables a sane multi-threaded prefetch implementation. UMD prefetch benchmarks see 10% to 50% improvement in prefetch performance on BMG depending on PCIe bus speed.
Once parallel fault processing is available, the pagefault queue can be unified into a single queue with multiple workers pulling faults to process. A single queue then allows a sensible pagefault cache to be implemented, so that multiple faults targeting the same region can be batched together and acknowledged in, ideally, a single pass. This saves CPU cycles during pagefault handling and improves overall throughput of the fault handler.
Significant improvements in UMD pagefault benchmarks can be seen when utilizing this caching.
v3:
- Fix kunit build (CI)
v4:
- Actually fix kunit build (CI)
v5:
- Address v4 feedback, rebase, fix several CI bugs
v6:
- Address Sashiko feedback: https://sashiko.dev/#/patchset/20260723214743.1498125-1-matthew.brost%40intel.com
- CI fixes
v7:
- Dedicated prefetch work queue (Sashiko, arch choice)
- Clean up unwind paths (Sashiko)
v8:
- Still fighting valid Sashiko issues (unwind, locking issues, reset flows)
Matt
Matthew Brost (12):
drm/xe: Fine grained page fault locking
drm/xe: Allow prefetch-only VM bind IOCTLs to use VM read lock drm/xe: Thread prefetch of SVM ranges
drm/xe: Use a single page-fault queue with multiple workers drm/xe: Add num_pf_work modparam
drm/xe: Engine class and instance into a u8
drm/xe: Track pagefault worker runtime
drm/xe: Chain page faults via queue-resident cache to avoid fault
storms
drm/xe: Add pagefault chaining stats
drm/xe: Add debugfs pagefault_info
drm/xe: batch CT pagefault acks with periodic flush drm/xe: Track parallel page fault activity in GT stats
drivers/gpu/drm/xe/xe_debugfs.c | 11 + drivers/gpu/drm/xe/xe_defaults.h | 1 + drivers/gpu/drm/xe/xe_device.c | 14 +- drivers/gpu/drm/xe/xe_device_types.h | 23 +- drivers/gpu/drm/xe/xe_gt_stats.c | 7 + drivers/gpu/drm/xe/xe_gt_stats_types.h | 22 + drivers/gpu/drm/xe/xe_guc_ct.c | 95 +++- drivers/gpu/drm/xe/xe_guc_ct.h | 36 +- drivers/gpu/drm/xe/xe_guc_pagefault.c | 39 +- drivers/gpu/drm/xe/xe_guc_types.h | 6 + drivers/gpu/drm/xe/xe_module.c | 4 + drivers/gpu/drm/xe/xe_module.h | 1 + drivers/gpu/drm/xe/xe_pagefault.c | 689 ++++++++++++++++++++---- drivers/gpu/drm/xe/xe_pagefault.h | 74 +++ drivers/gpu/drm/xe/xe_pagefault_types.h | 112 +++- drivers/gpu/drm/xe/xe_svm.c | 146 +++-- drivers/gpu/drm/xe/xe_svm.h | 59 +- drivers/gpu/drm/xe/xe_userptr.c | 19 +- drivers/gpu/drm/xe/xe_vm.c | 285 +++++++--- drivers/gpu/drm/xe/xe_vm_types.h | 42 +- 20 files changed, 1376 insertions(+), 309 deletions(-)
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 1/12 drm/xe: Fine grained page fault locking
Enable page faults to be serviced while holding vm->lock in read mode.
Introduce additional locks to:
- Ensure only one page fault thread services a given range or VMA
- Serialize SVM garbage collection
- Protect SVM range insertion and removal
While these locks may contend during page faults, expensive operations like migration can now run in parallel within a single VM.
In addition to new locking, ranges must be reference-counted after lookup, as another thread could immediately remove them from the GPU SVM tree, potentially dropping the last reference.
Lastly, decouple the VM’s ASID from the page fault queue selection to allow parallel page fault handling within the same VM.
Lays the groundwork for prefetch IOCTLs to use threaded migration too.
--- drivers/gpu/drm/xe/xe_device_types.h | 2 + drivers/gpu/drm/xe/xe_pagefault.c | 102 ++++++++++++---------- drivers/gpu/drm/xe/xe_svm.c | 122 +++++++++++++++++++-------- drivers/gpu/drm/xe/xe_svm.h | 44 ++++++++++ drivers/gpu/drm/xe/xe_userptr.c | 19 ++++- drivers/gpu/drm/xe/xe_vm.c | 106 +++++++++++++---------- drivers/gpu/drm/xe/xe_vm_types.h | 29 +++++-- 7 files changed, 293 insertions(+), 131 deletions(-)
drivers/gpu/drm/xe/xe_device_types.h +2 -0
@@ -303,6 +303,8 @@ struct xe_device {struct xarray asid_to_vm;/** @usm.next_asid: next ASID, used to cyclical alloc asids */ u32 next_asid; +/** @usm.current_pf_queue: current page fault queue */ +u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock;/** @usm.pf_wq: page fault work queue, unbound, high priority */
drivers/gpu/drm/xe/xe_pagefault.c +56 -46
155 lines not on this page · raw message
drivers/gpu/drm/xe/xe_svm.c +88 -34
307 lines not on this page · raw message
drivers/gpu/drm/xe/xe_svm.h +44 -0
80 lines not on this page · raw message
drivers/gpu/drm/xe/xe_userptr.c +18 -1
@@ -57,6 +57,23 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm) list_empty(&vm->userptr.invalidated)) ? 0 : -EAGAIN; } +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +static bool __xe_vma_userptr_lockdep (struct xe_userptr_vma *uvma) +{ +struct xe_vma *vma = &uvma->vma; +struct xe_vm *vm =xe_vma_vm (vma); + +return lockdep_is_held_type (&vm->lock,0 ) || + (lockdep_is_held_type (&vm->lock,1 ) && +lockdep_is_held_type (&vma->fault_lock,0 )); +} + +#define xe_vma_userptr_lockdep (uvma) \ + lockdep_assert(__xe_vma_userptr_lockdep(uvma)) +#else +#define xe_vma_userptr_lockdep (uvma) +#endif +int xe_vma_userptr_pin_pages (struct xe_userptr_vma *uvma) {struct xe_vma *vma = &uvma->vma;
@@ -68,7 +85,7 @@ int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma) .allow_mixed = true, }; -lockdep_assert_held (&vm->lock); +xe_vma_userptr_lockdep (uvma);xe_assert (xe,xe_vma_is_userptr (vma));if (vma->gpuva.flags &XE_VMA_DESTROYED )
drivers/gpu/drm/xe/xe_vm.c +63 -43
302 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +22 -7
@@ -132,6 +132,12 @@ struct xe_vma {struct work_struct destroy_work; }; +/** +* @fault_lock: Synchronizes fault processing. Locking order: inside +* vm->lock, outside dma-resv. +*/ +struct mutex fault_lock; +/** * @tile_invalidated: Tile mask of binding are invalidated for this VMA. * protected by BO's resv and for userptrs, vm->svm.gpusvm.notifier_lock in
@@ -214,13 +220,27 @@ struct xe_vm {struct {/** @svm.gpusvm: base GPUSVM used to track fault allocations */ struct drm_gpusvm gpusvm; +/** +* @svm.range_lock: Protects insertion and removal of ranges +* from GPU SVM tree. +*/ +struct mutex range_lock;/** * @svm.garbage_collector: Garbage collector which is used unmap * SVM range's GPU bindings and destroy the ranges. */ struct { -/** @svm.garbage_collector.lock: Protect's range list */ -spinlock_t lock; +/** +* @svm.garbage_collector.lock: Ensures only one thread +* runs the garbage collector at a time. Locking order: +* inside vm->lock, outside range->lock and dma-resv. +*/ +struct mutex lock; +/** +* @svm.garbage_collector.list_lock: Protect's range +* list +*/ +spinlock_t list_lock;/** * @svm.garbage_collector.range_list: List of SVM ranges * in the garbage collector.
@@ -350,11 +370,6 @@ struct xe_vm {struct {/** @asid: address space ID, unique to each VM */ u32 asid; -/** -* @last_fault_vma: Last fault VMA, used for fast lookup when we -* get a flood of faults to the same VMA -*/ -struct xe_vma *last_fault_vma; } usm;/** @error_capture: allow to track errors */
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 2/12 drm/xe: Allow prefetch-only VM bind IOCTLs to use VM read lock
In reply to Matthew BrostPrefetch-only VM bind IOCTLs do not modify VMAs or use userptr pages. Downgrade vm->lock to read mode once setup is complete.
Lays the groundwork for prefetch IOCTLs to use threaded migration.
--- drivers/gpu/drm/xe/xe_vm.c | 36 +++++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_vm_types.h | 2 ++ 2 files changed, 33 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +31 -5
@@ -2451,10 +2451,12 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, .map.gem.offset = bo_offset_or_userptr, }; + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; ops =drm_gpuvm_sm_map_ops_create (&vm->gpuvm, &map_req);break ; }case DRM_XE_VM_BIND_OP_UNMAP : + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; ops =drm_gpuvm_sm_unmap_ops_create (&vm->gpuvm, addr, range);break ;case DRM_XE_VM_BIND_OP_PREFETCH :
@@ -2463,6 +2465,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,case DRM_XE_VM_BIND_OP_UNMAP_ALL :xe_assert (vm->xe, bo); + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; err =xe_bo_lock (bo, true);if (err)return ERR_PTR (err);
@@ -2523,6 +2526,9 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,u8 id, tile_mask =0 ;u32 i; +if (xe_vma_is_userptr (vma)) + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; +if (!xe_vma_is_cpu_addr_mirror (vma)) { op->prefetch.region = prefetch_region;continue ;
@@ -2708,10 +2714,12 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op) {int err =0 ; -xe_vm_assert_write_mode_or_garbage_collector (vm); +lockdep_assert_held (&vm->lock);switch (op->base.op) {case DRM_GPUVA_OP_MAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); + err |=xe_vm_insert_vma (vm, op->map.vma);if (!err) op->flags |=XE_VMA_OP_COMMITTED ;
@@ -2721,6 +2729,8 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op)u8 tile_present =gpuva_to_vma (op->base.remap.unmap->va)->tile_present; +xe_vm_assert_write_mode_or_garbage_collector (vm); +prep_vma_destroy (vm,gpuva_to_vma (op->base.remap.unmap->va), true); op->flags |=XE_VMA_OP_COMMITTED ;
@@ -2755,6 +2765,8 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op)break ; }case DRM_GPUVA_OP_UNMAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); +prep_vma_destroy (vm,gpuva_to_vma (op->base.unmap.va), true); op->flags |=XE_VMA_OP_COMMITTED ;break ;
@@ -2979,10 +2991,12 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, bool post_commit, bool prev_post_commit, bool next_post_commit) { -xe_vm_assert_write_mode_or_garbage_collector (vm ); + lockdep_assert_held(&vm->lock);switch (op->base.op) {case DRM_GPUVA_OP_MAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (op->map.vma) {prep_vma_destroy (vm, op->map.vma, post_commit);xe_vma_destroy_unlocked (op->map.vma);
@@ -2992,6 +3006,8 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, {struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va); +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (vma) {xe_svm_notifier_lock (vm); vma->gpuva.flags &= ~XE_VMA_DESTROYED ;
@@ -3005,6 +3021,8 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, {struct xe_vma *vma =gpuva_to_vma (op->base.remap.unmap->va); +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (op->remap.prev) {prep_vma_destroy (vm, op->remap.prev, prev_post_commit);xe_vma_destroy_unlocked (op->remap.prev);
@@ -3589,7 +3607,7 @@ static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,struct dma_fence *fence;int err =0 ; -lockdep_assert_held_write (&vm->lock); +lockdep_assert_held (&vm->lock); xe_validation_guard(&ctx, &vm->xe->val, &exec, ((struct xe_val_flags) {
@@ -3912,7 +3930,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u32 num_syncs, num_ufence =0 ;struct xe_sync_entry *syncs =NULL ;struct drm_xe_vm_bind_op *bind_ops =NULL ; -struct xe_vma_ops vops; +struct xe_vma_ops vops = { .flags =0 , };struct dma_fence *fence;int err;int i;
@@ -4087,6 +4105,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto unwind_ops; } +if (!(vops.flags &XE_VMA_OPS_FLAG_MODIFIES_GPUVA )) { + vops.flags |=XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ; +downgrade_write (&vm->lock); + } + err =xe_vma_ops_alloc (&vops, args->num_binds >1 );if (err) goto unwind_ops;
@@ -4123,7 +4146,10 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) free_bos:kvfree (bos); release_vm_lock: -up_write (&vm->lock); +if (vops.flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ) +up_read (&vm->lock); +else +up_write (&vm->lock); put_exec_queue:if (q)xe_exec_queue_put (q);
drivers/gpu/drm/xe/xe_vm_types.h +2 -0
@@ -561,6 +561,8 @@ struct xe_vma_ops {#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2)#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3)#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */
Signature · 1 line
2.34.1
Re: [PATCH v8 02/12] drm/xe: Allow prefetch-only VM bind IOCTLs to use VM read lock
Quoted 6 lines · Matthew Brost
Prefetch-only VM bind IOCTLs do not modify VMAs or use userptr pages. Downgrade
vm->lockto read mode once setup is complete.Lays the groundwork for prefetch IOCTLs to use threaded migration.
Signed-off-by Matthew Brost <[email protected]>
Quoted 162 lines
--- drivers/gpu/drm/xe/xe_vm.c | 36 +++++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_vm_types.h | 2 ++ 2 files changed, 33 insertions(+), 5 deletions(-)drivers/gpu/drm/xe/xe_vm.c +31
-5@@ -2451,10 +2451,12 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, .map.gem.offset = bo_offset_or_userptr, }; + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; ops =drm_gpuvm_sm_map_ops_create (&vm->gpuvm, &map_req);break ; }case DRM_XE_VM_BIND_OP_UNMAP : + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; ops =drm_gpuvm_sm_unmap_ops_create (&vm->gpuvm, addr, range);break ;case DRM_XE_VM_BIND_OP_PREFETCH :@@ -2463,6 +2465,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,case DRM_XE_VM_BIND_OP_UNMAP_ALL :xe_assert (vm->xe, bo); + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; err =xe_bo_lock (bo, true);if (err)return ERR_PTR (err);@@ -2523,6 +2526,9 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,u8 id, tile_mask =0 ;u32 i; +if (xe_vma_is_userptr (vma)) + vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ; +if (!xe_vma_is_cpu_addr_mirror (vma)) { op->prefetch.region = prefetch_region;continue ;@@ -2708,10 +2714,12 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op) {int err =0 ; -xe_vm_assert_write_mode_or_garbage_collector (vm); +lockdep_assert_held (&vm->lock);switch (op->base.op) {case DRM_GPUVA_OP_MAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); + err |=xe_vm_insert_vma (vm, op->map.vma);if (!err) op->flags |=XE_VMA_OP_COMMITTED ;@@ -2721,6 +2729,8 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op)u8 tile_present =gpuva_to_vma (op->base.remap.unmap->va)->tile_present; +xe_vm_assert_write_mode_or_garbage_collector (vm); +prep_vma_destroy (vm,gpuva_to_vma (op->base.remap.unmap->va), true); op->flags |=XE_VMA_OP_COMMITTED ;@@ -2755,6 +2765,8 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op)break ; }case DRM_GPUVA_OP_UNMAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); +prep_vma_destroy (vm,gpuva_to_vma (op->base.unmap.va), true); op->flags |=XE_VMA_OP_COMMITTED ;break ;@@ -2979,10 +2991,12 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, bool post_commit, bool prev_post_commit, bool next_post_commit) { -xe_vm_assert_write_mode_or_garbage_collector (vm ); + lockdep_assert_held(&vm->lock);switch (op->base.op) {case DRM_GPUVA_OP_MAP : +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (op->map.vma) {prep_vma_destroy (vm, op->map.vma, post_commit);xe_vma_destroy_unlocked (op->map.vma);@@ -2992,6 +3006,8 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, {struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va); +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (vma) {xe_svm_notifier_lock (vm); vma->gpuva.flags &= ~XE_VMA_DESTROYED ;@@ -3005,6 +3021,8 @@ static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op, {struct xe_vma *vma =gpuva_to_vma (op->base.remap.unmap->va); +xe_vm_assert_write_mode_or_garbage_collector (vm); +if (op->remap.prev) {prep_vma_destroy (vm, op->remap.prev, prev_post_commit);xe_vma_destroy_unlocked (op->remap.prev);@@ -3589,7 +3607,7 @@ static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,struct dma_fence *fence;int err =0 ; -lockdep_assert_held_write (&vm->lock); +lockdep_assert_held (&vm->lock); xe_validation_guard(&ctx, &vm->xe->val, &exec, ((struct xe_val_flags) {@@ -3912,7 +3930,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u32 num_syncs, num_ufence =0 ;struct xe_sync_entry *syncs =NULL ;struct drm_xe_vm_bind_op *bind_ops =NULL ; -struct xe_vma_ops vops; +struct xe_vma_ops vops = { .flags =0 , };struct dma_fence *fence;int err;int i;@@ -4087,6 +4105,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto unwind_ops; } +if (!(vops.flags &XE_VMA_OPS_FLAG_MODIFIES_GPUVA )) { + vops.flags |=XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ; +downgrade_write (&vm->lock); + } + err =xe_vma_ops_alloc (&vops, args->num_binds >1 );if (err) goto unwind_ops;@@ -4123,7 +4146,10 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) free_bos:kvfree (bos); release_vm_lock: -up_write (&vm->lock); +if (vops.flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ) +up_read (&vm->lock); +else +up_write (&vm->lock); put_exec_queue:if (q)xe_exec_queue_put (q);drivers/gpu/drm/xe/xe_vm_types.h +2
-0@@ -561,6 +561,8 @@ struct xe_vma_ops {#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2)#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3)#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 3/12 drm/xe: Thread prefetch of SVM ranges
In reply to Matthew BrostThe migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running xe_exec_system_allocator --r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.
Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain down_read() on vm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.
v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need for down_read_trylock() based lock avoidance in the page fault and garbage collector paths.
--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)
drivers/gpu/drm/xe/xe_device_types.h +4 -2
@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging
drivers/gpu/drm/xe/xe_pagefault.c +20 -9
@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));
@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }
@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/**
@@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)
@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }
@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",
drivers/gpu/drm/xe/xe_svm.c +6 -2
@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)
@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region.
@@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;
@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;
drivers/gpu/drm/xe/xe_svm.h +4 -2
@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);
@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }
drivers/gpu/drm/xe/xe_vm.c +112 -35
205 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +8 -7
@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */
Signature · 1 line
2.34.1
Re: [PATCH v8 03/12] drm/xe: Thread prefetch of SVM ranges
Quoted 274 lines · Matthew Brost
The migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running
xe_exec_system_allocator--r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain
down_read()onvm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need fordown_read_trylock()based lock avoidance in the page fault and garbage collector paths.Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +4
-2@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging drivers/gpu/drm/xe/xe_pagefault.c +20
-9@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/** @@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",drivers/gpu/drm/xe/xe_svm.c +6
-2@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region. @@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;drivers/gpu/drm/xe/xe_svm.h +4
-2@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }drivers/gpu/drm/xe/xe_vm.c +34
-2@@ -2525,6 +2525,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,struct drm_pagemap *dpagemap =NULL ;u8 id, tile_mask =0 ;u32 i; +bool valid_pages;if (xe_vma_is_userptr (vma)) vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ;@@ -2569,9 +2570,11 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, goto unwind_prefetch_ops; } -if (xe_svm_range_validate (vm, svm_range, tile_mask, dpagemap)) { +if (xe_svm_range_validate (vm, svm_range, tile_mask, + dpagemap, &valid_pages)) {xe_svm_range_debug (svm_range,"PREFETCH - RANGE IS VALID" );xe_svm_range_put (svm_range); +xe_assert (vm->xe, valid_pages); goto check_next_range; }@@ -2586,6 +2589,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, op->prefetch_range.ranges_count++; vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH ; +if (valid_pages) + vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE ;xe_svm_range_debug (svm_range,"PREFETCH - RANGE CREATED" ); check_next_range: if (range_end > xe_svm_range_end(svm_range) &&@@ -3151,16 +3156,80 @@ static int check_ufence(struct xe_vma *vma)return 0 ; } -static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_op *op) +struct prefetch_thread { +struct work_struct work; +struct drm_gpusvm_ctx *ctx; +struct xe_vma *vma; +struct xe_svm_range *svm_range; +struct drm_pagemap *dpagemap; +int err; +}; + +static void prefetch_thread_func (struct prefetch_thread *thread) +{ +struct xe_vma *vma = thread->vma; +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_svm_range *svm_range = thread->svm_range; +struct drm_pagemap *dpagemap = thread->dpagemap; +int err =0 ; + +guard (mutex)(&svm_range->lock); + +if (xe_svm_range_is_removed (svm_range)) +return ; + +if (!dpagemap) +xe_svm_range_migrate_to_smem (vm, svm_range); + +if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { + drm_dbg(&vm->xe->drm, + "Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n",
drivers/gpu/drm/xe/xe_vm.c static int check_ufence(struct xe_vma *vma) · line 3196
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ dpagemap ? dpagemap->drm->unique :"system" , + xe_svm_range_start(svm_range), xe_svm_range_end(svm_range)); + } + +if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { + err =xe_svm_alloc_vram (svm_range, thread->ctx, dpagemap); +if (err) { +drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +return ;
We should set the error code like below with
thread->err = err;
to propagate the error to user space.
Rest LGTM.
Francois
Quoted 168 lines
drivers/gpu/drm/xe/xe_vm.c +68
-33+ } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + } + + err =xe_svm_range_get_pages (vm, svm_range, thread->ctx); +if (err) { +drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) + err =0 ; + thread->err = err; +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +} + +static void prefetch_work_func (struct work_struct *w) +{ +struct prefetch_thread *thread = + container_of(w, struct prefetch_thread, work); + +prefetch_thread_func (thread); +} + +static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops, +struct xe_vma_op *op) {bool devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va);struct drm_pagemap *dpagemap = op->prefetch_range.dpagemap; -int err =0 ; -struct xe_svm_range *svm_range;struct drm_gpusvm_ctx ctx = {}; +struct prefetch_thread stack_thread, *thread, *prefetches;unsigned long i; +int err =0 , idx =0 ; +bool skip_threads;if (!xe_vma_is_cpu_addr_mirror (vma))return 0 ;@@ -3170,42 +3239,49 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_op *op) ctx.check_pages_threshold = devmem_possible ?SZ_64K :0 ; ctx.device_private_page_owner =xe_svm_private_page_owner (vm, !dpagemap); -/* TODO: Threading the migration */ -xa_for_each (&op->prefetch_range.range, i, svm_range) { -guard (mutex)(&svm_range->lock); - -if (xe_svm_range_is_removed (svm_range)) -continue ; + skip_threads = op->prefetch_range.ranges_count ==1 || + (!dpagemap && !(vops->flags & +XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + thread = skip_threads ? &stack_thread :NULL ; -if (!dpagemap) -xe_svm_range_migrate_to_smem (vm, svm_range); +if (!skip_threads) { + prefetches =kvmalloc_array (op->prefetch_range.ranges_count, +sizeof (*prefetches),GFP_KERNEL ); +if (!prefetches) +return -ENOMEM ; + } -if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { -drm_dbg (&vm->xe->drm, -"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , - dpagemap ? dpagemap->drm->unique :"system" , -xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); +xa_for_each (&op->prefetch_range.range, i, svm_range) { +if (!skip_threads) { + thread = prefetches + idx++; +INIT_WORK (&thread->work, prefetch_work_func); } -if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { - err =xe_svm_alloc_vram (svm_range, &ctx, dpagemap); -if (err) { -drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -return -ENODATA ; - } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + thread->ctx = &ctx; + thread->vma = vma; + thread->svm_range = svm_range; + thread->dpagemap = dpagemap; + thread->err =0 ; + +if (skip_threads) { +prefetch_thread_func (thread); +if (thread->err) +return thread->err; + }else { +queue_work (vm->xe->usm.prefetch_wq, &thread->work); } + } - err =xe_svm_range_get_pages (vm, svm_range, &ctx); -if (err) { -drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) - err = -ENODATA ; -return err; +if (!skip_threads) { +for (i =0 ; i < idx; ++i) { + thread = prefetches + i; + +flush_work (&thread->work); +if (thread->err && !err) + err = thread->err; } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +kvfree (prefetches); }return err;@@ -3336,7 +3412,8 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,return err; } -static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm, +struct xe_vma_ops *vops) {struct xe_vma_op *op;int err;@@ -3346,7 +3423,7 @@ static int vm_bind_ioctl_ops_prefetch_ranges(struct xe_vm *vm, struct xe_vma_opslist_for_each_entry (op, &vops->list, link) {if (op->base.op ==DRM_GPUVA_OP_PREFETCH ) { - err =prefetch_ranges (vm, op); + err =prefetch_ranges (vm, vops, op);if (err)return err; }drivers/gpu/drm/xe/xe_vm_types.h +8
-7@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v8 03/12] drm/xe: Thread prefetch of SVM ranges
Quoted 288 lines · Francois Dugast
Quoted 274 lines · Matthew Brost
The migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running
xe_exec_system_allocator--r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain
down_read()onvm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need fordown_read_trylock()based lock avoidance in the page fault and garbage collector paths.Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +4
-2@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging drivers/gpu/drm/xe/xe_pagefault.c +20
-9@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/** @@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",drivers/gpu/drm/xe/xe_svm.c +6
-2@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region. @@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;drivers/gpu/drm/xe/xe_svm.h +4
-2@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }drivers/gpu/drm/xe/xe_vm.c +34
-2@@ -2525,6 +2525,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,struct drm_pagemap *dpagemap =NULL ;u8 id, tile_mask =0 ;u32 i; +bool valid_pages;if (xe_vma_is_userptr (vma)) vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ;@@ -2569,9 +2570,11 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, goto unwind_prefetch_ops; } -if (xe_svm_range_validate (vm, svm_range, tile_mask, dpagemap)) { +if (xe_svm_range_validate (vm, svm_range, tile_mask, + dpagemap, &valid_pages)) {xe_svm_range_debug (svm_range,"PREFETCH - RANGE IS VALID" );xe_svm_range_put (svm_range); +xe_assert (vm->xe, valid_pages); goto check_next_range; }@@ -2586,6 +2589,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, op->prefetch_range.ranges_count++; vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH ; +if (valid_pages) + vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE ;xe_svm_range_debug (svm_range,"PREFETCH - RANGE CREATED" ); check_next_range: if (range_end > xe_svm_range_end(svm_range) &&@@ -3151,16 +3156,80 @@ static int check_ufence(struct xe_vma *vma)return 0 ; } -static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_op *op) +struct prefetch_thread { +struct work_struct work; +struct drm_gpusvm_ctx *ctx; +struct xe_vma *vma; +struct xe_svm_range *svm_range; +struct drm_pagemap *dpagemap; +int err; +}; + +static void prefetch_thread_func (struct prefetch_thread *thread) +{ +struct xe_vma *vma = thread->vma; +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_svm_range *svm_range = thread->svm_range; +struct drm_pagemap *dpagemap = thread->dpagemap; +int err =0 ; + +guard (mutex)(&svm_range->lock); + +if (xe_svm_range_is_removed (svm_range)) +return ; + +if (!dpagemap) +xe_svm_range_migrate_to_smem (vm, svm_range); + +if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { + drm_dbg(&vm->xe->drm, + "Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n",drivers/gpu/drm/xe/xe_vm.c static int check_ufence(struct xe_vma *vma) · line 3196
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ dpagemap ? dpagemap->drm->unique :"system" , + xe_svm_range_start(svm_range), xe_svm_range_end(svm_range)); + } + +if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { + err =xe_svm_alloc_vram (svm_range, thread->ctx, dpagemap); +if (err) { +drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +return ;We should set the error code like below with
Quoted 3 lines
thread->err = err;to propagate the error to user space.
So this was part an error handling fix - here this error occurs most likely from racing CPU access and you don't want the error to be sent back to user space, rather just don't commit the prefetch bind for the affected range(s). Later in the pipeline should handle this but it actually is 100% right there either as it will return -EAGAIN (retry, possible livelock) or -ENODATA (squashes error in IOCTL, possible leak in memory). Let me try fixing this part a bit better in a seperate patch but this is all pre-existing issues actually so I don't think it worth holding up the series.
Matt
Quoted 173 lines
Rest LGTM.
Francois
Quoted 168 lines
drivers/gpu/drm/xe/xe_vm.c +68
-33+ } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + } + + err =xe_svm_range_get_pages (vm, svm_range, thread->ctx); +if (err) { +drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) + err =0 ; + thread->err = err; +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +} + +static void prefetch_work_func (struct work_struct *w) +{ +struct prefetch_thread *thread = + container_of(w, struct prefetch_thread, work); + +prefetch_thread_func (thread); +} + +static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops, +struct xe_vma_op *op) {bool devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va);struct drm_pagemap *dpagemap = op->prefetch_range.dpagemap; -int err =0 ; -struct xe_svm_range *svm_range;struct drm_gpusvm_ctx ctx = {}; +struct prefetch_thread stack_thread, *thread, *prefetches;unsigned long i; +int err =0 , idx =0 ; +bool skip_threads;if (!xe_vma_is_cpu_addr_mirror (vma))return 0 ;@@ -3170,42 +3239,49 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_op *op) ctx.check_pages_threshold = devmem_possible ?SZ_64K :0 ; ctx.device_private_page_owner =xe_svm_private_page_owner (vm, !dpagemap); -/* TODO: Threading the migration */ -xa_for_each (&op->prefetch_range.range, i, svm_range) { -guard (mutex)(&svm_range->lock); - -if (xe_svm_range_is_removed (svm_range)) -continue ; + skip_threads = op->prefetch_range.ranges_count ==1 || + (!dpagemap && !(vops->flags & +XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + thread = skip_threads ? &stack_thread :NULL ; -if (!dpagemap) -xe_svm_range_migrate_to_smem (vm, svm_range); +if (!skip_threads) { + prefetches =kvmalloc_array (op->prefetch_range.ranges_count, +sizeof (*prefetches),GFP_KERNEL ); +if (!prefetches) +return -ENOMEM ; + } -if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { -drm_dbg (&vm->xe->drm, -"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , - dpagemap ? dpagemap->drm->unique :"system" , -xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); +xa_for_each (&op->prefetch_range.range, i, svm_range) { +if (!skip_threads) { + thread = prefetches + idx++; +INIT_WORK (&thread->work, prefetch_work_func); } -if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { - err =xe_svm_alloc_vram (svm_range, &ctx, dpagemap); -if (err) { -drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -return -ENODATA ; - } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + thread->ctx = &ctx; + thread->vma = vma; + thread->svm_range = svm_range; + thread->dpagemap = dpagemap; + thread->err =0 ; + +if (skip_threads) { +prefetch_thread_func (thread); +if (thread->err) +return thread->err; + }else { +queue_work (vm->xe->usm.prefetch_wq, &thread->work); } + } - err =xe_svm_range_get_pages (vm, svm_range, &ctx); -if (err) { -drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) - err = -ENODATA ; -return err; +if (!skip_threads) { +for (i =0 ; i < idx; ++i) { + thread = prefetches + i; + +flush_work (&thread->work); +if (thread->err && !err) + err = thread->err; } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +kvfree (prefetches); }return err;@@ -3336,7 +3412,8 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,return err; } -static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm, +struct xe_vma_ops *vops) {struct xe_vma_op *op;int err;@@ -3346,7 +3423,7 @@ static int vm_bind_ioctl_ops_prefetch_ranges(struct xe_vm *vm, struct xe_vma_opslist_for_each_entry (op, &vops->list, link) {if (op->base.op ==DRM_GPUVA_OP_PREFETCH ) { - err =prefetch_ranges (vm, op); + err =prefetch_ranges (vm, vops, op);if (err)return err; }drivers/gpu/drm/xe/xe_vm_types.h +8
-7@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v8 03/12] drm/xe: Thread prefetch of SVM ranges
Quoted 293 lines · Matthew Brost
Quoted 288 lines · Francois Dugast
Quoted 274 lines · Matthew Brost
The migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running
xe_exec_system_allocator--r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain
down_read()onvm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need fordown_read_trylock()based lock avoidance in the page fault and garbage collector paths.Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +4
-2@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging drivers/gpu/drm/xe/xe_pagefault.c +20
-9@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/** @@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",drivers/gpu/drm/xe/xe_svm.c +6
-2@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region. @@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;drivers/gpu/drm/xe/xe_svm.h +4
-2@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }drivers/gpu/drm/xe/xe_vm.c +34
-2@@ -2525,6 +2525,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,struct drm_pagemap *dpagemap =NULL ;u8 id, tile_mask =0 ;u32 i; +bool valid_pages;if (xe_vma_is_userptr (vma)) vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ;@@ -2569,9 +2570,11 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, goto unwind_prefetch_ops; } -if (xe_svm_range_validate (vm, svm_range, tile_mask, dpagemap)) { +if (xe_svm_range_validate (vm, svm_range, tile_mask, + dpagemap, &valid_pages)) {xe_svm_range_debug (svm_range,"PREFETCH - RANGE IS VALID" );xe_svm_range_put (svm_range); +xe_assert (vm->xe, valid_pages); goto check_next_range; }@@ -2586,6 +2589,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, op->prefetch_range.ranges_count++; vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH ; +if (valid_pages) + vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE ;xe_svm_range_debug (svm_range,"PREFETCH - RANGE CREATED" ); check_next_range: if (range_end > xe_svm_range_end(svm_range) &&@@ -3151,16 +3156,80 @@ static int check_ufence(struct xe_vma *vma)return 0 ; } -static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_op *op) +struct prefetch_thread { +struct work_struct work; +struct drm_gpusvm_ctx *ctx; +struct xe_vma *vma; +struct xe_svm_range *svm_range; +struct drm_pagemap *dpagemap; +int err; +}; + +static void prefetch_thread_func (struct prefetch_thread *thread) +{ +struct xe_vma *vma = thread->vma; +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_svm_range *svm_range = thread->svm_range; +struct drm_pagemap *dpagemap = thread->dpagemap; +int err =0 ; + +guard (mutex)(&svm_range->lock); + +if (xe_svm_range_is_removed (svm_range)) +return ; + +if (!dpagemap) +xe_svm_range_migrate_to_smem (vm, svm_range); + +if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { + drm_dbg(&vm->xe->drm, + "Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n",drivers/gpu/drm/xe/xe_vm.c static int check_ufence(struct xe_vma *vma) · line 3196
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ dpagemap ? dpagemap->drm->unique :"system" , + xe_svm_range_start(svm_range), xe_svm_range_end(svm_range)); + } + +if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { + err =xe_svm_alloc_vram (svm_range, thread->ctx, dpagemap); +if (err) { +drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +return ;We should set the error code like below with
Quoted 3 lines
thread->err = err;to propagate the error to user space.
Quoted 10 lines
So this was part an error handling fix - here this error occurs most likely from racing CPU access and you don't want the error to be sent back to user space, rather just don't commit the prefetch bind for the affected range(s). Later in the pipeline should handle this but it actually is 100% right there either as it will return
-EAGAIN(retry, possible livelock) or-ENODATA(squashes error in IOCTL, possible leak in memory). Let me try fixing this part a bit better in a seperate patch but this is all pre-existing issues actually so I don't think it worth holding up the series.
Actually a partial commit while it would ideal, creates another set of problems too. Let me revert this to returning an sential error like the previous -ENODATA behavior which is unwind safe though.
Matt
Quoted 175 lines
Matt
Quoted 173 lines
Rest LGTM.
Francois
Quoted 168 lines
drivers/gpu/drm/xe/xe_vm.c +68
-33+ } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + } + + err =xe_svm_range_get_pages (vm, svm_range, thread->ctx); +if (err) { +drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) + err =0 ; + thread->err = err; +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +} + +static void prefetch_work_func (struct work_struct *w) +{ +struct prefetch_thread *thread = + container_of(w, struct prefetch_thread, work); + +prefetch_thread_func (thread); +} + +static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops, +struct xe_vma_op *op) {bool devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va);struct drm_pagemap *dpagemap = op->prefetch_range.dpagemap; -int err =0 ; -struct xe_svm_range *svm_range;struct drm_gpusvm_ctx ctx = {}; +struct prefetch_thread stack_thread, *thread, *prefetches;unsigned long i; +int err =0 , idx =0 ; +bool skip_threads;if (!xe_vma_is_cpu_addr_mirror (vma))return 0 ;@@ -3170,42 +3239,49 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_op *op) ctx.check_pages_threshold = devmem_possible ?SZ_64K :0 ; ctx.device_private_page_owner =xe_svm_private_page_owner (vm, !dpagemap); -/* TODO: Threading the migration */ -xa_for_each (&op->prefetch_range.range, i, svm_range) { -guard (mutex)(&svm_range->lock); - -if (xe_svm_range_is_removed (svm_range)) -continue ; + skip_threads = op->prefetch_range.ranges_count ==1 || + (!dpagemap && !(vops->flags & +XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + thread = skip_threads ? &stack_thread :NULL ; -if (!dpagemap) -xe_svm_range_migrate_to_smem (vm, svm_range); +if (!skip_threads) { + prefetches =kvmalloc_array (op->prefetch_range.ranges_count, +sizeof (*prefetches),GFP_KERNEL ); +if (!prefetches) +return -ENOMEM ; + } -if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { -drm_dbg (&vm->xe->drm, -"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , - dpagemap ? dpagemap->drm->unique :"system" , -xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); +xa_for_each (&op->prefetch_range.range, i, svm_range) { +if (!skip_threads) { + thread = prefetches + idx++; +INIT_WORK (&thread->work, prefetch_work_func); } -if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { - err =xe_svm_alloc_vram (svm_range, &ctx, dpagemap); -if (err) { -drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -return -ENODATA ; - } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + thread->ctx = &ctx; + thread->vma = vma; + thread->svm_range = svm_range; + thread->dpagemap = dpagemap; + thread->err =0 ; + +if (skip_threads) { +prefetch_thread_func (thread); +if (thread->err) +return thread->err; + }else { +queue_work (vm->xe->usm.prefetch_wq, &thread->work); } + } - err =xe_svm_range_get_pages (vm, svm_range, &ctx); -if (err) { -drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) - err = -ENODATA ; -return err; +if (!skip_threads) { +for (i =0 ; i < idx; ++i) { + thread = prefetches + i; + +flush_work (&thread->work); +if (thread->err && !err) + err = thread->err; } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +kvfree (prefetches); }return err;@@ -3336,7 +3412,8 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,return err; } -static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm, +struct xe_vma_ops *vops) {struct xe_vma_op *op;int err;@@ -3346,7 +3423,7 @@ static int vm_bind_ioctl_ops_prefetch_ranges(struct xe_vm *vm, struct xe_vma_opslist_for_each_entry (op, &vops->list, link) {if (op->base.op ==DRM_GPUVA_OP_PREFETCH ) { - err =prefetch_ranges (vm, op); + err =prefetch_ranges (vm, vops, op);if (err)return err; }drivers/gpu/drm/xe/xe_vm_types.h +8
-7@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Re: [PATCH v8 03/12] drm/xe: Thread prefetch of SVM ranges
In reply to Matthew BrostQuoted 35 lines · Matthew Brost
The migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running
xe_exec_system_allocator--r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain
down_read()onvm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need fordown_read_trylock()based lock avoidance in the page fault and garbage collector paths.
Quoted 4 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>
I believe we should go with "Assisted-by" instead of "Co-authored-by".
This also applies to patch #8 in the series.
Francois
Quoted 413 lines
--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +4
-2@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging drivers/gpu/drm/xe/xe_pagefault.c +20
-9@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/** @@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",drivers/gpu/drm/xe/xe_svm.c +6
-2@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region. @@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;drivers/gpu/drm/xe/xe_svm.h +4
-2@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }drivers/gpu/drm/xe/xe_vm.c +112
-35@@ -2525,6 +2525,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,struct drm_pagemap *dpagemap =NULL ;u8 id, tile_mask =0 ;u32 i; +bool valid_pages;if (xe_vma_is_userptr (vma)) vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ;@@ -2569,9 +2570,11 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, goto unwind_prefetch_ops; } -if (xe_svm_range_validate (vm, svm_range, tile_mask, dpagemap)) { +if (xe_svm_range_validate (vm, svm_range, tile_mask, + dpagemap, &valid_pages)) {xe_svm_range_debug (svm_range,"PREFETCH - RANGE IS VALID" );xe_svm_range_put (svm_range); +xe_assert (vm->xe, valid_pages); goto check_next_range; }@@ -2586,6 +2589,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, op->prefetch_range.ranges_count++; vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH ; +if (valid_pages) + vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE ;xe_svm_range_debug (svm_range,"PREFETCH - RANGE CREATED" ); check_next_range: if (range_end > xe_svm_range_end(svm_range) &&@@ -3151,16 +3156,80 @@ static int check_ufence(struct xe_vma *vma)return 0 ; } -static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_op *op) +struct prefetch_thread { +struct work_struct work; +struct drm_gpusvm_ctx *ctx; +struct xe_vma *vma; +struct xe_svm_range *svm_range; +struct drm_pagemap *dpagemap; +int err; +}; + +static void prefetch_thread_func (struct prefetch_thread *thread) +{ +struct xe_vma *vma = thread->vma; +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_svm_range *svm_range = thread->svm_range; +struct drm_pagemap *dpagemap = thread->dpagemap; +int err =0 ; + +guard (mutex)(&svm_range->lock); + +if (xe_svm_range_is_removed (svm_range)) +return ; + +if (!dpagemap) +xe_svm_range_migrate_to_smem (vm, svm_range); + +if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { +drm_dbg (&vm->xe->drm, +"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , + dpagemap ? dpagemap->drm->unique :"system" , +xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); + } + +if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { + err =xe_svm_alloc_vram (svm_range, thread->ctx, dpagemap); +if (err) { +drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + } + + err =xe_svm_range_get_pages (vm, svm_range, thread->ctx); +if (err) { +drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) + err =0 ; + thread->err = err; +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +} + +static void prefetch_work_func (struct work_struct *w) +{ +struct prefetch_thread *thread = + container_of(w, struct prefetch_thread, work); + +prefetch_thread_func (thread); +} + +static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops, +struct xe_vma_op *op) {bool devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va);struct drm_pagemap *dpagemap = op->prefetch_range.dpagemap; -int err =0 ; -struct xe_svm_range *svm_range;struct drm_gpusvm_ctx ctx = {}; +struct prefetch_thread stack_thread, *thread, *prefetches;unsigned long i; +int err =0 , idx =0 ; +bool skip_threads;if (!xe_vma_is_cpu_addr_mirror (vma))return 0 ;@@ -3170,42 +3239,49 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_op *op) ctx.check_pages_threshold = devmem_possible ?SZ_64K :0 ; ctx.device_private_page_owner =xe_svm_private_page_owner (vm, !dpagemap); -/* TODO: Threading the migration */ -xa_for_each (&op->prefetch_range.range, i, svm_range) { -guard (mutex)(&svm_range->lock); - -if (xe_svm_range_is_removed (svm_range)) -continue ; + skip_threads = op->prefetch_range.ranges_count ==1 || + (!dpagemap && !(vops->flags & +XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + thread = skip_threads ? &stack_thread :NULL ; -if (!dpagemap) -xe_svm_range_migrate_to_smem (vm, svm_range); +if (!skip_threads) { + prefetches =kvmalloc_array (op->prefetch_range.ranges_count, +sizeof (*prefetches),GFP_KERNEL ); +if (!prefetches) +return -ENOMEM ; + } -if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { -drm_dbg (&vm->xe->drm, -"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , - dpagemap ? dpagemap->drm->unique :"system" , -xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); +xa_for_each (&op->prefetch_range.range, i, svm_range) { +if (!skip_threads) { + thread = prefetches + idx++; +INIT_WORK (&thread->work, prefetch_work_func); } -if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { - err =xe_svm_alloc_vram (svm_range, &ctx, dpagemap); -if (err) { -drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -return -ENODATA ; - } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + thread->ctx = &ctx; + thread->vma = vma; + thread->svm_range = svm_range; + thread->dpagemap = dpagemap; + thread->err =0 ; + +if (skip_threads) { +prefetch_thread_func (thread); +if (thread->err) +return thread->err; + }else { +queue_work (vm->xe->usm.prefetch_wq, &thread->work); } + } - err =xe_svm_range_get_pages (vm, svm_range, &ctx); -if (err) { -drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) - err = -ENODATA ; -return err; +if (!skip_threads) { +for (i =0 ; i < idx; ++i) { + thread = prefetches + i; + +flush_work (&thread->work); +if (thread->err && !err) + err = thread->err; } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +kvfree (prefetches); }return err;@@ -3336,7 +3412,8 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,return err; } -static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm, +struct xe_vma_ops *vops) {struct xe_vma_op *op;int err;@@ -3346,7 +3423,7 @@ static int vm_bind_ioctl_ops_prefetch_ranges(struct xe_vm *vm, struct xe_vma_opslist_for_each_entry (op, &vops->list, link) {if (op->base.op ==DRM_GPUVA_OP_PREFETCH ) { - err =prefetch_ranges (vm, op); + err =prefetch_ranges (vm, vops, op);if (err)return err; }drivers/gpu/drm/xe/xe_vm_types.h +8
-7@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v8 03/12] drm/xe: Thread prefetch of SVM ranges
Quoted 36 lines · Francois Dugast
Quoted 35 lines · Matthew Brost
The migrate_vma_* functions are very CPU-intensive; as a result, prefetching SVM ranges is limited by CPU performance rather than paging copy engine bandwidth. To accelerate SVM range prefetching, the step that calls migrate_vma_* is now threaded. A dedicated prefetch workqueue is used for threading so prefetch work is never mixed with page fault or garbage collector work.
Running
xe_exec_system_allocator--r prefetch-benchmark, which tests 64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with this patch on drm-tip. Enabling high SLPC further increases throughput to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both of these optimizations are upcoming.Since the dedicated prefetch workqueue is not shared with page fault or SVM garbage collector work, page fault servicing and garbage collection can keep using a plain
down_read()onvm->lock: there is no risk of a blocked reader starving a worker that a pending writer is waiting to flush, because that flushing is now confined to the separate prefetch workqueue.v2:
- Use dedicated prefetch workqueue
- Pick dedicated prefetch thread count based on profiling
- Skip threaded prefetch for only 1 range or if prefetching to SRAM
- Fully tested
v3:
- Use page fault work queue
v4:
- Go back to a dedicated prefetch workqueue (usm.prefetch_wq) rather than reusing the page fault workqueue (usm.pagefault_wq), so threaded prefetches and page fault / garbage collector work no longer contend for the same workqueue. This removes the need fordown_read_trylock()based lock avoidance in the page fault and garbage collector paths.
Quoted 9 lines
Quoted 4 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>I believe we should go with "Assisted-by" instead of "Co-authored-by".
This also applies to patch #8 in the series.
Yes, that was an oversight. I fixed somethings on the tip of the series and had copilot squash the fixes into the correct patches and this got added. Will adjust.
Matt
Quoted 415 lines
Francois
Quoted 413 lines
--- drivers/gpu/drm/xe/xe_device_types.h | 6 +- drivers/gpu/drm/xe/xe_pagefault.c | 29 ++++-- drivers/gpu/drm/xe/xe_svm.c | 8 +- drivers/gpu/drm/xe/xe_svm.h | 6 +- drivers/gpu/drm/xe/xe_vm.c | 147 ++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 15 +-- 6 files changed, 154 insertions(+), 57 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +4
-2@@ -307,8 +307,10 @@ struct xe_device {u32 current_pf_queue;/** @usm.lock: protects UM state */ struct rw_semaphore lock; -/** @usm.pf_wq: page fault work queue, unbound, high priority */ -struct workqueue_struct *pf_wq; +/** @usm.pagefault_wq: page fault work queue, unbound, high priority */ +struct workqueue_struct *pagefault_wq; +/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ +struct workqueue_struct *prefetch_wq;/* * We pick 4 here because, in the current implementation, it * yields the best bandwidth utilization of the kernel paging drivers/gpu/drm/xe/xe_pagefault.c +20
-9@@ -303,8 +303,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) err =xe_pagefault_service (&pf);if (err) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf);xe_pagefault_print (&pf);xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err));@@ -318,7 +318,7 @@ static void xe_pagefault_queue_work(struct work_struct *w) pf.producer.ops->ack_fault (&pf, err);if (time_after (jiffies, threshold)) { -queue_work (gt_to_xe (pf.gt)->usm.pf_wq, w); +queue_work (gt_to_xe (pf.gt)->usm.pagefault_wq, w);break ; } }@@ -376,7 +376,8 @@ static void xe_pagefault_fini(void *arg) {struct xe_device *xe = arg; -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +destroy_workqueue (xe->usm.pagefault_wq); }/** @@ -394,12 +395,20 @@ int xe_pagefault_init(struct xe_device *xe)if (!xe->info.has_usm)return 0 ; - xe->usm.pf_wq =alloc_workqueue ("xe_page_fault_work_queue" , -WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_QUEUE_COUNT ); -if (!xe->usm.pf_wq) + xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" , +WQ_UNBOUND |WQ_HIGHPRI , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.pagefault_wq)return -ENOMEM ; + xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" , +WQ_UNBOUND , +XE_PAGEFAULT_QUEUE_COUNT ); +if (!xe->usm.prefetch_wq) { + err = -ENOMEM ; + goto err_pagefault_wq; + } +for (i =0 ; i <XE_PAGEFAULT_QUEUE_COUNT ; ++i) { err =xe_pagefault_queue_init (xe, xe->usm.pf_queue + i);if (err)@@ -409,7 +418,9 @@ int xe_pagefault_init(struct xe_device *xe)return devm_add_action_or_reset (xe->drm.dev, xe_pagefault_fini, xe); err_out: -destroy_workqueue (xe->usm.pf_wq); +destroy_workqueue (xe->usm.prefetch_wq); +err_pagefault_wq: +destroy_workqueue (xe->usm.pagefault_wq);return err; }@@ -495,7 +506,7 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf)memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; -queue_work (xe->usm.pf_wq, &pf_queue->worker); +queue_work (xe->usm.pagefault_wq, &pf_queue->worker); }else { drm_warn(&xe->drm, "PageFault Queue (%d) full, shouldn't be possible\n",drivers/gpu/drm/xe/xe_svm.c +6
-2@@ -148,7 +148,7 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, &vm->svm.garbage_collector.range_list);spin_unlock (&vm->svm.garbage_collector.list_lock); -queue_work (xe->usm.pf_wq, &vm->svm.garbage_collector.work); +queue_work (xe->usm.pagefault_wq, &vm->svm.garbage_collector.work); }static void xe_svm_tlb_inval_count_stats_incr (struct xe_gt *gt)@@ -1051,6 +1051,7 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)* @tile_mask: Mask representing the tiles to be checked * @dpagemap: if !%NULL, the range is expected to be present * in device memory identified by this parameter. +* @valid_pages: Pages are valid, result written back to caller * * The xe_svm_range_validate() function checks if a range is * valid and located in the desired memory region. @@ -1059,7 +1060,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)*/ bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {bool ret;@@ -1071,6 +1073,8 @@ bool xe_svm_range_validate(struct xe_vm *vm,else ret = ret && !range->pages.dpagemap; + *valid_pages =xe_svm_range_pages_valid (range); +xe_svm_notifier_unlock (vm);return ret;drivers/gpu/drm/xe/xe_svm.h +4
-2@@ -134,7 +134,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range);bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,const struct drm_pagemap *dpagemap); +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages);u64 xe_svm_find_vma_start (struct xe_vm *vm,u64 addr,u64 end,struct xe_vma *vma);@@ -376,7 +377,8 @@ void xe_svm_range_migrate_to_smem(struct xe_vm *vm, struct xe_svm_range *range)static inline bool xe_svm_range_validate (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_mask,bool devmem_preferred) +u8 tile_mask,const struct drm_pagemap *dpagemap, +bool *valid_pages) {return false; }drivers/gpu/drm/xe/xe_vm.c +112
-35@@ -2525,6 +2525,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops,struct drm_pagemap *dpagemap =NULL ;u8 id, tile_mask =0 ;u32 i; +bool valid_pages;if (xe_vma_is_userptr (vma)) vops->flags |=XE_VMA_OPS_FLAG_MODIFIES_GPUVA ;@@ -2569,9 +2570,11 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, goto unwind_prefetch_ops; } -if (xe_svm_range_validate (vm, svm_range, tile_mask, dpagemap)) { +if (xe_svm_range_validate (vm, svm_range, tile_mask, + dpagemap, &valid_pages)) {xe_svm_range_debug (svm_range,"PREFETCH - RANGE IS VALID" );xe_svm_range_put (svm_range); +xe_assert (vm->xe, valid_pages); goto check_next_range; }@@ -2586,6 +2589,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, op->prefetch_range.ranges_count++; vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH ; +if (valid_pages) + vops->flags |=XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE ;xe_svm_range_debug (svm_range,"PREFETCH - RANGE CREATED" ); check_next_range: if (range_end > xe_svm_range_end(svm_range) &&@@ -3151,16 +3156,80 @@ static int check_ufence(struct xe_vma *vma)return 0 ; } -static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_op *op) +struct prefetch_thread { +struct work_struct work; +struct drm_gpusvm_ctx *ctx; +struct xe_vma *vma; +struct xe_svm_range *svm_range; +struct drm_pagemap *dpagemap; +int err; +}; + +static void prefetch_thread_func (struct prefetch_thread *thread) +{ +struct xe_vma *vma = thread->vma; +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_svm_range *svm_range = thread->svm_range; +struct drm_pagemap *dpagemap = thread->dpagemap; +int err =0 ; + +guard (mutex)(&svm_range->lock); + +if (xe_svm_range_is_removed (svm_range)) +return ; + +if (!dpagemap) +xe_svm_range_migrate_to_smem (vm, svm_range); + +if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { +drm_dbg (&vm->xe->drm, +"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , + dpagemap ? dpagemap->drm->unique :"system" , +xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); + } + +if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { + err =xe_svm_alloc_vram (svm_range, thread->ctx, dpagemap); +if (err) { +drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + } + + err =xe_svm_range_get_pages (vm, svm_range, thread->ctx); +if (err) { +drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , + vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); +if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) + err =0 ; + thread->err = err; +return ; + } +xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +} + +static void prefetch_work_func (struct work_struct *w) +{ +struct prefetch_thread *thread = + container_of(w, struct prefetch_thread, work); + +prefetch_thread_func (thread); +} + +static int prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops, +struct xe_vma_op *op) {bool devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va);struct drm_pagemap *dpagemap = op->prefetch_range.dpagemap; -int err =0 ; -struct xe_svm_range *svm_range;struct drm_gpusvm_ctx ctx = {}; +struct prefetch_thread stack_thread, *thread, *prefetches;unsigned long i; +int err =0 , idx =0 ; +bool skip_threads;if (!xe_vma_is_cpu_addr_mirror (vma))return 0 ;@@ -3170,42 +3239,49 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_op *op) ctx.check_pages_threshold = devmem_possible ?SZ_64K :0 ; ctx.device_private_page_owner =xe_svm_private_page_owner (vm, !dpagemap); -/* TODO: Threading the migration */ -xa_for_each (&op->prefetch_range.range, i, svm_range) { -guard (mutex)(&svm_range->lock); - -if (xe_svm_range_is_removed (svm_range)) -continue ; + skip_threads = op->prefetch_range.ranges_count ==1 || + (!dpagemap && !(vops->flags & +XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + thread = skip_threads ? &stack_thread :NULL ; -if (!dpagemap) -xe_svm_range_migrate_to_smem (vm, svm_range); +if (!skip_threads) { + prefetches =kvmalloc_array (op->prefetch_range.ranges_count, +sizeof (*prefetches),GFP_KERNEL ); +if (!prefetches) +return -ENOMEM ; + } -if (IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )) { -drm_dbg (&vm->xe->drm, -"Prefetch pagemap is %s start 0x%016lx end 0x%016lx\n" , - dpagemap ? dpagemap->drm->unique :"system" , -xe_svm_range_start (svm_range),xe_svm_range_end (svm_range)); +xa_for_each (&op->prefetch_range.range, i, svm_range) { +if (!skip_threads) { + thread = prefetches + idx++; +INIT_WORK (&thread->work, prefetch_work_func); } -if (xe_svm_range_needs_migrate_to_vram (svm_range, vma, dpagemap)) { - err =xe_svm_alloc_vram (svm_range, &ctx, dpagemap); -if (err) { -drm_dbg (&vm->xe->drm,"VRAM allocation failed, retry from userspace, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -return -ENODATA ; - } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE MIGRATED TO VRAM" ); + thread->ctx = &ctx; + thread->vma = vma; + thread->svm_range = svm_range; + thread->dpagemap = dpagemap; + thread->err =0 ; + +if (skip_threads) { +prefetch_thread_func (thread); +if (thread->err) +return thread->err; + }else { +queue_work (vm->xe->usm.prefetch_wq, &thread->work); } + } - err =xe_svm_range_get_pages (vm, svm_range, &ctx); -if (err) { -drm_dbg (&vm->xe->drm,"Get pages failed, asid=%u, gpusvm=%p, errno=%pe\n" , - vm->usm.asid, &vm->svm.gpusvm,ERR_PTR (err)); -if (err == -EOPNOTSUPP || err == -EFAULT || err == -EPERM ) - err = -ENODATA ; -return err; +if (!skip_threads) { +for (i =0 ; i < idx; ++i) { + thread = prefetches + i; + +flush_work (&thread->work); +if (thread->err && !err) + err = thread->err; } -xe_svm_range_debug (svm_range,"PREFETCH - RANGE GET PAGES DONE" ); +kvfree (prefetches); }return err;@@ -3336,7 +3412,8 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,return err; } -static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm,struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_prefetch_ranges (struct xe_vm *vm, +struct xe_vma_ops *vops) {struct xe_vma_op *op;int err;@@ -3346,7 +3423,7 @@ static int vm_bind_ioctl_ops_prefetch_ranges(struct xe_vm *vm, struct xe_vma_opslist_for_each_entry (op, &vops->list, link) {if (op->base.op ==DRM_GPUVA_OP_PREFETCH ) { - err =prefetch_ranges (vm, op); + err =prefetch_ranges (vm, vops, op);if (err)return err; }drivers/gpu/drm/xe/xe_vm_types.h +8
-7@@ -556,13 +556,14 @@ struct xe_vma_ops {/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ];/** @flag: signify the properties within xe_vma_ops*/ -#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) -#define XE_VMA_OPS_FLAG_MADVISE BIT(1) -#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) -#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) -#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) -#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) -#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0) +#define XE_VMA_OPS_FLAG_MADVISE BIT(1) +#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2) +#define XE_VMA_OPS_FLAG_SKIP_TLB_WAIT BIT(3) +#define XE_VMA_OPS_FLAG_ALLOW_SVM_UNMAP BIT(4) +#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5) +#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6) +#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 4/12 drm/xe: Use a single page-fault queue with multiple workers
In reply to Matthew BrostWith fine-grained page-fault locking, it no longer makes sense to maintain multiple page-fault queues, as we no longer hash queues based on the VM’s ASID. Multiple workers can pull page faults from a single queue, eliminating any head-of-queue blocking. Refactor the structures and code to use a single shared queue.
--- drivers/gpu/drm/xe/xe_device_types.h | 12 +++--- drivers/gpu/drm/xe/xe_pagefault.c | 51 +++++++++++++------------ drivers/gpu/drm/xe/xe_pagefault_types.h | 17 ++++++++- 3 files changed, 49 insertions(+), 31 deletions(-)
drivers/gpu/drm/xe/xe_device_types.h +7 -5
@@ -303,8 +303,8 @@ struct xe_device {struct xarray asid_to_vm;/** @usm.next_asid: next ASID, used to cyclical alloc asids */ u32 next_asid; -/** @usm.current_pf_queue: current page fault queue */ -u32 current_pf_queue; +/** @usm.current_pf_work: current page fault work item */ +u32 current_pf_work;/** @usm.lock: protects UM state */ struct rw_semaphore lock;/** @usm.pagefault_wq: page fault work queue, unbound, high priority */
@@ -316,9 +316,11 @@ struct xe_device {* yields the best bandwidth utilization of the kernel paging * engine. */ -#define XE_PAGEFAULT_QUEUE_COUNT 4 -/** @usm.pf_queue: Page fault queues */ -struct xe_pagefault_queue pf_queue[XE_PAGEFAULT_QUEUE_COUNT ]; +#define XE_PAGEFAULT_WORK_COUNT 4 +/** @usm.pf_workers: Page fault workers */ +struct xe_pagefault_work pf_workers[XE_PAGEFAULT_WORK_COUNT ]; +/** @usm.pf_queue: Page fault queue */ +struct xe_pagefault_queue pf_queue;#if IS_ENABLED (CONFIG_DRM_XE_PAGEMAP )/** @usm.dpagemap_shrinker: Shrinker for unused pagemaps */ struct drm_pagemap_shrinker *dpagemap_shrinker;
drivers/gpu/drm/xe/xe_pagefault.c +27 -24
116 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pagefault_types.h +15 -2
@@ -131,8 +131,21 @@ struct xe_pagefault_queue {u32 tail;/** @lock: protects page fault queue */ spinlock_t lock; -/** @worker: to process page faults */ - struct work_struct worker; +}; + +/** +* struct xe_pagefault_work - Xe page fault work item (consumer) +* +* Represents a worker that pops a &struct xe_pagefault from the page fault +* queue and processes it. +*/ +struct xe_pagefault_work { +/** @xe: Back-pointer to the Xe device */ +struct xe_device *xe; +/** @id: Identifier for this work item */ +int id; +/** @work: Work item used to process the page fault */ +struct work_struct work; };#endif
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 5/12 drm/xe: Add num_pf_work modparam
In reply to Matthew BrostAdd a module parameter to control the number of page-fault work threads, making it easy to experiment with how different numbers of work threads impact performance.
--- drivers/gpu/drm/xe/xe_defaults.h | 1 + drivers/gpu/drm/xe/xe_device.c | 14 ++++++++++++-- drivers/gpu/drm/xe/xe_device_types.h | 11 ++++------- drivers/gpu/drm/xe/xe_module.c | 4 ++++ drivers/gpu/drm/xe/xe_module.h | 1 + drivers/gpu/drm/xe/xe_pagefault.c | 8 ++++---- drivers/gpu/drm/xe/xe_vm.c | 3 ++- 7 files changed, 28 insertions(+), 14 deletions(-)
drivers/gpu/drm/xe/xe_defaults.h +1 -0
@@ -22,5 +22,6 @@#define XE_DEFAULT_WEDGED_MODE XE_WEDGED_MODE_UPON_CRITICAL_ERROR#define XE_DEFAULT_WEDGED_MODE_STR "upon-critical-error"#define XE_DEFAULT_SVM_NOTIFIER_SIZE 512 +#define XE_DEFAULT_NUM_PF_WORK 2#endif
drivers/gpu/drm/xe/xe_device.c +12 -2
@@ -513,6 +513,17 @@ struct xe_device *xe_device_create(struct pci_dev *pdev) }ALLOW_ERROR_INJECTION (xe_device_create,ERRNO );/* See xe_pci_probe() */ +static void xe_device_parse_modparam (struct xe_device *xe) +{ + xe->atomic_svm_timeslice_ms =5 ; + xe->min_run_period_lr_ms =5 ; + xe->info.num_pf_work = xe_modparam.num_pf_work; +if (xe->info.num_pf_work <1 ) + xe->info.num_pf_work =1 ; +else if (xe->info.num_pf_work >XE_PAGEFAULT_WORK_MAX ) + xe->info.num_pf_work =XE_PAGEFAULT_WORK_MAX ; +} +/** * xe_device_init_early() - Initialize a new &xe_device instance * @xe: the &xe_device to initialize
@@ -539,8 +550,7 @@ int xe_device_init_early(struct xe_device *xe)if (err)return err; - xe->atomic_svm_timeslice_ms =5 ; - xe->min_run_period_lr_ms =5 ; +xe_device_parse_modparam (xe); err =xe_irq_init (xe);if (err)
drivers/gpu/drm/xe/xe_device_types.h +4 -7
@@ -123,6 +123,8 @@ struct xe_device {u8 revid;/** @info.step: stepping information for each IP */ struct xe_step_info step; +/** @info.num_pf_work: Number of page fault work thread */ +int num_pf_work;/** @info.dma_mask_size: DMA address bits */ u8 dma_mask_size;/** @info.vram_flags: Vram flags */
@@ -311,14 +313,9 @@ struct xe_device {struct workqueue_struct *pagefault_wq;/** @usm.prefetch_wq: threaded prefetch work queue, unbound */ struct workqueue_struct *prefetch_wq; -/* -* We pick 4 here because, in the current implementation, it -* yields the best bandwidth utilization of the kernel paging -* engine. -*/ -#define XE_PAGEFAULT_WORK_COUNT 4 +#define XE_PAGEFAULT_WORK_MAX 8/** @usm.pf_workers: Page fault workers */ -struct xe_pagefault_work pf_workers[XE_PAGEFAULT_WORK_COUNT ]; +struct xe_pagefault_work pf_workers[XE_PAGEFAULT_WORK_MAX ];/** @usm.pf_queue: Page fault queue */ struct xe_pagefault_queue pf_queue;#if IS_ENABLED (CONFIG_DRM_XE_PAGEMAP )
drivers/gpu/drm/xe/xe_module.c +4 -0
@@ -29,6 +29,7 @@ struct xe_modparam xe_modparam = { .max_vfs = XE_DEFAULT_MAX_VFS, #endif .wedged_mode = XE_DEFAULT_WEDGED_MODE, + .num_pf_work = XE_DEFAULT_NUM_PF_WORK, .svm_notifier_size = XE_DEFAULT_SVM_NOTIFIER_SIZE, /* the rest are 0 by default */ };
@@ -81,6 +82,9 @@ MODULE_PARM_DESC(wedged_mode,"Module's default policy for the wedged mode (0=never, 1=upon-critical-error, 2=upon-any-hang-no-reset " "[default=" XE_DEFAULT_WEDGED_MODE_STR "])"); +module_param_named (num_pf_work, xe_modparam.num_pf_work, uint,0600 ); +MODULE_PARM_DESC (num_pf_work,"Number of page fault work threads, default=2, min=1, max=8" ); +static int xe_check_nomodeset (void ) {if (drm_firmware_drivers_only ())
drivers/gpu/drm/xe/xe_module.h +1 -0
@@ -23,6 +23,7 @@ struct xe_modparam {unsigned int max_vfs;#endif unsigned int wedged_mode; +unsigned int num_pf_work;u32 svm_notifier_size; };
drivers/gpu/drm/xe/xe_pagefault.c +4 -4
@@ -398,13 +398,13 @@ int xe_pagefault_init(struct xe_device *xe) xe->usm.pagefault_wq =alloc_workqueue ("xe_page_fault_work_queue" ,WQ_UNBOUND |WQ_HIGHPRI , -XE_PAGEFAULT_WORK_COUNT ); + xe->info.num_pf_work);if (!xe->usm.pagefault_wq)return -ENOMEM ; xe->usm.prefetch_wq =alloc_workqueue ("xe_prefetch_work_queue" ,WQ_UNBOUND , -XE_PAGEFAULT_WORK_COUNT ); + xe->info.num_pf_work);if (!xe->usm.prefetch_wq) { err = -ENOMEM ; goto err_pagefault_wq;
@@ -414,7 +414,7 @@ int xe_pagefault_init(struct xe_device *xe)if (err) goto err_out; -for (i =0 ; i <XE_PAGEFAULT_WORK_COUNT ; ++i) { +for (i =0 ; i < xe->info.num_pf_work; ++i) {struct xe_pagefault_work *pf_work = xe->usm.pf_workers + i; pf_work->xe = xe;
@@ -482,7 +482,7 @@ static int xe_pagefault_work_index(struct xe_device *xe) {lockdep_assert_held (&xe->usm.pf_queue.lock); -return xe->usm.current_pf_work++ %XE_PAGEFAULT_WORK_COUNT ; +return xe->usm.current_pf_work++ % xe->info.num_pf_work; }/**
drivers/gpu/drm/xe/xe_vm.c +2 -1
@@ -3242,7 +3242,8 @@ static int prefetch_ranges(struct xe_vm *vm, struct xe_vma_ops *vops, skip_threads = op->prefetch_range.ranges_count ==1 || (!dpagemap && !(vops->flags &XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE )) || - !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ); + !(vops->flags &XE_VMA_OPS_FLAG_DOWNGRADE_LOCK ) || + vm->xe->info.num_pf_work ==1 ; thread = skip_threads ? &stack_thread :NULL ;if (!skip_threads) {
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 6/12 drm/xe: Engine class and instance into a u8
In reply to Matthew BrostPack the engine class and instance fields into a single u8 to save space in struct xe_pagefault. This also makes future extensions easier.
--- drivers/gpu/drm/xe/xe_guc_pagefault.c | 7 +++++-- drivers/gpu/drm/xe/xe_pagefault.c | 12 ++++++++---- drivers/gpu/drm/xe/xe_pagefault_types.h | 10 ++++++---- drivers/gpu/drm/xe/xe_vm.c | 7 +++++-- 4 files changed, 24 insertions(+), 12 deletions(-)
drivers/gpu/drm/xe/xe_guc_pagefault.c +5 -2
@@ -89,8 +89,11 @@ int xe_guc_pagefault_handler(struct xe_guc *guc, u32 *msg, u32 len) FIELD_GET(PFD_FAULT_LEVEL, msg[0])) |FIELD_PREP (XE_PAGEFAULT_TYPE_MASK ,FIELD_GET (PFD_FAULT_TYPE , msg[2 ])); - pf.consumer.engine_class =FIELD_GET (PFD_ENG_CLASS , msg[0 ]); - pf.consumer.engine_instance =FIELD_GET (PFD_ENG_INSTANCE , msg[0 ]); + pf.consumer.engine_class_instance = +FIELD_PREP (XE_PAGEFAULT_ENGINE_CLASS_MASK , +FIELD_GET (PFD_ENG_CLASS , msg[0 ])) | +FIELD_PREP (XE_PAGEFAULT_ENGINE_INSTANCE_MASK , +FIELD_GET (PFD_ENG_INSTANCE , msg[0 ])); pf.producer.private = guc; pf.producer.ops = &guc_pagefault_ops;
drivers/gpu/drm/xe/xe_pagefault.c +8 -4
@@ -239,13 +239,16 @@ static bool xe_pagefault_queue_pop(struct xe_pagefault_queue *pf_queue,static void xe_pagefault_print (struct xe_pagefault *pf) { +u8 engine_class =FIELD_GET (XE_PAGEFAULT_ENGINE_CLASS_MASK , + pf->consumer.engine_class_instance); + xe_gt_info(pf->gt, "\n\tASID: %d\n" "\tFaulted Address: 0x%08x%08x\n" "\tFaultType: %lu\n" "\tAccessType: %lu\n" "\tFaultLevel: %lu\n" "\tEngineClass: %d %s\n" - "\tEngineInstance: %d\n", + "\tEngineInstance: %lu\n", pf->consumer.asid, upper_32_bits(pf->consumer.page_addr), lower_32_bits(pf->consumer.page_addr),
@@ -255,9 +258,10 @@ static void xe_pagefault_print(struct xe_pagefault *pf) pf->consumer.access_type),FIELD_GET (XE_PAGEFAULT_LEVEL_MASK , pf->consumer.fault_type_level), - pf->consumer.engine_class, -xe_hw_engine_class_to_str (pf->consumer.engine_class), - pf->consumer.engine_instance); + engine_class, +xe_hw_engine_class_to_str (engine_class), +FIELD_GET (XE_PAGEFAULT_ENGINE_INSTANCE_MASK , + pf->consumer.engine_class_instance)); }static void xe_pagefault_save_to_vm (struct xe_device *xe,struct xe_pagefault *pf)
drivers/gpu/drm/xe/xe_pagefault_types.h +6 -4
@@ -82,10 +82,12 @@ struct xe_pagefault {#define XE_PAGEFAULT_TYPE_LEVEL_NACK 0xff/* Producer indicates nack fault */ #define XE_PAGEFAULT_LEVEL_MASK GENMASK(3, 0)#define XE_PAGEFAULT_TYPE_MASK GENMASK(7, 4) -/** @consumer.engine_class: engine class */ -u8 engine_class; -/** @consumer.engine_instance: engine instance */ -u8 engine_instance; +/** @consumer.engine_class_instance: engine class and instance */ +u8 engine_class_instance; +#define XE_PAGEFAULT_ENGINE_CLASS_MASK GENMASK(3, 0) +#define XE_PAGEFAULT_ENGINE_INSTANCE_MASK GENMASK(7, 4) +/** @pad: alignment padding */ +u8 pad;/** @consumer.reserved: reserved bits for future expansion */ u64 reserved; } consumer;
drivers/gpu/drm/xe/xe_vm.c +5 -2
@@ -615,10 +615,13 @@ void xe_vm_add_fault_entry_pf(struct xe_vm *vm, struct xe_pagefault *pf) {struct xe_vm_fault_entry *e;struct xe_hw_engine *hwe; +u8 engine_class =FIELD_GET (XE_PAGEFAULT_ENGINE_CLASS_MASK , + pf->consumer.engine_class_instance); +u8 engine_instance =FIELD_GET (XE_PAGEFAULT_ENGINE_INSTANCE_MASK , + pf->consumer.engine_class_instance);/* Do not report faults on reserved engines */ - hwe =xe_gt_hw_engine (pf->gt, pf->consumer.engine_class, - pf->consumer.engine_instance, false); + hwe =xe_gt_hw_engine (pf->gt, engine_class, engine_instance, false);if (!hwe ||xe_hw_engine_is_reserved (hwe))return ;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 7/12 drm/xe: Track pagefault worker runtime
In reply to Matthew BrostAdd a GT stat measuring total time spent servicing pagefault workqueue iterations. The counter accumulates the runtime of the pagefault worker in microseconds, allowing correlation of fault storms and chaining behavior with CPU time spent in the fault handler.
--- drivers/gpu/drm/xe/xe_gt_stats.c | 1 + drivers/gpu/drm/xe/xe_gt_stats_types.h | 3 +++ drivers/gpu/drm/xe/xe_pagefault.c | 8 ++++++++ 3 files changed, 12 insertions(+)
drivers/gpu/drm/xe/xe_gt_stats.c +1 -0
@@ -101,6 +101,7 @@ static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {DEF_STAT_STR (SVM_TLB_INVAL_US ,"svm_tlb_inval_us" ),DEF_STAT_STR (VMA_PAGEFAULT_COUNT ,"vma_pagefault_count" ),DEF_STAT_STR (VMA_PAGEFAULT_KB ,"vma_pagefault_kb" ), +DEF_STAT_STR (PAGEFAULT_US ,"pagefault_us" ),DEF_STAT_STR (INVALID_PREFETCH_PAGEFAULT_COUNT ,"invalid_prefetch_pagefault_count" ),DEF_STAT_STR (SVM_4K_PAGEFAULT_COUNT ,"svm_4K_pagefault_count" ), DEF_STAT_STR(SVM_64K_PAGEFAULT_COUNT, "svm_64K_pagefault_count"),
drivers/gpu/drm/xe/xe_gt_stats_types.h +3 -0
@@ -22,6 +22,8 @@* handled. * @XE_GT_STATS_ID_VMA_PAGEFAULT_KB: Size (KiB) of VMAs involved in * buffer-object page fault handling. +* @XE_GT_STATS_ID_PAGEFAULT_US: Cumulative time (µs) handling of all page +* faults. * @XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT: GPU prefetch faults for * addresses with no valid backing. *
@@ -133,6 +135,7 @@ enum xe_gt_stats_id {XE_GT_STATS_ID_SVM_TLB_INVAL_US ,XE_GT_STATS_ID_VMA_PAGEFAULT_COUNT ,XE_GT_STATS_ID_VMA_PAGEFAULT_KB , +XE_GT_STATS_ID_PAGEFAULT_US ,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,XE_GT_STATS_ID_SVM_4K_PAGEFAULT_COUNT , XE_GT_STATS_ID_SVM_64K_PAGEFAULT_COUNT,
drivers/gpu/drm/xe/xe_pagefault.c +8 -0
@@ -296,6 +296,8 @@ static void xe_pagefault_queue_work(struct work_struct *w)struct xe_device *xe = pf_work->xe;struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue;struct xe_pagefault pf; +ktime_t start =xe_gt_stats_ktime_get (); +struct xe_gt *gt =NULL ;unsigned long threshold;#define USM_QUEUE_MAX_RUNTIME_MS 20
@@ -307,6 +309,7 @@ static void xe_pagefault_queue_work(struct work_struct *w)if (!pf.gt)/* Fault squashed during reset */ continue ; + gt = pf.gt; err =xe_pagefault_service (&pf);if (err) {if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) {
@@ -329,6 +332,11 @@ static void xe_pagefault_queue_work(struct work_struct *w) } }#undef USM_QUEUE_MAX_RUNTIME_MS + +if (gt) +xe_gt_stats_incr (xe_root_mmio_gt (gt_to_xe (gt)), +XE_GT_STATS_ID_PAGEFAULT_US , +xe_gt_stats_ktime_us_delta (start)); } static int xe_pagefault_queue_init(struct xe_device *xe,
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 8/12 drm/xe: Chain page faults via queue-resident cache to avoid fault storms
In reply to Matthew BrostSome Xe platforms can generate pagefault storms where many faults target the same address range in a short time window (e.g. many EU threads faulting the same page). The current worker/locking model effectively serializes faults for a given range and repeatedly performs VMA/range lookups for each fault, which creates head-of-queue blocking and wastes CPU in the hot path.
Introduce a page fault chaining cache that coalesces faults targeting the samr ASID and address range.
Each worker tracks the active fault range it is servicing. Fault entries reside in stable queue storage, allowing the IRQ handler to match new faults against the worker cache and directly chain cache hits onto the active entry without allocation or waiting for dequeue. Once the leading fault completes, the worker acknowledges the entire chain.
A small allocation state is added to each entry so queue, worker, and IRQ pathd can safely reference the same fault object. This prevents reuse while the fault is active and guarantees that chained faults remain valid until acknowledged.
Fault handlers also record the serviced range so subsequent faults can be acknowledged without re-running the full resolution path.
This removes repeated fault resolution during fault storms and significantly improves forward progress in SVM workloads.
Since threaded prefetches now use a dedicated prefetch workqueue (usm.prefetch_wq) rather than sharing the page fault workqueue, page fault servicing can no longer deadlock on vm->lock, so this cache does not need an -EAGAIN retry path for a failed vm->lock acquisition.
--- drivers/gpu/drm/xe/xe_pagefault.c | 435 +++++++++++++++++++++--- drivers/gpu/drm/xe/xe_pagefault.h | 71 ++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 81 +++-- drivers/gpu/drm/xe/xe_svm.c | 16 +- drivers/gpu/drm/xe/xe_svm.h | 9 +- 5 files changed, 527 insertions(+), 85 deletions(-)
drivers/gpu/drm/xe/xe_pagefault.c +381 -54
@@ -35,6 +35,70 @@* xe_pagefault.c implements the consumer layer. */ +/** +* DOC: Xe page fault cache +* +* Some Xe hardware can trigger “fault storms,” which are many page faults to +* the same address within a short period of time. An example is many EU threads +* faulting on the same page simultaneously. With the current page fault locking +* structure, only one page fault for a given address range can be processed at +* a time. This causes head-of-queue blocking across workers, killing +* parallelism. If the page fault handler must repeatedly look up resources +* (VMAs, ranges) to determine that the pages are valid for each fault in the +* storm, the time complexity grows rapidly. +* +* To address this, each page fault worker maintains a cache of the active fault +* being processed. Subsequent faults that hit in the cache are chained to the +* pending fault, and all chained faults are acknowledged once the initial fault +* completes. This alleviates head-of-queue blocking and quickly chains faults +* in the upper layers, avoiding expensive lookups in the main fault-handling +* path. +* +* Faults are buffered in the page fault queue in a way that provides stable +* storage for outstanding faults. In particular, faults may be chained directly +* while still resident in the queue storage (i.e., outside the worker’s current +* head/tail dequeue position). This allows the IRQ handler to match newly +* arrived faults against the per-worker cache and immediately chain cache hits +* onto the active fault under the queue lock, without allocating memory or +* waiting for the worker to pop the fault first. +* +* A per-fault state field is used to assert correctness of these invariants. +* The state tracks whether an entry is free, queued, chained, or currently +* active. Transitions are performed under the page fault queue lock, and the +* worker acknowledges faults by walking the chain and returning entries to the +* free state once they are complete. +*/ + +/** +* enum xe_pagefault_alloc_state - lifetime state for a page fault queue entry +* @XE_PAGEFAULT_ALLOC_STATE_FREE: +* Entry is unused and may be overwritten by the producer, consumer retry +* or requeue.. +* @XE_PAGEFAULT_ALLOC_STATE_QUEUED: +* Entry has been enqueued and may be dequeued by a worker. +* @XE_PAGEFAULT_ALLOC_STATE_ACTIVE: +* Entry has been dequeued and is the worker's currently serviced fault. +* The worker may attach additional faults to it via consumer.next. +* @XE_PAGEFAULT_ALLOC_STATE_CHAINED: +* Entry is not independently serviced; it has been chained onto an +* ACTIVE entry via consumer.next and will be acknowledged when the +* leading fault completes. +* +* The page fault queue provides stable storage for outstanding faults so the +* IRQ handler can chain new cache hits directly onto a worker's active fault. +* Because entries may remain referenced outside the consumer dequeue window, +* the producer must only write into entries in the FREE state. +* +* State transitions are protected by the page fault queue lock. Workers return +* entries to FREE after acknowledging the fault (either as ACTIVE or CHAINED). +*/ +enum xe_pagefault_alloc_state { +XE_PAGEFAULT_ALLOC_STATE_FREE =0 , +XE_PAGEFAULT_ALLOC_STATE_QUEUED =1 , +XE_PAGEFAULT_ALLOC_STATE_CHAINED =2 , +XE_PAGEFAULT_ALLOC_STATE_ACTIVE =3 , +}; +static int xe_pagefault_entry_size (void ) {/*
@@ -77,7 +141,7 @@ static int xe_pagefault_begin(struct drm_exec *exec, struct xe_vma *vma, }static int xe_pagefault_handle_vma (struct xe_gt *gt,struct xe_vma *vma, -bool atomic) +struct xe_pagefault *pf,bool atomic) {struct xe_vm *vm =xe_vma_vm (vma);struct xe_tile *tile =gt_to_tile (gt);
@@ -102,8 +166,11 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma,/* Check if VMA is valid, opportunistic check only */ if (xe_vm_has_valid_gpu_mapping (tile, vma->tile_present, - vma->tile_invalidated) && !atomic) + vma->tile_invalidated) && !atomic) { +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma));return 0 ; + } do { if (xe_vma_is_userptr(vma) &&
@@ -141,6 +208,10 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma, }while (err == -EAGAIN );if (!err) { +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma)); +dma_fence_wait (fence, false);dma_fence_put (fence); }
@@ -208,10 +279,10 @@ static int xe_pagefault_service(struct xe_pagefault *pf) atomic =xe_pagefault_access_is_atomic (pf->consumer.access_type);if (xe_vma_is_cpu_addr_mirror (vma)) - err =xe_svm_handle_pagefault (vm, vma, gt, + err =xe_svm_handle_pagefault (vm, vma, pf, gt, pf->consumer.page_addr, atomic);else - err =xe_pagefault_handle_vma (gt, vma, atomic); + err =xe_pagefault_handle_vma (gt, vma, pf, atomic); unlock_vm:up_read (&vm->lock);
@@ -220,21 +291,221 @@ static int xe_pagefault_service(struct xe_pagefault *pf)return err; } -static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, -struct xe_pagefault *pf) +#define XE_PAGEFAULT_CACHE_START_INVALID U64_MAX +#define xe_pagefault_cache_start_invalidate (val) \ + (val = XE_PAGEFAULT_CACHE_START_INVALID) + +static void +xe_pagefault_cache_invalidate (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault_work *pf_work) { -bool found_fault = false; +lockdep_assert_held (&pf_queue->lock); + +xe_pagefault_cache_start_invalidate (pf_work->cache.start); +} + +static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return CIRC_SPACE (pf_queue->head, pf_queue->tail, + pf_queue->size) <=xe_pagefault_entry_size (); +} + +static struct xe_pagefault * +xe_pagefault_queue_add (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *lpf; + +lockdep_assert_held (&pf_queue->lock); + +do { +/* Not possible, warn on and drop page fault */ +if (WARN_ON (xe_pagefault_queue_full (pf_queue))) +return NULL ; -spin_lock_irq (&pf_queue->lock); -if (pf_queue->tail != pf_queue->head) { -memcpy (pf, pf_queue->data + pf_queue->tail,sizeof (*pf)); - pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + lpf = (pf_queue->data + pf_queue->head); + pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; - found_fault = true; + }while (lpf->consumer.alloc_state !=XE_PAGEFAULT_ALLOC_STATE_FREE ); + +xe_assert (xe, lpf != pf); +memcpy (lpf, pf,sizeof (*pf)); + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_QUEUED ; + +return lpf; +} + +static struct xe_pagefault * +xe_pagefault_queue_unchain_requeue (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf,struct xe_gt *gt) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *next = pf->consumer.next, *lpf; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_CHAINED ); + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + lpf->consumer.fault_type_level |=XE_PAGEFAULT_REQUEUE_MASK ; + } + +return next; +} + +static bool xe_pagefault_match (struct xe_pagefault *pf,u64 start, +u64 end,u64 cache_asid) +{ +struct xe_device *xe =gt_to_xe (pf->gt); +u64 page_addr = pf->consumer.page_addr; +u32 pf_asid = pf->consumer.asid; + +xe_assert (xe, pf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_FREE ); + +return page_addr >= start && page_addr < end && + pf_asid == cache_asid; +} + +static bool xe_pagefault_try_chain (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +bool requeue =FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + pf->consumer.fault_type_level); +int i; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_QUEUED ); + +/* +* If this is a retry, we may already have a chain attached. In that +* case, we cannot hit in the cache because chains cannot easily be +* combined. +*/ +if (pf->consumer.next) +return false; + +for (i =0 , pf_work = xe->usm.pf_workers; + i < xe->info.num_pf_work; ++i, ++pf_work) { +u64 start = pf_work->cache.start; +u64 end = requeue ? start +SZ_4K : pf_work->cache.end; +u32 asid = pf_work->cache.asid; + +if (xe_pagefault_match (pf, start, end, asid)) { +xe_assert (xe, pf_work->cache.pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_CHAINED ; + pf->consumer.next = pf_work->cache.pf->consumer.next; + pf_work->cache.pf->consumer.next = pf; + +return true; + } + } + +return false; +} + +static void xe_pagefault_queue_advance (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + + pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + pf_queue->size; +} + +static struct xe_pagefault * +xe_pagefault_queue_tail_fault (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->data + pf_queue->tail; +} + +static bool xe_pagefault_queue_empty (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->head == pf_queue->tail; +} + +static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault **pf,int id) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +struct xe_pagefault *lpf; +size_t align =SZ_2M ; + +guard (spinlock_irq)(&pf_queue->lock); + +for (*pf =NULL ; !*pf;) { +if (xe_pagefault_queue_empty (pf_queue)) +return false; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); +xe_pagefault_queue_advance (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) +continue ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) +continue ; + + *pf = lpf;/* Hand back page fault for processing */ + } + +/* +* No cache hit; allocate a new cache entry. We assume most faults +* within a 2M range will hit the same pages. If this assumption proves +* false, the mismatched fault is requeued after the initial fault is +* acknowledged. +*/ + pf_work = xe->usm.pf_workers + id; +if (FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + lpf->consumer.fault_type_level)) + align =SZ_4K ; + pf_work->cache.start =ALIGN_DOWN (lpf->consumer.page_addr, align); + pf_work->cache.end = pf_work->cache.start + align; + pf_work->cache.asid = lpf->consumer.asid; + pf_work->cache.pf = lpf; + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; + +/* Drain queue until empty or new fault found */ +while (1 ) { +if (xe_pagefault_queue_empty (pf_queue)) +break ; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) { +xe_pagefault_queue_advance (pf_queue); +continue ; + } + +if (!xe_pagefault_try_chain (pf_queue, lpf)) +break ; + +xe_pagefault_queue_advance (pf_queue); } -spin_unlock_irq (&pf_queue->lock); -return found_fault; +return true; }static void xe_pagefault_print (struct xe_pagefault *pf)
@@ -295,36 +566,83 @@ static void xe_pagefault_queue_work(struct work_struct *w)container_of (w,typeof (*pf_work), work);struct xe_device *xe = pf_work->xe;struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue; -struct xe_pagefault pf; +struct xe_pagefault *pf;ktime_t start =xe_gt_stats_ktime_get (); -struct xe_gt *gt =NULL ;unsigned long threshold; +u64 cache_start =XE_PAGEFAULT_CACHE_START_INVALID , cache_end =0 ; +u32 cache_asid =0 ;#define USM_QUEUE_MAX_RUNTIME_MS 20 threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS ); -while (xe_pagefault_queue_pop (pf_queue, &pf)) { -int err; +while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +struct xe_gt *gt = pf->gt; +u32 asid = pf->consumer.asid; +int err =0 ; +bool invalidated = false; -if (!pf.gt)/* Fault squashed during reset */ -continue ; +/* Last fault same address, ack immediately */ +if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) + goto ack_fault; + + err =xe_pagefault_service (pf); - gt = pf.gt; - err =xe_pagefault_service (&pf);if (err) { -if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf); -xe_pagefault_print (&pf); -xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" , +if (!(pf->consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (gt), pf); +xe_pagefault_cache_start_invalidate (cache_start); +xe_pagefault_print (pf); +xe_gt_info (pf->gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); }else { -xe_gt_stats_incr (pf.gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); -xe_gt_dbg (pf.gt,"Prefetch Fault response: Unsuccessful %pe\n" , +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); +xe_gt_dbg (pf->gt,"Prefetch Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); } + }else { +/* Cache valid fault locally */ + cache_start =xe_pagefault_start_addr (pf); + cache_end =xe_pagefault_end_addr (pf); + cache_asid = asid; } - pf.producer.ops->ack_fault (&pf, err); +ack_fault: +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, pf == pf_work->cache.pf); + +while (pf) { +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->producer.ops->ack_fault (pf, err); + +spin_lock_irq (&pf_queue->lock); + +if (!invalidated) { + invalidated = true; +xe_pagefault_cache_invalidate (pf_queue, + pf_work); + } + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf = pf->consumer.next; + +/* +* Requeue chained faults which do not match the last +* fault processed +*/ +while (pf && !xe_pagefault_match (pf, cache_start, + cache_end, cache_asid)) + pf =xe_pagefault_queue_unchain_requeue (pf_queue, pf, gt); + + +/* Ensure resets are safe */ +if (pf) + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; +spin_unlock_irq (&pf_queue->lock); + }if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);
@@ -333,10 +651,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) }#undef USM_QUEUE_MAX_RUNTIME_MS -if (gt) -xe_gt_stats_incr (xe_root_mmio_gt (gt_to_xe (gt)), -XE_GT_STATS_ID_PAGEFAULT_US , -xe_gt_stats_ktime_us_delta (start)); +xe_gt_stats_incr (xe_root_mmio_gt (xe),XE_GT_STATS_ID_PAGEFAULT_US , +xe_gt_stats_ktime_us_delta (start)); } static int xe_pagefault_queue_init(struct xe_device *xe,
@@ -431,6 +747,7 @@ int xe_pagefault_init(struct xe_device *xe) pf_work->xe = xe; pf_work->id = i; +xe_pagefault_cache_start_invalidate (pf_work->cache.start);INIT_WORK (&pf_work->work, xe_pagefault_queue_work); }
@@ -454,15 +771,23 @@ static void xe_pagefault_queue_reset(struct xe_device *xe, struct xe_gt *gt,/* Squash all pending faults on the GT */ -spin_lock_irq (&pf_queue->lock); -for (i = pf_queue->tail; i != pf_queue->head; - i = (i +xe_pagefault_entry_size ()) % pf_queue->size) { +guard (spinlock_irq)(&pf_queue->lock); + +for (i =0 ; i < pf_queue->size; i +=xe_pagefault_entry_size ()) {struct xe_pagefault *pf = pf_queue->data + i; +bool active = pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; -if (pf->gt == gt) - pf->gt =NULL ; +if (pf->gt != gt || active) { +if (active) + pf->consumer.next =NULL ; +continue ; + } + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf->consumer.next =NULL ; } -spin_unlock_irq (&pf_queue->lock); }/**
@@ -478,14 +803,6 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt)xe_pagefault_queue_reset (xe, gt, &xe->usm.pf_queue); } -static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) -{ -lockdep_assert_held (&pf_queue->lock); - -return CIRC_SPACE (pf_queue->head, pf_queue->tail, pf_queue->size) <= -xe_pagefault_entry_size (); -} -/* * This function can race with multiple page fault producers, but worst case we * stick a page fault on the same queue for consumption.
@@ -511,18 +828,28 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf) {struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue;unsigned long flags; -int work_index;bool full;spin_lock_irqsave (&pf_queue->lock, flags); - work_index =xe_pagefault_work_index (xe); full =xe_pagefault_queue_full (pf_queue);if (!full) { -memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); - pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % - pf_queue->size; -queue_work (xe->usm.pagefault_wq, - &xe->usm.pf_workers[work_index].work); +struct xe_pagefault *lpf; +bool empty =xe_pagefault_queue_empty (pf_queue); + + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) { +if (empty) +xe_pagefault_queue_advance (pf_queue); + }else { +int work_index =xe_pagefault_work_index (xe); + +queue_work (xe->usm.pagefault_wq, + &xe->usm.pf_workers[work_index].work); + } + } }else {drm_warn (&xe->drm,"PageFault Queue full, shouldn't be possible\n" );
drivers/gpu/drm/xe/xe_pagefault.h +71 -0
81 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pagefault_types.h +59 -22
108 lines not on this page · raw message
drivers/gpu/drm/xe/xe_svm.c +11 -5
@@ -15,6 +15,7 @@#include "xe_gt_stats.h" #include "xe_migrate.h" #include "xe_module.h" +#include "xe_pagefault.h" #include "xe_pm.h" #include "xe_pt.h" #include "xe_svm.h"
@@ -1264,8 +1265,8 @@ DECL_SVM_RANGE_US_STATS(bind, BIND) DECL_SVM_RANGE_US_STATS(fault, PAGEFAULT)static int __xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool need_vram) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool need_vram) {int devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );
@@ -1424,6 +1425,10 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,xe_svm_range_bind_us_stats_incr (gt, range, bind_start); out: +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_svm_range_start (range)); +xe_pagefault_set_end_addr (pf,xe_svm_range_end (range)); +xe_svm_range_fault_us_stats_incr (gt, range, start);mutex_unlock (&range->lock);drm_gpusvm_range_put (&range->base);
@@ -1446,6 +1451,7 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* xe_svm_handle_pagefault() - SVM handle page fault * @vm: The VM. * @vma: The CPU address mirror VMA. +* @pf: Pagefault structure * @gt: The gt upon the fault occurred. * @fault_addr: The GPU fault address. * @atomic: The fault atomic access bit.
@@ -1456,8 +1462,8 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* Return: 0 on success, negative error code on error. */ int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {int need_vram, ret; retry:
@@ -1465,7 +1471,7 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,if (need_vram <0 )return need_vram; - ret =__xe_svm_handle_pagefault (vm, vma, gt, fault_addr, + ret =__xe_svm_handle_pagefault (vm, vma, pf, gt, fault_addr, need_vram ? true : false);if (ret == -EAGAIN ) {/*
drivers/gpu/drm/xe/xe_svm.h +5 -4
@@ -21,6 +21,7 @@ struct drm_file;struct xe_bo ;struct xe_gt ;struct xe_device ; +struct xe_pagefault ;struct xe_vram_region ;struct xe_tile ;struct xe_vm ;
@@ -109,8 +110,8 @@ void xe_svm_fini(struct xe_vm *vm);void xe_svm_close (struct xe_vm *vm);int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic); +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic);bool xe_svm_has_mapping (struct xe_vm *vm,u64 start,u64 end);
@@ -298,8 +299,8 @@ void xe_svm_close(struct xe_vm *vm)static inline int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v8 08/12] drm/xe: Chain page faults via queue-resident cache to avoid fault storms
Hi,
Patch size makes review a bit longer but I cannot suggest a good way to break it down.
Quoted 9 lines · Matthew Brost
Some Xe platforms can generate pagefault storms where many faults target the same address range in a short time window (e.g. many EU threads faulting the same page). The current worker/locking model effectively serializes faults for a given range and repeatedly performs VMA/range lookups for each fault, which creates head-of-queue blocking and wastes CPU in the hot path.
Introduce a page fault chaining cache that coalesces faults targeting the samr ASID and address range.
s/samr/same/
Quoted 9 lines
Each worker tracks the active fault range it is servicing. Fault entries reside in stable queue storage, allowing the IRQ handler to match new faults against the worker cache and directly chain cache hits onto the active entry without allocation or waiting for dequeue. Once the leading fault completes, the worker acknowledges the entire chain.
A small allocation state is added to each entry so queue, worker, and IRQ pathd can safely reference the same fault object. This prevents
s/pathd/paths/
Quoted 9 lines
reuse while the fault is active and guarantees that chained faults remain valid until acknowledged.
Fault handlers also record the serviced range so subsequent faults can be acknowledged without re-running the full resolution path.
This removes repeated fault resolution during fault storms and significantly improves forward progress in SVM workloads.
Quoted 8 lines
Since threaded prefetches now use a dedicated prefetch workqueue (usm.prefetch_wq) rather than sharing the page fault workqueue, page fault servicing can no longer deadlock on
vm->lock, so this cache does not need an-EAGAINretry path for a failedvm->lockacquisition.Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>
Please remove this "Co-authored-by".
(Already mentioned in patch #3 but repeating here not to forget)
Quoted 214 lines
--- drivers/gpu/drm/xe/xe_pagefault.c | 435 +++++++++++++++++++++--- drivers/gpu/drm/xe/xe_pagefault.h | 71 ++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 81 +++-- drivers/gpu/drm/xe/xe_svm.c | 16 +- drivers/gpu/drm/xe/xe_svm.h | 9 +- 5 files changed, 527 insertions(+), 85 deletions(-)drivers/gpu/drm/xe/xe_pagefault.c +145
-12@@ -35,6 +35,70 @@* xe_pagefault.c implements the consumer layer. */ +/** +* DOC: Xe page fault cache +* +* Some Xe hardware can trigger “fault storms,” which are many page faults to +* the same address within a short period of time. An example is many EU threads +* faulting on the same page simultaneously. With the current page fault locking +* structure, only one page fault for a given address range can be processed at +* a time. This causes head-of-queue blocking across workers, killing +* parallelism. If the page fault handler must repeatedly look up resources +* (VMAs, ranges) to determine that the pages are valid for each fault in the +* storm, the time complexity grows rapidly. +* +* To address this, each page fault worker maintains a cache of the active fault +* being processed. Subsequent faults that hit in the cache are chained to the +* pending fault, and all chained faults are acknowledged once the initial fault +* completes. This alleviates head-of-queue blocking and quickly chains faults +* in the upper layers, avoiding expensive lookups in the main fault-handling +* path. +* +* Faults are buffered in the page fault queue in a way that provides stable +* storage for outstanding faults. In particular, faults may be chained directly +* while still resident in the queue storage (i.e., outside the worker’s current +* head/tail dequeue position). This allows the IRQ handler to match newly +* arrived faults against the per-worker cache and immediately chain cache hits +* onto the active fault under the queue lock, without allocating memory or +* waiting for the worker to pop the fault first. +* +* A per-fault state field is used to assert correctness of these invariants. +* The state tracks whether an entry is free, queued, chained, or currently +* active. Transitions are performed under the page fault queue lock, and the +* worker acknowledges faults by walking the chain and returning entries to the +* free state once they are complete. +*/ + +/** +* enum xe_pagefault_alloc_state - lifetime state for a page fault queue entry +* @XE_PAGEFAULT_ALLOC_STATE_FREE: +* Entry is unused and may be overwritten by the producer, consumer retry +* or requeue.. +* @XE_PAGEFAULT_ALLOC_STATE_QUEUED: +* Entry has been enqueued and may be dequeued by a worker. +* @XE_PAGEFAULT_ALLOC_STATE_ACTIVE: +* Entry has been dequeued and is the worker's currently serviced fault. +* The worker may attach additional faults to it via consumer.next. +* @XE_PAGEFAULT_ALLOC_STATE_CHAINED: +* Entry is not independently serviced; it has been chained onto an +* ACTIVE entry via consumer.next and will be acknowledged when the +* leading fault completes. +* +* The page fault queue provides stable storage for outstanding faults so the +* IRQ handler can chain new cache hits directly onto a worker's active fault. +* Because entries may remain referenced outside the consumer dequeue window, +* the producer must only write into entries in the FREE state. +* +* State transitions are protected by the page fault queue lock. Workers return +* entries to FREE after acknowledging the fault (either as ACTIVE or CHAINED). +*/ +enum xe_pagefault_alloc_state { +XE_PAGEFAULT_ALLOC_STATE_FREE =0 , +XE_PAGEFAULT_ALLOC_STATE_QUEUED =1 , +XE_PAGEFAULT_ALLOC_STATE_CHAINED =2 , +XE_PAGEFAULT_ALLOC_STATE_ACTIVE =3 , +}; +static int xe_pagefault_entry_size (void ) {/* @@ -77,7 +141,7 @@ static int xe_pagefault_begin(struct drm_exec *exec, struct xe_vma *vma, }static int xe_pagefault_handle_vma (struct xe_gt *gt,struct xe_vma *vma, -bool atomic) +struct xe_pagefault *pf,bool atomic) {struct xe_vm *vm =xe_vma_vm (vma);struct xe_tile *tile =gt_to_tile (gt);@@ -102,8 +166,11 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma,/* Check if VMA is valid, opportunistic check only */ if (xe_vm_has_valid_gpu_mapping (tile, vma->tile_present, - vma->tile_invalidated) && !atomic) + vma->tile_invalidated) && !atomic) { +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma));return 0 ; + } do { if (xe_vma_is_userptr(vma) &&@@ -141,6 +208,10 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma, }while (err == -EAGAIN );if (!err) { +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma)); +dma_fence_wait (fence, false);dma_fence_put (fence); }@@ -208,10 +279,10 @@ static int xe_pagefault_service(struct xe_pagefault *pf) atomic =xe_pagefault_access_is_atomic (pf->consumer.access_type);if (xe_vma_is_cpu_addr_mirror (vma)) - err =xe_svm_handle_pagefault (vm, vma, gt, + err =xe_svm_handle_pagefault (vm, vma, pf, gt, pf->consumer.page_addr, atomic);else - err =xe_pagefault_handle_vma (gt, vma, atomic); + err =xe_pagefault_handle_vma (gt, vma, pf, atomic); unlock_vm:up_read (&vm->lock);@@ -220,21 +291,221 @@ static int xe_pagefault_service(struct xe_pagefault *pf)return err; } -static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, -struct xe_pagefault *pf) +#define XE_PAGEFAULT_CACHE_START_INVALID U64_MAX +#define xe_pagefault_cache_start_invalidate (val) \ + (val = XE_PAGEFAULT_CACHE_START_INVALID) + +static void +xe_pagefault_cache_invalidate (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault_work *pf_work) { -bool found_fault = false; +lockdep_assert_held (&pf_queue->lock); + +xe_pagefault_cache_start_invalidate (pf_work->cache.start); +} + +static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return CIRC_SPACE (pf_queue->head, pf_queue->tail, + pf_queue->size) <=xe_pagefault_entry_size (); +} + +static struct xe_pagefault * +xe_pagefault_queue_add (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *lpf; + +lockdep_assert_held (&pf_queue->lock); + +do { +/* Not possible, warn on and drop page fault */ +if (WARN_ON (xe_pagefault_queue_full (pf_queue))) +return NULL ; -spin_lock_irq (&pf_queue->lock); -if (pf_queue->tail != pf_queue->head) { -memcpy (pf, pf_queue->data + pf_queue->tail,sizeof (*pf)); - pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + lpf = (pf_queue->data + pf_queue->head); + pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; - found_fault = true; + }while (lpf->consumer.alloc_state !=XE_PAGEFAULT_ALLOC_STATE_FREE ); + +xe_assert (xe, lpf != pf); +memcpy (lpf, pf,sizeof (*pf)); + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_QUEUED ; + +return lpf; +} + +static struct xe_pagefault * +xe_pagefault_queue_unchain_requeue (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf,struct xe_gt *gt) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *next = pf->consumer.next, *lpf; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_CHAINED ); + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + lpf->consumer.fault_type_level |=XE_PAGEFAULT_REQUEUE_MASK ; + } + +return next; +} + +static bool xe_pagefault_match (struct xe_pagefault *pf,u64 start, +u64 end,u64 cache_asid) +{
drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 376
Quoted 10 lines
drivers/gpu/drm/xe/xe_pagefault.c +10
-0+struct xe_device *xe =gt_to_xe (pf->gt); +u64 page_addr = pf->consumer.page_addr; +u32 pf_asid = pf->consumer.asid; + +xe_assert (xe, pf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_FREE ); + +return page_addr >= start && page_addr < end && + pf_asid == cache_asid; +}
Sashiko's comment here seems valid: access type should be taken into consideration when evaluating if the page faults match (atomic).
Quoted 217 lines
drivers/gpu/drm/xe/xe_pagefault.c +176
-17+ +static bool xe_pagefault_try_chain (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +bool requeue =FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + pf->consumer.fault_type_level); +int i; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_QUEUED ); + +/* +* If this is a retry, we may already have a chain attached. In that +* case, we cannot hit in the cache because chains cannot easily be +* combined. +*/ +if (pf->consumer.next) +return false; + +for (i =0 , pf_work = xe->usm.pf_workers; + i < xe->info.num_pf_work; ++i, ++pf_work) { +u64 start = pf_work->cache.start; +u64 end = requeue ? start +SZ_4K : pf_work->cache.end; +u32 asid = pf_work->cache.asid; + +if (xe_pagefault_match (pf, start, end, asid)) { +xe_assert (xe, pf_work->cache.pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_CHAINED ; + pf->consumer.next = pf_work->cache.pf->consumer.next; + pf_work->cache.pf->consumer.next = pf; + +return true; + } + } + +return false; +} + +static void xe_pagefault_queue_advance (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + + pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + pf_queue->size; +} + +static struct xe_pagefault * +xe_pagefault_queue_tail_fault (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->data + pf_queue->tail; +} + +static bool xe_pagefault_queue_empty (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->head == pf_queue->tail; +} + +static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault **pf,int id) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +struct xe_pagefault *lpf; +size_t align =SZ_2M ; + +guard (spinlock_irq)(&pf_queue->lock); + +for (*pf =NULL ; !*pf;) { +if (xe_pagefault_queue_empty (pf_queue)) +return false; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); +xe_pagefault_queue_advance (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) +continue ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) +continue ; + + *pf = lpf;/* Hand back page fault for processing */ + } + +/* +* No cache hit; allocate a new cache entry. We assume most faults +* within a 2M range will hit the same pages. If this assumption proves +* false, the mismatched fault is requeued after the initial fault is +* acknowledged. +*/ + pf_work = xe->usm.pf_workers + id; +if (FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + lpf->consumer.fault_type_level)) + align =SZ_4K ; + pf_work->cache.start =ALIGN_DOWN (lpf->consumer.page_addr, align); + pf_work->cache.end = pf_work->cache.start + align; + pf_work->cache.asid = lpf->consumer.asid; + pf_work->cache.pf = lpf; + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; + +/* Drain queue until empty or new fault found */ +while (1 ) { +if (xe_pagefault_queue_empty (pf_queue)) +break ; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) { +xe_pagefault_queue_advance (pf_queue); +continue ; + } + +if (!xe_pagefault_try_chain (pf_queue, lpf)) +break ; + +xe_pagefault_queue_advance (pf_queue); } -spin_unlock_irq (&pf_queue->lock); -return found_fault; +return true; }static void xe_pagefault_print (struct xe_pagefault *pf)@@ -295,36 +566,83 @@ static void xe_pagefault_queue_work(struct work_struct *w)container_of (w,typeof (*pf_work), work);struct xe_device *xe = pf_work->xe;struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue; -struct xe_pagefault pf; +struct xe_pagefault *pf;ktime_t start =xe_gt_stats_ktime_get (); -struct xe_gt *gt =NULL ;unsigned long threshold; +u64 cache_start =XE_PAGEFAULT_CACHE_START_INVALID , cache_end =0 ; +u32 cache_asid =0 ;#define USM_QUEUE_MAX_RUNTIME_MS 20 threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS ); -while (xe_pagefault_queue_pop (pf_queue, &pf)) { -int err; +while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +struct xe_gt *gt = pf->gt; +u32 asid = pf->consumer.asid; +int err =0 ; +bool invalidated = false; -if (!pf.gt)/* Fault squashed during reset */ -continue ; +/* Last fault same address, ack immediately */ +if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) + goto ack_fault; + + err =xe_pagefault_service (pf); - gt = pf.gt; - err =xe_pagefault_service (&pf);if (err) { -if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf); -xe_pagefault_print (&pf); -xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" , +if (!(pf->consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (gt), pf); +xe_pagefault_cache_start_invalidate (cache_start); +xe_pagefault_print (pf); +xe_gt_info (pf->gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); }else { -xe_gt_stats_incr (pf.gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); -xe_gt_dbg (pf.gt,"Prefetch Fault response: Unsuccessful %pe\n" , +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); +xe_gt_dbg (pf->gt,"Prefetch Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); } + }else { +/* Cache valid fault locally */ + cache_start =xe_pagefault_start_addr (pf); + cache_end =xe_pagefault_end_addr (pf); + cache_asid = asid; } - pf.producer.ops->ack_fault (&pf, err); +ack_fault: +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, pf == pf_work->cache.pf); + +while (pf) { +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->producer.ops->ack_fault (pf, err); + +spin_lock_irq (&pf_queue->lock); + +if (!invalidated) { + invalidated = true; +xe_pagefault_cache_invalidate (pf_queue, + pf_work); + } + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf = pf->consumer.next;
drivers/gpu/drm/xe/xe_pagefault.c static void xe_pagefault_queue_work(struct work_struct *w) · line 639
Quoted 10 lines
drivers/gpu/drm/xe/xe_pagefault.c +10
-0+ +/* +* Requeue chained faults which do not match the last +* fault processed +*/ +while (pf && !xe_pagefault_match (pf, cache_start, + cache_end, cache_asid)) + pf =xe_pagefault_queue_unchain_requeue (pf_queue, pf, gt); + +
Extra blank line.
Quoted 337 lines
drivers/gpu/drm/xe/xe_pagefault.c +40
-25+/* Ensure resets are safe */ +if (pf) + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; +spin_unlock_irq (&pf_queue->lock); + }if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);@@ -333,10 +651,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) }#undef USM_QUEUE_MAX_RUNTIME_MS -if (gt) -xe_gt_stats_incr (xe_root_mmio_gt (gt_to_xe (gt)), -XE_GT_STATS_ID_PAGEFAULT_US , -xe_gt_stats_ktime_us_delta (start)); +xe_gt_stats_incr (xe_root_mmio_gt (xe),XE_GT_STATS_ID_PAGEFAULT_US , +xe_gt_stats_ktime_us_delta (start)); } static int xe_pagefault_queue_init(struct xe_device *xe,@@ -431,6 +747,7 @@ int xe_pagefault_init(struct xe_device *xe) pf_work->xe = xe; pf_work->id = i; +xe_pagefault_cache_start_invalidate (pf_work->cache.start);INIT_WORK (&pf_work->work, xe_pagefault_queue_work); }@@ -454,15 +771,23 @@ static void xe_pagefault_queue_reset(struct xe_device *xe, struct xe_gt *gt,/* Squash all pending faults on the GT */ -spin_lock_irq (&pf_queue->lock); -for (i = pf_queue->tail; i != pf_queue->head; - i = (i +xe_pagefault_entry_size ()) % pf_queue->size) { +guard (spinlock_irq)(&pf_queue->lock); + +for (i =0 ; i < pf_queue->size; i +=xe_pagefault_entry_size ()) {struct xe_pagefault *pf = pf_queue->data + i; +bool active = pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; -if (pf->gt == gt) - pf->gt =NULL ; +if (pf->gt != gt || active) { +if (active) + pf->consumer.next =NULL ; +continue ; + } + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf->consumer.next =NULL ; } -spin_unlock_irq (&pf_queue->lock); }/** @@ -478,14 +803,6 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt)xe_pagefault_queue_reset (xe, gt, &xe->usm.pf_queue); } -static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) -{ -lockdep_assert_held (&pf_queue->lock); - -return CIRC_SPACE (pf_queue->head, pf_queue->tail, pf_queue->size) <= -xe_pagefault_entry_size (); -} -/* * This function can race with multiple page fault producers, but worst case we * stick a page fault on the same queue for consumption. @@ -511,18 +828,28 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf) {struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue;unsigned long flags; -int work_index;bool full;spin_lock_irqsave (&pf_queue->lock, flags); - work_index =xe_pagefault_work_index (xe); full =xe_pagefault_queue_full (pf_queue);if (!full) { -memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); - pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % - pf_queue->size; -queue_work (xe->usm.pagefault_wq, - &xe->usm.pf_workers[work_index].work); +struct xe_pagefault *lpf; +bool empty =xe_pagefault_queue_empty (pf_queue); + + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) { +if (empty) +xe_pagefault_queue_advance (pf_queue); + }else { +int work_index =xe_pagefault_work_index (xe); + +queue_work (xe->usm.pagefault_wq, + &xe->usm.pf_workers[work_index].work); + } + } }else {drm_warn (&xe->drm,"PageFault Queue full, shouldn't be possible\n" );drivers/gpu/drm/xe/xe_pagefault.h +71
-0@@ -6,6 +6,8 @@#ifndef _XE_PAGEFAULT_H_#define _XE_PAGEFAULT_H_ +#include "xe_pagefault_types.h" +struct xe_device ;struct xe_gt ;struct xe_pagefault ;@@ -16,4 +18,73 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt);int xe_pagefault_handler (struct xe_device *xe,struct xe_pagefault *pf); +#define XE_PAGEFAULT_END_ADDR_MASK (~0xfffull) + +/** +* xe_pagefault_set_end_addr() - store serviced range end for a pagefault +* @pf: Pagefault entry +* @end_addr: Inclusive end address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +* +* The end address shares storage with other consumer metadata and therefore +* must be masked with %XE_PAGEFAULT_END_ADDR_MASK before storing. Bits outside +* the mask are reserved for internal state tracking and must be preserved. +*/ +static inline void +xe_pagefault_set_end_addr (struct xe_pagefault *pf,u64 end_addr) +{ + pf->consumer.end_addr &= ~XE_PAGEFAULT_END_ADDR_MASK ; + pf->consumer.end_addr |= end_addr; +} + +/** +* xe_pagefault_end_addr() - read serviced range end for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive end address of the range previously recorded by +* xe_pagefault_set_end_addr(). Only the bits covered by +* %XE_PAGEFAULT_END_ADDR_MASK are returned; other bits in the storage are +* reserved for internal state. +* +* Return: End address of the serviced fault range. +*/ +static inline u64 xe_pagefault_end_addr (struct xe_pagefault *pf) +{ +return pf->consumer.end_addr &XE_PAGEFAULT_END_ADDR_MASK ; +} + +#undef XE_PAGEFAULT_END_ADDR_MASK + +/** +* xe_pagefault_set_start_addr() - store serviced range start for a pagefault +* @pf: Pagefault entry +* @start_addr: Start address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +*/ +static inline void +xe_pagefault_set_start_addr (struct xe_pagefault *pf,u64 start_addr) +{ + pf->consumer.page_addr = start_addr; +} + +/** +* xe_pagefault_start_addr() - read serviced range start for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive start address of the range previously recorded by +* xe_pagefault_set_start_addr(). +* +* Return: Start address of the serviced fault range. +*/ +static inline u64 xe_pagefault_start_addr (struct xe_pagefault *pf) +{ +return pf->consumer.page_addr; +} +#endif drivers/gpu/drm/xe/xe_pagefault_types.h +59
-22@@ -60,36 +60,58 @@ struct xe_pagefault {/** * @consumer: State for the software handling the fault. Populated by * the producer and may be modified by the consumer to communicate -* information back to the producer upon fault acknowledgment. +* information back to the producer upon fault acknowledgment. After +* fault acknowledgment, the producer should only access consumer fields +* via well defined helpers. */ struct { -/** @consumer.page_addr: address of page fault */ -u64 page_addr; -/** @consumer.asid: address space ID */ -u32 asid;/** -* @consumer.access_type: access type and prefetch flag packed -* into a u8. +* @consumer.page_addr: address of page fault, populated by +* consumer after fault completion */ -u8 access_type; +u64 page_addr; +union { +struct { +/** +* @consumer.alloc_state: page fault allocation +* state +*/ +u8 alloc_state; +/** +* @consumer.access_type: access type, u8 rather +* than enum to keep size compact +*/ +u8 access_type;#define XE_PAGEFAULT_ACCESS_TYPE_MASK GENMASK(1, 0)#define XE_PAGEFAULT_ACCESS_PREFETCH BIT(7) -/** -* @consumer.fault_type_level: fault type and level, u8 rather -* than enum to keep size compact -*/ -u8 fault_type_level; +/** +* @consumer.fault_type_level: fault type and +* level, u8 rather than enum to keep size +* compact +*/ +u8 fault_type_level;#define XE_PAGEFAULT_TYPE_LEVEL_NACK 0xff/* Producer indicates nack fault */ -#define XE_PAGEFAULT_LEVEL_MASK GENMASK(3, 0) -#define XE_PAGEFAULT_TYPE_MASK GENMASK(7, 4) -/** @consumer.engine_class_instance: engine class and instance */ -u8 engine_class_instance; +#define XE_PAGEFAULT_LEVEL_MASK GENMASK(2, 0) +#define XE_PAGEFAULT_TYPE_MASK GENMASK(6, 3) +#define XE_PAGEFAULT_REQUEUE_MASK BIT(7) +/** @consumer.engine_class_instance: engine class and instance */ +u8 engine_class_instance;#define XE_PAGEFAULT_ENGINE_CLASS_MASK GENMASK(3, 0)#define XE_PAGEFAULT_ENGINE_INSTANCE_MASK GENMASK(7, 4) -/** @pad: alignment padding */ -u8 pad; -/** @consumer.reserved: reserved bits for future expansion */ -u64 reserved; +/** @consumer.asid: address space ID */ +u32 asid; + }; +/** +* @consumer.end_addr: end address of page fault, +* populated by consumer after fault completion +*/ +u64 end_addr; + }; +/** +* @consumer.next: next pagefault chained to this fault, +* protected by pf_queue lock +*/ +struct xe_pagefault *next; } consumer;/** * @producer: State for the producer (i.e., HW/FW interface). Populated @@ -131,7 +153,7 @@ struct xe_pagefault_queue {u32 head;/** @tail: Tail pointer in bytes, moved by consumer, protected by @lock */ u32 tail; -/** @lock: protects page fault queue */ +/** @lock: protects page fault queue, workers caches */ spinlock_t lock; };@@ -146,6 +168,21 @@ struct xe_pagefault_work {struct xe_device *xe;/** @id: Identifier for this work item */ int id; +/** +* @cache: Page fault cache for the currently processed fault +* +* Protected by the page fault queue lock. +*/ +struct { +/** @cache.start: Start address of the current page fault */ +u64 start; +/** @cache.end: End address of the current page fault */ +u64 end; +/** @cache.asid: Address space ID of the current page fault */ +u32 asid; +/** @cache.pf: Pointer to the current page fault */ +struct xe_pagefault *pf; + } cache;/** @work: Work item used to process the page fault */ struct work_struct work; };drivers/gpu/drm/xe/xe_svm.c +3
-2@@ -15,6 +15,7 @@#include "xe_gt_stats.h" #include "xe_migrate.h" #include "xe_module.h" +#include "xe_pagefault.h" #include "xe_pm.h" #include "xe_pt.h" #include "xe_svm.h" @@ -1264,8 +1265,8 @@ DECL_SVM_RANGE_US_STATS(bind, BIND) DECL_SVM_RANGE_US_STATS(fault, PAGEFAULT)static int __xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool need_vram) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool need_vram) {int devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );
drivers/gpu/drm/xe/xe_svm.c static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, · line 1431
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm.c +4
-0@@ -1424,6 +1425,10 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,xe_svm_range_bind_us_stats_incr (gt, range, bind_start); out: +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_svm_range_start (range)); +xe_pagefault_set_end_addr (pf,xe_svm_range_end (range)); +
I have not checked the GT stats from later patches in the series but are we able to chain enough faults by setting start/end so late during page fault handling?
Would there be a way to move it closer to the beginning of __xe_svm_handle_pagefault()?
Francois
Quoted 68 lines
drivers/gpu/drm/xe/xe_svm.c +4
-3xe_svm_range_fault_us_stats_incr (gt, range, start);mutex_unlock (&range->lock);drm_gpusvm_range_put (&range->base);@@ -1446,6 +1451,7 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* xe_svm_handle_pagefault() - SVM handle page fault * @vm: The VM. * @vma: The CPU address mirror VMA. +* @pf: Pagefault structure * @gt: The gt upon the fault occurred. * @fault_addr: The GPU fault address. * @atomic: The fault atomic access bit. @@ -1456,8 +1462,8 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* Return: 0 on success, negative error code on error. */ int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {int need_vram, ret; retry:@@ -1465,7 +1471,7 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,if (need_vram <0 )return need_vram; - ret =__xe_svm_handle_pagefault (vm, vma, gt, fault_addr, + ret =__xe_svm_handle_pagefault (vm, vma, pf, gt, fault_addr, need_vram ? true : false);if (ret == -EAGAIN ) {/* drivers/gpu/drm/xe/xe_svm.h +5
-4@@ -21,6 +21,7 @@ struct drm_file;struct xe_bo ;struct xe_gt ;struct xe_device ; +struct xe_pagefault ;struct xe_vram_region ;struct xe_tile ;struct xe_vm ;@@ -109,8 +110,8 @@ void xe_svm_fini(struct xe_vm *vm);void xe_svm_close (struct xe_vm *vm);int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic); +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic);bool xe_svm_has_mapping (struct xe_vm *vm,u64 start,u64 end);@@ -298,8 +299,8 @@ void xe_svm_close(struct xe_vm *vm)static inline int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {return 0 ; }Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v8 08/12] drm/xe: Chain page faults via queue-resident cache to avoid fault storms
Quoted 13 lines · Francois Dugast
Hi,
Patch size makes review a bit longer but I cannot suggest a good way to break it down.
Quoted 7 lines · Matthew Brost
Some Xe platforms can generate pagefault storms where many faults target the same address range in a short time window (e.g. many EU threads faulting the same page). The current worker/locking model effectively serializes faults for a given range and repeatedly performs VMA/range lookups for each fault, which creates head-of-queue blocking and wastes CPU in the hot path.
Quoted 5 lines
Quoted 2 lines
Introduce a page fault chaining cache that coalesces faults targeting the samr ASID and address range.
s/samr/same/
+1
Quoted 7 lines
Quoted 7 lines
Each worker tracks the active fault range it is servicing. Fault entries reside in stable queue storage, allowing the IRQ handler to match new faults against the worker cache and directly chain cache hits onto the active entry without allocation or waiting for dequeue. Once the leading fault completes, the worker acknowledges the entire chain.
Quoted 5 lines
Quoted 2 lines
A small allocation state is added to each entry so queue, worker, and IRQ pathd can safely reference the same fault object. This prevents
s/pathd/paths/
+1
Quoted 14 lines
Quoted 14 lines
reuse while the fault is active and guarantees that chained faults remain valid until acknowledged.
Fault handlers also record the serviced range so subsequent faults can be acknowledged without re-running the full resolution path.
This removes repeated fault resolution during fault storms and significantly improves forward progress in SVM workloads.
Since threaded prefetches now use a dedicated prefetch workqueue (usm.prefetch_wq) rather than sharing the page fault workqueue, page fault servicing can no longer deadlock on
vm->lock, so this cache does not need an-EAGAINretry path for a failedvm->lockacquisition.
Quoted 8 lines
Quoted 3 lines
Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>Please remove this "Co-authored-by".
(Already mentioned in patch #3 but repeating here not to forget)
+1
Quoted 219 lines
Quoted 219 lines
--- drivers/gpu/drm/xe/xe_pagefault.c | 435 +++++++++++++++++++++--- drivers/gpu/drm/xe/xe_pagefault.h | 71 ++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 81 +++-- drivers/gpu/drm/xe/xe_svm.c | 16 +- drivers/gpu/drm/xe/xe_svm.h | 9 +- 5 files changed, 527 insertions(+), 85 deletions(-)drivers/gpu/drm/xe/xe_pagefault.c +150
-12@@ -35,6 +35,70 @@* xe_pagefault.c implements the consumer layer. */ +/** +* DOC: Xe page fault cache +* +* Some Xe hardware can trigger “fault storms,” which are many page faults to +* the same address within a short period of time. An example is many EU threads +* faulting on the same page simultaneously. With the current page fault locking +* structure, only one page fault for a given address range can be processed at +* a time. This causes head-of-queue blocking across workers, killing +* parallelism. If the page fault handler must repeatedly look up resources +* (VMAs, ranges) to determine that the pages are valid for each fault in the +* storm, the time complexity grows rapidly. +* +* To address this, each page fault worker maintains a cache of the active fault +* being processed. Subsequent faults that hit in the cache are chained to the +* pending fault, and all chained faults are acknowledged once the initial fault +* completes. This alleviates head-of-queue blocking and quickly chains faults +* in the upper layers, avoiding expensive lookups in the main fault-handling +* path. +* +* Faults are buffered in the page fault queue in a way that provides stable +* storage for outstanding faults. In particular, faults may be chained directly +* while still resident in the queue storage (i.e., outside the worker’s current +* head/tail dequeue position). This allows the IRQ handler to match newly +* arrived faults against the per-worker cache and immediately chain cache hits +* onto the active fault under the queue lock, without allocating memory or +* waiting for the worker to pop the fault first. +* +* A per-fault state field is used to assert correctness of these invariants. +* The state tracks whether an entry is free, queued, chained, or currently +* active. Transitions are performed under the page fault queue lock, and the +* worker acknowledges faults by walking the chain and returning entries to the +* free state once they are complete. +*/ + +/** +* enum xe_pagefault_alloc_state - lifetime state for a page fault queue entry +* @XE_PAGEFAULT_ALLOC_STATE_FREE: +* Entry is unused and may be overwritten by the producer, consumer retry +* or requeue.. +* @XE_PAGEFAULT_ALLOC_STATE_QUEUED: +* Entry has been enqueued and may be dequeued by a worker. +* @XE_PAGEFAULT_ALLOC_STATE_ACTIVE: +* Entry has been dequeued and is the worker's currently serviced fault. +* The worker may attach additional faults to it via consumer.next. +* @XE_PAGEFAULT_ALLOC_STATE_CHAINED: +* Entry is not independently serviced; it has been chained onto an +* ACTIVE entry via consumer.next and will be acknowledged when the +* leading fault completes. +* +* The page fault queue provides stable storage for outstanding faults so the +* IRQ handler can chain new cache hits directly onto a worker's active fault. +* Because entries may remain referenced outside the consumer dequeue window, +* the producer must only write into entries in the FREE state. +* +* State transitions are protected by the page fault queue lock. Workers return +* entries to FREE after acknowledging the fault (either as ACTIVE or CHAINED). +*/ +enum xe_pagefault_alloc_state { +XE_PAGEFAULT_ALLOC_STATE_FREE =0 , +XE_PAGEFAULT_ALLOC_STATE_QUEUED =1 , +XE_PAGEFAULT_ALLOC_STATE_CHAINED =2 , +XE_PAGEFAULT_ALLOC_STATE_ACTIVE =3 , +}; +static int xe_pagefault_entry_size (void ) {/* @@ -77,7 +141,7 @@ static int xe_pagefault_begin(struct drm_exec *exec, struct xe_vma *vma, }static int xe_pagefault_handle_vma (struct xe_gt *gt,struct xe_vma *vma, -bool atomic) +struct xe_pagefault *pf,bool atomic) {struct xe_vm *vm =xe_vma_vm (vma);struct xe_tile *tile =gt_to_tile (gt);@@ -102,8 +166,11 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma,/* Check if VMA is valid, opportunistic check only */ if (xe_vm_has_valid_gpu_mapping (tile, vma->tile_present, - vma->tile_invalidated) && !atomic) + vma->tile_invalidated) && !atomic) { +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma));return 0 ; + } do { if (xe_vma_is_userptr(vma) &&@@ -141,6 +208,10 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma, }while (err == -EAGAIN );if (!err) { +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma)); +dma_fence_wait (fence, false);dma_fence_put (fence); }@@ -208,10 +279,10 @@ static int xe_pagefault_service(struct xe_pagefault *pf) atomic =xe_pagefault_access_is_atomic (pf->consumer.access_type);if (xe_vma_is_cpu_addr_mirror (vma)) - err =xe_svm_handle_pagefault (vm, vma, gt, + err =xe_svm_handle_pagefault (vm, vma, pf, gt, pf->consumer.page_addr, atomic);else - err =xe_pagefault_handle_vma (gt, vma, atomic); + err =xe_pagefault_handle_vma (gt, vma, pf, atomic); unlock_vm:up_read (&vm->lock);@@ -220,21 +291,221 @@ static int xe_pagefault_service(struct xe_pagefault *pf)return err; } -static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, -struct xe_pagefault *pf) +#define XE_PAGEFAULT_CACHE_START_INVALID U64_MAX +#define xe_pagefault_cache_start_invalidate (val) \ + (val = XE_PAGEFAULT_CACHE_START_INVALID) + +static void +xe_pagefault_cache_invalidate (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault_work *pf_work) { -bool found_fault = false; +lockdep_assert_held (&pf_queue->lock); + +xe_pagefault_cache_start_invalidate (pf_work->cache.start); +} + +static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return CIRC_SPACE (pf_queue->head, pf_queue->tail, + pf_queue->size) <=xe_pagefault_entry_size (); +} + +static struct xe_pagefault * +xe_pagefault_queue_add (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *lpf; + +lockdep_assert_held (&pf_queue->lock); + +do { +/* Not possible, warn on and drop page fault */ +if (WARN_ON (xe_pagefault_queue_full (pf_queue))) +return NULL ; -spin_lock_irq (&pf_queue->lock); -if (pf_queue->tail != pf_queue->head) { -memcpy (pf, pf_queue->data + pf_queue->tail,sizeof (*pf)); - pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + lpf = (pf_queue->data + pf_queue->head); + pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; - found_fault = true; + }while (lpf->consumer.alloc_state !=XE_PAGEFAULT_ALLOC_STATE_FREE ); + +xe_assert (xe, lpf != pf); +memcpy (lpf, pf,sizeof (*pf)); + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_QUEUED ; + +return lpf; +} + +static struct xe_pagefault * +xe_pagefault_queue_unchain_requeue (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf,struct xe_gt *gt) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *next = pf->consumer.next, *lpf; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_CHAINED ); + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + lpf->consumer.fault_type_level |=XE_PAGEFAULT_REQUEUE_MASK ; + } + +return next; +} + +static bool xe_pagefault_match (struct xe_pagefault *pf,u64 start, +u64 end,u64 cache_asid) +{ +struct xe_device *xe =gt_to_xe (pf->gt); +u64 page_addr = pf->consumer.page_addr; +u32 pf_asid = pf->consumer.asid; + + xe_assert(xe, pf->consumer.alloc_state !=
Quoted 9 lines
drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 376
Quoted 5 lines
drivers/gpu/drm/xe/xe_pagefault.c +5
-0+ XE_PAGEFAULT_ALLOC_STATE_FREE); + +return page_addr >= start && page_addr < end && + pf_asid == cache_asid; +}Sashiko's comment here seems valid: access type should be taken into consideration when evaluating if the page faults match (atomic).
I don't think that is a major concern. Generally, when we move memory, you end up with the highest permission level available. Atomics are an exception because we typically have to move the memory. We more or less always attempt a move, but we try much harder for atomics.
If, for some reason, we encounter a read or write fault, fail to move the memory, and then chain an atomic fault and acknowledgment, the worst-case outcome is that the atomic operation will fault again, and we'll handle it at that point.
Quoted 221 lines
Quoted 221 lines
drivers/gpu/drm/xe/xe_pagefault.c +180
-17+ +static bool xe_pagefault_try_chain (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +bool requeue =FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + pf->consumer.fault_type_level); +int i; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_QUEUED ); + +/* +* If this is a retry, we may already have a chain attached. In that +* case, we cannot hit in the cache because chains cannot easily be +* combined. +*/ +if (pf->consumer.next) +return false; + +for (i =0 , pf_work = xe->usm.pf_workers; + i < xe->info.num_pf_work; ++i, ++pf_work) { +u64 start = pf_work->cache.start; +u64 end = requeue ? start +SZ_4K : pf_work->cache.end; +u32 asid = pf_work->cache.asid; + +if (xe_pagefault_match (pf, start, end, asid)) { +xe_assert (xe, pf_work->cache.pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_CHAINED ; + pf->consumer.next = pf_work->cache.pf->consumer.next; + pf_work->cache.pf->consumer.next = pf; + +return true; + } + } + +return false; +} + +static void xe_pagefault_queue_advance (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + + pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + pf_queue->size; +} + +static struct xe_pagefault * +xe_pagefault_queue_tail_fault (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->data + pf_queue->tail; +} + +static bool xe_pagefault_queue_empty (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->head == pf_queue->tail; +} + +static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault **pf,int id) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +struct xe_pagefault *lpf; +size_t align =SZ_2M ; + +guard (spinlock_irq)(&pf_queue->lock); + +for (*pf =NULL ; !*pf;) { +if (xe_pagefault_queue_empty (pf_queue)) +return false; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); +xe_pagefault_queue_advance (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) +continue ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) +continue ; + + *pf = lpf;/* Hand back page fault for processing */ + } + +/* +* No cache hit; allocate a new cache entry. We assume most faults +* within a 2M range will hit the same pages. If this assumption proves +* false, the mismatched fault is requeued after the initial fault is +* acknowledged. +*/ + pf_work = xe->usm.pf_workers + id; +if (FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + lpf->consumer.fault_type_level)) + align =SZ_4K ; + pf_work->cache.start =ALIGN_DOWN (lpf->consumer.page_addr, align); + pf_work->cache.end = pf_work->cache.start + align; + pf_work->cache.asid = lpf->consumer.asid; + pf_work->cache.pf = lpf; + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; + +/* Drain queue until empty or new fault found */ +while (1 ) { +if (xe_pagefault_queue_empty (pf_queue)) +break ; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) { +xe_pagefault_queue_advance (pf_queue); +continue ; + } + +if (!xe_pagefault_try_chain (pf_queue, lpf)) +break ; + +xe_pagefault_queue_advance (pf_queue); } -spin_unlock_irq (&pf_queue->lock); -return found_fault; +return true; }static void xe_pagefault_print (struct xe_pagefault *pf)@@ -295,36 +566,83 @@ static void xe_pagefault_queue_work(struct work_struct *w)container_of (w,typeof (*pf_work), work);struct xe_device *xe = pf_work->xe;struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue; -struct xe_pagefault pf; +struct xe_pagefault *pf;ktime_t start =xe_gt_stats_ktime_get (); -struct xe_gt *gt =NULL ;unsigned long threshold; +u64 cache_start =XE_PAGEFAULT_CACHE_START_INVALID , cache_end =0 ; +u32 cache_asid =0 ;#define USM_QUEUE_MAX_RUNTIME_MS 20 threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS ); -while (xe_pagefault_queue_pop (pf_queue, &pf)) { -int err; +while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +struct xe_gt *gt = pf->gt; +u32 asid = pf->consumer.asid; +int err =0 ; +bool invalidated = false; -if (!pf.gt)/* Fault squashed during reset */ -continue ; +/* Last fault same address, ack immediately */ +if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) + goto ack_fault; + + err =xe_pagefault_service (pf); - gt = pf.gt; - err =xe_pagefault_service (&pf);if (err) { -if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf); -xe_pagefault_print (&pf); -xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" , +if (!(pf->consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (gt), pf); +xe_pagefault_cache_start_invalidate (cache_start); +xe_pagefault_print (pf); +xe_gt_info (pf->gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); }else { -xe_gt_stats_incr (pf.gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); -xe_gt_dbg (pf.gt,"Prefetch Fault response: Unsuccessful %pe\n" , +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); +xe_gt_dbg (pf->gt,"Prefetch Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); } + }else { +/* Cache valid fault locally */ + cache_start =xe_pagefault_start_addr (pf); + cache_end =xe_pagefault_end_addr (pf); + cache_asid = asid; } - pf.producer.ops->ack_fault (&pf, err); +ack_fault: +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, pf == pf_work->cache.pf); + +while (pf) { +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->producer.ops->ack_fault (pf, err); + +spin_lock_irq (&pf_queue->lock); + +if (!invalidated) { + invalidated = true; +xe_pagefault_cache_invalidate (pf_queue, + pf_work); + } + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf = pf->consumer.next; + +/* +* Requeue chained faults which do not match the last +* fault processed
Quoted 9 lines
drivers/gpu/drm/xe/xe_pagefault.c static void xe_pagefault_queue_work(struct work_struct *w) · line 639
Quoted 6 lines
drivers/gpu/drm/xe/xe_pagefault.c +6
-0+*/ +while (pf && !xe_pagefault_match (pf, cache_start, + cache_end, cache_asid)) + pf =xe_pagefault_queue_unchain_requeue (pf_queue, pf, gt); + +Extra blank line.
Will fix.
Quoted 339 lines
Quoted 339 lines
drivers/gpu/drm/xe/xe_pagefault.c +40
-25+/* Ensure resets are safe */ +if (pf) + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; +spin_unlock_irq (&pf_queue->lock); + }if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);@@ -333,10 +651,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) }#undef USM_QUEUE_MAX_RUNTIME_MS -if (gt) -xe_gt_stats_incr (xe_root_mmio_gt (gt_to_xe (gt)), -XE_GT_STATS_ID_PAGEFAULT_US , -xe_gt_stats_ktime_us_delta (start)); +xe_gt_stats_incr (xe_root_mmio_gt (xe),XE_GT_STATS_ID_PAGEFAULT_US , +xe_gt_stats_ktime_us_delta (start)); } static int xe_pagefault_queue_init(struct xe_device *xe,@@ -431,6 +747,7 @@ int xe_pagefault_init(struct xe_device *xe) pf_work->xe = xe; pf_work->id = i; +xe_pagefault_cache_start_invalidate (pf_work->cache.start);INIT_WORK (&pf_work->work, xe_pagefault_queue_work); }@@ -454,15 +771,23 @@ static void xe_pagefault_queue_reset(struct xe_device *xe, struct xe_gt *gt,/* Squash all pending faults on the GT */ -spin_lock_irq (&pf_queue->lock); -for (i = pf_queue->tail; i != pf_queue->head; - i = (i +xe_pagefault_entry_size ()) % pf_queue->size) { +guard (spinlock_irq)(&pf_queue->lock); + +for (i =0 ; i < pf_queue->size; i +=xe_pagefault_entry_size ()) {struct xe_pagefault *pf = pf_queue->data + i; +bool active = pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; -if (pf->gt == gt) - pf->gt =NULL ; +if (pf->gt != gt || active) { +if (active) + pf->consumer.next =NULL ; +continue ; + } + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf->consumer.next =NULL ; } -spin_unlock_irq (&pf_queue->lock); }/** @@ -478,14 +803,6 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt)xe_pagefault_queue_reset (xe, gt, &xe->usm.pf_queue); } -static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) -{ -lockdep_assert_held (&pf_queue->lock); - -return CIRC_SPACE (pf_queue->head, pf_queue->tail, pf_queue->size) <= -xe_pagefault_entry_size (); -} -/* * This function can race with multiple page fault producers, but worst case we * stick a page fault on the same queue for consumption. @@ -511,18 +828,28 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf) {struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue;unsigned long flags; -int work_index;bool full;spin_lock_irqsave (&pf_queue->lock, flags); - work_index =xe_pagefault_work_index (xe); full =xe_pagefault_queue_full (pf_queue);if (!full) { -memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); - pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % - pf_queue->size; -queue_work (xe->usm.pagefault_wq, - &xe->usm.pf_workers[work_index].work); +struct xe_pagefault *lpf; +bool empty =xe_pagefault_queue_empty (pf_queue); + + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) { +if (empty) +xe_pagefault_queue_advance (pf_queue); + }else { +int work_index =xe_pagefault_work_index (xe); + +queue_work (xe->usm.pagefault_wq, + &xe->usm.pf_workers[work_index].work); + } + } }else {drm_warn (&xe->drm,"PageFault Queue full, shouldn't be possible\n" );drivers/gpu/drm/xe/xe_pagefault.h +71
-0@@ -6,6 +6,8 @@#ifndef _XE_PAGEFAULT_H_#define _XE_PAGEFAULT_H_ +#include "xe_pagefault_types.h" +struct xe_device ;struct xe_gt ;struct xe_pagefault ;@@ -16,4 +18,73 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt);int xe_pagefault_handler (struct xe_device *xe,struct xe_pagefault *pf); +#define XE_PAGEFAULT_END_ADDR_MASK (~0xfffull) + +/** +* xe_pagefault_set_end_addr() - store serviced range end for a pagefault +* @pf: Pagefault entry +* @end_addr: Inclusive end address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +* +* The end address shares storage with other consumer metadata and therefore +* must be masked with %XE_PAGEFAULT_END_ADDR_MASK before storing. Bits outside +* the mask are reserved for internal state tracking and must be preserved. +*/ +static inline void +xe_pagefault_set_end_addr (struct xe_pagefault *pf,u64 end_addr) +{ + pf->consumer.end_addr &= ~XE_PAGEFAULT_END_ADDR_MASK ; + pf->consumer.end_addr |= end_addr; +} + +/** +* xe_pagefault_end_addr() - read serviced range end for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive end address of the range previously recorded by +* xe_pagefault_set_end_addr(). Only the bits covered by +* %XE_PAGEFAULT_END_ADDR_MASK are returned; other bits in the storage are +* reserved for internal state. +* +* Return: End address of the serviced fault range. +*/ +static inline u64 xe_pagefault_end_addr (struct xe_pagefault *pf) +{ +return pf->consumer.end_addr &XE_PAGEFAULT_END_ADDR_MASK ; +} + +#undef XE_PAGEFAULT_END_ADDR_MASK + +/** +* xe_pagefault_set_start_addr() - store serviced range start for a pagefault +* @pf: Pagefault entry +* @start_addr: Start address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +*/ +static inline void +xe_pagefault_set_start_addr (struct xe_pagefault *pf,u64 start_addr) +{ + pf->consumer.page_addr = start_addr; +} + +/** +* xe_pagefault_start_addr() - read serviced range start for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive start address of the range previously recorded by +* xe_pagefault_set_start_addr(). +* +* Return: Start address of the serviced fault range. +*/ +static inline u64 xe_pagefault_start_addr (struct xe_pagefault *pf) +{ +return pf->consumer.page_addr; +} +#endif drivers/gpu/drm/xe/xe_pagefault_types.h +59
-22@@ -60,36 +60,58 @@ struct xe_pagefault {/** * @consumer: State for the software handling the fault. Populated by * the producer and may be modified by the consumer to communicate -* information back to the producer upon fault acknowledgment. +* information back to the producer upon fault acknowledgment. After +* fault acknowledgment, the producer should only access consumer fields +* via well defined helpers. */ struct { -/** @consumer.page_addr: address of page fault */ -u64 page_addr; -/** @consumer.asid: address space ID */ -u32 asid;/** -* @consumer.access_type: access type and prefetch flag packed -* into a u8. +* @consumer.page_addr: address of page fault, populated by +* consumer after fault completion */ -u8 access_type; +u64 page_addr; +union { +struct { +/** +* @consumer.alloc_state: page fault allocation +* state +*/ +u8 alloc_state; +/** +* @consumer.access_type: access type, u8 rather +* than enum to keep size compact +*/ +u8 access_type;#define XE_PAGEFAULT_ACCESS_TYPE_MASK GENMASK(1, 0)#define XE_PAGEFAULT_ACCESS_PREFETCH BIT(7) -/** -* @consumer.fault_type_level: fault type and level, u8 rather -* than enum to keep size compact -*/ -u8 fault_type_level; +/** +* @consumer.fault_type_level: fault type and +* level, u8 rather than enum to keep size +* compact +*/ +u8 fault_type_level;#define XE_PAGEFAULT_TYPE_LEVEL_NACK 0xff/* Producer indicates nack fault */ -#define XE_PAGEFAULT_LEVEL_MASK GENMASK(3, 0) -#define XE_PAGEFAULT_TYPE_MASK GENMASK(7, 4) -/** @consumer.engine_class_instance: engine class and instance */ -u8 engine_class_instance; +#define XE_PAGEFAULT_LEVEL_MASK GENMASK(2, 0) +#define XE_PAGEFAULT_TYPE_MASK GENMASK(6, 3) +#define XE_PAGEFAULT_REQUEUE_MASK BIT(7) +/** @consumer.engine_class_instance: engine class and instance */ +u8 engine_class_instance;#define XE_PAGEFAULT_ENGINE_CLASS_MASK GENMASK(3, 0)#define XE_PAGEFAULT_ENGINE_INSTANCE_MASK GENMASK(7, 4) -/** @pad: alignment padding */ -u8 pad; -/** @consumer.reserved: reserved bits for future expansion */ -u64 reserved; +/** @consumer.asid: address space ID */ +u32 asid; + }; +/** +* @consumer.end_addr: end address of page fault, +* populated by consumer after fault completion +*/ +u64 end_addr; + }; +/** +* @consumer.next: next pagefault chained to this fault, +* protected by pf_queue lock +*/ +struct xe_pagefault *next; } consumer;/** * @producer: State for the producer (i.e., HW/FW interface). Populated @@ -131,7 +153,7 @@ struct xe_pagefault_queue {u32 head;/** @tail: Tail pointer in bytes, moved by consumer, protected by @lock */ u32 tail; -/** @lock: protects page fault queue */ +/** @lock: protects page fault queue, workers caches */ spinlock_t lock; };@@ -146,6 +168,21 @@ struct xe_pagefault_work {struct xe_device *xe;/** @id: Identifier for this work item */ int id; +/** +* @cache: Page fault cache for the currently processed fault +* +* Protected by the page fault queue lock. +*/ +struct { +/** @cache.start: Start address of the current page fault */ +u64 start; +/** @cache.end: End address of the current page fault */ +u64 end; +/** @cache.asid: Address space ID of the current page fault */ +u32 asid; +/** @cache.pf: Pointer to the current page fault */ +struct xe_pagefault *pf; + } cache;/** @work: Work item used to process the page fault */ struct work_struct work; };drivers/gpu/drm/xe/xe_svm.c +3
-2@@ -15,6 +15,7 @@#include "xe_gt_stats.h" #include "xe_migrate.h" #include "xe_module.h" +#include "xe_pagefault.h" #include "xe_pm.h" #include "xe_pt.h" #include "xe_svm.h" @@ -1264,8 +1265,8 @@ DECL_SVM_RANGE_US_STATS(bind, BIND) DECL_SVM_RANGE_US_STATS(fault, PAGEFAULT)static int __xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool need_vram) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool need_vram) {int devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );@@ -1424,6 +1425,10 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,xe_svm_range_bind_us_stats_incr (gt, range, bind_start);
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm.c static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, · line 1431
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm.c +4
-0out: +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_svm_range_start (range)); +xe_pagefault_set_end_addr (pf,xe_svm_range_end (range)); +I have not checked the GT stats from later patches in the series but are we able to chain enough faults by setting start/end so late during page fault handling?
This doesn't affect the chain-building process. We blindly attempt to build chains on 2 MB-aligned boundaries. This is about communicating back that we serviced for the fault and determining whether faults in the chain should be acknowledged or retried. As a result, the placement doesn't matter.
I had considered dynamically updating the chain size/window after looking up the range or VMA, but that isn't implemented in this series. If we went that route, then yes, we'd want to do it immediately after the range/VMA lookup. We'd also probably want to use try-locking on the VMA lock to avoid HoQ blocking.
The chain reconstruction is a bit tricky here, which is why I haven't implemented it yet.
Matt
Quoted 73 lines
Would there be a way to move it closer to the beginning of
__xe_svm_handle_pagefault()?Francois
Quoted 68 lines
drivers/gpu/drm/xe/xe_svm.c +4
-3xe_svm_range_fault_us_stats_incr (gt, range, start);mutex_unlock (&range->lock);drm_gpusvm_range_put (&range->base);@@ -1446,6 +1451,7 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* xe_svm_handle_pagefault() - SVM handle page fault * @vm: The VM. * @vma: The CPU address mirror VMA. +* @pf: Pagefault structure * @gt: The gt upon the fault occurred. * @fault_addr: The GPU fault address. * @atomic: The fault atomic access bit. @@ -1456,8 +1462,8 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* Return: 0 on success, negative error code on error. */ int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {int need_vram, ret; retry:@@ -1465,7 +1471,7 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,if (need_vram <0 )return need_vram; - ret =__xe_svm_handle_pagefault (vm, vma, gt, fault_addr, + ret =__xe_svm_handle_pagefault (vm, vma, pf, gt, fault_addr, need_vram ? true : false);if (ret == -EAGAIN ) {/* drivers/gpu/drm/xe/xe_svm.h +5
-4@@ -21,6 +21,7 @@ struct drm_file;struct xe_bo ;struct xe_gt ;struct xe_device ; +struct xe_pagefault ;struct xe_vram_region ;struct xe_tile ;struct xe_vm ;@@ -109,8 +110,8 @@ void xe_svm_fini(struct xe_vm *vm);void xe_svm_close (struct xe_vm *vm);int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic); +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic);bool xe_svm_has_mapping (struct xe_vm *vm,u64 start,u64 end);@@ -298,8 +299,8 @@ void xe_svm_close(struct xe_vm *vm)static inline int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {return 0 ; }Signature · 1 line
2.34.1
Re: [PATCH v8 08/12] drm/xe: Chain page faults via queue-resident cache to avoid fault storms
Quoted 296 lines · Matthew Brost
Quoted 13 lines · Francois Dugast
Hi,
Patch size makes review a bit longer but I cannot suggest a good way to break it down.
Quoted 7 lines · Matthew Brost
Some Xe platforms can generate pagefault storms where many faults target the same address range in a short time window (e.g. many EU threads faulting the same page). The current worker/locking model effectively serializes faults for a given range and repeatedly performs VMA/range lookups for each fault, which creates head-of-queue blocking and wastes CPU in the hot path.
Quoted 5 lines
Quoted 2 lines
Introduce a page fault chaining cache that coalesces faults targeting the samr ASID and address range.
s/samr/same/
+1
Quoted 7 lines
Quoted 7 lines
Each worker tracks the active fault range it is servicing. Fault entries reside in stable queue storage, allowing the IRQ handler to match new faults against the worker cache and directly chain cache hits onto the active entry without allocation or waiting for dequeue. Once the leading fault completes, the worker acknowledges the entire chain.
Quoted 5 lines
Quoted 2 lines
A small allocation state is added to each entry so queue, worker, and IRQ pathd can safely reference the same fault object. This prevents
s/pathd/paths/
+1
Quoted 14 lines
Quoted 14 lines
reuse while the fault is active and guarantees that chained faults remain valid until acknowledged.
Fault handlers also record the serviced range so subsequent faults can be acknowledged without re-running the full resolution path.
This removes repeated fault resolution during fault storms and significantly improves forward progress in SVM workloads.
Since threaded prefetches now use a dedicated prefetch workqueue (usm.prefetch_wq) rather than sharing the page fault workqueue, page fault servicing can no longer deadlock on
vm->lock, so this cache does not need an-EAGAINretry path for a failedvm->lockacquisition.Quoted 8 lines
Quoted 3 lines
Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>Co-authored-by Copilot <[email protected]>Please remove this "Co-authored-by".
(Already mentioned in patch #3 but repeating here not to forget)
+1
Quoted 219 lines
Quoted 219 lines
--- drivers/gpu/drm/xe/xe_pagefault.c | 435 +++++++++++++++++++++--- drivers/gpu/drm/xe/xe_pagefault.h | 71 ++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 81 +++-- drivers/gpu/drm/xe/xe_svm.c | 16 +- drivers/gpu/drm/xe/xe_svm.h | 9 +- 5 files changed, 527 insertions(+), 85 deletions(-)drivers/gpu/drm/xe/xe_pagefault.c +150
-12@@ -35,6 +35,70 @@* xe_pagefault.c implements the consumer layer. */ +/** +* DOC: Xe page fault cache +* +* Some Xe hardware can trigger “fault storms,” which are many page faults to +* the same address within a short period of time. An example is many EU threads +* faulting on the same page simultaneously. With the current page fault locking +* structure, only one page fault for a given address range can be processed at +* a time. This causes head-of-queue blocking across workers, killing +* parallelism. If the page fault handler must repeatedly look up resources +* (VMAs, ranges) to determine that the pages are valid for each fault in the +* storm, the time complexity grows rapidly. +* +* To address this, each page fault worker maintains a cache of the active fault +* being processed. Subsequent faults that hit in the cache are chained to the +* pending fault, and all chained faults are acknowledged once the initial fault +* completes. This alleviates head-of-queue blocking and quickly chains faults +* in the upper layers, avoiding expensive lookups in the main fault-handling +* path. +* +* Faults are buffered in the page fault queue in a way that provides stable +* storage for outstanding faults. In particular, faults may be chained directly +* while still resident in the queue storage (i.e., outside the worker’s current +* head/tail dequeue position). This allows the IRQ handler to match newly +* arrived faults against the per-worker cache and immediately chain cache hits +* onto the active fault under the queue lock, without allocating memory or +* waiting for the worker to pop the fault first. +* +* A per-fault state field is used to assert correctness of these invariants. +* The state tracks whether an entry is free, queued, chained, or currently +* active. Transitions are performed under the page fault queue lock, and the +* worker acknowledges faults by walking the chain and returning entries to the +* free state once they are complete. +*/ + +/** +* enum xe_pagefault_alloc_state - lifetime state for a page fault queue entry +* @XE_PAGEFAULT_ALLOC_STATE_FREE: +* Entry is unused and may be overwritten by the producer, consumer retry +* or requeue.. +* @XE_PAGEFAULT_ALLOC_STATE_QUEUED: +* Entry has been enqueued and may be dequeued by a worker. +* @XE_PAGEFAULT_ALLOC_STATE_ACTIVE: +* Entry has been dequeued and is the worker's currently serviced fault. +* The worker may attach additional faults to it via consumer.next. +* @XE_PAGEFAULT_ALLOC_STATE_CHAINED: +* Entry is not independently serviced; it has been chained onto an +* ACTIVE entry via consumer.next and will be acknowledged when the +* leading fault completes. +* +* The page fault queue provides stable storage for outstanding faults so the +* IRQ handler can chain new cache hits directly onto a worker's active fault. +* Because entries may remain referenced outside the consumer dequeue window, +* the producer must only write into entries in the FREE state. +* +* State transitions are protected by the page fault queue lock. Workers return +* entries to FREE after acknowledging the fault (either as ACTIVE or CHAINED). +*/ +enum xe_pagefault_alloc_state { +XE_PAGEFAULT_ALLOC_STATE_FREE =0 , +XE_PAGEFAULT_ALLOC_STATE_QUEUED =1 , +XE_PAGEFAULT_ALLOC_STATE_CHAINED =2 , +XE_PAGEFAULT_ALLOC_STATE_ACTIVE =3 , +}; +static int xe_pagefault_entry_size (void ) {/* @@ -77,7 +141,7 @@ static int xe_pagefault_begin(struct drm_exec *exec, struct xe_vma *vma, }static int xe_pagefault_handle_vma (struct xe_gt *gt,struct xe_vma *vma, -bool atomic) +struct xe_pagefault *pf,bool atomic) {struct xe_vm *vm =xe_vma_vm (vma);struct xe_tile *tile =gt_to_tile (gt);@@ -102,8 +166,11 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma,/* Check if VMA is valid, opportunistic check only */ if (xe_vm_has_valid_gpu_mapping (tile, vma->tile_present, - vma->tile_invalidated) && !atomic) + vma->tile_invalidated) && !atomic) { +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma));return 0 ; + } do { if (xe_vma_is_userptr(vma) &&@@ -141,6 +208,10 @@ static int xe_pagefault_handle_vma(struct xe_gt *gt, struct xe_vma *vma, }while (err == -EAGAIN );if (!err) { +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_vma_start (vma)); +xe_pagefault_set_end_addr (pf,xe_vma_end (vma)); +dma_fence_wait (fence, false);dma_fence_put (fence); }@@ -208,10 +279,10 @@ static int xe_pagefault_service(struct xe_pagefault *pf) atomic =xe_pagefault_access_is_atomic (pf->consumer.access_type);if (xe_vma_is_cpu_addr_mirror (vma)) - err =xe_svm_handle_pagefault (vm, vma, gt, + err =xe_svm_handle_pagefault (vm, vma, pf, gt, pf->consumer.page_addr, atomic);else - err =xe_pagefault_handle_vma (gt, vma, atomic); + err =xe_pagefault_handle_vma (gt, vma, pf, atomic); unlock_vm:up_read (&vm->lock);@@ -220,21 +291,221 @@ static int xe_pagefault_service(struct xe_pagefault *pf)return err; } -static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, -struct xe_pagefault *pf) +#define XE_PAGEFAULT_CACHE_START_INVALID U64_MAX +#define xe_pagefault_cache_start_invalidate (val) \ + (val = XE_PAGEFAULT_CACHE_START_INVALID) + +static void +xe_pagefault_cache_invalidate (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault_work *pf_work) { -bool found_fault = false; +lockdep_assert_held (&pf_queue->lock); + +xe_pagefault_cache_start_invalidate (pf_work->cache.start); +} + +static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return CIRC_SPACE (pf_queue->head, pf_queue->tail, + pf_queue->size) <=xe_pagefault_entry_size (); +} + +static struct xe_pagefault * +xe_pagefault_queue_add (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *lpf; + +lockdep_assert_held (&pf_queue->lock); + +do { +/* Not possible, warn on and drop page fault */ +if (WARN_ON (xe_pagefault_queue_full (pf_queue))) +return NULL ; -spin_lock_irq (&pf_queue->lock); -if (pf_queue->tail != pf_queue->head) { -memcpy (pf, pf_queue->data + pf_queue->tail,sizeof (*pf)); - pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + lpf = (pf_queue->data + pf_queue->head); + pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % pf_queue->size; - found_fault = true; + }while (lpf->consumer.alloc_state !=XE_PAGEFAULT_ALLOC_STATE_FREE ); + +xe_assert (xe, lpf != pf); +memcpy (lpf, pf,sizeof (*pf)); + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_QUEUED ; + +return lpf; +} + +static struct xe_pagefault * +xe_pagefault_queue_unchain_requeue (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf,struct xe_gt *gt) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault *next = pf->consumer.next, *lpf; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_CHAINED ); + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + lpf->consumer.fault_type_level |=XE_PAGEFAULT_REQUEUE_MASK ; + } + +return next; +} + +static bool xe_pagefault_match (struct xe_pagefault *pf,u64 start, +u64 end,u64 cache_asid) +{ +struct xe_device *xe =gt_to_xe (pf->gt); +u64 page_addr = pf->consumer.page_addr; +u32 pf_asid = pf->consumer.asid; + + xe_assert(xe, pf->consumer.alloc_state !=Quoted 9 lines
drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 376
Quoted 5 lines
drivers/gpu/drm/xe/xe_pagefault.c +5
-0+ XE_PAGEFAULT_ALLOC_STATE_FREE); + +return page_addr >= start && page_addr < end && + pf_asid == cache_asid; +}Sashiko's comment here seems valid: access type should be taken into consideration when evaluating if the page faults match (atomic).
I don't think that is a major concern. Generally, when we move memory, you end up with the highest permission level available. Atomics are an exception because we typically have to move the memory. We more or less always attempt a move, but we try much harder for atomics.
Quoted 4 lines
If, for some reason, we encounter a read or write fault, fail to move the memory, and then chain an atomic fault and acknowledgment, the worst-case outcome is that the atomic operation will fault again, and we'll handle it at that point.
Sounds reasonable, thanks for clarifying.
Francois
Quoted 673 lines
Quoted 222 lines
Quoted 222 lines
drivers/gpu/drm/xe/xe_pagefault.c +180
-17+ +static bool xe_pagefault_try_chain (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault *pf) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +bool requeue =FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + pf->consumer.fault_type_level); +int i; + +lockdep_assert_held (&pf_queue->lock); +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_QUEUED ); + +/* +* If this is a retry, we may already have a chain attached. In that +* case, we cannot hit in the cache because chains cannot easily be +* combined. +*/ +if (pf->consumer.next) +return false; + +for (i =0 , pf_work = xe->usm.pf_workers; + i < xe->info.num_pf_work; ++i, ++pf_work) { +u64 start = pf_work->cache.start; +u64 end = requeue ? start +SZ_4K : pf_work->cache.end; +u32 asid = pf_work->cache.asid; + +if (xe_pagefault_match (pf, start, end, asid)) { +xe_assert (xe, pf_work->cache.pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_CHAINED ; + pf->consumer.next = pf_work->cache.pf->consumer.next; + pf_work->cache.pf->consumer.next = pf; + +return true; + } + } + +return false; +} + +static void xe_pagefault_queue_advance (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + + pf_queue->tail = (pf_queue->tail +xe_pagefault_entry_size ()) % + pf_queue->size; +} + +static struct xe_pagefault * +xe_pagefault_queue_tail_fault (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->data + pf_queue->tail; +} + +static bool xe_pagefault_queue_empty (struct xe_pagefault_queue *pf_queue) +{ +lockdep_assert_held (&pf_queue->lock); + +return pf_queue->head == pf_queue->tail; +} + +static bool xe_pagefault_queue_pop (struct xe_pagefault_queue *pf_queue, +struct xe_pagefault **pf,int id) +{ +struct xe_device *xe =container_of (pf_queue,typeof (*xe), + usm.pf_queue); +struct xe_pagefault_work *pf_work; +struct xe_pagefault *lpf; +size_t align =SZ_2M ; + +guard (spinlock_irq)(&pf_queue->lock); + +for (*pf =NULL ; !*pf;) { +if (xe_pagefault_queue_empty (pf_queue)) +return false; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); +xe_pagefault_queue_advance (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) +continue ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) +continue ; + + *pf = lpf;/* Hand back page fault for processing */ + } + +/* +* No cache hit; allocate a new cache entry. We assume most faults +* within a 2M range will hit the same pages. If this assumption proves +* false, the mismatched fault is requeued after the initial fault is +* acknowledged. +*/ + pf_work = xe->usm.pf_workers + id; +if (FIELD_GET (XE_PAGEFAULT_REQUEUE_MASK , + lpf->consumer.fault_type_level)) + align =SZ_4K ; + pf_work->cache.start =ALIGN_DOWN (lpf->consumer.page_addr, align); + pf_work->cache.end = pf_work->cache.start + align; + pf_work->cache.asid = lpf->consumer.asid; + pf_work->cache.pf = lpf; + lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; + +/* Drain queue until empty or new fault found */ +while (1 ) { +if (xe_pagefault_queue_empty (pf_queue)) +break ; + + lpf =xe_pagefault_queue_tail_fault (pf_queue); + +if (lpf->consumer.alloc_state != +XE_PAGEFAULT_ALLOC_STATE_QUEUED ) { +xe_pagefault_queue_advance (pf_queue); +continue ; + } + +if (!xe_pagefault_try_chain (pf_queue, lpf)) +break ; + +xe_pagefault_queue_advance (pf_queue); } -spin_unlock_irq (&pf_queue->lock); -return found_fault; +return true; }static void xe_pagefault_print (struct xe_pagefault *pf)@@ -295,36 +566,83 @@ static void xe_pagefault_queue_work(struct work_struct *w)container_of (w,typeof (*pf_work), work);struct xe_device *xe = pf_work->xe;struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue; -struct xe_pagefault pf; +struct xe_pagefault *pf;ktime_t start =xe_gt_stats_ktime_get (); -struct xe_gt *gt =NULL ;unsigned long threshold; +u64 cache_start =XE_PAGEFAULT_CACHE_START_INVALID , cache_end =0 ; +u32 cache_asid =0 ;#define USM_QUEUE_MAX_RUNTIME_MS 20 threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS ); -while (xe_pagefault_queue_pop (pf_queue, &pf)) { -int err; +while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +struct xe_gt *gt = pf->gt; +u32 asid = pf->consumer.asid; +int err =0 ; +bool invalidated = false; -if (!pf.gt)/* Fault squashed during reset */ -continue ; +/* Last fault same address, ack immediately */ +if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) + goto ack_fault; + + err =xe_pagefault_service (pf); - gt = pf.gt; - err =xe_pagefault_service (&pf);if (err) { -if (!(pf.consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { -xe_pagefault_save_to_vm (gt_to_xe (pf.gt), &pf); -xe_pagefault_print (&pf); -xe_gt_info (pf.gt,"Fault response: Unsuccessful %pe\n" , +if (!(pf->consumer.access_type &XE_PAGEFAULT_ACCESS_PREFETCH )) { +xe_pagefault_save_to_vm (gt_to_xe (gt), pf); +xe_pagefault_cache_start_invalidate (cache_start); +xe_pagefault_print (pf); +xe_gt_info (pf->gt,"Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); }else { -xe_gt_stats_incr (pf.gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); -xe_gt_dbg (pf.gt,"Prefetch Fault response: Unsuccessful %pe\n" , +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT ,1 ); +xe_gt_dbg (pf->gt,"Prefetch Fault response: Unsuccessful %pe\n" ,ERR_PTR (err)); } + }else { +/* Cache valid fault locally */ + cache_start =xe_pagefault_start_addr (pf); + cache_end =xe_pagefault_end_addr (pf); + cache_asid = asid; } - pf.producer.ops->ack_fault (&pf, err); +ack_fault: +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, pf == pf_work->cache.pf); + +while (pf) { +xe_assert (xe, pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); + + pf->producer.ops->ack_fault (pf, err); + +spin_lock_irq (&pf_queue->lock); + +if (!invalidated) { + invalidated = true; +xe_pagefault_cache_invalidate (pf_queue, + pf_work); + } + + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf = pf->consumer.next; + +/* +* Requeue chained faults which do not match the last +* fault processed Quoted 9 lines
drivers/gpu/drm/xe/xe_pagefault.c static void xe_pagefault_queue_work(struct work_struct *w) · line 639
Quoted 6 lines
drivers/gpu/drm/xe/xe_pagefault.c +6
-0+*/ +while (pf && !xe_pagefault_match (pf, cache_start, + cache_end, cache_asid)) + pf =xe_pagefault_queue_unchain_requeue (pf_queue, pf, gt); + +Extra blank line.
Will fix.
Quoted 339 lines
Quoted 339 lines
drivers/gpu/drm/xe/xe_pagefault.c +40
-25+/* Ensure resets are safe */ +if (pf) + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; +spin_unlock_irq (&pf_queue->lock); + }if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);@@ -333,10 +651,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) }#undef USM_QUEUE_MAX_RUNTIME_MS -if (gt) -xe_gt_stats_incr (xe_root_mmio_gt (gt_to_xe (gt)), -XE_GT_STATS_ID_PAGEFAULT_US , -xe_gt_stats_ktime_us_delta (start)); +xe_gt_stats_incr (xe_root_mmio_gt (xe),XE_GT_STATS_ID_PAGEFAULT_US , +xe_gt_stats_ktime_us_delta (start)); } static int xe_pagefault_queue_init(struct xe_device *xe,@@ -431,6 +747,7 @@ int xe_pagefault_init(struct xe_device *xe) pf_work->xe = xe; pf_work->id = i; +xe_pagefault_cache_start_invalidate (pf_work->cache.start);INIT_WORK (&pf_work->work, xe_pagefault_queue_work); }@@ -454,15 +771,23 @@ static void xe_pagefault_queue_reset(struct xe_device *xe, struct xe_gt *gt,/* Squash all pending faults on the GT */ -spin_lock_irq (&pf_queue->lock); -for (i = pf_queue->tail; i != pf_queue->head; - i = (i +xe_pagefault_entry_size ()) % pf_queue->size) { +guard (spinlock_irq)(&pf_queue->lock); + +for (i =0 ; i < pf_queue->size; i +=xe_pagefault_entry_size ()) {struct xe_pagefault *pf = pf_queue->data + i; +bool active = pf->consumer.alloc_state == +XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; -if (pf->gt == gt) - pf->gt =NULL ; +if (pf->gt != gt || active) { +if (active) + pf->consumer.next =NULL ; +continue ; + } + + pf->consumer.alloc_state = +XE_PAGEFAULT_ALLOC_STATE_FREE ; + pf->consumer.next =NULL ; } -spin_unlock_irq (&pf_queue->lock); }/** @@ -478,14 +803,6 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt)xe_pagefault_queue_reset (xe, gt, &xe->usm.pf_queue); } -static bool xe_pagefault_queue_full (struct xe_pagefault_queue *pf_queue) -{ -lockdep_assert_held (&pf_queue->lock); - -return CIRC_SPACE (pf_queue->head, pf_queue->tail, pf_queue->size) <= -xe_pagefault_entry_size (); -} -/* * This function can race with multiple page fault producers, but worst case we * stick a page fault on the same queue for consumption. @@ -511,18 +828,28 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf) {struct xe_pagefault_queue *pf_queue = &xe->usm.pf_queue;unsigned long flags; -int work_index;bool full;spin_lock_irqsave (&pf_queue->lock, flags); - work_index =xe_pagefault_work_index (xe); full =xe_pagefault_queue_full (pf_queue);if (!full) { -memcpy (pf_queue->data + pf_queue->head, pf,sizeof (*pf)); - pf_queue->head = (pf_queue->head +xe_pagefault_entry_size ()) % - pf_queue->size; -queue_work (xe->usm.pagefault_wq, - &xe->usm.pf_workers[work_index].work); +struct xe_pagefault *lpf; +bool empty =xe_pagefault_queue_empty (pf_queue); + + lpf =xe_pagefault_queue_add (pf_queue, pf); +if (lpf) { + lpf->consumer.next =NULL ; + +if (xe_pagefault_try_chain (pf_queue, lpf)) { +if (empty) +xe_pagefault_queue_advance (pf_queue); + }else { +int work_index =xe_pagefault_work_index (xe); + +queue_work (xe->usm.pagefault_wq, + &xe->usm.pf_workers[work_index].work); + } + } }else {drm_warn (&xe->drm,"PageFault Queue full, shouldn't be possible\n" );drivers/gpu/drm/xe/xe_pagefault.h +71
-0@@ -6,6 +6,8 @@#ifndef _XE_PAGEFAULT_H_#define _XE_PAGEFAULT_H_ +#include "xe_pagefault_types.h" +struct xe_device ;struct xe_gt ;struct xe_pagefault ;@@ -16,4 +18,73 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt);int xe_pagefault_handler (struct xe_device *xe,struct xe_pagefault *pf); +#define XE_PAGEFAULT_END_ADDR_MASK (~0xfffull) + +/** +* xe_pagefault_set_end_addr() - store serviced range end for a pagefault +* @pf: Pagefault entry +* @end_addr: Inclusive end address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +* +* The end address shares storage with other consumer metadata and therefore +* must be masked with %XE_PAGEFAULT_END_ADDR_MASK before storing. Bits outside +* the mask are reserved for internal state tracking and must be preserved. +*/ +static inline void +xe_pagefault_set_end_addr (struct xe_pagefault *pf,u64 end_addr) +{ + pf->consumer.end_addr &= ~XE_PAGEFAULT_END_ADDR_MASK ; + pf->consumer.end_addr |= end_addr; +} + +/** +* xe_pagefault_end_addr() - read serviced range end for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive end address of the range previously recorded by +* xe_pagefault_set_end_addr(). Only the bits covered by +* %XE_PAGEFAULT_END_ADDR_MASK are returned; other bits in the storage are +* reserved for internal state. +* +* Return: End address of the serviced fault range. +*/ +static inline u64 xe_pagefault_end_addr (struct xe_pagefault *pf) +{ +return pf->consumer.end_addr &XE_PAGEFAULT_END_ADDR_MASK ; +} + +#undef XE_PAGEFAULT_END_ADDR_MASK + +/** +* xe_pagefault_set_start_addr() - store serviced range start for a pagefault +* @pf: Pagefault entry +* @start_addr: Start address of the serviced fault range +* +* The pagefault consumer stores the resolved fault range so subsequent faults +* hitting the same range can be immediately acknowledged without re-running +* the full fault handling path. +*/ +static inline void +xe_pagefault_set_start_addr (struct xe_pagefault *pf,u64 start_addr) +{ + pf->consumer.page_addr = start_addr; +} + +/** +* xe_pagefault_start_addr() - read serviced range start for a pagefault +* @pf: Pagefault entry +* +* Returns the inclusive start address of the range previously recorded by +* xe_pagefault_set_start_addr(). +* +* Return: Start address of the serviced fault range. +*/ +static inline u64 xe_pagefault_start_addr (struct xe_pagefault *pf) +{ +return pf->consumer.page_addr; +} +#endif drivers/gpu/drm/xe/xe_pagefault_types.h +59
-22@@ -60,36 +60,58 @@ struct xe_pagefault {/** * @consumer: State for the software handling the fault. Populated by * the producer and may be modified by the consumer to communicate -* information back to the producer upon fault acknowledgment. +* information back to the producer upon fault acknowledgment. After +* fault acknowledgment, the producer should only access consumer fields +* via well defined helpers. */ struct { -/** @consumer.page_addr: address of page fault */ -u64 page_addr; -/** @consumer.asid: address space ID */ -u32 asid;/** -* @consumer.access_type: access type and prefetch flag packed -* into a u8. +* @consumer.page_addr: address of page fault, populated by +* consumer after fault completion */ -u8 access_type; +u64 page_addr; +union { +struct { +/** +* @consumer.alloc_state: page fault allocation +* state +*/ +u8 alloc_state; +/** +* @consumer.access_type: access type, u8 rather +* than enum to keep size compact +*/ +u8 access_type;#define XE_PAGEFAULT_ACCESS_TYPE_MASK GENMASK(1, 0)#define XE_PAGEFAULT_ACCESS_PREFETCH BIT(7) -/** -* @consumer.fault_type_level: fault type and level, u8 rather -* than enum to keep size compact -*/ -u8 fault_type_level; +/** +* @consumer.fault_type_level: fault type and +* level, u8 rather than enum to keep size +* compact +*/ +u8 fault_type_level;#define XE_PAGEFAULT_TYPE_LEVEL_NACK 0xff/* Producer indicates nack fault */ -#define XE_PAGEFAULT_LEVEL_MASK GENMASK(3, 0) -#define XE_PAGEFAULT_TYPE_MASK GENMASK(7, 4) -/** @consumer.engine_class_instance: engine class and instance */ -u8 engine_class_instance; +#define XE_PAGEFAULT_LEVEL_MASK GENMASK(2, 0) +#define XE_PAGEFAULT_TYPE_MASK GENMASK(6, 3) +#define XE_PAGEFAULT_REQUEUE_MASK BIT(7) +/** @consumer.engine_class_instance: engine class and instance */ +u8 engine_class_instance;#define XE_PAGEFAULT_ENGINE_CLASS_MASK GENMASK(3, 0)#define XE_PAGEFAULT_ENGINE_INSTANCE_MASK GENMASK(7, 4) -/** @pad: alignment padding */ -u8 pad; -/** @consumer.reserved: reserved bits for future expansion */ -u64 reserved; +/** @consumer.asid: address space ID */ +u32 asid; + }; +/** +* @consumer.end_addr: end address of page fault, +* populated by consumer after fault completion +*/ +u64 end_addr; + }; +/** +* @consumer.next: next pagefault chained to this fault, +* protected by pf_queue lock +*/ +struct xe_pagefault *next; } consumer;/** * @producer: State for the producer (i.e., HW/FW interface). Populated @@ -131,7 +153,7 @@ struct xe_pagefault_queue {u32 head;/** @tail: Tail pointer in bytes, moved by consumer, protected by @lock */ u32 tail; -/** @lock: protects page fault queue */ +/** @lock: protects page fault queue, workers caches */ spinlock_t lock; };@@ -146,6 +168,21 @@ struct xe_pagefault_work {struct xe_device *xe;/** @id: Identifier for this work item */ int id; +/** +* @cache: Page fault cache for the currently processed fault +* +* Protected by the page fault queue lock. +*/ +struct { +/** @cache.start: Start address of the current page fault */ +u64 start; +/** @cache.end: End address of the current page fault */ +u64 end; +/** @cache.asid: Address space ID of the current page fault */ +u32 asid; +/** @cache.pf: Pointer to the current page fault */ +struct xe_pagefault *pf; + } cache;/** @work: Work item used to process the page fault */ struct work_struct work; };drivers/gpu/drm/xe/xe_svm.c +3
-2@@ -15,6 +15,7 @@#include "xe_gt_stats.h" #include "xe_migrate.h" #include "xe_module.h" +#include "xe_pagefault.h" #include "xe_pm.h" #include "xe_pt.h" #include "xe_svm.h" @@ -1264,8 +1265,8 @@ DECL_SVM_RANGE_US_STATS(bind, BIND) DECL_SVM_RANGE_US_STATS(fault, PAGEFAULT)static int __xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool need_vram) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool need_vram) {int devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP );@@ -1424,6 +1425,10 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,xe_svm_range_bind_us_stats_incr (gt, range, bind_start);Quoted 9 lines
drivers/gpu/drm/xe/xe_svm.c static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, · line 1431
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm.c +4
-0out: +/* Give hint to immediately ack faults */ +xe_pagefault_set_start_addr (pf,xe_svm_range_start (range)); +xe_pagefault_set_end_addr (pf,xe_svm_range_end (range)); +I have not checked the GT stats from later patches in the series but are we able to chain enough faults by setting start/end so late during page fault handling?
This doesn't affect the chain-building process. We blindly attempt to build chains on 2 MB-aligned boundaries. This is about communicating back that we serviced for the fault and determining whether faults in the chain should be acknowledged or retried. As a result, the placement doesn't matter.
I had considered dynamically updating the chain size/window after looking up the range or VMA, but that isn't implemented in this series. If we went that route, then yes, we'd want to do it immediately after the range/VMA lookup. We'd also probably want to use try-locking on the VMA lock to avoid HoQ blocking.
The chain reconstruction is a bit tricky here, which is why I haven't implemented it yet.
Matt
Quoted 73 lines
Would there be a way to move it closer to the beginning of
__xe_svm_handle_pagefault()?Francois
Quoted 68 lines
drivers/gpu/drm/xe/xe_svm.c +4
-3xe_svm_range_fault_us_stats_incr (gt, range, start);mutex_unlock (&range->lock);drm_gpusvm_range_put (&range->base);@@ -1446,6 +1451,7 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* xe_svm_handle_pagefault() - SVM handle page fault * @vm: The VM. * @vma: The CPU address mirror VMA. +* @pf: Pagefault structure * @gt: The gt upon the fault occurred. * @fault_addr: The GPU fault address. * @atomic: The fault atomic access bit. @@ -1456,8 +1462,8 @@ static int __xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,* Return: 0 on success, negative error code on error. */ int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {int need_vram, ret; retry:@@ -1465,7 +1471,7 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma,if (need_vram <0 )return need_vram; - ret =__xe_svm_handle_pagefault (vm, vma, gt, fault_addr, + ret =__xe_svm_handle_pagefault (vm, vma, pf, gt, fault_addr, need_vram ? true : false);if (ret == -EAGAIN ) {/* drivers/gpu/drm/xe/xe_svm.h +5
-4@@ -21,6 +21,7 @@ struct drm_file;struct xe_bo ;struct xe_gt ;struct xe_device ; +struct xe_pagefault ;struct xe_vram_region ;struct xe_tile ;struct xe_vm ;@@ -109,8 +110,8 @@ void xe_svm_fini(struct xe_vm *vm);void xe_svm_close (struct xe_vm *vm);int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic); +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic);bool xe_svm_has_mapping (struct xe_vm *vm,u64 start,u64 end);@@ -298,8 +299,8 @@ void xe_svm_close(struct xe_vm *vm)static inline int xe_svm_handle_pagefault (struct xe_vm *vm,struct xe_vma *vma, -struct xe_gt *gt,u64 fault_addr, -bool atomic) +struct xe_pagefault *pf,struct xe_gt *gt, +u64 fault_addr,bool atomic) {return 0 ; }Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 9/12 drm/xe: Add pagefault chaining stats
In reply to Matthew BrostAdd GT stats to quantify pagefault chaining behavior during fault storms.
Track total chained faults, faults chained directly from the IRQ handler, cases where IRQ chaining also drained the queue, chained faults that had to be requeued due to range mismatch, and cases where the last serviced range allowed immediate ack.
--- drivers/gpu/drm/xe/xe_gt_stats.c | 5 +++++ drivers/gpu/drm/xe/xe_gt_stats_types.h | 16 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 19 +++++++++++++++++-- 3 files changed, 38 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_gt_stats.c +5 -0
@@ -95,6 +95,11 @@ void xe_gt_stats_incr(struct xe_gt *gt, const enum xe_gt_stats_id id, int incr)#define DEF_STAT_STR (ID , name) [XE_GT_STATS_ID_##ID] = namestatic const char *const stat_description[__XE_GT_STATS_NUM_IDS] = { +DEF_STAT_STR (CHAIN_PAGEFAULT_COUNT ,"chain_pagefault_count" ), +DEF_STAT_STR (CHAIN_IRQ_PAGEFAULT_COUNT ,"chain_irq_pagefault_count" ), +DEF_STAT_STR (CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT ,"chain_drain_irq_pagefault_count" ), +DEF_STAT_STR (CHAIN_MISMATCH_PAGEFAULT_COUNT ,"chain_mismatch_pagefault_count" ), +DEF_STAT_STR (LAST_PAGEFAULT_COUNT ,"last_pagefault_count" ),DEF_STAT_STR (SVM_PAGEFAULT_COUNT ,"svm_pagefault_count" ),DEF_STAT_STR (TLB_INVAL ,"tlb_inval_count" ),DEF_STAT_STR (SVM_TLB_INVAL_COUNT ,"svm_tlb_inval_count" ),
drivers/gpu/drm/xe/xe_gt_stats_types.h +16 -0
@@ -10,6 +10,17 @@/** * enum xe_gt_stats_id - GT statistics identifiers +* @XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT: Total page faults chained onto an +* in-flight fault instead of being queued separately. +* @XE_GT_STATS_ID_CHAIN_IRQ_PAGEFAULT_COUNT: Page faults chained directly from +* the IRQ handler. +* @XE_GT_STATS_ID_CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT: IRQ-handler chained faults +* that also drained the fault queue. +* @XE_GT_STATS_ID_CHAIN_MISMATCH_PAGEFAULT_COUNT: Chained faults requeued +* because their fault range did not match the fault they were chained onto. +* @XE_GT_STATS_ID_LAST_PAGEFAULT_COUNT: Faults whose range matched the last +* serviced range, allowing an immediate ack. +* * @XE_GT_STATS_ID_SVM_PAGEFAULT_COUNT: Total SVM page faults handled. * @XE_GT_STATS_ID_TLB_INVAL: Total GPU Translation Lookaside Buffer (TLB) * invalidations issued.
@@ -129,6 +140,11 @@* See Documentation/gpu/xe/xe_gt_stats.rst. */ enum xe_gt_stats_id { +XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT , +XE_GT_STATS_ID_CHAIN_IRQ_PAGEFAULT_COUNT , +XE_GT_STATS_ID_CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT , +XE_GT_STATS_ID_CHAIN_MISMATCH_PAGEFAULT_COUNT , +XE_GT_STATS_ID_LAST_PAGEFAULT_COUNT ,XE_GT_STATS_ID_SVM_PAGEFAULT_COUNT ,XE_GT_STATS_ID_TLB_INVAL ,XE_GT_STATS_ID_SVM_TLB_INVAL_COUNT ,
drivers/gpu/drm/xe/xe_pagefault.c +17 -2
@@ -351,6 +351,8 @@ xe_pagefault_queue_unchain_requeue(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_CHAINED ); +xe_gt_stats_incr (gt,XE_GT_STATS_ID_CHAIN_MISMATCH_PAGEFAULT_COUNT ,1 ); + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_FREE ; lpf =xe_pagefault_queue_add (pf_queue, pf);if (lpf) {
@@ -407,6 +409,10 @@ static bool xe_pagefault_try_chain(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf_work->cache.pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_gt_stats_incr (pf->gt, +XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT , +1 ); + pf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_CHAINED ; pf->consumer.next = pf_work->cache.pf->consumer.next;
@@ -582,8 +588,10 @@ static void xe_pagefault_queue_work(struct work_struct *w)bool invalidated = false;/* Last fault same address, ack immediately */ -if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) +if (xe_pagefault_match (pf, cache_start, cache_end, cache_asid)) { +xe_gt_stats_incr (gt,XE_GT_STATS_ID_LAST_PAGEFAULT_COUNT ,1 ); goto ack_fault; + } err =xe_pagefault_service (pf);
@@ -841,8 +849,15 @@ int xe_pagefault_handler(struct xe_device *xe, struct xe_pagefault *pf) lpf->consumer.next =NULL ;if (xe_pagefault_try_chain (pf_queue, lpf)) { -if (empty) +xe_gt_stats_incr (pf->gt, +XE_GT_STATS_ID_CHAIN_IRQ_PAGEFAULT_COUNT , +1 ); +if (empty) { +xe_gt_stats_incr (pf->gt, +XE_GT_STATS_ID_CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT , +1 );xe_pagefault_queue_advance (pf_queue); + } }else {int work_index =xe_pagefault_work_index (xe);
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 10/12 drm/xe: Add debugfs pagefault_info
In reply to Matthew BrostAdd a debugfs entry to dump Xe page fault queue state. The output includes queue geometry (entry size, total size, head/tail), per-entry allocation state counts, and whether each page fault worker cache is currently valid.
This is intended to help debug page fault storms, chaining, and retry behaviour without needing tracing.
--- drivers/gpu/drm/xe/xe_debugfs.c | 11 +++++ drivers/gpu/drm/xe/xe_pagefault.c | 67 +++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.h | 3 ++ 3 files changed, 81 insertions(+)
drivers/gpu/drm/xe/xe_debugfs.c +11 -0
@@ -22,6 +22,7 @@#include "xe_guc_ads.h" #include "xe_hw_engine.h" #include "xe_mmio.h" +#include "xe_pagefault.h" #include "xe_pcode.h" #include "xe_pm.h" #include "xe_psmi.h"
@@ -194,6 +195,15 @@ static int sriov_info(struct seq_file *m, void *data)return 0 ; } +static int pagefault_info (struct seq_file *m,void *data) +{ +struct xe_device *xe =node_to_xe (m->private); +struct drm_printer p =drm_seq_file_printer (m); + +xe_pagefault_print_info (xe, &p); +return 0 ; +} +static int workarounds (struct xe_device *xe,struct drm_printer *p) {guard (xe_pm_runtime)(xe);
@@ -285,6 +295,7 @@ static const struct drm_info_list debugfs_list[] = { {"info", info, 0}, { .name = "sriov_info", .show = sriov_info, }, { .name = "workarounds", .show = workaround_info, }, + { .name = "pagefault_info", .show = pagefault_info, }, }; static const struct drm_info_list pcode_info_debugfs[] = {
drivers/gpu/drm/xe/xe_pagefault.c +67 -0
82 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pagefault.h +3 -0
@@ -8,6 +8,7 @@#include "xe_pagefault_types.h" +struct drm_printer ;struct xe_device ;struct xe_gt ;struct xe_pagefault ;
@@ -18,6 +19,8 @@ void xe_pagefault_reset(struct xe_device *xe, struct xe_gt *gt);int xe_pagefault_handler (struct xe_device *xe,struct xe_pagefault *pf); +void xe_pagefault_print_info (struct xe_device *xe,struct drm_printer *p); +#define XE_PAGEFAULT_END_ADDR_MASK (~0xfffull)/**
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 11/12 drm/xe: batch CT pagefault acks with periodic flush
In reply to Matthew BrostPagefault storms can generate long chains of acknowledgments back to the GuC. Sending each ack as a full CT submission forces a barrier, descriptor update and doorbell per fault.
Extend xe_guc_ct_send_locked() with a “write-only” mode that copies the message into the H2G ring but defers publishing the descriptor and ringing the doorbell. Add xe_guc_ct_send_flush() to publish pending writes and notify GuC once per batch. Wire this into the pagefault producer via new ack_fault_begin/ack_fault_end callbacks and CT lock wrappers.
To avoid excessive flush latency while still amortizing MMIO costs, use a simple periodic flush heuristic for GuC pagefault acks: batch most acks as write-only and force a publish at a fixed interval (e.g., every 16th ack), with a final flush at end-of-batch.
Also increase the H2G CTB size to 16K to better absorb bursts.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 95 +++++++++++++++++++------ drivers/gpu/drm/xe/xe_guc_ct.h | 36 +++++++++- drivers/gpu/drm/xe/xe_guc_pagefault.c | 32 ++++++++- drivers/gpu/drm/xe/xe_guc_types.h | 6 ++ drivers/gpu/drm/xe/xe_pagefault.c | 12 +++- drivers/gpu/drm/xe/xe_pagefault_types.h | 14 ++++ 6 files changed, 170 insertions(+), 25 deletions(-)
drivers/gpu/drm/xe/xe_guc_ct.c +73 -22
@@ -265,7 +265,7 @@ static bool g2h_fence_needs_alloc(struct g2h_fence *g2h_fence)#define CTB_DESC_SIZE ALIGN(sizeof(struct guc_ct_buffer_desc), SZ_2K)#define CTB_H2G_BUFFER_OFFSET (CTB_DESC_SIZE * 2)#define CTB_G2H_BUFFER_OFFSET (CTB_DESC_SIZE * 2) -#define CTB_H2G_BUFFER_SIZE (SZ_4K) +#define CTB_H2G_BUFFER_SIZE (SZ_16K)#define CTB_H2G_BUFFER_DWORDS (CTB_H2G_BUFFER_SIZE / sizeof(u32))#define CTB_G2H_BUFFER_SIZE (SZ_128K)#define CTB_G2H_BUFFER_DWORDS (CTB_G2H_BUFFER_SIZE / sizeof(u32))
@@ -939,7 +939,7 @@ static bool vf_action_can_safely_fail(struct xe_device *xe, u32 action)#define H2G_CT_HEADERS (GUC_CTB_HDR_LEN + 1)/* one DW CTB header and one DW HxG header */ static int h2g_write (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 ct_fence_value,bool want_response) +u32 ct_fence_value,bool want_response,bool write_only) {struct xe_device *xe =ct_to_xe (ct);struct xe_gt *gt =ct_to_gt (ct);
@@ -956,7 +956,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (gt, full_len <=GUC_CTB_MSG_MAX_LEN );if (IS_ENABLED (CONFIG_DRM_XE_DEBUG )) { -u32 desc_tail =desc_read (xe, h2g, tail);u32 desc_head =desc_read (xe, h2g, head);u32 desc_status;
@@ -966,12 +965,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, goto corrupted; } -if (tail != desc_tail) { -desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_MISMATCH ); -xe_gt_err (gt,"CT write: tail was modified %u != %u\n" , desc_tail, tail); - goto corrupted; - } -if (tail > h2g->info.size) {desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_OVERFLOW ); xe_gt_err(gt, "CT write: tail out of range: %u vs %u\n",
@@ -993,7 +986,10 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, (h2g->info.size - tail) * sizeof(u32));h2g_reserve_space (ct, (h2g->info.size - tail)); h2g->info.tail =0 ; -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }return -EAGAIN ; }
@@ -1024,14 +1020,15 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,/* Write H2G ensuring visible before descriptor update */ xe_map_memcpy_to (xe, &map,0 , cmd,H2G_CT_HEADERS *sizeof (u32));xe_map_memcpy_to (xe, &map,H2G_CT_HEADERS *sizeof (u32), action, len *sizeof (u32)); -xe_device_wmb (xe); -/* Update local copies */ h2g->info.tail = (tail + full_len) % h2g->info.size;h2g_reserve_space (ct, full_len);/* Update descriptor */ -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }/* * desc_read() performs an VRAM read which serializes the CPU and drains
@@ -1052,7 +1049,7 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,static int __guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);u16 seqno;
@@ -1112,7 +1109,7 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action,if (unlikely (ret)) goto out_unlock; - ret =h2g_write (ct, action, len, seqno, !!g2h_fence); + ret =h2g_write (ct, action, len, seqno, !!g2h_fence, write_only);if (unlikely (ret)) {if (ret == -EAGAIN ) goto retry;
@@ -1120,7 +1117,8 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, }__g2h_reserve_space (ct, g2h_len, num_g2h); -xe_guc_notify (ct_to_guc (ct)); +if (!write_only) +xe_guc_notify (ct_to_guc (ct)); out_unlock:if (g2h_len)spin_unlock_irq (&ct->fast_lock);
@@ -1196,7 +1194,7 @@ static bool guc_ct_send_wait_for_retry(struct xe_guc_ct *ct, u32 len,static int guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);unsigned int sleep_period_ms =1 ;
@@ -1209,9 +1207,10 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len, try_again: ret =__guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, - g2h_fence); + g2h_fence, write_only);if (unlikely (ret == -EBUSY )) { +xe_guc_ct_send_flush (ct);if (!guc_ct_send_wait_for_retry (ct, len, g2h_len, g2h_fence, &sleep_period_ms, &sleep_total_ms)) goto broken;
@@ -1235,7 +1234,8 @@ static int guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (ct_to_gt (ct), !g2h_len || !g2h_fence);mutex_lock (&ct->lock); - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence); + ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence, + false);mutex_unlock (&ct->lock);return ret;
@@ -1283,25 +1283,76 @@ int xe_guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,return ret; } +/** +* xe_guc_ct_send_locked() - submit a GuC CT H2G message with CT lock held +* @ct: GuC CT object +* @action: payload dwords (HxG header dword is expected at @action[-1]) +* @len: number of payload dwords in @action +* @write_only: defer publishing/doorbell for batching +* +* Sends a single H2G message to the GuC CT buffer while the caller already +* holds @ct->lock. +* +* If @write_only is false, the function completes the submission immediately: +* it makes the payload visible to the device, updates the H2G descriptor and +* rings the GuC doorbell. +* +* If @write_only is true, the message payload is copied into the H2G ring and +* the software tail is advanced, but the descriptor update and doorbell are +* deferred so multiple messages can be batched. In this mode, the caller must +* eventually call xe_guc_ct_send_flush() (still holding @ct->lock) to publish +* the descriptor and notify the GuC. On internal retry paths (-EBUSY), the +* implementation may force a flush to ensure forward progress. +* +* Return: 0 on success, negative errno on failure. +* +* Locking: +* Must be called with @ct->lock held. +*/ int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h) +bool write_only) {int ret; - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , write_only);if (ret == -EDEADLK )kick_reset (ct);return ret; } +/** +* xe_guc_ct_send_flush() - flush pending GuC CT H2G writes +* @ct: GuC CT instance +* +* Some callers batch multiple H2G writes using xe_guc_ct_send_locked() in +* "write-only" mode (i.e., queue the message payloads but defer ringing the +* doorbell / updating the CT descriptor). This helper completes the submission +* by ensuring the payload writes are visible to the device, updating the H2G +* descriptor, and ringing the GuC CT doorbell. +* +* Locking: +* Must be called with @ct->lock held. +*/ +void xe_guc_ct_send_flush (struct xe_guc_ct *ct) +{ +struct xe_device *xe =ct_to_xe (ct); +struct guc_ctb *h2g = &ct->ctbs.h2g; + +lockdep_assert_held (&ct->lock); + +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); +xe_guc_notify (ct_to_guc (ct)); +} +int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len) {int ret;lockdep_assert_held (&ct->lock); - ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , false);if (ret == -EDEADLK )kick_reset (ct);
drivers/gpu/drm/xe/xe_guc_ct.h +35 -1
@@ -54,7 +54,7 @@ static inline void xe_guc_ct_irq_handler(struct xe_guc_ct *ct)int xe_guc_ct_send (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h);int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h); +bool write_only);int xe_guc_ct_send_recv (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 *response_buffer); static inline int
@@ -63,6 +63,8 @@ xe_guc_ct_send_block(struct xe_guc_ct *ct, const u32 *action, u32 len)return xe_guc_ct_send_recv (ct, action, len,NULL ); } +void xe_guc_ct_send_flush (struct xe_guc_ct *ct); +/* This is only version of the send CT you can call from a G2H handler */ int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len);
@@ -87,4 +89,36 @@ static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct) wake_up_all(&ct->wq); } +/** +* xe_guc_ct_lock() - take the GuC CT mutex +* @ct: GuC CT object +* +* Wrapper around mutex_lock(&ct->lock) for cases where CT operations need to be +* performed from contexts that want an explicit "CT locked" pair without +* exporting the lock itself. +* +* Return/Locking: +* Acquires @ct->lock. +*/ +static inline void xe_guc_ct_lock (struct xe_guc_ct *ct) +__acquires(&ct->lock) +{ +mutex_lock (&ct->lock); +} + +/** +* xe_guc_ct_unlock() - release the GuC CT mutex +* @ct: GuC CT object +* +* Counterpart to xe_guc_ct_lock(). +* +* Locking: +* Releases @ct->lock. +*/ +static inline void xe_guc_ct_unlock (struct xe_guc_ct *ct) +__releases(&ct->lock) +{ +mutex_unlock (&ct->lock); +} +#endif
drivers/gpu/drm/xe/xe_guc_pagefault.c +31 -1
@@ -10,6 +10,22 @@#include "xe_pagefault.h" #include "xe_pagefault_types.h" +#define XE_GUC_PAGEFAULT_FLUSH_PERIOD BIT(4)/* Sixteen */ + +static void guc_ack_fault_begin (void *private) +{ +struct xe_guc *guc = private; + +xe_guc_ct_lock (&guc->ct); + +BUILD_BUG_ON (((XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ) & +XE_GUC_PAGEFAULT_FLUSH_PERIOD ) !=0 ); + +/* Ack the 2nd, then 18th, etc... */ + guc->pagefault_ack_counter = +XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ; +} +static void guc_ack_fault (struct xe_pagefault *pf,int err) {u32 vfid =FIELD_GET (PFD_VFID , pf->producer.msg[2 ]);
@@ -36,12 +52,26 @@ static void guc_ack_fault(struct xe_pagefault *pf, int err) FIELD_PREP(PFR_PDATA, pdata), };struct xe_guc *guc = pf->producer.private; +bool write_only = guc->pagefault_ack_counter++ & + (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ); + +xe_guc_ct_send_locked (&guc->ct, action,ARRAY_SIZE (action), + write_only); +} + +static void guc_ack_fault_end (void *private) +{ +struct xe_guc *guc = private; -xe_guc_ct_send (&guc->ct, action,ARRAY_SIZE (action),0 ,0 ); +if ((guc->pagefault_ack_counter & (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 )) !=1 ) +xe_guc_ct_send_flush (&guc->ct); +xe_guc_ct_unlock (&guc->ct); }static const struct xe_pagefault_ops guc_pagefault_ops = { + .ack_fault_begin = guc_ack_fault_begin, .ack_fault = guc_ack_fault, + .ack_fault_end = guc_ack_fault_end, };/**
drivers/gpu/drm/xe/xe_guc_types.h +6 -0
@@ -122,6 +122,12 @@ struct xe_guc {struct xe_reg notify_reg;/** @params: Control params for fw initialization */ u32 params[GUC_CTL_MAX_DWORDS ]; + +/** +* @pagefault_ack_counter: Counter to determine when periodically ack +* pagefaults in a batch. +*/ + u32 pagefault_ack_counter; };#endif
drivers/gpu/drm/xe/xe_pagefault.c +11 -1
@@ -412,6 +412,10 @@ static bool xe_pagefault_try_chain(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf_work->cache.pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +if (pf->producer.private != + pf_work->cache.pf->producer.private) +continue ; +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT ,1 );
@@ -585,6 +589,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS );while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +const struct xe_pagefault_ops *ops = pf->producer.ops; +void *private = pf->producer.private;struct xe_gt *gt = pf->gt;u32 asid = pf->consumer.asid;int err =0 ;
@@ -622,11 +628,14 @@ static void xe_pagefault_queue_work(struct work_struct *w) XE_PAGEFAULT_ALLOC_STATE_ACTIVE);xe_assert (xe, pf == pf_work->cache.pf); + ops->ack_fault_begin (private);while (pf) {xe_assert (xe, pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, ops == pf->producer.ops); +xe_assert (xe, gt == pf->gt); - pf->producer.ops->ack_fault (pf, err); + ops->ack_fault (pf, err);spin_lock_irq (&pf_queue->lock);
@@ -654,6 +663,7 @@ static void xe_pagefault_queue_work(struct work_struct *w)XE_PAGEFAULT_ALLOC_STATE_ACTIVE ;spin_unlock_irq (&pf_queue->lock); } + ops->ack_fault_end (private);if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);
drivers/gpu/drm/xe/xe_pagefault_types.h +14 -0
@@ -33,6 +33,13 @@ enum xe_pagefault_type {/** struct xe_pagefault_ops - Xe pagefault ops (producer) */ struct xe_pagefault_ops { +/** +* @ack_fault_begin: Ack fault begin +* @private: producer private data +* +* Page fault producer begins acknowledgment from the consumer. +*/ +void (*ack_fault_begin)(void *private);/** * @ack_fault: Ack fault * @pf: Page fault
@@ -42,6 +49,13 @@ struct xe_pagefault_ops {* sends the result to the HW/FW interface. */ void (*ack_fault)(struct xe_pagefault *pf,int err); +/** +* @ack_fault_end: Ack fault end +* @private: producer private data +* +* Page fault producer ends acknowledgment from the consumer. +*/ +void (*ack_fault_end)(void *private); };/**
Signature · 1 line
2.34.1
Re: [PATCH v8 11/12] drm/xe: batch CT pagefault acks with periodic flush
LGTM, just a couple of nits below.
Quoted 5 lines · Matthew Brost
Pagefault storms can generate long chains of acknowledgments back to the GuC. Sending each ack as a full CT submission forces a barrier, descriptor update and doorbell per fault.
Extend
xe_guc_ct_send_locked()with a “write-only” mode that copies the
Wondering if calling the argument "defer_flush" instead of "write_only" would be a better fit, and also hint the caller to call xe_guc_ct_send_flush().
Quoted 55 lines
message into the H2G ring but defers publishing the descriptor and ringing the doorbell. Add
xe_guc_ct_send_flush()to publish pending writes and notify GuC once per batch. Wire this into the pagefault producer via new ack_fault_begin/ack_fault_end callbacks and CT lock wrappers.To avoid excessive flush latency while still amortizing MMIO costs, use a simple periodic flush heuristic for GuC pagefault acks: batch most acks as write-only and force a publish at a fixed interval (e.g., every 16th ack), with a final flush at end-of-batch.
Also increase the H2G CTB size to 16K to better absorb bursts.
Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>--- drivers/gpu/drm/xe/xe_guc_ct.c | 95 +++++++++++++++++++------ drivers/gpu/drm/xe/xe_guc_ct.h | 36 +++++++++- drivers/gpu/drm/xe/xe_guc_pagefault.c | 32 ++++++++- drivers/gpu/drm/xe/xe_guc_types.h | 6 ++ drivers/gpu/drm/xe/xe_pagefault.c | 12 +++- drivers/gpu/drm/xe/xe_pagefault_types.h | 14 ++++ 6 files changed, 170 insertions(+), 25 deletions(-)drivers/gpu/drm/xe/xe_guc_ct.c +2
-3@@ -265,7 +265,7 @@ static bool g2h_fence_needs_alloc(struct g2h_fence *g2h_fence)#define CTB_DESC_SIZE ALIGN(sizeof(struct guc_ct_buffer_desc), SZ_2K)#define CTB_H2G_BUFFER_OFFSET (CTB_DESC_SIZE * 2)#define CTB_G2H_BUFFER_OFFSET (CTB_DESC_SIZE * 2) -#define CTB_H2G_BUFFER_SIZE (SZ_4K) +#define CTB_H2G_BUFFER_SIZE (SZ_16K)#define CTB_H2G_BUFFER_DWORDS (CTB_H2G_BUFFER_SIZE / sizeof(u32))#define CTB_G2H_BUFFER_SIZE (SZ_128K)#define CTB_G2H_BUFFER_DWORDS (CTB_G2H_BUFFER_SIZE / sizeof(u32))@@ -939,7 +939,7 @@ static bool vf_action_can_safely_fail(struct xe_device *xe, u32 action)#define H2G_CT_HEADERS (GUC_CTB_HDR_LEN + 1)/* one DW CTB header and one DW HxG header */ static int h2g_write (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 ct_fence_value,bool want_response) +u32 ct_fence_value,bool want_response,bool write_only) {struct xe_device *xe =ct_to_xe (ct);struct xe_gt *gt =ct_to_gt (ct);@@ -956,7 +956,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (gt, full_len <=GUC_CTB_MSG_MAX_LEN );if (IS_ENABLED (CONFIG_DRM_XE_DEBUG )) { -u32 desc_tail =desc_read (xe, h2g, tail);u32 desc_head =desc_read (xe, h2g, head);u32 desc_status;
drivers/gpu/drm/xe/xe_guc_ct.c static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, · line 973
Quoted 8 lines
drivers/gpu/drm/xe/xe_guc_ct.c +0
-5@@ -966,12 +965,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, goto corrupted; } -if (tail != desc_tail) { -desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_MISMATCH ); -xe_gt_err (gt,"CT write: tail was modified %u != %u\n" , desc_tail, tail); - goto corrupted; - }
Could we keep that check when !write_only?
Quoted 205 lines
drivers/gpu/drm/xe/xe_guc_ct.c +71
-14-if (tail > h2g->info.size) {desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_OVERFLOW ); xe_gt_err(gt, "CT write: tail out of range: %u vs %u\n",@@ -993,7 +986,10 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, (h2g->info.size - tail) * sizeof(u32));h2g_reserve_space (ct, (h2g->info.size - tail)); h2g->info.tail =0 ; -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }return -EAGAIN ; }@@ -1024,14 +1020,15 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,/* Write H2G ensuring visible before descriptor update */ xe_map_memcpy_to (xe, &map,0 , cmd,H2G_CT_HEADERS *sizeof (u32));xe_map_memcpy_to (xe, &map,H2G_CT_HEADERS *sizeof (u32), action, len *sizeof (u32)); -xe_device_wmb (xe); -/* Update local copies */ h2g->info.tail = (tail + full_len) % h2g->info.size;h2g_reserve_space (ct, full_len);/* Update descriptor */ -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }/* * desc_read() performs an VRAM read which serializes the CPU and drains @@ -1052,7 +1049,7 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,static int __guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);u16 seqno;@@ -1112,7 +1109,7 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action,if (unlikely (ret)) goto out_unlock; - ret =h2g_write (ct, action, len, seqno, !!g2h_fence); + ret =h2g_write (ct, action, len, seqno, !!g2h_fence, write_only);if (unlikely (ret)) {if (ret == -EAGAIN ) goto retry;@@ -1120,7 +1117,8 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, }__g2h_reserve_space (ct, g2h_len, num_g2h); -xe_guc_notify (ct_to_guc (ct)); +if (!write_only) +xe_guc_notify (ct_to_guc (ct)); out_unlock:if (g2h_len)spin_unlock_irq (&ct->fast_lock);@@ -1196,7 +1194,7 @@ static bool guc_ct_send_wait_for_retry(struct xe_guc_ct *ct, u32 len,static int guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);unsigned int sleep_period_ms =1 ;@@ -1209,9 +1207,10 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len, try_again: ret =__guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, - g2h_fence); + g2h_fence, write_only);if (unlikely (ret == -EBUSY )) { +xe_guc_ct_send_flush (ct);if (!guc_ct_send_wait_for_retry (ct, len, g2h_len, g2h_fence, &sleep_period_ms, &sleep_total_ms)) goto broken;@@ -1235,7 +1234,8 @@ static int guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (ct_to_gt (ct), !g2h_len || !g2h_fence);mutex_lock (&ct->lock); - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence); + ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence, + false);mutex_unlock (&ct->lock);return ret;@@ -1283,25 +1283,76 @@ int xe_guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,return ret; } +/** +* xe_guc_ct_send_locked() - submit a GuC CT H2G message with CT lock held +* @ct: GuC CT object +* @action: payload dwords (HxG header dword is expected at @action[-1]) +* @len: number of payload dwords in @action +* @write_only: defer publishing/doorbell for batching +* +* Sends a single H2G message to the GuC CT buffer while the caller already +* holds @ct->lock. +* +* If @write_only is false, the function completes the submission immediately: +* it makes the payload visible to the device, updates the H2G descriptor and +* rings the GuC doorbell. +* +* If @write_only is true, the message payload is copied into the H2G ring and +* the software tail is advanced, but the descriptor update and doorbell are +* deferred so multiple messages can be batched. In this mode, the caller must +* eventually call xe_guc_ct_send_flush() (still holding @ct->lock) to publish +* the descriptor and notify the GuC. On internal retry paths (-EBUSY), the +* implementation may force a flush to ensure forward progress. +* +* Return: 0 on success, negative errno on failure. +* +* Locking: +* Must be called with @ct->lock held. +*/ int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h) +bool write_only) {int ret; - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , write_only);if (ret == -EDEADLK )kick_reset (ct);return ret; } +/** +* xe_guc_ct_send_flush() - flush pending GuC CT H2G writes +* @ct: GuC CT instance +* +* Some callers batch multiple H2G writes using xe_guc_ct_send_locked() in +* "write-only" mode (i.e., queue the message payloads but defer ringing the +* doorbell / updating the CT descriptor). This helper completes the submission +* by ensuring the payload writes are visible to the device, updating the H2G +* descriptor, and ringing the GuC CT doorbell. +* +* Locking: +* Must be called with @ct->lock held. +*/ +void xe_guc_ct_send_flush (struct xe_guc_ct *ct) +{ +struct xe_device *xe =ct_to_xe (ct); +struct guc_ctb *h2g = &ct->ctbs.h2g; + +lockdep_assert_held (&ct->lock); + +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); +xe_guc_notify (ct_to_guc (ct)); +} +int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len) {int ret;lockdep_assert_held (&ct->lock); - ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , false);if (ret == -EDEADLK )kick_reset (ct);drivers/gpu/drm/xe/xe_guc_ct.h +8
-1@@ -54,7 +54,7 @@ static inline void xe_guc_ct_irq_handler(struct xe_guc_ct *ct)int xe_guc_ct_send (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h);int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h); +bool write_only);int xe_guc_ct_send_recv (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 *response_buffer); static inline int@@ -63,6 +63,8 @@ xe_guc_ct_send_block(struct xe_guc_ct *ct, const u32 *action, u32 len)return xe_guc_ct_send_recv (ct, action, len,NULL ); } +void xe_guc_ct_send_flush (struct xe_guc_ct *ct); +/* This is only version of the send CT you can call from a G2H handler */ int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len);@@ -87,4 +89,36 @@ static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct)wake_up_all (&ct->wq); } +/** +* xe_guc_ct_lock() - take the GuC CT mutex +* @ct: GuC CT object +* +* Wrapper around mutex_lock(&ct->lock) for cases where CT operations need to be
drivers/gpu/drm/xe/xe_guc_ct.h static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct) · line 106
Quoted 10 lines
drivers/gpu/drm/xe/xe_guc_ct.h +10
-0+* performed from contexts that want an explicit "CT locked" pair without +* exporting the lock itself. +* +* Return/Locking: +* Acquires @ct->lock. +*/ +static inline void xe_guc_ct_lock (struct xe_guc_ct *ct) +__acquires(&ct->lock) +{ +mutex_lock (&ct->lock);
Now it would be cleaner to add "#include <linux/mutex.h>" in xe_guc_ct.h.
Francois
Quoted 173 lines
drivers/gpu/drm/xe/xe_guc_ct.h +17
-0+} + +/** +* xe_guc_ct_unlock() - release the GuC CT mutex +* @ct: GuC CT object +* +* Counterpart to xe_guc_ct_lock(). +* +* Locking: +* Releases @ct->lock. +*/ +static inline void xe_guc_ct_unlock (struct xe_guc_ct *ct) +__releases(&ct->lock) +{ +mutex_unlock (&ct->lock); +} +#endif drivers/gpu/drm/xe/xe_guc_pagefault.c +31
-1@@ -10,6 +10,22 @@#include "xe_pagefault.h" #include "xe_pagefault_types.h" +#define XE_GUC_PAGEFAULT_FLUSH_PERIOD BIT(4)/* Sixteen */ + +static void guc_ack_fault_begin (void *private) +{ +struct xe_guc *guc = private; + +xe_guc_ct_lock (&guc->ct); + +BUILD_BUG_ON (((XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ) & +XE_GUC_PAGEFAULT_FLUSH_PERIOD ) !=0 ); + +/* Ack the 2nd, then 18th, etc... */ + guc->pagefault_ack_counter = +XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ; +} +static void guc_ack_fault (struct xe_pagefault *pf,int err) {u32 vfid =FIELD_GET (PFD_VFID , pf->producer.msg[2 ]);@@ -36,12 +52,26 @@ static void guc_ack_fault(struct xe_pagefault *pf, int err) FIELD_PREP(PFR_PDATA, pdata), };struct xe_guc *guc = pf->producer.private; +bool write_only = guc->pagefault_ack_counter++ & + (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ); + +xe_guc_ct_send_locked (&guc->ct, action,ARRAY_SIZE (action), + write_only); +} + +static void guc_ack_fault_end (void *private) +{ +struct xe_guc *guc = private; -xe_guc_ct_send (&guc->ct, action,ARRAY_SIZE (action),0 ,0 ); +if ((guc->pagefault_ack_counter & (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 )) !=1 ) +xe_guc_ct_send_flush (&guc->ct); +xe_guc_ct_unlock (&guc->ct); }static const struct xe_pagefault_ops guc_pagefault_ops = { + .ack_fault_begin = guc_ack_fault_begin, .ack_fault = guc_ack_fault, + .ack_fault_end = guc_ack_fault_end, };/** drivers/gpu/drm/xe/xe_guc_types.h +6
-0@@ -122,6 +122,12 @@ struct xe_guc {struct xe_reg notify_reg;/** @params: Control params for fw initialization */ u32 params[GUC_CTL_MAX_DWORDS ]; + +/** +* @pagefault_ack_counter: Counter to determine when periodically ack +* pagefaults in a batch. +*/ + u32 pagefault_ack_counter; };#endif drivers/gpu/drm/xe/xe_pagefault.c +11
-1@@ -412,6 +412,10 @@ static bool xe_pagefault_try_chain(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf_work->cache.pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +if (pf->producer.private != + pf_work->cache.pf->producer.private) +continue ; +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT ,1 );@@ -585,6 +589,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS );while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +const struct xe_pagefault_ops *ops = pf->producer.ops; +void *private = pf->producer.private;struct xe_gt *gt = pf->gt;u32 asid = pf->consumer.asid;int err =0 ;@@ -622,11 +628,14 @@ static void xe_pagefault_queue_work(struct work_struct *w) XE_PAGEFAULT_ALLOC_STATE_ACTIVE);xe_assert (xe, pf == pf_work->cache.pf); + ops->ack_fault_begin (private);while (pf) {xe_assert (xe, pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, ops == pf->producer.ops); +xe_assert (xe, gt == pf->gt); - pf->producer.ops->ack_fault (pf, err); + ops->ack_fault (pf, err);spin_lock_irq (&pf_queue->lock);@@ -654,6 +663,7 @@ static void xe_pagefault_queue_work(struct work_struct *w)XE_PAGEFAULT_ALLOC_STATE_ACTIVE ;spin_unlock_irq (&pf_queue->lock); } + ops->ack_fault_end (private);if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);drivers/gpu/drm/xe/xe_pagefault_types.h +14
-0@@ -33,6 +33,13 @@ enum xe_pagefault_type {/** struct xe_pagefault_ops - Xe pagefault ops (producer) */ struct xe_pagefault_ops { +/** +* @ack_fault_begin: Ack fault begin +* @private: producer private data +* +* Page fault producer begins acknowledgment from the consumer. +*/ +void (*ack_fault_begin)(void *private);/** * @ack_fault: Ack fault * @pf: Page fault @@ -42,6 +49,13 @@ struct xe_pagefault_ops {* sends the result to the HW/FW interface. */ void (*ack_fault)(struct xe_pagefault *pf,int err); +/** +* @ack_fault_end: Ack fault end +* @private: producer private data +* +* Page fault producer ends acknowledgment from the consumer. +*/ +void (*ack_fault_end)(void *private); };/** Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v8 11/12] drm/xe: batch CT pagefault acks with periodic flush
Quoted 70 lines · Francois Dugast
LGTM, just a couple of nits below.
Quoted 5 lines · Matthew Brost
Pagefault storms can generate long chains of acknowledgments back to the GuC. Sending each ack as a full CT submission forces a barrier, descriptor update and doorbell per fault.
Extend
xe_guc_ct_send_locked()with a “write-only” mode that copies theWondering if calling the argument "
defer_flush" instead of "write_only" would be a better fit, and also hint the caller to callxe_guc_ct_send_flush().Quoted 58 lines
message into the H2G ring but defers publishing the descriptor and ringing the doorbell. Add
xe_guc_ct_send_flush()to publish pending writes and notify GuC once per batch. Wire this into the pagefault producer via new ack_fault_begin/ack_fault_end callbacks and CT lock wrappers.To avoid excessive flush latency while still amortizing MMIO costs, use a simple periodic flush heuristic for GuC pagefault acks: batch most acks as write-only and force a publish at a fixed interval (e.g., every 16th ack), with a final flush at end-of-batch.
Also increase the H2G CTB size to 16K to better absorb bursts.
Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>--- drivers/gpu/drm/xe/xe_guc_ct.c | 95 +++++++++++++++++++------ drivers/gpu/drm/xe/xe_guc_ct.h | 36 +++++++++- drivers/gpu/drm/xe/xe_guc_pagefault.c | 32 ++++++++- drivers/gpu/drm/xe/xe_guc_types.h | 6 ++ drivers/gpu/drm/xe/xe_pagefault.c | 12 +++- drivers/gpu/drm/xe/xe_pagefault_types.h | 14 ++++ 6 files changed, 170 insertions(+), 25 deletions(-)drivers/gpu/drm/xe/xe_guc_ct.c +2
-3@@ -265,7 +265,7 @@ static bool g2h_fence_needs_alloc(struct g2h_fence *g2h_fence)#define CTB_DESC_SIZE ALIGN(sizeof(struct guc_ct_buffer_desc), SZ_2K)#define CTB_H2G_BUFFER_OFFSET (CTB_DESC_SIZE * 2)#define CTB_G2H_BUFFER_OFFSET (CTB_DESC_SIZE * 2) -#define CTB_H2G_BUFFER_SIZE (SZ_4K) +#define CTB_H2G_BUFFER_SIZE (SZ_16K)#define CTB_H2G_BUFFER_DWORDS (CTB_H2G_BUFFER_SIZE / sizeof(u32))#define CTB_G2H_BUFFER_SIZE (SZ_128K)#define CTB_G2H_BUFFER_DWORDS (CTB_G2H_BUFFER_SIZE / sizeof(u32))@@ -939,7 +939,7 @@ static bool vf_action_can_safely_fail(struct xe_device *xe, u32 action)#define H2G_CT_HEADERS (GUC_CTB_HDR_LEN + 1)/* one DW CTB header and one DW HxG header */ static int h2g_write (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 ct_fence_value,bool want_response) +u32 ct_fence_value,bool want_response,bool write_only) {struct xe_device *xe =ct_to_xe (ct);struct xe_gt *gt =ct_to_gt (ct);@@ -956,7 +956,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (gt, full_len <=GUC_CTB_MSG_MAX_LEN );if (IS_ENABLED (CONFIG_DRM_XE_DEBUG )) { -u32 desc_tail =desc_read (xe, h2g, tail);u32 desc_head =desc_read (xe, h2g, head);u32 desc_status;@@ -966,12 +965,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, goto corrupted; }
Quoted 8 lines
drivers/gpu/drm/xe/xe_guc_ct.c static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, · line 973
Quoted 5 lines
drivers/gpu/drm/xe/xe_guc_ct.c +0
-5-if (tail != desc_tail) { -desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_MISMATCH ); -xe_gt_err (gt,"CT write: tail was modified %u != %u\n" , desc_tail, tail); - goto corrupted; - }Could we keep that check when !write_only?
Not without a bigger change. Consider write_only followed by !write_only, the tail != desc_tail immediately mismatches.
We'd have to add 3 states:
- Normal (check)
- Defer flush (no check)
- Issue flush (no check)
Not sure if it is worth the complexity, but change add if this if you want.
Quoted 209 lines
Quoted 209 lines
drivers/gpu/drm/xe/xe_guc_ct.c +71
-14-if (tail > h2g->info.size) {desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_OVERFLOW ); xe_gt_err(gt, "CT write: tail out of range: %u vs %u\n",@@ -993,7 +986,10 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, (h2g->info.size - tail) * sizeof(u32));h2g_reserve_space (ct, (h2g->info.size - tail)); h2g->info.tail =0 ; -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }return -EAGAIN ; }@@ -1024,14 +1020,15 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,/* Write H2G ensuring visible before descriptor update */ xe_map_memcpy_to (xe, &map,0 , cmd,H2G_CT_HEADERS *sizeof (u32));xe_map_memcpy_to (xe, &map,H2G_CT_HEADERS *sizeof (u32), action, len *sizeof (u32)); -xe_device_wmb (xe); -/* Update local copies */ h2g->info.tail = (tail + full_len) % h2g->info.size;h2g_reserve_space (ct, full_len);/* Update descriptor */ -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }/* * desc_read() performs an VRAM read which serializes the CPU and drains @@ -1052,7 +1049,7 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,static int __guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);u16 seqno;@@ -1112,7 +1109,7 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action,if (unlikely (ret)) goto out_unlock; - ret =h2g_write (ct, action, len, seqno, !!g2h_fence); + ret =h2g_write (ct, action, len, seqno, !!g2h_fence, write_only);if (unlikely (ret)) {if (ret == -EAGAIN ) goto retry;@@ -1120,7 +1117,8 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, }__g2h_reserve_space (ct, g2h_len, num_g2h); -xe_guc_notify (ct_to_guc (ct)); +if (!write_only) +xe_guc_notify (ct_to_guc (ct)); out_unlock:if (g2h_len)spin_unlock_irq (&ct->fast_lock);@@ -1196,7 +1194,7 @@ static bool guc_ct_send_wait_for_retry(struct xe_guc_ct *ct, u32 len,static int guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);unsigned int sleep_period_ms =1 ;@@ -1209,9 +1207,10 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len, try_again: ret =__guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, - g2h_fence); + g2h_fence, write_only);if (unlikely (ret == -EBUSY )) { +xe_guc_ct_send_flush (ct);if (!guc_ct_send_wait_for_retry (ct, len, g2h_len, g2h_fence, &sleep_period_ms, &sleep_total_ms)) goto broken;@@ -1235,7 +1234,8 @@ static int guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (ct_to_gt (ct), !g2h_len || !g2h_fence);mutex_lock (&ct->lock); - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence); + ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence, + false);mutex_unlock (&ct->lock);return ret;@@ -1283,25 +1283,76 @@ int xe_guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,return ret; } +/** +* xe_guc_ct_send_locked() - submit a GuC CT H2G message with CT lock held +* @ct: GuC CT object +* @action: payload dwords (HxG header dword is expected at @action[-1]) +* @len: number of payload dwords in @action +* @write_only: defer publishing/doorbell for batching +* +* Sends a single H2G message to the GuC CT buffer while the caller already +* holds @ct->lock. +* +* If @write_only is false, the function completes the submission immediately: +* it makes the payload visible to the device, updates the H2G descriptor and +* rings the GuC doorbell. +* +* If @write_only is true, the message payload is copied into the H2G ring and +* the software tail is advanced, but the descriptor update and doorbell are +* deferred so multiple messages can be batched. In this mode, the caller must +* eventually call xe_guc_ct_send_flush() (still holding @ct->lock) to publish +* the descriptor and notify the GuC. On internal retry paths (-EBUSY), the +* implementation may force a flush to ensure forward progress. +* +* Return: 0 on success, negative errno on failure. +* +* Locking: +* Must be called with @ct->lock held. +*/ int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h) +bool write_only) {int ret; - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , write_only);if (ret == -EDEADLK )kick_reset (ct);return ret; } +/** +* xe_guc_ct_send_flush() - flush pending GuC CT H2G writes +* @ct: GuC CT instance +* +* Some callers batch multiple H2G writes using xe_guc_ct_send_locked() in +* "write-only" mode (i.e., queue the message payloads but defer ringing the +* doorbell / updating the CT descriptor). This helper completes the submission +* by ensuring the payload writes are visible to the device, updating the H2G +* descriptor, and ringing the GuC CT doorbell. +* +* Locking: +* Must be called with @ct->lock held. +*/ +void xe_guc_ct_send_flush (struct xe_guc_ct *ct) +{ +struct xe_device *xe =ct_to_xe (ct); +struct guc_ctb *h2g = &ct->ctbs.h2g; + +lockdep_assert_held (&ct->lock); + +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); +xe_guc_notify (ct_to_guc (ct)); +} +int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len) {int ret;lockdep_assert_held (&ct->lock); - ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , false);if (ret == -EDEADLK )kick_reset (ct);drivers/gpu/drm/xe/xe_guc_ct.h +12
-1@@ -54,7 +54,7 @@ static inline void xe_guc_ct_irq_handler(struct xe_guc_ct *ct)int xe_guc_ct_send (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h);int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h); +bool write_only);int xe_guc_ct_send_recv (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 *response_buffer); static inline int@@ -63,6 +63,8 @@ xe_guc_ct_send_block(struct xe_guc_ct *ct, const u32 *action, u32 len)return xe_guc_ct_send_recv (ct, action, len,NULL ); } +void xe_guc_ct_send_flush (struct xe_guc_ct *ct); +/* This is only version of the send CT you can call from a G2H handler */ int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len);@@ -87,4 +89,36 @@ static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct)wake_up_all (&ct->wq); } +/** +* xe_guc_ct_lock() - take the GuC CT mutex +* @ct: GuC CT object +* +* Wrapper around mutex_lock(&ct->lock) for cases where CT operations need to be +* performed from contexts that want an explicit "CT locked" pair without +* exporting the lock itself. +* +* Return/Locking:
Quoted 9 lines
drivers/gpu/drm/xe/xe_guc_ct.h static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct) · line 106
Quoted 6 lines
drivers/gpu/drm/xe/xe_guc_ct.h +6
-0+* Acquires @ct->lock. +*/ +static inline void xe_guc_ct_lock (struct xe_guc_ct *ct) +__acquires(&ct->lock) +{ +mutex_lock (&ct->lock);Now it would be cleaner to add "#include <
linux/mutex.h>" inxe_guc_ct.h.
Sure.
Matt
Quoted 175 lines
Francois
Quoted 173 lines
drivers/gpu/drm/xe/xe_guc_ct.h +17
-0+} + +/** +* xe_guc_ct_unlock() - release the GuC CT mutex +* @ct: GuC CT object +* +* Counterpart to xe_guc_ct_lock(). +* +* Locking: +* Releases @ct->lock. +*/ +static inline void xe_guc_ct_unlock (struct xe_guc_ct *ct) +__releases(&ct->lock) +{ +mutex_unlock (&ct->lock); +} +#endif drivers/gpu/drm/xe/xe_guc_pagefault.c +31
-1@@ -10,6 +10,22 @@#include "xe_pagefault.h" #include "xe_pagefault_types.h" +#define XE_GUC_PAGEFAULT_FLUSH_PERIOD BIT(4)/* Sixteen */ + +static void guc_ack_fault_begin (void *private) +{ +struct xe_guc *guc = private; + +xe_guc_ct_lock (&guc->ct); + +BUILD_BUG_ON (((XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ) & +XE_GUC_PAGEFAULT_FLUSH_PERIOD ) !=0 ); + +/* Ack the 2nd, then 18th, etc... */ + guc->pagefault_ack_counter = +XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ; +} +static void guc_ack_fault (struct xe_pagefault *pf,int err) {u32 vfid =FIELD_GET (PFD_VFID , pf->producer.msg[2 ]);@@ -36,12 +52,26 @@ static void guc_ack_fault(struct xe_pagefault *pf, int err) FIELD_PREP(PFR_PDATA, pdata), };struct xe_guc *guc = pf->producer.private; +bool write_only = guc->pagefault_ack_counter++ & + (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ); + +xe_guc_ct_send_locked (&guc->ct, action,ARRAY_SIZE (action), + write_only); +} + +static void guc_ack_fault_end (void *private) +{ +struct xe_guc *guc = private; -xe_guc_ct_send (&guc->ct, action,ARRAY_SIZE (action),0 ,0 ); +if ((guc->pagefault_ack_counter & (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 )) !=1 ) +xe_guc_ct_send_flush (&guc->ct); +xe_guc_ct_unlock (&guc->ct); }static const struct xe_pagefault_ops guc_pagefault_ops = { + .ack_fault_begin = guc_ack_fault_begin, .ack_fault = guc_ack_fault, + .ack_fault_end = guc_ack_fault_end, };/** drivers/gpu/drm/xe/xe_guc_types.h +6
-0@@ -122,6 +122,12 @@ struct xe_guc {struct xe_reg notify_reg;/** @params: Control params for fw initialization */ u32 params[GUC_CTL_MAX_DWORDS ]; + +/** +* @pagefault_ack_counter: Counter to determine when periodically ack +* pagefaults in a batch. +*/ + u32 pagefault_ack_counter; };#endif drivers/gpu/drm/xe/xe_pagefault.c +11
-1@@ -412,6 +412,10 @@ static bool xe_pagefault_try_chain(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf_work->cache.pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +if (pf->producer.private != + pf_work->cache.pf->producer.private) +continue ; +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT ,1 );@@ -585,6 +589,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS );while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +const struct xe_pagefault_ops *ops = pf->producer.ops; +void *private = pf->producer.private;struct xe_gt *gt = pf->gt;u32 asid = pf->consumer.asid;int err =0 ;@@ -622,11 +628,14 @@ static void xe_pagefault_queue_work(struct work_struct *w) XE_PAGEFAULT_ALLOC_STATE_ACTIVE);xe_assert (xe, pf == pf_work->cache.pf); + ops->ack_fault_begin (private);while (pf) {xe_assert (xe, pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, ops == pf->producer.ops); +xe_assert (xe, gt == pf->gt); - pf->producer.ops->ack_fault (pf, err); + ops->ack_fault (pf, err);spin_lock_irq (&pf_queue->lock);@@ -654,6 +663,7 @@ static void xe_pagefault_queue_work(struct work_struct *w)XE_PAGEFAULT_ALLOC_STATE_ACTIVE ;spin_unlock_irq (&pf_queue->lock); } + ops->ack_fault_end (private);if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);drivers/gpu/drm/xe/xe_pagefault_types.h +14
-0@@ -33,6 +33,13 @@ enum xe_pagefault_type {/** struct xe_pagefault_ops - Xe pagefault ops (producer) */ struct xe_pagefault_ops { +/** +* @ack_fault_begin: Ack fault begin +* @private: producer private data +* +* Page fault producer begins acknowledgment from the consumer. +*/ +void (*ack_fault_begin)(void *private);/** * @ack_fault: Ack fault * @pf: Page fault @@ -42,6 +49,13 @@ struct xe_pagefault_ops {* sends the result to the HW/FW interface. */ void (*ack_fault)(struct xe_pagefault *pf,int err); +/** +* @ack_fault_end: Ack fault end +* @private: producer private data +* +* Page fault producer ends acknowledgment from the consumer. +*/ +void (*ack_fault_end)(void *private); };/** Signature · 1 line
2.34.1
Re: [PATCH v8 11/12] drm/xe: batch CT pagefault acks with periodic flush
Quoted 83 lines · Matthew Brost
Quoted 70 lines · Francois Dugast
LGTM, just a couple of nits below.
Quoted 5 lines · Matthew Brost
Pagefault storms can generate long chains of acknowledgments back to the GuC. Sending each ack as a full CT submission forces a barrier, descriptor update and doorbell per fault.
Extend
xe_guc_ct_send_locked()with a “write-only” mode that copies theWondering if calling the argument "
defer_flush" instead of "write_only" would be a better fit, and also hint the caller to callxe_guc_ct_send_flush().Quoted 58 lines
message into the H2G ring but defers publishing the descriptor and ringing the doorbell. Add
xe_guc_ct_send_flush()to publish pending writes and notify GuC once per batch. Wire this into the pagefault producer via new ack_fault_begin/ack_fault_end callbacks and CT lock wrappers.To avoid excessive flush latency while still amortizing MMIO costs, use a simple periodic flush heuristic for GuC pagefault acks: batch most acks as write-only and force a publish at a fixed interval (e.g., every 16th ack), with a final flush at end-of-batch.
Also increase the H2G CTB size to 16K to better absorb bursts.
Assisted-by ChatGPT:gpt-5 # DocumentationSigned-off-by Matthew Brost <[email protected]>--- drivers/gpu/drm/xe/xe_guc_ct.c | 95 +++++++++++++++++++------ drivers/gpu/drm/xe/xe_guc_ct.h | 36 +++++++++- drivers/gpu/drm/xe/xe_guc_pagefault.c | 32 ++++++++- drivers/gpu/drm/xe/xe_guc_types.h | 6 ++ drivers/gpu/drm/xe/xe_pagefault.c | 12 +++- drivers/gpu/drm/xe/xe_pagefault_types.h | 14 ++++ 6 files changed, 170 insertions(+), 25 deletions(-)drivers/gpu/drm/xe/xe_guc_ct.c +2
-3@@ -265,7 +265,7 @@ static bool g2h_fence_needs_alloc(struct g2h_fence *g2h_fence)#define CTB_DESC_SIZE ALIGN(sizeof(struct guc_ct_buffer_desc), SZ_2K)#define CTB_H2G_BUFFER_OFFSET (CTB_DESC_SIZE * 2)#define CTB_G2H_BUFFER_OFFSET (CTB_DESC_SIZE * 2) -#define CTB_H2G_BUFFER_SIZE (SZ_4K) +#define CTB_H2G_BUFFER_SIZE (SZ_16K)#define CTB_H2G_BUFFER_DWORDS (CTB_H2G_BUFFER_SIZE / sizeof(u32))#define CTB_G2H_BUFFER_SIZE (SZ_128K)#define CTB_G2H_BUFFER_DWORDS (CTB_G2H_BUFFER_SIZE / sizeof(u32))@@ -939,7 +939,7 @@ static bool vf_action_can_safely_fail(struct xe_device *xe, u32 action)#define H2G_CT_HEADERS (GUC_CTB_HDR_LEN + 1)/* one DW CTB header and one DW HxG header */ static int h2g_write (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 ct_fence_value,bool want_response) +u32 ct_fence_value,bool want_response,bool write_only) {struct xe_device *xe =ct_to_xe (ct);struct xe_gt *gt =ct_to_gt (ct);@@ -956,7 +956,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (gt, full_len <=GUC_CTB_MSG_MAX_LEN );if (IS_ENABLED (CONFIG_DRM_XE_DEBUG )) { -u32 desc_tail =desc_read (xe, h2g, tail);u32 desc_head =desc_read (xe, h2g, head);u32 desc_status;@@ -966,12 +965,6 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, goto corrupted; }Quoted 8 lines
drivers/gpu/drm/xe/xe_guc_ct.c static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, · line 973
Quoted 5 lines
drivers/gpu/drm/xe/xe_guc_ct.c +0
-5-if (tail != desc_tail) { -desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_MISMATCH ); -xe_gt_err (gt,"CT write: tail was modified %u != %u\n" , desc_tail, tail); - goto corrupted; - }Could we keep that check when !write_only?
Not without a bigger change. Consider
write_onlyfollowed by !write_only, the tail !=desc_tailimmediately mismatches.
Quoted 8 lines
We'd have to add 3 states:
- Normal (check)
- Defer flush (no check)
- Issue flush (no check)Not sure if it is worth the complexity, but change add if this if you want.
Agreed, probably not worth the extra complexity in that case.
Francois
Quoted 399 lines
Quoted 210 lines
Quoted 210 lines
drivers/gpu/drm/xe/xe_guc_ct.c +71
-14-if (tail > h2g->info.size) {desc_write (xe, h2g, status, desc_status |GUC_CTB_STATUS_OVERFLOW ); xe_gt_err(gt, "CT write: tail out of range: %u vs %u\n",@@ -993,7 +986,10 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len, (h2g->info.size - tail) * sizeof(u32));h2g_reserve_space (ct, (h2g->info.size - tail)); h2g->info.tail =0 ; -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }return -EAGAIN ; }@@ -1024,14 +1020,15 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,/* Write H2G ensuring visible before descriptor update */ xe_map_memcpy_to (xe, &map,0 , cmd,H2G_CT_HEADERS *sizeof (u32));xe_map_memcpy_to (xe, &map,H2G_CT_HEADERS *sizeof (u32), action, len *sizeof (u32)); -xe_device_wmb (xe); -/* Update local copies */ h2g->info.tail = (tail + full_len) % h2g->info.size;h2g_reserve_space (ct, full_len);/* Update descriptor */ -desc_write (xe, h2g, tail, h2g->info.tail); +if (!write_only) { +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); + }/* * desc_read() performs an VRAM read which serializes the CPU and drains @@ -1052,7 +1049,7 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,static int __guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);u16 seqno;@@ -1112,7 +1109,7 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action,if (unlikely (ret)) goto out_unlock; - ret =h2g_write (ct, action, len, seqno, !!g2h_fence); + ret =h2g_write (ct, action, len, seqno, !!g2h_fence, write_only);if (unlikely (ret)) {if (ret == -EAGAIN ) goto retry;@@ -1120,7 +1117,8 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, }__g2h_reserve_space (ct, g2h_len, num_g2h); -xe_guc_notify (ct_to_guc (ct)); +if (!write_only) +xe_guc_notify (ct_to_guc (ct)); out_unlock:if (g2h_len)spin_unlock_irq (&ct->fast_lock);@@ -1196,7 +1194,7 @@ static bool guc_ct_send_wait_for_retry(struct xe_guc_ct *ct, u32 len,static int guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h, -struct g2h_fence *g2h_fence) +struct g2h_fence *g2h_fence,bool write_only) {struct xe_gt *gt =ct_to_gt (ct);unsigned int sleep_period_ms =1 ;@@ -1209,9 +1207,10 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len, try_again: ret =__guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, - g2h_fence); + g2h_fence, write_only);if (unlikely (ret == -EBUSY )) { +xe_guc_ct_send_flush (ct);if (!guc_ct_send_wait_for_retry (ct, len, g2h_len, g2h_fence, &sleep_period_ms, &sleep_total_ms)) goto broken;@@ -1235,7 +1234,8 @@ static int guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,xe_gt_assert (ct_to_gt (ct), !g2h_len || !g2h_fence);mutex_lock (&ct->lock); - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence); + ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h, g2h_fence, + false);mutex_unlock (&ct->lock);return ret;@@ -1283,25 +1283,76 @@ int xe_guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len,return ret; } +/** +* xe_guc_ct_send_locked() - submit a GuC CT H2G message with CT lock held +* @ct: GuC CT object +* @action: payload dwords (HxG header dword is expected at @action[-1]) +* @len: number of payload dwords in @action +* @write_only: defer publishing/doorbell for batching +* +* Sends a single H2G message to the GuC CT buffer while the caller already +* holds @ct->lock. +* +* If @write_only is false, the function completes the submission immediately: +* it makes the payload visible to the device, updates the H2G descriptor and +* rings the GuC doorbell. +* +* If @write_only is true, the message payload is copied into the H2G ring and +* the software tail is advanced, but the descriptor update and doorbell are +* deferred so multiple messages can be batched. In this mode, the caller must +* eventually call xe_guc_ct_send_flush() (still holding @ct->lock) to publish +* the descriptor and notify the GuC. On internal retry paths (-EBUSY), the +* implementation may force a flush to ensure forward progress. +* +* Return: 0 on success, negative errno on failure. +* +* Locking: +* Must be called with @ct->lock held. +*/ int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h) +bool write_only) {int ret; - ret =guc_ct_send_locked (ct, action, len, g2h_len, num_g2h,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , write_only);if (ret == -EDEADLK )kick_reset (ct);return ret; } +/** +* xe_guc_ct_send_flush() - flush pending GuC CT H2G writes +* @ct: GuC CT instance +* +* Some callers batch multiple H2G writes using xe_guc_ct_send_locked() in +* "write-only" mode (i.e., queue the message payloads but defer ringing the +* doorbell / updating the CT descriptor). This helper completes the submission +* by ensuring the payload writes are visible to the device, updating the H2G +* descriptor, and ringing the GuC CT doorbell. +* +* Locking: +* Must be called with @ct->lock held. +*/ +void xe_guc_ct_send_flush (struct xe_guc_ct *ct) +{ +struct xe_device *xe =ct_to_xe (ct); +struct guc_ctb *h2g = &ct->ctbs.h2g; + +lockdep_assert_held (&ct->lock); + +xe_device_wmb (xe); +desc_write (xe, h2g, tail, h2g->info.tail); +xe_guc_notify (ct_to_guc (ct)); +} +int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len) {int ret;lockdep_assert_held (&ct->lock); - ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL ); + ret =guc_ct_send_locked (ct, action, len,0 ,0 ,NULL , false);if (ret == -EDEADLK )kick_reset (ct);drivers/gpu/drm/xe/xe_guc_ct.h +12
-1@@ -54,7 +54,7 @@ static inline void xe_guc_ct_irq_handler(struct xe_guc_ct *ct)int xe_guc_ct_send (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 g2h_len,u32 num_g2h);int xe_guc_ct_send_locked (struct xe_guc_ct *ct,const u32 *action,u32 len, -u32 g2h_len,u32 num_g2h); +bool write_only);int xe_guc_ct_send_recv (struct xe_guc_ct *ct,const u32 *action,u32 len,u32 *response_buffer); static inline int@@ -63,6 +63,8 @@ xe_guc_ct_send_block(struct xe_guc_ct *ct, const u32 *action, u32 len)return xe_guc_ct_send_recv (ct, action, len,NULL ); } +void xe_guc_ct_send_flush (struct xe_guc_ct *ct); +/* This is only version of the send CT you can call from a G2H handler */ int xe_guc_ct_send_g2h_handler (struct xe_guc_ct *ct,const u32 *action,u32 len);@@ -87,4 +89,36 @@ static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct)wake_up_all (&ct->wq); } +/** +* xe_guc_ct_lock() - take the GuC CT mutex +* @ct: GuC CT object +* +* Wrapper around mutex_lock(&ct->lock) for cases where CT operations need to be +* performed from contexts that want an explicit "CT locked" pair without +* exporting the lock itself. +* +* Return/Locking: Quoted 9 lines
drivers/gpu/drm/xe/xe_guc_ct.h static inline void xe_guc_ct_wake_waiters(struct xe_guc_ct *ct) · line 106
Quoted 6 lines
drivers/gpu/drm/xe/xe_guc_ct.h +6
-0+* Acquires @ct->lock. +*/ +static inline void xe_guc_ct_lock (struct xe_guc_ct *ct) +__acquires(&ct->lock) +{ +mutex_lock (&ct->lock);Now it would be cleaner to add "#include <
linux/mutex.h>" inxe_guc_ct.h.Sure.
Matt
Quoted 175 lines
Francois
Quoted 173 lines
drivers/gpu/drm/xe/xe_guc_ct.h +17
-0+} + +/** +* xe_guc_ct_unlock() - release the GuC CT mutex +* @ct: GuC CT object +* +* Counterpart to xe_guc_ct_lock(). +* +* Locking: +* Releases @ct->lock. +*/ +static inline void xe_guc_ct_unlock (struct xe_guc_ct *ct) +__releases(&ct->lock) +{ +mutex_unlock (&ct->lock); +} +#endif drivers/gpu/drm/xe/xe_guc_pagefault.c +31
-1@@ -10,6 +10,22 @@#include "xe_pagefault.h" #include "xe_pagefault_types.h" +#define XE_GUC_PAGEFAULT_FLUSH_PERIOD BIT(4)/* Sixteen */ + +static void guc_ack_fault_begin (void *private) +{ +struct xe_guc *guc = private; + +xe_guc_ct_lock (&guc->ct); + +BUILD_BUG_ON (((XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ) & +XE_GUC_PAGEFAULT_FLUSH_PERIOD ) !=0 ); + +/* Ack the 2nd, then 18th, etc... */ + guc->pagefault_ack_counter = +XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ; +} +static void guc_ack_fault (struct xe_pagefault *pf,int err) {u32 vfid =FIELD_GET (PFD_VFID , pf->producer.msg[2 ]);@@ -36,12 +52,26 @@ static void guc_ack_fault(struct xe_pagefault *pf, int err) FIELD_PREP(PFR_PDATA, pdata), };struct xe_guc *guc = pf->producer.private; +bool write_only = guc->pagefault_ack_counter++ & + (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 ); + +xe_guc_ct_send_locked (&guc->ct, action,ARRAY_SIZE (action), + write_only); +} + +static void guc_ack_fault_end (void *private) +{ +struct xe_guc *guc = private; -xe_guc_ct_send (&guc->ct, action,ARRAY_SIZE (action),0 ,0 ); +if ((guc->pagefault_ack_counter & (XE_GUC_PAGEFAULT_FLUSH_PERIOD -1 )) !=1 ) +xe_guc_ct_send_flush (&guc->ct); +xe_guc_ct_unlock (&guc->ct); }static const struct xe_pagefault_ops guc_pagefault_ops = { + .ack_fault_begin = guc_ack_fault_begin, .ack_fault = guc_ack_fault, + .ack_fault_end = guc_ack_fault_end, };/** drivers/gpu/drm/xe/xe_guc_types.h +6
-0@@ -122,6 +122,12 @@ struct xe_guc {struct xe_reg notify_reg;/** @params: Control params for fw initialization */ u32 params[GUC_CTL_MAX_DWORDS ]; + +/** +* @pagefault_ack_counter: Counter to determine when periodically ack +* pagefaults in a batch. +*/ + u32 pagefault_ack_counter; };#endif drivers/gpu/drm/xe/xe_pagefault.c +11
-1@@ -412,6 +412,10 @@ static bool xe_pagefault_try_chain(struct xe_pagefault_queue *pf_queue,xe_assert (xe, pf_work->cache.pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +if (pf->producer.private != + pf_work->cache.pf->producer.private) +continue ; +xe_gt_stats_incr (pf->gt,XE_GT_STATS_ID_CHAIN_PAGEFAULT_COUNT ,1 );@@ -585,6 +589,8 @@ static void xe_pagefault_queue_work(struct work_struct *w) threshold = jiffies +msecs_to_jiffies (USM_QUEUE_MAX_RUNTIME_MS );while (xe_pagefault_queue_pop (pf_queue, &pf, pf_work->id)) { +const struct xe_pagefault_ops *ops = pf->producer.ops; +void *private = pf->producer.private;struct xe_gt *gt = pf->gt;u32 asid = pf->consumer.asid;int err =0 ;@@ -622,11 +628,14 @@ static void xe_pagefault_queue_work(struct work_struct *w) XE_PAGEFAULT_ALLOC_STATE_ACTIVE);xe_assert (xe, pf == pf_work->cache.pf); + ops->ack_fault_begin (private);while (pf) {xe_assert (xe, pf->consumer.alloc_state ==XE_PAGEFAULT_ALLOC_STATE_ACTIVE ); +xe_assert (xe, ops == pf->producer.ops); +xe_assert (xe, gt == pf->gt); - pf->producer.ops->ack_fault (pf, err); + ops->ack_fault (pf, err);spin_lock_irq (&pf_queue->lock);@@ -654,6 +663,7 @@ static void xe_pagefault_queue_work(struct work_struct *w)XE_PAGEFAULT_ALLOC_STATE_ACTIVE ;spin_unlock_irq (&pf_queue->lock); } + ops->ack_fault_end (private);if (time_after (jiffies, threshold)) {queue_work (xe->usm.pagefault_wq, w);drivers/gpu/drm/xe/xe_pagefault_types.h +14
-0@@ -33,6 +33,13 @@ enum xe_pagefault_type {/** struct xe_pagefault_ops - Xe pagefault ops (producer) */ struct xe_pagefault_ops { +/** +* @ack_fault_begin: Ack fault begin +* @private: producer private data +* +* Page fault producer begins acknowledgment from the consumer. +*/ +void (*ack_fault_begin)(void *private);/** * @ack_fault: Ack fault * @pf: Page fault @@ -42,6 +49,13 @@ struct xe_pagefault_ops {* sends the result to the HW/FW interface. */ void (*ack_fault)(struct xe_pagefault *pf,int err); +/** +* @ack_fault_end: Ack fault end +* @private: producer private data +* +* Page fault producer ends acknowledgment from the consumer. +*/ +void (*ack_fault_end)(void *private); };/** Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v8 12/12 drm/xe: Track parallel page fault activity in GT stats
In reply to Matthew BrostAdd a new GT statistic, PARALLEL_PAGEFAULT_COUNT, to record when multiple page fault workers are active concurrently.
When a worker dequeues a fault, scan peer workers for an active cache entry and increment the counter if another fault is already in flight. This provides basic visibility into parallel fault handling behavior for performance analysis and tuning.
--- drivers/gpu/drm/xe/xe_gt_stats.c | 1 + drivers/gpu/drm/xe/xe_gt_stats_types.h | 3 +++ drivers/gpu/drm/xe/xe_pagefault.c | 18 +++++++++++++++++- 3 files changed, 21 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_gt_stats.c +1 -0
@@ -99,6 +99,7 @@ static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {DEF_STAT_STR (CHAIN_IRQ_PAGEFAULT_COUNT ,"chain_irq_pagefault_count" ),DEF_STAT_STR (CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT ,"chain_drain_irq_pagefault_count" ),DEF_STAT_STR (CHAIN_MISMATCH_PAGEFAULT_COUNT ,"chain_mismatch_pagefault_count" ), +DEF_STAT_STR (PARALLEL_PAGEFAULT_COUNT ,"parallel_pagefault_count" ),DEF_STAT_STR (LAST_PAGEFAULT_COUNT ,"last_pagefault_count" ),DEF_STAT_STR (SVM_PAGEFAULT_COUNT ,"svm_pagefault_count" ), DEF_STAT_STR(TLB_INVAL, "tlb_inval_count"),
drivers/gpu/drm/xe/xe_gt_stats_types.h +3 -0
@@ -18,6 +18,8 @@* that also drained the fault queue. * @XE_GT_STATS_ID_CHAIN_MISMATCH_PAGEFAULT_COUNT: Chained faults requeued * because their fault range did not match the fault they were chained onto. +* @XE_GT_STATS_ID_PARALLEL_PAGEFAULT_COUNT: Faults dequeued while another page +* fault worker was already handling a fault concurrently. * @XE_GT_STATS_ID_LAST_PAGEFAULT_COUNT: Faults whose range matched the last * serviced range, allowing an immediate ack. *
@@ -144,6 +146,7 @@ enum xe_gt_stats_id {XE_GT_STATS_ID_CHAIN_IRQ_PAGEFAULT_COUNT ,XE_GT_STATS_ID_CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT ,XE_GT_STATS_ID_CHAIN_MISMATCH_PAGEFAULT_COUNT , +XE_GT_STATS_ID_PARALLEL_PAGEFAULT_COUNT ,XE_GT_STATS_ID_LAST_PAGEFAULT_COUNT ,XE_GT_STATS_ID_SVM_PAGEFAULT_COUNT , XE_GT_STATS_ID_TLB_INVAL,
drivers/gpu/drm/xe/xe_pagefault.c +17 -1
@@ -460,9 +460,10 @@ static bool xe_pagefault_queue_pop(struct xe_pagefault_queue *pf_queue, {struct xe_device *xe =container_of (pf_queue,typeof (*xe), usm.pf_queue); -struct xe_pagefault_work *pf_work; +struct xe_pagefault_work *pf_work, *__pf_work;struct xe_pagefault *lpf;size_t align =SZ_2M ; +int i;guard (spinlock_irq)(&pf_queue->lock);
@@ -499,6 +500,21 @@ static bool xe_pagefault_queue_pop(struct xe_pagefault_queue *pf_queue, pf_work->cache.pf = lpf; lpf->consumer.alloc_state =XE_PAGEFAULT_ALLOC_STATE_ACTIVE ; +for (i =0 , __pf_work = xe->usm.pf_workers; + i < xe->info.num_pf_work; ++i, ++__pf_work) { +u64 cache_start = __pf_work->cache.start; + +if (__pf_work == pf_work) +continue ; + +if (cache_start !=XE_PAGEFAULT_CACHE_START_INVALID ) { +xe_gt_stats_incr (xe_root_mmio_gt (xe), +XE_GT_STATS_ID_PARALLEL_PAGEFAULT_COUNT , +1 ); +break ; + } + } +/* Drain queue until empty or new fault found */ while (1 ) {if (xe_pagefault_queue_empty (pf_queue))
Signature · 1 line
2.34.1
✗ CI.checkpatch: warning for Fine grained fault locking, threaded prefetch, storm cache (rev8)
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Fine grained fault locking, threaded prefetch, storm cache (rev8)
URL : https://patchwork.freedesktop.org/series/162167/
State : warning
== Summary ==
+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 061140b9bc586ae7f40abc1249c97e1cc72d1b9d
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 6127c7fa172994ce8b15dfc4b8ae8f93e86e14e8
Author: Matthew Brost <[email protected]> Date: Fri Jul 24 16:26:01 2026 -0700
drm/xe: Track parallel page fault activity in GT stats
Add a new GT statistic, PARALLEL_PAGEFAULT_COUNT, to record when multiple page fault workers are active concurrently.
When a worker dequeues a fault, scan peer workers for an active cache entry and increment the counter if another fault is already in flight. This provides basic visibility into parallel fault handling behavior for performance analysis and tuning.
+ /mt/dim checkpatch 03696c1dc9eabc0658baefcff8787dab48c89d93 drm-intel 0f0603b3c74b drm/xe: Fine grained page fault locking
-:609: CHECK:UNCOMMENTED_DEFINITION: struct mutex definition without comment #609: FILE: drivers/gpu/drm/xe/xe_svm.h:253:
+ struct mutex lock;
total: 0 errors, 0 warnings, 1 checks, 935 lines checked 642cbbc4d719 drm/xe: Allow prefetch-only VM bind IOCTLs to use VM read lock c7aa31bd85b5 drm/xe: Thread prefetch of SVM ranges
-:47: WARNING:BAD_SIGN_OFF: Non-standard signature: Co-authored-by:
#47:
total: 0 errors, 1 warnings, 0 checks, 356 lines checked9c9b2f274ffe drm/xe: Use a single page-fault queue with multiple workers 0f3c5ada6164 drm/xe: Add num_pf_work modparame456d7cdceab drm/xe: Engine class and instance into a u8ef9ac1c6a6fb drm/xe: Track pagefault worker runtimeac0b4177ea0b drm/xe: Chain page faults via queue-resident cache to avoid fault storms -:41: WARNING:BAD_SIGN_OFF: Non-standard signature: Co-authored-by:
#41:
-:443: WARNING:LONG_LINE: line length of 109 exceeds 100 columns #443: FILE: drivers/gpu/drm/xe/xe_pagefault.c:598:
+ xe_gt_stats_incr(pf->gt, XE_GT_STATS_ID_INVALID_PREFETCH_PAGEFAULT_COUNT, 1);
-:485: CHECK:LINE_SPACING: Please don't use multiple blank lines #485: FILE: drivers/gpu/drm/xe/xe_pagefault.c:639:
+
+
total: 0 errors, 2 warnings, 1 checks, 806 lines checked 4e6703849361 drm/xe: Add pagefault chaining stats -:112: WARNING:LONG_LINE: line length of 104 exceeds 100 columns #112: FILE: drivers/gpu/drm/xe/xe_pagefault.c:857:
+ XE_GT_STATS_ID_CHAIN_DRAIN_IRQ_PAGEFAULT_COUNT,
total: 0 errors, 1 warnings, 0 checks, 84 lines checked b50ac559ad31 drm/xe: Add debugfs pagefault_info 8e8d86f2c0d3 drm/xe: batch CT pagefault acks with periodic flush 6127c7fa1729 drm/xe: Track parallel page fault activity in GT stats
Patchwork
CI✓ CI.KUnit: success for Fine grained fault locking, threaded prefetch, storm cache (rev8)
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Fine grained fault locking, threaded prefetch, storm cache (rev8)
URL : https://patchwork.freedesktop.org/series/162167/
State : success
== Summary ==
+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig [23:32:22] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [23:32:26] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 [23:32:59] Starting KUnit Kernel (1/1)... [23:32:59] ============================================================ Running tests with: $ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt [23:32:59] ================== guc_buf (11 subtests) =================== [23:32:59] [PASSED] test_smallest [23:32:59] [PASSED] test_largest [23:32:59] [PASSED] test_granular [23:32:59] [PASSED] test_unique [23:32:59] [PASSED] test_overlap [23:32:59] [PASSED] test_reusable [23:32:59] [PASSED] test_too_big [23:32:59] [PASSED] test_flush [23:32:59] [PASSED] test_lookup [23:32:59] [PASSED] test_data [23:32:59] [PASSED] test_class [23:32:59] ===================== [PASSED] guc_buf ===================== [23:32:59] =================== guc_dbm (7 subtests) =================== [23:32:59] [PASSED] test_empty [23:32:59] [PASSED] test_default [23:32:59] ======================== test_size ======================== [23:32:59] [PASSED] 4 [23:32:59] [PASSED] 8 [23:32:59] [PASSED] 32 [23:32:59] [PASSED] 256 [23:32:59] ==================== [PASSED] test_size ==================== [23:32:59] ======================= test_reuse ======================== [23:32:59] [PASSED] 4 [23:32:59] [PASSED] 8 [23:32:59] [PASSED] 32 [23:32:59] [PASSED] 256 [23:32:59] =================== [PASSED] test_reuse ==================== [23:32:59] =================== test_range_overlap ==================== [23:32:59] [PASSED] 4 [23:32:59] [PASSED] 8 [23:32:59] [PASSED] 32 [23:32:59] [PASSED] 256 [23:32:59] =============== [PASSED] test_range_overlap ================ [23:32:59] =================== test_range_compact ==================== [23:32:59] [PASSED] 4 [23:32:59] [PASSED] 8 [23:32:59] [PASSED] 32 [23:32:59] [PASSED] 256 [23:32:59] =============== [PASSED] test_range_compact ================ [23:32:59] ==================== test_range_spare ===================== [23:32:59] [PASSED] 4 [23:32:59] [PASSED] 8 [23:32:59] [PASSED] 32 [23:32:59] [PASSED] 256 [23:32:59] ================ [PASSED] test_range_spare ================= [23:32:59] ===================== [PASSED] guc_dbm ===================== [23:32:59] =================== guc_idm (6 subtests) =================== [23:32:59] [PASSED] bad_init [23:32:59] [PASSED] no_init [23:32:59] [PASSED] init_fini [23:32:59] [PASSED] check_used [23:32:59] [PASSED] check_quota [23:32:59] [PASSED] check_all [23:32:59] ===================== [PASSED] guc_idm ===================== [23:32:59] =============== guc_klv_helpers (9 subtests) =============== [23:32:59] [PASSED] test_count [23:32:59] [PASSED] test_encode_u32 [23:32:59] [PASSED] test_encode_u64 [23:32:59] [PASSED] test_encode_string [23:32:59] [PASSED] test_encode_object_raw [23:32:59] [PASSED] test_encode_object_klv [23:32:59] [PASSED] test_encode_object_nested [23:32:59] [PASSED] test_encode_object_basic [23:32:59] [PASSED] test_print [23:32:59] ================= [PASSED] guc_klv_helpers ================= [23:32:59] ================== no_relay (3 subtests) =================== [23:32:59] [PASSED] xe_drops_guc2pf_if_not_ready [23:32:59] [PASSED] xe_drops_guc2vf_if_not_ready [23:32:59] [PASSED] xe_rejects_send_if_not_ready [23:32:59] ==================== [PASSED] no_relay ===================== [23:32:59] ================== pf_relay (14 subtests) ================== [23:32:59] [PASSED] pf_rejects_guc2pf_too_short [23:32:59] [PASSED] pf_rejects_guc2pf_too_long [23:32:59] [PASSED] pf_rejects_guc2pf_no_payload [23:32:59] [PASSED] pf_fails_no_payload [23:32:59] [PASSED] pf_fails_bad_origin [23:32:59] [PASSED] pf_fails_bad_type [23:32:59] [PASSED] pf_txn_reports_error [23:32:59] [PASSED] pf_txn_sends_pf2guc [23:32:59] [PASSED] pf_sends_pf2guc [23:32:59] [SKIPPED] pf_loopback_nop (requires CONFIG_DRM_XE_DEBUG_SRIOV) [23:32:59] [SKIPPED] pf_loopback_echo (requires CONFIG_DRM_XE_DEBUG_SRIOV) [23:32:59] [SKIPPED] pf_loopback_fail (requires CONFIG_DRM_XE_DEBUG_SRIOV) [23:32:59] [SKIPPED] pf_loopback_busy (requires CONFIG_DRM_XE_DEBUG_SRIOV) [23:32:59] [SKIPPED] pf_loopback_retry (requires CONFIG_DRM_XE_DEBUG_SRIOV) [23:32:59] ==================== [PASSED] pf_relay ===================== [23:32:59] ================== vf_relay (3 subtests) =================== [23:32:59] [PASSED] vf_rejects_guc2vf_too_short [23:32:59] [PASSED] vf_rejects_guc2vf_too_long [23:32:59] [PASSED] vf_rejects_guc2vf_no_payload [23:32:59] ==================== [PASSED] vf_relay ===================== [23:32:59] ================ pf_gt_config (9 subtests) ================= [23:32:59] [PASSED] fair_contexts_1vf [23:32:59] [PASSED] fair_doorbells_1vf [23:32:59] [PASSED] fair_ggtt_1vf [23:32:59] ====================== fair_vram_1vf ====================== [23:32:59] [PASSED] 3.50 GiB [23:32:59] [PASSED] 11.5 GiB [23:32:59] [PASSED] 15.5 GiB [23:32:59] [PASSED] 31.5 GiB [23:32:59] [PASSED] 63.5 GiB [23:32:59] [PASSED] 1.91 GiB [23:32:59] ================== [PASSED] fair_vram_1vf ================== [23:32:59] ================ fair_vram_1vf_admin_only ================= [23:32:59] [PASSED] 3.50 GiB [23:32:59] [PASSED] 11.5 GiB [23:32:59] [PASSED] 15.5 GiB [23:32:59] [PASSED] 31.5 GiB [23:32:59] [PASSED] 63.5 GiB [23:32:59] [PASSED] 1.91 GiB [23:32:59] ============ [PASSED] fair_vram_1vf_admin_only ============= [23:32:59] ====================== fair_contexts ====================== [23:32:59] [PASSED] 1 VF [23:32:59] [PASSED] 2 VFs [23:32:59] [PASSED] 3 VFs [23:32:59] [PASSED] 4 VFs [23:32:59] [PASSED] 5 VFs [23:32:59] [PASSED] 6 VFs [23:32:59] [PASSED] 7 VFs [23:32:59] [PASSED] 8 VFs [23:32:59] [PASSED] 9 VFs [23:32:59] [PASSED] 10 VFs [23:32:59] [PASSED] 11 VFs [23:32:59] [PASSED] 12 VFs [23:32:59] [PASSED] 13 VFs [23:32:59] [PASSED] 14 VFs [23:32:59] [PASSED] 15 VFs [23:32:59] [PASSED] 16 VFs [23:32:59] [PASSED] 17 VFs [23:32:59] [PASSED] 18 VFs [23:32:59] [PASSED] 19 VFs [23:32:59] [PASSED] 20 VFs [23:32:59] [PASSED] 21 VFs [23:32:59] [PASSED] 22 VFs [23:32:59] [PASSED] 23 VFs [23:32:59] [PASSED] 24 VFs [23:32:59] [PASSED] 25 VFs [23:32:59] [PASSED] 26 VFs [23:32:59] [PASSED] 27 VFs [23:32:59] [PASSED] 28 VFs [23:32:59] [PASSED] 29 VFs [23:32:59] [PASSED] 30 VFs [23:32:59] [PASSED] 31 VFs [23:32:59] [PASSED] 32 VFs [23:32:59] [PASSED] 33 VFs [23:32:59] [PASSED] 34 VFs [23:32:59] [PASSED] 35 VFs [23:32:59] [PASSED] 36 VFs [23:32:59] [PASSED] 37 VFs [23:32:59] [PASSED] 38 VFs [23:32:59] [PASSED] 39 VFs [23:32:59] [PASSED] 40 VFs [23:32:59] [PASSED] 41 VFs [23:32:59] [PASSED] 42 VFs [23:32:59] [PASSED] 43 VFs [23:32:59] [PASSED] 44 VFs [23:32:59] [PASSED] 45 VFs [23:32:59] [PASSED] 46 VFs [23:32:59] [PASSED] 47 VFs [23:32:59] [PASSED] 48 VFs [23:32:59] [PASSED] 49 VFs [23:32:59] [PASSED] 50 VFs [23:32:59] [PASSED] 51 VFs [23:32:59] [PASSED] 52 VFs [23:32:59] [PASSED] 53 VFs [23:32:59] [PASSED] 54 VFs [23:32:59] [PASSED] 55 VFs [23:32:59] [PASSED] 56 VFs [23:32:59] [PASSED] 57 VFs [23:32:59] [PASSED] 58 VFs [23:32:59] [PASSED] 59 VFs [23:32:59] [PASSED] 60 VFs [23:32:59] [PASSED] 61 VFs [23:32:59] [PASSED] 62 VFs [23:32:59] [PASSED] 63 VFs [23:32:59] ================== [PASSED] fair_contexts ================== [23:32:59] ===================== fair_doorbells ====================== [23:32:59] [PASSED] 1 VF [23:32:59] [PASSED] 2 VFs [23:32:59] [PASSED] 3 VFs [23:32:59] [PASSED] 4 VFs [23:32:59] [PASSED] 5 VFs [23:32:59] [PASSED] 6 VFs [23:32:59] [PASSED] 7 VFs [23:32:59] [PASSED] 8 VFs [23:32:59] [PASSED] 9 VFs [23:32:59] [PASSED] 10 VFs [23:32:59] [PASSED] 11 VFs [23:32:59] [PASSED] 12 VFs [23:32:59] [PASSED] 13 VFs [23:32:59] [PASSED] 14 VFs [23:32:59] [PASSED] 15 VFs [23:32:59] [PASSED] 16 VFs [23:32:59] [PASSED] 17 VFs [23:32:59] [PASSED] 18 VFs [23:32:59] [PASSED] 19 VFs [23:32:59] [PASSED] 20 VFs [23:32:59] [PASSED] 21 VFs [23:32:59] [PASSED] 22 VFs [23:32:59] [PASSED] 23 VFs [23:32:59] [PASSED] 24 VFs [23:32:59] [PASSED] 25 VFs [23:32:59] [PASSED] 26 VFs [23:32:59] [PASSED] 27 VFs [23:32:59] [PASSED] 28 VFs [23:32:59] [PASSED] 29 VFs [23:32:59] [PASSED] 30 VFs [23:32:59] [PASSED] 31 VFs [23:32:59] [PASSED] 32 VFs [23:32:59] [PASSED] 33 VFs [23:32:59] [PASSED] 34 VFs [23:32:59] [PASSED] 35 VFs [23:32:59] [PASSED] 36 VFs [23:32:59] [PASSED] 37 VFs [23:32:59] [PASSED] 38 VFs [23:32:59] [PASSED] 39 VFs [23:32:59] [PASSED] 40 VFs [23:32:59] [PASSED] 41 VFs [23:32:59] [PASSED] 42 VFs [23:32:59] [PASSED] 43 VFs [23:32:59] [PASSED] 44 VFs [23:32:59] [PASSED] 45 VFs [23:32:59] [PASSED] 46 VFs [23:32:59] [PASSED] 47 VFs [23:32:59] [PASSED] 48 VFs [23:32:59] [PASSED] 49 VFs [23:32:59] [PASSED] 50 VFs [23:32:59] [PASSED] 51 VFs [23:32:59] [PASSED] 52 VFs [23:32:59] [PASSED] 53 VFs [23:32:59] [PASSED] 54 VFs [23:32:59] [PASSED] 55 VFs [23:32:59] [PASSED] 56 VFs [23:32:59] [PASSED] 57 VFs [23:32:59] [PASSED] 58 VFs [23:32:59] [PASSED] 59 VFs [23:32:59] [PASSED] 60 VFs [23:32:59] [PASSED] 61 VFs [23:32:59] [PASSED] 62 VFs [23:32:59] [PASSED] 63 VFs [23:32:59] ================= [PASSED] fair_doorbells ================== [23:32:59] ======================== fair_ggtt ======================== [23:32:59] [PASSED] 1 VF [23:32:59] [PASSED] 2 VFs [23:32:59] [PASSED] 3 VFs [23:32:59] [PASSED] 4 VFs [23:32:59] [PASSED] 5 VFs [23:32:59] [PASSED] 6 VFs [23:32:59] [PASSED] 7 VFs [23:32:59] [PASSED] 8 VFs [23:32:59] [PASSED] 9 VFs [23:32:59] [PASSED] 10 VFs [23:32:59] [PASSED] 11 VFs [23:32:59] [PASSED] 12 VFs [23:32:59] [PASSED] 13 VFs [23:32:59] [PASSED] 14 VFs [23:32:59] [PASSED] 15 VFs [23:32:59] [PASSED] 16 VFs [23:32:59] [PASSED] 17 VFs [23:32:59] [PASSED] 18 VFs [23:32:59] [PASSED] 19 VFs [23:32:59] [PASSED] 20 VFs [23:32:59] [PASSED] 21 VFs [23:32:59] [PASSED] 22 VFs [23:32:59] [PASSED] 23 VFs [23:32:59] [PASSED] 24 VFs [23:32:59] [PASSED] 25 VFs [23:32:59] [PASSED] 26 VFs [23:32:59] [PASSED] 27 VFs [23:32:59] [PASSED] 28 VFs [23:32:59] [PASSED] 29 VFs [23:32:59] [PASSED] 30 VFs [23:32:59] [PASSED] 31 VFs [23:32:59] [PASSED] 32 VFs [23:32:59] [PASSED] 33 VFs [23:32:59] [PASSED] 34 VFs [23:32:59] [PASSED] 35 VFs [23:32:59] [PASSED] 36 VFs [23:32:59] [PASSED] 37 VFs [23:32:59] [PASSED] 38 VFs [23:32:59] [PASSED] 39 VFs [23:32:59] [PASSED] 40 VFs [23:32:59] [PASSED] 41 VFs [23:32:59] [PASSED] 42 VFs [23:32:59] [PASSED] 43 VFs [23:32:59] [PASSED] 44 VFs [23:32:59] [PASSED] 45 VFs [23:32:59] [PASSED] 46 VFs [23:32:59] [PASSED] 47 VFs [23:32:59] [PASSED] 48 VFs [23:32:59] [PASSED] 49 VFs [23:32:59] [PASSED] 50 VFs [23:32:59] [PASSED] 51 VFs [23:32:59] [PASSED] 52 VFs [23:32:59] [PASSED] 53 VFs [23:32:59] [PASSED] 54 VFs [23:32:59] [PASSED] 55 VFs [23:32:59] [PASSED] 56 VFs [23:32:59] [PASSED] 57 VFs [23:32:59] [PASSED] 58 VFs [23:32:59] [PASSED] 59 VFs [23:32:59] [PASSED] 60 VFs [23:32:59] [PASSED] 61 VFs [23:32:59] [PASSED] 62 VFs [23:32:59] [PASSED] 63 VFs [23:32:59] ==================== [PASSED] fair_ggtt ==================== [23:32:59] ======================== fair_vram ======================== [23:32:59] [PASSED] 1 VF [23:32:59] [PASSED] 2 VFs [23:32:59] [PASSED] 3 VFs [23:32:59] [PASSED] 4 VFs [23:32:59] [PASSED] 5 VFs [23:32:59] [PASSED] 6 VFs [23:32:59] [PASSED] 7 VFs [23:32:59] [PASSED] 8 VFs [23:32:59] [PASSED] 9 VFs [23:32:59] [PASSED] 10 VFs [23:32:59] [PASSED] 11 VFs [23:32:59] [PASSED] 12 VFs [23:32:59] [PASSED] 13 VFs [23:32:59] [PASSED] 14 VFs [23:32:59] [PASSED] 15 VFs [23:32:59] [PASSED] 16 VFs [23:32:59] [PASSED] 17 VFs [23:32:59] [PASSED] 18 VFs [23:32:59] [PASSED] 19 VFs [23:32:59] [PASSED] 20 VFs [23:32:59] [PASSED] 21 VFs [23:32:59] [PASSED] 22 VFs [23:32:59] [PASSED] 23 VFs [23:32:59] [PASSED] 24 VFs [23:32:59] [PASSED] 25 VFs [23:32:59] [PASSED] 26 VFs [23:32:59] [PASSED] 27 VFs [23:32:59] [PASSED] 28 VFs [23:32:59] [PASSED] 29 VFs [23:32:59] [PASSED] 30 VFs [23:32:59] [PASSED] 31 VFs [23:32:59] [PASSED] 32 VFs [23:32:59] [PASSED] 33 VFs [23:32:59] [PASSED] 34 VFs [23:32:59] [PASSED] 35 VFs [23:32:59] [PASSED] 36 VFs [23:32:59] [PASSED] 37 VFs [23:32:59] [PASSED] 38 VFs [23:32:59] [PASSED] 39 VFs [23:32:59] [PASSED] 40 VFs [23:32:59] [PASSED] 41 VFs [23:32:59] [PASSED] 42 VFs [23:32:59] [PASSED] 43 VFs [23:32:59] [PASSED] 44 VFs [23:32:59] [PASSED] 45 VFs [23:32:59] [PASSED] 46 VFs [23:32:59] [PASSED] 47 VFs [23:32:59] [PASSED] 48 VFs [23:32:59] [PASSED] 49 VFs [23:32:59] [PASSED] 50 VFs [23:32:59] [PASSED] 51 VFs [23:32:59] [PASSED] 52 VFs [23:32:59] [PASSED] 53 VFs [23:32:59] [PASSED] 54 VFs [23:32:59] [PASSED] 55 VFs [23:32:59] [PASSED] 56 VFs [23:32:59] [PASSED] 57 VFs [23:32:59] [PASSED] 58 VFs [23:32:59] [PASSED] 59 VFs [23:32:59] [PASSED] 60 VFs [23:32:59] [PASSED] 61 VFs [23:32:59] [PASSED] 62 VFs [23:32:59] [PASSED] 63 VFs [23:32:59] ==================== [PASSED] fair_vram ==================== [23:32:59] ================== [PASSED] pf_gt_config =================== [23:32:59] ===================== lmtt (1 subtest) ===================== [23:32:59] ======================== test_ops ========================= [23:32:59] [PASSED] 2-level [23:32:59] [PASSED] multi-level [23:32:59] ==================== [PASSED] test_ops ===================== [23:32:59] ====================== [PASSED] lmtt ======================= [23:32:59] ================= sriov_packet (1 subtest) ================= [23:32:59] [PASSED] test_descriptor_init [23:32:59] ================== [PASSED] sriov_packet =================== [23:32:59] ================= pf_service (11 subtests) ================= [23:32:59] [PASSED] pf_negotiate_any [23:32:59] [PASSED] pf_negotiate_base_match [23:32:59] [PASSED] pf_negotiate_base_newer [23:32:59] [PASSED] pf_negotiate_base_next [23:32:59] [SKIPPED] pf_negotiate_base_older (no older minor) [23:32:59] [PASSED] pf_negotiate_base_prev [23:32:59] [PASSED] pf_negotiate_latest_match [23:32:59] [PASSED] pf_negotiate_latest_newer [23:32:59] [PASSED] pf_negotiate_latest_next [23:32:59] [SKIPPED] pf_negotiate_latest_older (no older minor) [23:32:59] [SKIPPED] pf_negotiate_latest_prev (no prev major) [23:32:59] =================== [PASSED] pf_service ==================== [23:32:59] ================= xe_guc_g2g (2 subtests) ================== [23:32:59] ============== xe_live_guc_g2g_kunit_default ============== [23:32:59] ========= [SKIPPED] xe_live_guc_g2g_kunit_default ========== [23:32:59] ============== xe_live_guc_g2g_kunit_allmem =============== [23:32:59] ========== [SKIPPED] xe_live_guc_g2g_kunit_allmem ========== [23:32:59] =================== [SKIPPED] xe_guc_g2g =================== [23:32:59] =================== xe_mocs (2 subtests) =================== [23:32:59] ================ xe_live_mocs_kernel_kunit ================ [23:32:59] =========== [SKIPPED] xe_live_mocs_kernel_kunit ============ [23:32:59] ================ xe_live_mocs_reset_kunit ================= [23:32:59] ============ [SKIPPED] xe_live_mocs_reset_kunit ============ [23:32:59] ==================== [SKIPPED] xe_mocs ===================== [23:32:59] ================= xe_migrate (2 subtests) ================== [23:32:59] ================= xe_migrate_sanity_kunit ================= [23:32:59] ============ [SKIPPED] xe_migrate_sanity_kunit ============= [23:32:59] ================== xe_validate_ccs_kunit ================== [23:32:59] ============= [SKIPPED] xe_validate_ccs_kunit ============== [23:32:59] =================== [SKIPPED] xe_migrate =================== [23:32:59] ================== xe_dma_buf (1 subtest) ================== [23:32:59] ==================== xe_dma_buf_kunit ===================== [23:32:59] ================ [SKIPPED] xe_dma_buf_kunit ================ [23:32:59] =================== [SKIPPED] xe_dma_buf =================== [23:32:59] ================= xe_bo_shrink (1 subtest) ================= [23:32:59] =================== xe_bo_shrink_kunit ==================== [23:32:59] =============== [SKIPPED] xe_bo_shrink_kunit =============== [23:32:59] ================== [SKIPPED] xe_bo_shrink ================== [23:32:59] ==================== xe_bo (2 subtests) ==================== [23:32:59] ================== xe_ccs_migrate_kunit =================== [23:32:59] ============== [SKIPPED] xe_ccs_migrate_kunit ============== [23:32:59] ==================== xe_bo_evict_kunit ==================== [23:32:59] =============== [SKIPPED] xe_bo_evict_kunit ================ [23:32:59] ===================== [SKIPPED] xe_bo ====================== [23:32:59] ==================== args (13 subtests) ==================== [23:32:59] [PASSED] count_args_test [23:32:59] [PASSED] call_args_example [23:32:59] [PASSED] call_args_test [23:32:59] [PASSED] drop_first_arg_example [23:32:59] [PASSED] drop_first_arg_test [23:32:59] [PASSED] first_arg_example [23:32:59] [PASSED] first_arg_test [23:32:59] [PASSED] last_arg_example [23:32:59] [PASSED] last_arg_test [23:32:59] [PASSED] pick_arg_example [23:32:59] [PASSED] if_args_example [23:32:59] [PASSED] if_args_test [23:32:59] [PASSED] sep_comma_example [23:32:59] ====================== [PASSED] args ======================= [23:32:59] =================== xe_pci (3 subtests) ==================== [23:32:59] ==================== check_graphics_ip ==================== [23:32:59] [PASSED] 12.00 Xe_LP [23:32:59] [PASSED] 12.10 Xe_LP+ [23:32:59] [PASSED] 12.55 Xe_HPG [23:32:59] [PASSED] 12.60 Xe_HPC [23:32:59] [PASSED] 12.70 Xe_LPG [23:32:59] [PASSED] 12.71 Xe_LPG [23:32:59] [PASSED] 12.74 Xe_LPG+ [23:32:59] [PASSED] 20.01 Xe2_HPG [23:32:59] [PASSED] 20.02 Xe2_HPG [23:32:59] [PASSED] 20.04 Xe2_LPG [23:32:59] [PASSED] 30.00 Xe3_LPG [23:32:59] [PASSED] 30.01 Xe3_LPG [23:32:59] [PASSED] 30.03 Xe3_LPG [23:32:59] [PASSED] 30.04 Xe3_LPG [23:32:59] [PASSED] 30.05 Xe3_LPG [23:32:59] [PASSED] 35.10 Xe3p_LPG [23:32:59] [PASSED] 35.11 Xe3p_XPC [23:32:59] ================ [PASSED] check_graphics_ip ================ [23:32:59] ===================== check_media_ip ====================== [23:32:59] [PASSED] 12.00 Xe_M [23:32:59] [PASSED] 12.55 Xe_HPM [23:32:59] [PASSED] 13.00 Xe_LPM+ [23:32:59] [PASSED] 13.01 Xe2_HPM [23:32:59] [PASSED] 20.00 Xe2_LPM [23:32:59] [PASSED] 30.00 Xe3_LPM [23:32:59] [PASSED] 30.02 Xe3_LPM [23:32:59] [PASSED] 35.00 Xe3p_LPM [23:32:59] [PASSED] 35.03 Xe3p_HPM [23:32:59] ================= [PASSED] check_media_ip ================== [23:32:59] =================== check_platform_desc =================== [23:32:59] [PASSED] 0x9A60 (TIGERLAKE) [23:32:59] [PASSED] 0x9A68 (TIGERLAKE) [23:32:59] [PASSED] 0x9A70 (TIGERLAKE) [23:32:59] [PASSED] 0x9A40 (TIGERLAKE) [23:32:59] [PASSED] 0x9A49 (TIGERLAKE) [23:32:59] [PASSED] 0x9A59 (TIGERLAKE) [23:32:59] [PASSED] 0x9A78 (TIGERLAKE) [23:32:59] [PASSED] 0x9AC0 (TIGERLAKE) [23:32:59] [PASSED] 0x9AC9 (TIGERLAKE) [23:32:59] [PASSED] 0x9AD9 (TIGERLAKE) [23:32:59] [PASSED] 0x9AF8 (TIGERLAKE) [23:32:59] [PASSED] 0x4C80 (ROCKETLAKE) [23:32:59] [PASSED] 0x4C8A (ROCKETLAKE) [23:32:59] [PASSED] 0x4C8B (ROCKETLAKE) [23:32:59] [PASSED] 0x4C8C (ROCKETLAKE) [23:32:59] [PASSED] 0x4C90 (ROCKETLAKE) [23:32:59] [PASSED] 0x4C9A (ROCKETLAKE) [23:32:59] [PASSED] 0x4680 (ALDERLAKE_S) [23:32:59] [PASSED] 0x4682 (ALDERLAKE_S) [23:32:59] [PASSED] 0x4688 (ALDERLAKE_S) [23:32:59] [PASSED] 0x468A (ALDERLAKE_S) [23:32:59] [PASSED] 0x468B (ALDERLAKE_S) [23:32:59] [PASSED] 0x4690 (ALDERLAKE_S) [23:32:59] [PASSED] 0x4692 (ALDERLAKE_S) [23:32:59] [PASSED] 0x4693 (ALDERLAKE_S) [23:32:59] [PASSED] 0x46A0 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46A1 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46A2 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46A3 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46A6 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46A8 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46AA (ALDERLAKE_P) [23:32:59] [PASSED] 0x462A (ALDERLAKE_P) [23:32:59] [PASSED] 0x4626 (ALDERLAKE_P) [23:32:59] [PASSED] 0x4628 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46B0 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46B1 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46B2 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46B3 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46C0 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46C1 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46C2 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46C3 (ALDERLAKE_P) [23:32:59] [PASSED] 0x46D0 (ALDERLAKE_N) [23:32:59] [PASSED] 0x46D1 (ALDERLAKE_N) [23:32:59] [PASSED] 0x46D2 (ALDERLAKE_N) [23:32:59] [PASSED] 0x46D3 (ALDERLAKE_N) [23:32:59] [PASSED] 0x46D4 (ALDERLAKE_N) [23:32:59] [PASSED] 0xA721 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7A1 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7A9 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7AC (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7AD (ALDERLAKE_P) [23:32:59] [PASSED] 0xA720 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7A0 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7A8 (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7AA (ALDERLAKE_P) [23:32:59] [PASSED] 0xA7AB (ALDERLAKE_P) [23:32:59] [PASSED] 0xA780 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA781 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA782 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA783 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA788 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA789 (ALDERLAKE_S) [23:32:59] [PASSED] 0xA78A (ALDERLAKE_S) [23:32:59] [PASSED] 0xA78B (ALDERLAKE_S) [23:32:59] [PASSED] 0x4905 (DG1) [23:32:59] [PASSED] 0x4906 (DG1) [23:32:59] [PASSED] 0x4907 (DG1) [23:32:59] [PASSED] 0x4908 (DG1) [23:32:59] [PASSED] 0x4909 (DG1) [23:32:59] [PASSED] 0x56C0 (DG2) [23:32:59] [PASSED] 0x56C2 (DG2) [23:32:59] [PASSED] 0x56C1 (DG2) [23:32:59] [PASSED] 0x7D51 (METEORLAKE) [23:32:59] [PASSED] 0x7DD1 (METEORLAKE) [23:32:59] [PASSED] 0x7D41 (METEORLAKE) [23:32:59] [PASSED] 0x7D67 (METEORLAKE) [23:32:59] [PASSED] 0xB640 (METEORLAKE) [23:32:59] [PASSED] 0x56A0 (DG2) [23:32:59] [PASSED] 0x56A1 (DG2) [23:32:59] [PASSED] 0x56A2 (DG2) [23:32:59] [PASSED] 0x56BE (DG2) [23:32:59] [PASSED] 0x56BF (DG2) [23:32:59] [PASSED] 0x5690 (DG2) [23:32:59] [PASSED] 0x5691 (DG2) [23:32:59] [PASSED] 0x5692 (DG2) [23:32:59] [PASSED] 0x56A5 (DG2) [23:32:59] [PASSED] 0x56A6 (DG2) [23:32:59] [PASSED] 0x56B0 (DG2) [23:32:59] [PASSED] 0x56B1 (DG2) [23:32:59] [PASSED] 0x56BA (DG2) [23:32:59] [PASSED] 0x56BB (DG2) [23:32:59] [PASSED] 0x56BC (DG2) [23:32:59] [PASSED] 0x56BD (DG2) [23:32:59] [PASSED] 0x5693 (DG2) [23:32:59] [PASSED] 0x5694 (DG2) [23:32:59] [PASSED] 0x5695 (DG2) [23:32:59] [PASSED] 0x56A3 (DG2) [23:32:59] [PASSED] 0x56A4 (DG2) [23:32:59] [PASSED] 0x56B2 (DG2) [23:32:59] [PASSED] 0x56B3 (DG2) [23:32:59] [PASSED] 0x5696 (DG2) [23:32:59] [PASSED] 0x5697 (DG2) [23:32:59] [PASSED] 0xB69 (PVC) [23:32:59] [PASSED] 0xB6E (PVC) [23:32:59] [PASSED] 0xBD4 (PVC) [23:32:59] [PASSED] 0xBD5 (PVC) [23:32:59] [PASSED] 0xBD6 (PVC) [23:32:59] [PASSED] 0xBD7 (PVC) [23:32:59] [PASSED] 0xBD8 (PVC) [23:32:59] [PASSED] 0xBD9 (PVC) [23:32:59] [PASSED] 0xBDA (PVC) [23:32:59] [PASSED] 0xBDB (PVC) [23:32:59] [PASSED] 0xBE0 (PVC) [23:32:59] [PASSED] 0xBE1 (PVC) [23:32:59] [PASSED] 0xBE5 (PVC) [23:32:59] [PASSED] 0x7D40 (METEORLAKE) [23:32:59] [PASSED] 0x7D45 (METEORLAKE) [23:32:59] [PASSED] 0x7D55 (METEORLAKE) [23:32:59] [PASSED] 0x7D60 (METEORLAKE) [23:32:59] [PASSED] 0x7DD5 (METEORLAKE) [23:32:59] [PASSED] 0x6420 (LUNARLAKE) [23:32:59] [PASSED] 0x64A0 (LUNARLAKE) [23:32:59] [PASSED] 0x64B0 (LUNARLAKE) [23:32:59] [PASSED] 0xE202 (BATTLEMAGE) [23:32:59] [PASSED] 0xE209 (BATTLEMAGE) [23:32:59] [PASSED] 0xE20B (BATTLEMAGE) [23:32:59] [PASSED] 0xE20C (BATTLEMAGE) [23:32:59] [PASSED] 0xE20D (BATTLEMAGE) [23:32:59] [PASSED] 0xE210 (BATTLEMAGE) [23:32:59] [PASSED] 0xE211 (BATTLEMAGE) [23:32:59] [PASSED] 0xE212 (BATTLEMAGE) [23:32:59] [PASSED] 0xE216 (BATTLEMAGE) [23:32:59] [PASSED] 0xE220 (BATTLEMAGE) [23:32:59] [PASSED] 0xE221 (BATTLEMAGE) [23:32:59] [PASSED] 0xE222 (BATTLEMAGE) [23:32:59] [PASSED] 0xE223 (BATTLEMAGE) [23:32:59] [PASSED] 0xB080 (PANTHERLAKE) [23:32:59] [PASSED] 0xB081 (PANTHERLAKE) [23:32:59] [PASSED] 0xB082 (PANTHERLAKE) [23:32:59] [PASSED] 0xB083 (PANTHERLAKE) [23:32:59] [PASSED] 0xB084 (PANTHERLAKE) [23:32:59] [PASSED] 0xB085 (PANTHERLAKE) [23:32:59] [PASSED] 0xB086 (PANTHERLAKE) [23:32:59] [PASSED] 0xB087 (PANTHERLAKE) [23:32:59] [PASSED] 0xB08F (PANTHERLAKE) [23:32:59] [PASSED] 0xB090 (PANTHERLAKE) [23:32:59] [PASSED] 0xB0A0 (PANTHERLAKE) [23:32:59] [PASSED] 0xB0B0 (PANTHERLAKE) [23:32:59] [PASSED] 0xFD80 (PANTHERLAKE) [23:32:59] [PASSED] 0xFD81 (PANTHERLAKE) [23:32:59] [PASSED] 0xD740 (NOVALAKE_S) [23:32:59] [PASSED] 0xD741 (NOVALAKE_S) [23:32:59] [PASSED] 0xD742 (NOVALAKE_S) [23:32:59] [PASSED] 0xD743 (NOVALAKE_S) [23:32:59] [PASSED] 0xD745 (NOVALAKE_S) [23:32:59] [PASSED] 0xD74A (NOVALAKE_S) [23:32:59] [PASSED] 0xD74B (NOVALAKE_S) [23:32:59] [PASSED] 0x674C (CRESCENTISLAND) [23:32:59] [PASSED] 0x674D (CRESCENTISLAND) [23:32:59] [PASSED] 0x674E (CRESCENTISLAND) [23:32:59] [PASSED] 0x674F (CRESCENTISLAND) [23:32:59] [PASSED] 0x6750 (CRESCENTISLAND) [23:32:59] [PASSED] 0xD750 (NOVALAKE_P) [23:32:59] [PASSED] 0xD751 (NOVALAKE_P) [23:32:59] [PASSED] 0xD752 (NOVALAKE_P) [23:32:59] [PASSED] 0xD753 (NOVALAKE_P) [23:32:59] [PASSED] 0xD754 (NOVALAKE_P) [23:32:59] [PASSED] 0xD755 (NOVALAKE_P) [23:32:59] [PASSED] 0xD756 (NOVALAKE_P) [23:32:59] [PASSED] 0xD757 (NOVALAKE_P) [23:32:59] [PASSED] 0xD75F (NOVALAKE_P) [23:32:59] =============== [PASSED] check_platform_desc =============== [23:32:59] ===================== [PASSED] xe_pci ====================== [23:32:59] ============= xe_rtp_tables_test (5 subtests) ============== [23:32:59] ================== xe_rtp_table_gt_test =================== [23:32:59] [PASSED] gt_was/14011060649 [23:32:59] [PASSED] gt_was/14011059788 [23:32:59] [PASSED] gt_was/14015795083 [23:32:59] [PASSED] gt_was/16021867713 [23:32:59] [PASSED] gt_was/14019449301 [23:32:59] [PASSED] gt_was/16028005424 [23:32:59] [PASSED] gt_was/14026578760 [23:32:59] [PASSED] gt_was/1409420604 [23:32:59] [PASSED] gt_was/1408615072 [23:32:59] [PASSED] gt_was/22010523718 [23:32:59] [PASSED] gt_was/14011006942 [23:32:59] [PASSED] gt_was/14014830051 [23:32:59] [PASSED] gt_was/18018781329 [23:32:59] [PASSED] gt_was/1509235366 [23:32:59] [PASSED] gt_was/18018781329 [23:32:59] [PASSED] gt_was/16016694945 [23:32:59] [PASSED] gt_was/14018575942 [23:32:59] [PASSED] gt_was/22016670082 [23:32:59] [PASSED] gt_was/22016670082 [23:32:59] [PASSED] gt_was/14017421178 [23:32:59] [PASSED] gt_was/16025250150 [23:32:59] [PASSED] gt_was/14021871409 [23:32:59] [PASSED] gt_was/16021865536 [23:32:59] [PASSED] gt_was/14021486841 [23:32:59] [PASSED] gt_was/14025160223 [23:32:59] [PASSED] gt_was/14026144927, 16029437861, 14026127056 [23:32:59] [PASSED] gt_was/14025635424 [23:32:59] [PASSED] gt_was/16028005424 [23:32:59] ============== [PASSED] xe_rtp_table_gt_test =============== [23:32:59] ================== xe_rtp_table_gt_test =================== [23:32:59] [PASSED] gt_tunings/Tuning: Blend Fill Caching Optimization Disable [23:32:59] [PASSED] gt_tunings/Tuning: 32B Access Enable [23:32:59] [PASSED] gt_tunings/Tuning: L3 cache [23:32:59] [PASSED] gt_tunings/Tuning: L3 cache - media [23:32:59] [PASSED] gt_tunings/Tuning: Compression Overfetch [23:32:59] [PASSED] gt_tunings/Tuning: Compression Overfetch - media [23:32:59] [PASSED] gt_tunings/Tuning: Enable compressible partial write overfetch in L3 [23:32:59] [PASSED] gt_tunings/Tuning: Enable compressible partial write overfetch in L3 - media [23:32:59] [PASSED] gt_tunings/Tuning: L2 Overfetch Compressible Only [23:32:59] [PASSED] gt_tunings/Tuning: L2 Overfetch Compressible Only - media [23:32:59] [PASSED] gt_tunings/Tuning: Stateless compression control [23:32:59] [PASSED] gt_tunings/Tuning: Stateless compression control - media [23:32:59] [PASSED] gt_tunings/Tuning: L3 RW flush all Cache [23:32:59] [PASSED] gt_tunings/Tuning: L3 RW flush all cache - media [23:32:59] [PASSED] gt_tunings/Tuning: Set STLB Bank Hash Mode to 4KB [23:32:59] ============== [PASSED] xe_rtp_table_gt_test =============== [23:32:59] ================== xe_rtp_table_oob_test ================== [23:32:59] [PASSED] oob_was/1607983814 [23:32:59] [PASSED] oob_was/16010904313 [23:32:59] [PASSED] oob_was/18022495364 [23:32:59] [PASSED] oob_was/22012773006 [23:32:59] [PASSED] oob_was/14014475959 [23:32:59] [PASSED] oob_was/22011391025 [23:32:59] [PASSED] oob_was/22012727170 [23:32:59] [PASSED] oob_was/22012727685 [23:32:59] [PASSED] oob_was/22016596838 [23:32:59] [PASSED] oob_was/18020744125 [23:32:59] [PASSED] oob_was/1409600907 [23:32:59] [PASSED] oob_was/22014953428 [23:32:59] [PASSED] oob_was/16017236439 [23:32:59] [PASSED] oob_was/14019821291 [23:32:59] [PASSED] oob_was/14015076503 [23:32:59] [PASSED] oob_was/14018913170 [23:32:59] [PASSED] oob_was/14018094691 [23:32:59] [PASSED] oob_was/18024947630 [23:32:59] [PASSED] oob_was/16022287689 [23:32:59] [PASSED] oob_was/13011645652 [23:32:59] [PASSED] oob_was/14022293748 [23:32:59] [PASSED] oob_was/22019794406 [23:32:59] [PASSED] oob_was/22019338487 [23:32:59] [PASSED] oob_was/16023588340 [23:32:59] [PASSED] oob_was/14019789679 [23:32:59] [PASSED] oob_was/14022866841 [23:32:59] [PASSED] oob_was/16021333562 [23:32:59] [PASSED] oob_was/14016712196 [23:32:59] [PASSED] oob_was/14015568240 [23:32:59] [PASSED] oob_was/18013179988 [23:32:59] [PASSED] oob_was/1508761755 [23:32:59] [PASSED] oob_was/16023105232 [23:32:59] [PASSED] oob_was/16026508708 [23:32:59] [PASSED] oob_was/14020001231 [23:32:59] [PASSED] oob_was/16023683509 [23:32:59] [PASSED] oob_was/14025515070 [23:32:59] [PASSED] oob_was/15015404425_disable [23:32:59] [PASSED] oob_was/16026007364 [23:32:59] [PASSED] oob_was/14020316580 [23:32:59] [PASSED] oob_was/14025883347 [23:32:59] [PASSED] oob_was/16029380221 [23:32:59] [PASSED] oob_was/22022079272 [23:32:59] [PASSED] oob_was/16029897822 [23:32:59] [PASSED] oob_was/14027054324 [23:32:59] ============== [PASSED] xe_rtp_table_oob_test ============== [23:32:59] ================ xe_rtp_table_dev_oob_test ================ [23:32:59] [PASSED] device_oob_was/22010954014 [23:32:59] [PASSED] device_oob_was/15015404425 [23:32:59] [PASSED] device_oob_was/22019338487_display [23:32:59] [PASSED] device_oob_was/14022085890 [23:32:59] [PASSED] device_oob_was/14026539277 [23:32:59] [PASSED] device_oob_was/14026633728 [23:32:59] [PASSED] device_oob_was/14026746987 [23:32:59] [PASSED] device_oob_was/14026779378 [23:32:59] ============ [PASSED] xe_rtp_table_dev_oob_test ============ [23:32:59] ========== xe_rtp_table_missing_upper_bound_test ========== [23:32:59] [PASSED] register_whitelist/WaAllowPMDepthAndInvocationCountAccessFromUMD, 1408556865 [23:32:59] [PASSED] register_whitelist/1508744258, 14012131227, 1808121037 [23:32:59] [PASSED] register_whitelist/1806527549 [23:32:59] [PASSED] register_whitelist/allow_read_ctx_timestamp [23:32:59] [PASSED] register_whitelist/allow_read_queue_timestamp [23:32:59] [PASSED] register_whitelist/16014440446 [23:32:59] [PASSED] register_whitelist/16017236439 [23:32:59] [PASSED] register_whitelist/16020183090 [23:32:59] [PASSED] register_whitelist/14024997852 [23:32:59] [PASSED] register_whitelist/14024997852 [23:32:59] ====== [PASSED] xe_rtp_table_missing_upper_bound_test ====== [23:32:59] =============== [PASSED] xe_rtp_tables_test ================ [23:32:59] =================== xe_rtp (3 subtests) ==================== [23:32:59] =================== xe_rtp_rules_tests ==================== [23:32:59] [PASSED] no [23:32:59] [PASSED] yes [23:32:59] [PASSED] no-and-no [23:32:59] [PASSED] no-and-yes [23:32:59] [PASSED] yes-and-no [23:32:59] [PASSED] yes-and-yes [23:32:59] [PASSED] no-or-no [23:32:59] [PASSED] no-or-yes [23:32:59] [PASSED] yes-or-no [23:32:59] [PASSED] yes-or-yes [23:32:59] [PASSED] no-yes-or-yes-no [23:32:59] [PASSED] no-yes-or-yes-yes [23:32:59] [PASSED] yes-yes-or-no-yes [23:32:59] [PASSED] yes-yes-or-yes-yes [23:32:59] [PASSED] no-no-or-yes-or-no [23:32:59] [PASSED] or [23:32:59] [PASSED] or-yes [23:32:59] [PASSED] or-no [23:32:59] [PASSED] yes-or [23:32:59] [PASSED] no-or [23:32:59] [PASSED] no-or-or-yes [23:32:59] [PASSED] yes-or-or-no [23:32:59] [PASSED] no-or-or-no [23:32:59] [PASSED] missing-context-engine-class [23:32:59] [PASSED] missing-context-engine-class-or-yes [23:32:59] [PASSED] missing-context-engine-class-or-or-yes [23:32:59] =============== [PASSED] xe_rtp_rules_tests ================ [23:32:59] =============== xe_rtp_process_to_sr_tests ================ [23:32:59] [PASSED] coalesce-same-reg [23:32:59] [PASSED] coalesce-same-reg-literal-and-func [23:32:59] [PASSED] no-match-no-add [23:32:59] [PASSED] two-regs-two-entries [23:32:59] [PASSED] clr-one-set-other [23:32:59] [PASSED] set-field [23:32:59] [PASSED] conflict-duplicate [23:32:59] [PASSED] conflict-not-disjoint [23:32:59] [PASSED] conflict-not-disjoint-literal-and-func [23:32:59] [PASSED] conflict-reg-type [23:32:59] [PASSED] bad-mcr-reg-forced-to-regular [23:32:59] [PASSED] bad-regular-reg-forced-to-mcr [23:32:59] =========== [PASSED] xe_rtp_process_to_sr_tests ============ [23:32:59] ================== xe_rtp_process_tests =================== [23:32:59] [PASSED] active1 [23:32:59] [PASSED] active2 [23:32:59] [PASSED] active-inactive [23:32:59] [PASSED] inactive-active [23:32:59] [PASSED] inactive-active-inactive [23:32:59] [PASSED] inactive-inactive-inactive [23:32:59] ============== [PASSED] xe_rtp_process_tests =============== [23:32:59] ===================== [PASSED] xe_rtp ====================== [23:32:59] ==================== xe_wa (1 subtest) ===================== [23:32:59] ======================== xe_wa_gt ========================= [23:32:59] [PASSED] TIGERLAKE B0 [23:32:59] [PASSED] DG1 A0 [23:32:59] [PASSED] DG1 B0 [23:32:59] [PASSED] ALDERLAKE_S A0 [23:32:59] [PASSED] ALDERLAKE_S B0 [23:32:59] [PASSED] ALDERLAKE_S C0 [23:32:59] [PASSED] ALDERLAKE_S D0 [23:32:59] [PASSED] ALDERLAKE_P A0 [23:32:59] [PASSED] ALDERLAKE_P B0 [23:32:59] [PASSED] ALDERLAKE_P C0 [23:32:59] [PASSED] ALDERLAKE_S RPLS D0 [23:32:59] [PASSED] ALDERLAKE_P RPLU E0 [23:32:59] [PASSED] DG2 G10 C0 [23:32:59] [PASSED] DG2 G11 B1 [23:32:59] [PASSED] DG2 G12 A1 [23:32:59] [PASSED] METEORLAKE 12.70(Xe_LPG) A0 13.00(Xe_LPM+) A0 [23:32:59] [PASSED] METEORLAKE 12.71(Xe_LPG) A0 13.00(Xe_LPM+) A0 [23:32:59] [PASSED] METEORLAKE 12.74(Xe_LPG+) A0 13.00(Xe_LPM+) A0 [23:32:59] [PASSED] LUNARLAKE 20.04(Xe2_LPG) A0 20.00(Xe2_LPM) A0 [23:32:59] [PASSED] LUNARLAKE 20.04(Xe2_LPG) B0 20.00(Xe2_LPM) A0 [23:32:59] [PASSED] BATTLEMAGE 20.01(Xe2_HPG) A0 13.01(Xe2_HPM) A1 [23:32:59] [PASSED] PANTHERLAKE 30.00(Xe3_LPG) A0 30.00(Xe3_LPM) A0 [23:32:59] ==================== [PASSED] xe_wa_gt ===================== [23:32:59] ====================== [PASSED] xe_wa ====================== [23:32:59] ============================================================ [23:32:59] Testing complete. Ran 742 tests: passed: 724, skipped: 18 [23:32:59] Elapsed time: 37.329s total, 4.301s configuring, 32.362s building, 0.641s running
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/tests/.kunitconfig [23:33:00] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [23:33:01] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 [23:33:27] Starting KUnit Kernel (1/1)... [23:33:27] ============================================================ Running tests with: $ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt [23:33:27] ============ drm_test_pick_cmdline (2 subtests) ============ [23:33:27] [PASSED] drm_test_pick_cmdline_res_1920_1080_60 [23:33:27] =============== drm_test_pick_cmdline_named =============== [23:33:27] [PASSED] NTSC [23:33:27] [PASSED] NTSC-J [23:33:27] [PASSED] PAL [23:33:27] [PASSED] PAL-M [23:33:27] =========== [PASSED] drm_test_pick_cmdline_named =========== [23:33:27] ============== [PASSED] drm_test_pick_cmdline ============== [23:33:27] == drm_test_atomic_get_connector_for_encoder (1 subtest) === [23:33:27] [PASSED] drm_test_drm_atomic_get_connector_for_encoder [23:33:27] ==== [PASSED] drm_test_atomic_get_connector_for_encoder ==== [23:33:27] =========== drm_validate_clone_mode (2 subtests) =========== [23:33:27] ============== drm_test_check_in_clone_mode =============== [23:33:27] [PASSED] in_clone_mode [23:33:27] [PASSED] not_in_clone_mode [23:33:27] ========== [PASSED] drm_test_check_in_clone_mode =========== [23:33:27] =============== drm_test_check_valid_clones =============== [23:33:27] [PASSED] not_in_clone_mode [23:33:27] [PASSED] valid_clone [23:33:27] [PASSED] invalid_clone [23:33:27] =========== [PASSED] drm_test_check_valid_clones =========== [23:33:27] ============= [PASSED] drm_validate_clone_mode ============= [23:33:27] ============= drm_validate_modeset (1 subtest) ============= [23:33:27] [PASSED] drm_test_check_connector_changed_modeset [23:33:27] ============== [PASSED] drm_validate_modeset =============== [23:33:27] ====== drm_test_bridge_get_current_state (1 subtest) ======= [23:33:27] [PASSED] drm_test_drm_bridge_get_current_state_atomic [23:33:27] ======== [PASSED] drm_test_bridge_get_current_state ======== [23:33:27] ====== drm_test_bridge_helper_reset_crtc (3 subtests) ====== [23:33:27] [PASSED] drm_test_drm_bridge_helper_reset_crtc_atomic [23:33:27] [PASSED] drm_test_drm_bridge_helper_reset_crtc_atomic_disabled [23:33:27] [PASSED] drm_test_drm_bridge_helper_hdmi_output_bus_fmts [23:33:27] ======== [PASSED] drm_test_bridge_helper_reset_crtc ======== [23:33:27] ============== drm_bridge_alloc (2 subtests) =============== [23:33:27] [PASSED] drm_test_drm_bridge_alloc_basic [23:33:27] [PASSED] drm_test_drm_bridge_alloc_get_put [23:33:27] ================ [PASSED] drm_bridge_alloc ================= [23:33:27] ============= drm_bridge_bus_fmt (5 subtests) ============== [23:33:27] [PASSED] drm_test_bridge_rgb_yuv_rgb [23:33:27] [PASSED] drm_test_bridge_must_convert_to_yuv444 [23:33:27] [PASSED] drm_test_bridge_hdmi_auto_rgb [23:33:27] [PASSED] drm_test_bridge_auto_first [23:33:27] [PASSED] drm_test_bridge_rgb_yuv_no_path [23:33:27] =============== [PASSED] drm_bridge_bus_fmt ================ [23:33:27] ============= drm_cmdline_parser (40 subtests) ============= [23:33:27] [PASSED] drm_test_cmdline_force_d_only [23:33:27] [PASSED] drm_test_cmdline_force_D_only_dvi [23:33:27] [PASSED] drm_test_cmdline_force_D_only_hdmi [23:33:27] [PASSED] drm_test_cmdline_force_D_only_not_digital [23:33:27] [PASSED] drm_test_cmdline_force_e_only [23:33:27] [PASSED] drm_test_cmdline_res [23:33:27] [PASSED] drm_test_cmdline_res_vesa [23:33:27] [PASSED] drm_test_cmdline_res_vesa_rblank [23:33:27] [PASSED] drm_test_cmdline_res_rblank [23:33:27] [PASSED] drm_test_cmdline_res_bpp [23:33:27] [PASSED] drm_test_cmdline_res_refresh [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_margins [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_force_off [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_analog [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_digital [23:33:27] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced_margins_force_on [23:33:27] [PASSED] drm_test_cmdline_res_margins_force_on [23:33:27] [PASSED] drm_test_cmdline_res_vesa_margins [23:33:27] [PASSED] drm_test_cmdline_name [23:33:27] [PASSED] drm_test_cmdline_name_bpp [23:33:27] [PASSED] drm_test_cmdline_name_option [23:33:27] [PASSED] drm_test_cmdline_name_bpp_option [23:33:27] [PASSED] drm_test_cmdline_rotate_0 [23:33:27] [PASSED] drm_test_cmdline_rotate_90 [23:33:27] [PASSED] drm_test_cmdline_rotate_180 [23:33:27] [PASSED] drm_test_cmdline_rotate_270 [23:33:27] [PASSED] drm_test_cmdline_hmirror [23:33:27] [PASSED] drm_test_cmdline_vmirror [23:33:27] [PASSED] drm_test_cmdline_margin_options [23:33:27] [PASSED] drm_test_cmdline_multiple_options [23:33:27] [PASSED] drm_test_cmdline_bpp_extra_and_option [23:33:27] [PASSED] drm_test_cmdline_extra_and_option [23:33:27] [PASSED] drm_test_cmdline_freestanding_options [23:33:27] [PASSED] drm_test_cmdline_freestanding_force_e_and_options [23:33:27] [PASSED] drm_test_cmdline_panel_orientation [23:33:27] ================ drm_test_cmdline_invalid ================= [23:33:27] [PASSED] margin_only [23:33:27] [PASSED] interlace_only [23:33:27] [PASSED] res_missing_x [23:33:27] [PASSED] res_missing_y [23:33:27] [PASSED] res_bad_y [23:33:27] [PASSED] res_missing_y_bpp [23:33:27] [PASSED] res_bad_bpp [23:33:27] [PASSED] res_bad_refresh [23:33:27] [PASSED] res_bpp_refresh_force_on_off [23:33:27] [PASSED] res_invalid_mode [23:33:27] [PASSED] res_bpp_wrong_place_mode [23:33:27] [PASSED] name_bpp_refresh [23:33:27] [PASSED] name_refresh [23:33:27] [PASSED] name_refresh_wrong_mode [23:33:27] [PASSED] name_refresh_invalid_mode [23:33:27] [PASSED] rotate_multiple [23:33:27] [PASSED] rotate_invalid_val [23:33:27] [PASSED] rotate_truncated [23:33:27] [PASSED] invalid_option [23:33:27] [PASSED] invalid_tv_option [23:33:27] [PASSED] truncated_tv_option [23:33:27] ============ [PASSED] drm_test_cmdline_invalid ============= [23:33:27] =============== drm_test_cmdline_tv_options =============== [23:33:27] [PASSED] NTSC [23:33:27] [PASSED] NTSC_443 [23:33:27] [PASSED] NTSC_J [23:33:27] [PASSED] PAL [23:33:27] [PASSED] PAL_M [23:33:27] [PASSED] PAL_N [23:33:27] [PASSED] SECAM [23:33:27] [PASSED] MONO_525 [23:33:27] [PASSED] MONO_625 [23:33:27] =========== [PASSED] drm_test_cmdline_tv_options =========== [23:33:27] =============== [PASSED] drm_cmdline_parser ================ [23:33:27] ========== drmm_connector_hdmi_init (20 subtests) ========== [23:33:27] [PASSED] drm_test_connector_hdmi_init_valid [23:33:27] [PASSED] drm_test_connector_hdmi_init_bpc_8 [23:33:27] [PASSED] drm_test_connector_hdmi_init_bpc_10 [23:33:27] [PASSED] drm_test_connector_hdmi_init_bpc_12 [23:33:27] [PASSED] drm_test_connector_hdmi_init_bpc_invalid [23:33:27] [PASSED] drm_test_connector_hdmi_init_bpc_null [23:33:27] [PASSED] drm_test_connector_hdmi_init_formats_empty [23:33:27] [PASSED] drm_test_connector_hdmi_init_formats_no_rgb [23:33:27] === drm_test_connector_hdmi_init_formats_yuv420_allowed === [23:33:27] [PASSED] supported_formats=0x9 yuv420_allowed=1 [23:33:27] [PASSED] supported_formats=0x9 yuv420_allowed=0 [23:33:27] [PASSED] supported_formats=0x5 yuv420_allowed=1 [23:33:27] [PASSED] supported_formats=0x5 yuv420_allowed=0 [23:33:27] === [PASSED] drm_test_connector_hdmi_init_formats_yuv420_allowed === [23:33:27] [PASSED] drm_test_connector_hdmi_init_null_ddc [23:33:27] [PASSED] drm_test_connector_hdmi_init_null_product [23:33:27] [PASSED] drm_test_connector_hdmi_init_null_vendor [23:33:27] [PASSED] drm_test_connector_hdmi_init_product_length_exact [23:33:27] [PASSED] drm_test_connector_hdmi_init_product_length_too_long [23:33:27] [PASSED] drm_test_connector_hdmi_init_product_valid [23:33:27] [PASSED] drm_test_connector_hdmi_init_vendor_length_exact [23:33:27] [PASSED] drm_test_connector_hdmi_init_vendor_length_too_long [23:33:27] [PASSED] drm_test_connector_hdmi_init_vendor_valid [23:33:27] ========= drm_test_connector_hdmi_init_type_valid ========= [23:33:27] [PASSED] HDMI-A [23:33:27] [PASSED] HDMI-B [23:33:27] ===== [PASSED] drm_test_connector_hdmi_init_type_valid ===== [23:33:27] ======== drm_test_connector_hdmi_init_type_invalid ======== [23:33:27] [PASSED] Unknown [23:33:27] [PASSED] VGA [23:33:27] [PASSED] DVI-I [23:33:27] [PASSED] DVI-D [23:33:27] [PASSED] DVI-A [23:33:27] [PASSED] Composite [23:33:27] [PASSED] SVIDEO [23:33:27] [PASSED] LVDS [23:33:27] [PASSED] Component [23:33:27] [PASSED] DIN [23:33:27] [PASSED] DP [23:33:27] [PASSED] TV [23:33:27] [PASSED] eDP [23:33:27] [PASSED] Virtual [23:33:27] [PASSED] DSI [23:33:27] [PASSED] DPI [23:33:27] [PASSED] Writeback [23:33:27] [PASSED] SPI [23:33:27] [PASSED] USB [23:33:27] ==== [PASSED] drm_test_connector_hdmi_init_type_invalid ==== [23:33:27] ============ [PASSED] drmm_connector_hdmi_init ============= [23:33:27] ============= drmm_connector_init (3 subtests) ============= [23:33:27] [PASSED] drm_test_drmm_connector_init [23:33:27] [PASSED] drm_test_drmm_connector_init_null_ddc [23:33:27] ========= drm_test_drmm_connector_init_type_valid ========= [23:33:27] [PASSED] Unknown [23:33:27] [PASSED] VGA [23:33:27] [PASSED] DVI-I [23:33:27] [PASSED] DVI-D [23:33:27] [PASSED] DVI-A [23:33:27] [PASSED] Composite [23:33:27] [PASSED] SVIDEO [23:33:27] [PASSED] LVDS [23:33:27] [PASSED] Component [23:33:27] [PASSED] DIN [23:33:27] [PASSED] DP [23:33:27] [PASSED] HDMI-A [23:33:27] [PASSED] HDMI-B [23:33:27] [PASSED] TV [23:33:27] [PASSED] eDP [23:33:27] [PASSED] Virtual [23:33:27] [PASSED] DSI [23:33:27] [PASSED] DPI [23:33:27] [PASSED] Writeback [23:33:27] [PASSED] SPI [23:33:27] [PASSED] USB [23:33:27] ===== [PASSED] drm_test_drmm_connector_init_type_valid ===== [23:33:27] =============== [PASSED] drmm_connector_init =============== [23:33:27] ========= drm_connector_dynamic_init (6 subtests) ========== [23:33:27] [PASSED] drm_test_drm_connector_dynamic_init [23:33:27] [PASSED] drm_test_drm_connector_dynamic_init_null_ddc [23:33:27] [PASSED] drm_test_drm_connector_dynamic_init_not_added [23:33:27] [PASSED] drm_test_drm_connector_dynamic_init_properties [23:33:27] ===== drm_test_drm_connector_dynamic_init_type_valid ====== [23:33:27] [PASSED] Unknown [23:33:27] [PASSED] VGA [23:33:27] [PASSED] DVI-I [23:33:27] [PASSED] DVI-D [23:33:27] [PASSED] DVI-A [23:33:27] [PASSED] Composite [23:33:27] [PASSED] SVIDEO [23:33:27] [PASSED] LVDS [23:33:27] [PASSED] Component [23:33:27] [PASSED] DIN [23:33:27] [PASSED] DP [23:33:27] [PASSED] HDMI-A [23:33:27] [PASSED] HDMI-B [23:33:27] [PASSED] TV [23:33:27] [PASSED] eDP [23:33:27] [PASSED] Virtual [23:33:27] [PASSED] DSI [23:33:27] [PASSED] DPI [23:33:27] [PASSED] Writeback [23:33:27] [PASSED] SPI [23:33:27] [PASSED] USB [23:33:27] = [PASSED] drm_test_drm_connector_dynamic_init_type_valid == [23:33:27] ======== drm_test_drm_connector_dynamic_init_name ========= [23:33:27] [PASSED] Unknown [23:33:27] [PASSED] VGA [23:33:27] [PASSED] DVI-I [23:33:27] [PASSED] DVI-D [23:33:27] [PASSED] DVI-A [23:33:27] [PASSED] Composite [23:33:27] [PASSED] SVIDEO [23:33:27] [PASSED] LVDS [23:33:27] [PASSED] Component [23:33:27] [PASSED] DIN [23:33:27] [PASSED] DP [23:33:27] [PASSED] HDMI-A [23:33:27] [PASSED] HDMI-B [23:33:27] [PASSED] TV [23:33:27] [PASSED] eDP [23:33:27] [PASSED] Virtual [23:33:27] [PASSED] DSI [23:33:27] [PASSED] DPI [23:33:27] [PASSED] Writeback [23:33:27] [PASSED] SPI [23:33:27] [PASSED] USB [23:33:27] ==== [PASSED] drm_test_drm_connector_dynamic_init_name ===== [23:33:27] =========== [PASSED] drm_connector_dynamic_init ============ [23:33:27] ==== drm_connector_dynamic_register_early (4 subtests) ===== [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_early_on_list [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_early_defer [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_early_no_init [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_early_no_mode_object [23:33:27] ====== [PASSED] drm_connector_dynamic_register_early ======= [23:33:27] ======= drm_connector_dynamic_register (7 subtests) ======== [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_on_list [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_no_defer [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_no_init [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_mode_object [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_sysfs [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_sysfs_name [23:33:27] [PASSED] drm_test_drm_connector_dynamic_register_debugfs [23:33:27] ========= [PASSED] drm_connector_dynamic_register ========== [23:33:27] = drm_connector_attach_broadcast_rgb_property (2 subtests) = [23:33:27] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property [23:33:27] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property_hdmi_connector [23:33:27] === [PASSED] drm_connector_attach_broadcast_rgb_property === [23:33:27] ========== drm_get_tv_mode_from_name (2 subtests) ========== [23:33:27] ========== drm_test_get_tv_mode_from_name_valid =========== [23:33:27] [PASSED] NTSC [23:33:27] [PASSED] NTSC-443 [23:33:27] [PASSED] NTSC-J [23:33:27] [PASSED] PAL [23:33:27] [PASSED] PAL-M [23:33:27] [PASSED] PAL-N [23:33:27] [PASSED] SECAM [23:33:27] [PASSED] Mono [23:33:27] ====== [PASSED] drm_test_get_tv_mode_from_name_valid ======= [23:33:27] [PASSED] drm_test_get_tv_mode_from_name_truncated [23:33:27] ============ [PASSED] drm_get_tv_mode_from_name ============ [23:33:27] = drm_test_connector_hdmi_compute_mode_clock (12 subtests) = [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc_vic_1 [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc_vic_1 [23:33:27] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_double [23:33:27] = drm_test_connector_hdmi_compute_mode_clock_yuv420_valid = [23:33:27] [PASSED] VIC 96 [23:33:27] [PASSED] VIC 97 [23:33:27] [PASSED] VIC 101 [23:33:27] [PASSED] VIC 102 [23:33:27] [PASSED] VIC 106 [23:33:27] [PASSED] VIC 107 [23:33:27] === [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_valid === [23:33:27] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_10_bpc [23:33:27] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_12_bpc [23:33:27] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_8_bpc [23:33:27] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_10_bpc [23:33:27] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_12_bpc [23:33:27] === [PASSED] drm_test_connector_hdmi_compute_mode_clock ==== [23:33:27] == drm_hdmi_connector_get_broadcast_rgb_name (2 subtests) == [23:33:27] === drm_test_drm_hdmi_connector_get_broadcast_rgb_name ==== [23:33:27] [PASSED] Automatic [23:33:27] [PASSED] Full [23:33:27] [PASSED] Limited 16:235 [23:33:27] === [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name === [23:33:27] [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name_invalid [23:33:27] ==== [PASSED] drm_hdmi_connector_get_broadcast_rgb_name ==== [23:33:27] == drm_hdmi_connector_get_output_format_name (2 subtests) == [23:33:27] === drm_test_drm_hdmi_connector_get_output_format_name ==== [23:33:27] [PASSED] RGB [23:33:27] [PASSED] YUV 4:2:0 [23:33:27] [PASSED] YUV 4:2:2 [23:33:27] [PASSED] YUV 4:4:4 [23:33:27] === [PASSED] drm_test_drm_hdmi_connector_get_output_format_name === [23:33:27] [PASSED] drm_test_drm_hdmi_connector_get_output_format_name_invalid [23:33:27] ==== [PASSED] drm_hdmi_connector_get_output_format_name ==== [23:33:27] ============= drm_damage_helper (21 subtests) ============== [23:33:27] [PASSED] drm_test_damage_iter_no_damage [23:33:27] [PASSED] drm_test_damage_iter_no_damage_fractional_src [23:33:27] [PASSED] drm_test_damage_iter_no_damage_src_moved [23:33:27] [PASSED] drm_test_damage_iter_no_damage_fractional_src_moved [23:33:27] [PASSED] drm_test_damage_iter_no_damage_not_visible [23:33:27] [PASSED] drm_test_damage_iter_no_damage_no_crtc [23:33:27] [PASSED] drm_test_damage_iter_no_damage_no_fb [23:33:27] [PASSED] drm_test_damage_iter_simple_damage [23:33:27] [PASSED] drm_test_damage_iter_single_damage [23:33:27] [PASSED] drm_test_damage_iter_single_damage_intersect_src [23:33:27] [PASSED] drm_test_damage_iter_single_damage_outside_src [23:33:27] [PASSED] drm_test_damage_iter_single_damage_fractional_src [23:33:27] [PASSED] drm_test_damage_iter_single_damage_intersect_fractional_src [23:33:27] [PASSED] drm_test_damage_iter_single_damage_outside_fractional_src [23:33:27] [PASSED] drm_test_damage_iter_single_damage_src_moved [23:33:27] [PASSED] drm_test_damage_iter_single_damage_fractional_src_moved [23:33:27] [PASSED] drm_test_damage_iter_damage [23:33:27] [PASSED] drm_test_damage_iter_damage_one_intersect [23:33:27] [PASSED] drm_test_damage_iter_damage_one_outside [23:33:27] [PASSED] drm_test_damage_iter_damage_src_moved [23:33:27] [PASSED] drm_test_damage_iter_damage_not_visible [23:33:27] ================ [PASSED] drm_damage_helper ================ [23:33:27] ============== drm_dp_mst_helper (3 subtests) ============== [23:33:27] ============== drm_test_dp_mst_calc_pbn_mode ============== [23:33:27] [PASSED] Clock 154000 BPP 30 DSC disabled [23:33:27] [PASSED] Clock 234000 BPP 30 DSC disabled [23:33:27] [PASSED] Clock 297000 BPP 24 DSC disabled [23:33:27] [PASSED] Clock 332880 BPP 24 DSC enabled [23:33:27] [PASSED] Clock 324540 BPP 24 DSC enabled [23:33:27] ========== [PASSED] drm_test_dp_mst_calc_pbn_mode ========== [23:33:27] ============== drm_test_dp_mst_calc_pbn_div =============== [23:33:27] [PASSED] Link rate 2000000 lane count 4 [23:33:27] [PASSED] Link rate 2000000 lane count 2 [23:33:27] [PASSED] Link rate 2000000 lane count 1 [23:33:27] [PASSED] Link rate 1350000 lane count 4 [23:33:27] [PASSED] Link rate 1350000 lane count 2 [23:33:27] [PASSED] Link rate 1350000 lane count 1 [23:33:27] [PASSED] Link rate 1000000 lane count 4 [23:33:27] [PASSED] Link rate 1000000 lane count 2 [23:33:27] [PASSED] Link rate 1000000 lane count 1 [23:33:27] [PASSED] Link rate 810000 lane count 4 [23:33:27] [PASSED] Link rate 810000 lane count 2 [23:33:27] [PASSED] Link rate 810000 lane count 1 [23:33:27] [PASSED] Link rate 540000 lane count 4 [23:33:27] [PASSED] Link rate 540000 lane count 2 [23:33:27] [PASSED] Link rate 540000 lane count 1 [23:33:27] [PASSED] Link rate 270000 lane count 4 [23:33:27] [PASSED] Link rate 270000 lane count 2 [23:33:27] [PASSED] Link rate 270000 lane count 1 [23:33:27] [PASSED] Link rate 162000 lane count 4 [23:33:27] [PASSED] Link rate 162000 lane count 2 [23:33:27] [PASSED] Link rate 162000 lane count 1 [23:33:27] ========== [PASSED] drm_test_dp_mst_calc_pbn_div =========== [23:33:27] ========= drm_test_dp_mst_sideband_msg_req_decode ========= [23:33:27] [PASSED] DP_ENUM_PATH_RESOURCES with port number [23:33:27] [PASSED] DP_POWER_UP_PHY with port number [23:33:27] [PASSED] DP_POWER_DOWN_PHY with port number [23:33:27] [PASSED] DP_ALLOCATE_PAYLOAD with SDP stream sinks [23:33:27] [PASSED] DP_ALLOCATE_PAYLOAD with port number [23:33:27] [PASSED] DP_ALLOCATE_PAYLOAD with VCPI [23:33:27] [PASSED] DP_ALLOCATE_PAYLOAD with PBN [23:33:27] [PASSED] DP_QUERY_PAYLOAD with port number [23:33:27] [PASSED] DP_QUERY_PAYLOAD with VCPI [23:33:27] [PASSED] DP_REMOTE_DPCD_READ with port number [23:33:27] [PASSED] DP_REMOTE_DPCD_READ with DPCD address [23:33:27] [PASSED] DP_REMOTE_DPCD_READ with max number of bytes [23:33:27] [PASSED] DP_REMOTE_DPCD_WRITE with port number [23:33:27] [PASSED] DP_REMOTE_DPCD_WRITE with DPCD address [23:33:27] [PASSED] DP_REMOTE_DPCD_WRITE with data array [23:33:27] [PASSED] DP_REMOTE_I2C_READ with port number [23:33:27] [PASSED] DP_REMOTE_I2C_READ with I2C device ID [23:33:27] [PASSED] DP_REMOTE_I2C_READ with transactions array [23:33:27] [PASSED] DP_REMOTE_I2C_WRITE with port number [23:33:27] [PASSED] DP_REMOTE_I2C_WRITE with I2C device ID [23:33:27] [PASSED] DP_REMOTE_I2C_WRITE with data array [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream ID [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with client ID [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream event [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with valid stream event [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream behavior [23:33:27] [PASSED] DP_QUERY_STREAM_ENC_STATUS with a valid stream behavior [23:33:27] ===== [PASSED] drm_test_dp_mst_sideband_msg_req_decode ===== [23:33:27] ================ [PASSED] drm_dp_mst_helper ================ [23:33:27] ================== drm_exec (7 subtests) =================== [23:33:27] [PASSED] sanitycheck [23:33:27] [PASSED] test_lock [23:33:27] [PASSED] test_lock_unlock [23:33:27] [PASSED] test_duplicates [23:33:27] [PASSED] test_prepare [23:33:27] [PASSED] test_prepare_array [23:33:27] [PASSED] test_multiple_loops [23:33:27] ==================== [PASSED] drm_exec ===================== [23:33:27] =========== drm_format_helper_test (17 subtests) =========== [23:33:27] ============== drm_test_fb_xrgb8888_to_gray8 ============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========== [PASSED] drm_test_fb_xrgb8888_to_gray8 ========== [23:33:27] ============= drm_test_fb_xrgb8888_to_rgb332 ============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb332 ========== [23:33:27] ============= drm_test_fb_xrgb8888_to_rgb565 ============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb565 ========== [23:33:27] ============ drm_test_fb_xrgb8888_to_xrgb1555 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_xrgb1555 ========= [23:33:27] ============ drm_test_fb_xrgb8888_to_argb1555 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_argb1555 ========= [23:33:27] ============ drm_test_fb_xrgb8888_to_rgba5551 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_rgba5551 ========= [23:33:27] ============= drm_test_fb_xrgb8888_to_rgb888 ============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb888 ========== [23:33:27] ============= drm_test_fb_xrgb8888_to_bgr888 ============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========= [PASSED] drm_test_fb_xrgb8888_to_bgr888 ========== [23:33:27] ============ drm_test_fb_xrgb8888_to_argb8888 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_argb8888 ========= [23:33:27] =========== drm_test_fb_xrgb8888_to_xrgb2101010 =========== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======= [PASSED] drm_test_fb_xrgb8888_to_xrgb2101010 ======= [23:33:27] =========== drm_test_fb_xrgb8888_to_argb2101010 =========== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======= [PASSED] drm_test_fb_xrgb8888_to_argb2101010 ======= [23:33:27] ============== drm_test_fb_xrgb8888_to_mono =============== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ========== [PASSED] drm_test_fb_xrgb8888_to_mono =========== [23:33:27] ==================== drm_test_fb_swab ===================== [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ================ [PASSED] drm_test_fb_swab ================= [23:33:27] ============ drm_test_fb_xrgb8888_to_xbgr8888 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_xbgr8888 ========= [23:33:27] ============ drm_test_fb_xrgb8888_to_abgr8888 ============= [23:33:27] [PASSED] single_pixel_source_buffer [23:33:27] [PASSED] single_pixel_clip_rectangle [23:33:27] [PASSED] well_known_colors [23:33:27] [PASSED] destination_pitch [23:33:27] ======== [PASSED] drm_test_fb_xrgb8888_to_abgr8888 ========= [23:33:27] ================= drm_test_fb_clip_offset ================= [23:33:27] [PASSED] pass through [23:33:27] [PASSED] horizontal offset [23:33:27] [PASSED] vertical offset [23:33:27] [PASSED] horizontal and vertical offset [23:33:27] [PASSED] horizontal offset (custom pitch) [23:33:27] [PASSED] vertical offset (custom pitch) [23:33:27] [PASSED] horizontal and vertical offset (custom pitch) [23:33:27] ============= [PASSED] drm_test_fb_clip_offset ============= [23:33:27] =================== drm_test_fb_memcpy ==================== [23:33:27] [PASSED] single_pixel_source_buffer: XR24 little-endian (0x34325258) [23:33:27] [PASSED] single_pixel_source_buffer: XRA8 little-endian (0x38415258) [23:33:27] [PASSED] single_pixel_source_buffer: YU24 little-endian (0x34325559) [23:33:27] [PASSED] single_pixel_clip_rectangle: XB24 little-endian (0x34324258) [23:33:27] [PASSED] single_pixel_clip_rectangle: XRA8 little-endian (0x38415258) [23:33:27] [PASSED] single_pixel_clip_rectangle: YU24 little-endian (0x34325559) [23:33:27] [PASSED] well_known_colors: XB24 little-endian (0x34324258) [23:33:27] [PASSED] well_known_colors: XRA8 little-endian (0x38415258) [23:33:27] [PASSED] well_known_colors: YU24 little-endian (0x34325559) [23:33:27] [PASSED] destination_pitch: XB24 little-endian (0x34324258) [23:33:27] [PASSED] destination_pitch: XRA8 little-endian (0x38415258) [23:33:27] [PASSED] destination_pitch: YU24 little-endian (0x34325559) [23:33:27] =============== [PASSED] drm_test_fb_memcpy ================ [23:33:27] ============= [PASSED] drm_format_helper_test ============== [23:33:27] ================= drm_format (18 subtests) ================= [23:33:27] [PASSED] drm_test_format_block_width_invalid [23:33:27] [PASSED] drm_test_format_block_width_one_plane [23:33:27] [PASSED] drm_test_format_block_width_two_plane [23:33:27] [PASSED] drm_test_format_block_width_three_plane [23:33:27] [PASSED] drm_test_format_block_width_tiled [23:33:27] [PASSED] drm_test_format_block_height_invalid [23:33:27] [PASSED] drm_test_format_block_height_one_plane [23:33:27] [PASSED] drm_test_format_block_height_two_plane [23:33:27] [PASSED] drm_test_format_block_height_three_plane [23:33:27] [PASSED] drm_test_format_block_height_tiled [23:33:27] [PASSED] drm_test_format_min_pitch_invalid [23:33:27] [PASSED] drm_test_format_min_pitch_one_plane_8bpp [23:33:27] [PASSED] drm_test_format_min_pitch_one_plane_16bpp [23:33:27] [PASSED] drm_test_format_min_pitch_one_plane_24bpp [23:33:27] [PASSED] drm_test_format_min_pitch_one_plane_32bpp [23:33:27] [PASSED] drm_test_format_min_pitch_two_plane [23:33:27] [PASSED] drm_test_format_min_pitch_three_plane_8bpp [23:33:27] [PASSED] drm_test_format_min_pitch_tiled [23:33:27] =================== [PASSED] drm_format ==================== [23:33:27] ============== drm_framebuffer (10 subtests) =============== [23:33:27] ========== drm_test_framebuffer_check_src_coords ========== [23:33:27] [PASSED] Success: source fits into fb [23:33:27] [PASSED] Fail: overflowing fb with x-axis coordinate [23:33:27] [PASSED] Fail: overflowing fb with y-axis coordinate [23:33:27] [PASSED] Fail: overflowing fb with source width [23:33:27] [PASSED] Fail: overflowing fb with source height [23:33:27] ====== [PASSED] drm_test_framebuffer_check_src_coords ====== [23:33:27] [PASSED] drm_test_framebuffer_cleanup [23:33:27] =============== drm_test_framebuffer_create =============== [23:33:27] [PASSED] ABGR8888 normal sizes [23:33:27] [PASSED] ABGR8888 max sizes [23:33:27] [PASSED] ABGR8888 pitch greater than min required [23:33:27] [PASSED] ABGR8888 pitch less than min required [23:33:27] [PASSED] ABGR8888 Invalid width [23:33:27] [PASSED] ABGR8888 Invalid buffer handle [23:33:27] [PASSED] No pixel format [23:33:27] [PASSED] ABGR8888 Width 0 [23:33:27] [PASSED] ABGR8888 Height 0 [23:33:27] [PASSED] ABGR8888 Out of bound height * pitch combination [23:33:27] [PASSED] ABGR8888 Large buffer offset [23:33:27] [PASSED] ABGR8888 Buffer offset for inexistent plane [23:33:27] [PASSED] ABGR8888 Invalid flag [23:33:27] [PASSED] ABGR8888 Set DRM_MODE_FB_MODIFIERS without modifiers [23:33:27] [PASSED] ABGR8888 Valid buffer modifier [23:33:27] [PASSED] ABGR8888 Invalid buffer modifier(DRM_FORMAT_MOD_SAMSUNG_64_32_TILE) [23:33:27] [PASSED] ABGR8888 Extra pitches without DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] ABGR8888 Extra pitches with DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] NV12 Normal sizes [23:33:27] [PASSED] NV12 Max sizes [23:33:27] [PASSED] NV12 Invalid pitch [23:33:27] [PASSED] NV12 Invalid modifier/missing DRM_MODE_FB_MODIFIERS flag [23:33:27] [PASSED] NV12 different modifier per-plane [23:33:27] [PASSED] NV12 with DRM_FORMAT_MOD_SAMSUNG_64_32_TILE [23:33:27] [PASSED] NV12 Valid modifiers without DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] NV12 Modifier for inexistent plane [23:33:27] [PASSED] NV12 Handle for inexistent plane [23:33:27] [PASSED] NV12 Handle for inexistent plane without DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] YVU420 DRM_MODE_FB_MODIFIERS set without modifier [23:33:27] [PASSED] YVU420 Normal sizes [23:33:27] [PASSED] YVU420 Max sizes [23:33:27] [PASSED] YVU420 Invalid pitch [23:33:27] [PASSED] YVU420 Different pitches [23:33:27] [PASSED] YVU420 Different buffer offsets/pitches [23:33:27] [PASSED] YVU420 Modifier set just for plane 0, without DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] YVU420 Modifier set just for planes 0, 1, without DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] YVU420 Modifier set just for plane 0, 1, with DRM_MODE_FB_MODIFIERS [23:33:27] [PASSED] YVU420 Valid modifier [23:33:27] [PASSED] YVU420 Different modifiers per plane [23:33:27] [PASSED] YVU420 Modifier for inexistent plane [23:33:27] [PASSED] YUV420_10BIT Invalid modifier(DRM_FORMAT_MOD_LINEAR) [23:33:27] [PASSED] X0L2 Normal sizes [23:33:27] [PASSED] X0L2 Max sizes [23:33:27] [PASSED] X0L2 Invalid pitch [23:33:27] [PASSED] X0L2 Pitch greater than minimum required [23:33:27] [PASSED] X0L2 Handle for inexistent plane [23:33:27] [PASSED] X0L2 Offset for inexistent plane, without DRM_MODE_FB_MODIFIERS set [23:33:27] [PASSED] X0L2 Modifier without DRM_MODE_FB_MODIFIERS set [23:33:27] [PASSED] X0L2 Valid modifier [23:33:27] [PASSED] X0L2 Modifier for inexistent plane [23:33:27] =========== [PASSED] drm_test_framebuffer_create =========== [23:33:27] [PASSED] drm_test_framebuffer_free [23:33:27] [PASSED] drm_test_framebuffer_init [23:33:27] [PASSED] drm_test_framebuffer_init_bad_format [23:33:27] [PASSED] drm_test_framebuffer_init_dev_mismatch [23:33:27] [PASSED] drm_test_framebuffer_lookup [23:33:27] [PASSED] drm_test_framebuffer_lookup_inexistent [23:33:27] [PASSED] drm_test_framebuffer_modifiers_not_supported [23:33:27] ================= [PASSED] drm_framebuffer ================= [23:33:27] ================ drm_gem_shmem (8 subtests) ================ [23:33:27] [PASSED] drm_gem_shmem_test_obj_create [23:33:27] [PASSED] drm_gem_shmem_test_obj_create_private [23:33:27] [PASSED] drm_gem_shmem_test_pin_pages [23:33:27] [PASSED] drm_gem_shmem_test_vmap [23:33:27] [PASSED] drm_gem_shmem_test_get_sg_table [23:33:27] [PASSED] drm_gem_shmem_test_get_pages_sgt [23:33:27] [PASSED] drm_gem_shmem_test_madvise [23:33:27] [PASSED] drm_gem_shmem_test_purge [23:33:27] ================== [PASSED] drm_gem_shmem ================== [23:33:27] === drm_atomic_helper_connector_hdmi_check (29 subtests) === [23:33:27] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode [23:33:27] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode_vic_1 [23:33:27] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode [23:33:27] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode_vic_1 [23:33:27] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode [23:33:27] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode_vic_1 [23:33:27] ====== drm_test_check_broadcast_rgb_cea_mode_yuv420 ======= [23:33:27] [PASSED] Automatic [23:33:27] [PASSED] Full [23:33:27] [PASSED] Limited 16:235 [23:33:27] == [PASSED] drm_test_check_broadcast_rgb_cea_mode_yuv420 === [23:33:27] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_changed [23:33:27] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_not_changed [23:33:27] [PASSED] drm_test_check_disable_connector [23:33:27] [PASSED] drm_test_check_hdmi_funcs_reject_rate [23:33:27] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_rgb [23:33:27] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_yuv420 [23:33:27] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_ignore_yuv422 [23:33:27] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_ignore_yuv420 [23:33:27] [PASSED] drm_test_check_driver_unsupported_fallback_yuv420 [23:33:27] [PASSED] drm_test_check_output_bpc_crtc_mode_changed [23:33:27] [PASSED] drm_test_check_output_bpc_crtc_mode_not_changed [23:33:27] [PASSED] drm_test_check_output_bpc_dvi [23:33:27] [PASSED] drm_test_check_output_bpc_format_vic_1 [23:33:27] [PASSED] drm_test_check_output_bpc_format_display_8bpc_only [23:33:27] [PASSED] drm_test_check_output_bpc_format_display_rgb_only [23:33:27] [PASSED] drm_test_check_output_bpc_format_driver_8bpc_only [23:33:27] [PASSED] drm_test_check_output_bpc_format_driver_rgb_only [23:33:27] [PASSED] drm_test_check_tmds_char_rate_rgb_8bpc [23:33:27] [PASSED] drm_test_check_tmds_char_rate_rgb_10bpc [23:33:27] [PASSED] drm_test_check_tmds_char_rate_rgb_12bpc [23:33:27] ============ drm_test_check_hdmi_color_format ============= [23:33:27] [PASSED] AUTO -> RGB [23:33:27] [PASSED] YCBCR422 -> YUV422 [23:33:27] [PASSED] YCBCR420 -> YUV420 [23:33:27] [PASSED] YCBCR444 -> YUV444 [23:33:27] [PASSED] RGB -> RGB [23:33:27] ======== [PASSED] drm_test_check_hdmi_color_format ========= [23:33:27] ======== drm_test_check_hdmi_color_format_420_only ======== [23:33:27] [PASSED] RGB should fail [23:33:27] [PASSED] YUV444 should fail [23:33:27] [PASSED] YUV422 should fail [23:33:27] [PASSED] YUV420 should work [23:33:27] ==== [PASSED] drm_test_check_hdmi_color_format_420_only ==== [23:33:27] ===== [PASSED] drm_atomic_helper_connector_hdmi_check ====== [23:33:27] === drm_atomic_helper_connector_hdmi_reset (6 subtests) ==== [23:33:27] [PASSED] drm_test_check_broadcast_rgb_value [23:33:27] [PASSED] drm_test_check_bpc_8_value [23:33:27] [PASSED] drm_test_check_bpc_10_value [23:33:27] [PASSED] drm_test_check_bpc_12_value [23:33:27] [PASSED] drm_test_check_format_value [23:33:27] [PASSED] drm_test_check_tmds_char_value [23:33:27] ===== [PASSED] drm_atomic_helper_connector_hdmi_reset ====== [23:33:27] = drm_atomic_helper_connector_hdmi_mode_valid (7 subtests) = [23:33:27] [PASSED] drm_test_check_mode_valid [23:33:27] [PASSED] drm_test_check_mode_valid_reject [23:33:27] [PASSED] drm_test_check_mode_valid_reject_rate [23:33:27] [PASSED] drm_test_check_mode_valid_reject_max_clock [23:33:27] [PASSED] drm_test_check_mode_valid_yuv420_only_max_clock [23:33:27] [PASSED] drm_test_check_mode_valid_reject_yuv420_only_connector [23:33:27] [PASSED] drm_test_check_mode_valid_accept_yuv420_also_connector_rgb [23:33:27] === [PASSED] drm_atomic_helper_connector_hdmi_mode_valid === [23:33:27] = drm_atomic_helper_connector_hdmi_infoframes (5 subtests) = [23:33:27] [PASSED] drm_test_check_infoframes [23:33:27] [PASSED] drm_test_check_reject_avi_infoframe [23:33:27] [PASSED] drm_test_check_reject_hdr_infoframe_bpc_8 [23:33:27] [PASSED] drm_test_check_reject_hdr_infoframe_bpc_10 [23:33:27] [PASSED] drm_test_check_reject_audio_infoframe [23:33:27] === [PASSED] drm_atomic_helper_connector_hdmi_infoframes === [23:33:27] ================= drm_managed (2 subtests) ================= [23:33:27] [PASSED] drm_test_managed_release_action [23:33:27] [PASSED] drm_test_managed_run_action [23:33:27] =================== [PASSED] drm_managed =================== [23:33:27] =================== drm_mm (6 subtests) ==================== [23:33:27] [PASSED] drm_test_mm_init [23:33:27] [PASSED] drm_test_mm_debug [23:33:27] [PASSED] drm_test_mm_align32 [23:33:27] [PASSED] drm_test_mm_align64 [23:33:27] [PASSED] drm_test_mm_lowest [23:33:27] [PASSED] drm_test_mm_highest [23:33:27] ===================== [PASSED] drm_mm ====================== [23:33:27] ============= drm_modes_analog_tv (5 subtests) ============= [23:33:27] [PASSED] drm_test_modes_analog_tv_mono_576i [23:33:27] [PASSED] drm_test_modes_analog_tv_ntsc_480i [23:33:27] [PASSED] drm_test_modes_analog_tv_ntsc_480i_inlined [23:33:27] [PASSED] drm_test_modes_analog_tv_pal_576i [23:33:27] [PASSED] drm_test_modes_analog_tv_pal_576i_inlined [23:33:27] =============== [PASSED] drm_modes_analog_tv =============== [23:33:27] ============== drm_plane_helper (2 subtests) =============== [23:33:27] =============== drm_test_check_plane_state ================ [23:33:27] [PASSED] clipping_simple [23:33:27] [PASSED] clipping_rotate_reflect [23:33:27] [PASSED] positioning_simple [23:33:27] [PASSED] upscaling [23:33:27] [PASSED] downscaling [23:33:27] [PASSED] rounding1 [23:33:27] [PASSED] rounding2 [23:33:27] [PASSED] rounding3 [23:33:27] [PASSED] rounding4 [23:33:27] =========== [PASSED] drm_test_check_plane_state ============ [23:33:27] =========== drm_test_check_invalid_plane_state ============ [23:33:27] [PASSED] positioning_invalid [23:33:27] [PASSED] upscaling_invalid [23:33:27] [PASSED] downscaling_invalid [23:33:27] ======= [PASSED] drm_test_check_invalid_plane_state ======== [23:33:27] ================ [PASSED] drm_plane_helper ================= [23:33:27] ====== drm_connector_helper_tv_get_modes (1 subtest) ======= [23:33:27] ====== drm_test_connector_helper_tv_get_modes_check ======= [23:33:27] [PASSED] None [23:33:27] [PASSED] PAL [23:33:27] [PASSED] NTSC [23:33:27] [PASSED] Both, NTSC Default [23:33:27] [PASSED] Both, PAL Default [23:33:27] [PASSED] Both, NTSC Default, with PAL on command-line [23:33:27] [PASSED] Both, PAL Default, with NTSC on command-line [23:33:27] == [PASSED] drm_test_connector_helper_tv_get_modes_check === [23:33:27] ======== [PASSED] drm_connector_helper_tv_get_modes ======== [23:33:27] ================== drm_rect (9 subtests) =================== [23:33:27] [PASSED] drm_test_rect_clip_scaled_div_by_zero [23:33:27] [PASSED] drm_test_rect_clip_scaled_not_clipped [23:33:27] [PASSED] drm_test_rect_clip_scaled_clipped [23:33:27] [PASSED] drm_test_rect_clip_scaled_signed_vs_unsigned [23:33:27] ================= drm_test_rect_intersect ================= [23:33:27] [PASSED] top-left x bottom-right: 2x2+1+1 x 2x2+0+0 [23:33:27] [PASSED] top-right x bottom-left: 2x2+0+0 x 2x2+1-1 [23:33:27] [PASSED] bottom-left x top-right: 2x2+1-1 x 2x2+0+0 [23:33:27] [PASSED] bottom-right x top-left: 2x2+0+0 x 2x2+1+1 [23:33:27] [PASSED] right x left: 2x1+0+0 x 3x1+1+0 [23:33:27] [PASSED] left x right: 3x1+1+0 x 2x1+0+0 [23:33:27] [PASSED] up x bottom: 1x2+0+0 x 1x3+0-1 [23:33:27] [PASSED] bottom x up: 1x3+0-1 x 1x2+0+0 [23:33:27] [PASSED] touching corner: 1x1+0+0 x 2x2+1+1 [23:33:27] [PASSED] touching side: 1x1+0+0 x 1x1+1+0 [23:33:27] [PASSED] equal rects: 2x2+0+0 x 2x2+0+0 [23:33:27] [PASSED] inside another: 2x2+0+0 x 1x1+1+1 [23:33:27] [PASSED] far away: 1x1+0+0 x 1x1+3+6 [23:33:27] [PASSED] points intersecting: 0x0+5+10 x 0x0+5+10 [23:33:27] [PASSED] points not intersecting: 0x0+0+0 x 0x0+5+10 [23:33:27] ============= [PASSED] drm_test_rect_intersect ============= [23:33:27] ================ drm_test_rect_calc_hscale ================ [23:33:27] [PASSED] normal use [23:33:27] [PASSED] out of max range [23:33:27] [PASSED] out of min range [23:33:27] [PASSED] zero dst [23:33:27] [PASSED] negative src [23:33:27] [PASSED] negative dst [23:33:27] ============ [PASSED] drm_test_rect_calc_hscale ============ [23:33:27] ================ drm_test_rect_calc_vscale ================ [23:33:27] [PASSED] normal use [23:33:27] [PASSED] out of max range [23:33:27] [PASSED] out of min range [23:33:27] [PASSED] zero dst [23:33:27] [PASSED] negative src [23:33:27] [PASSED] negative dst [23:33:27] ============ [PASSED] drm_test_rect_calc_vscale ============ [23:33:27] ================== drm_test_rect_rotate =================== [23:33:27] [PASSED] reflect-x [23:33:27] [PASSED] reflect-y [23:33:27] [PASSED] rotate-0 [23:33:27] [PASSED] rotate-90 [23:33:27] [PASSED] rotate-180 [23:33:27] [PASSED] rotate-270 [23:33:27] ============== [PASSED] drm_test_rect_rotate =============== [23:33:27] ================ drm_test_rect_rotate_inv ================= [23:33:27] [PASSED] reflect-x [23:33:27] [PASSED] reflect-y [23:33:27] [PASSED] rotate-0 [23:33:27] [PASSED] rotate-90 [23:33:27] [PASSED] rotate-180 [23:33:27] [PASSED] rotate-270 [23:33:27] ============ [PASSED] drm_test_rect_rotate_inv ============= [23:33:27] ==================== [PASSED] drm_rect ===================== [23:33:27] ============ drm_sysfb_modeset_test (1 subtest) ============ [23:33:27] ============ drm_test_sysfb_build_fourcc_list ============= [23:33:27] [PASSED] no native formats [23:33:27] [PASSED] XRGB8888 as native format [23:33:27] [PASSED] remove duplicates [23:33:27] [PASSED] convert alpha formats [23:33:27] [PASSED] random formats [23:33:27] ======== [PASSED] drm_test_sysfb_build_fourcc_list ========= [23:33:27] ============= [PASSED] drm_sysfb_modeset_test ============== [23:33:27] ================== drm_fixp (2 subtests) =================== [23:33:27] [PASSED] drm_test_int2fixp [23:33:27] [PASSED] drm_test_sm2fixp [23:33:27] ==================== [PASSED] drm_fixp ===================== [23:33:27] ============================================================ [23:33:27] Testing complete. Ran 637 tests: passed: 637 [23:33:27] Elapsed time: 27.173s total, 1.738s configuring, 25.221s building, 0.193s running
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/ttm/tests/.kunitconfig [23:33:27] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [23:33:29] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 [23:33:39] Starting KUnit Kernel (1/1)... [23:33:39] ============================================================ Running tests with: $ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt [23:33:39] ================= ttm_device (5 subtests) ================== [23:33:39] [PASSED] ttm_device_init_basic [23:33:39] [PASSED] ttm_device_init_multiple [23:33:39] [PASSED] ttm_device_fini_basic [23:33:39] [PASSED] ttm_device_init_no_vma_man [23:33:39] ================== ttm_device_init_pools ================== [23:33:39] [PASSED] No DMA allocations, no DMA32 required [23:33:39] [PASSED] DMA allocations, DMA32 required [23:33:39] [PASSED] No DMA allocations, DMA32 required [23:33:39] [PASSED] DMA allocations, no DMA32 required [23:33:39] ============== [PASSED] ttm_device_init_pools ============== [23:33:39] =================== [PASSED] ttm_device ==================== [23:33:39] ================== ttm_pool (8 subtests) =================== [23:33:39] ================== ttm_pool_alloc_basic =================== [23:33:39] [PASSED] One page [23:33:39] [PASSED] More than one page [23:33:39] [PASSED] Above the allocation limit [23:33:39] [PASSED] One page, with coherent DMA mappings enabled [23:33:39] [PASSED] Above the allocation limit, with coherent DMA mappings enabled [23:33:39] ============== [PASSED] ttm_pool_alloc_basic =============== [23:33:39] ============== ttm_pool_alloc_basic_dma_addr ============== [23:33:39] [PASSED] One page [23:33:39] [PASSED] More than one page [23:33:39] [PASSED] Above the allocation limit [23:33:39] [PASSED] One page, with coherent DMA mappings enabled [23:33:39] [PASSED] Above the allocation limit, with coherent DMA mappings enabled [23:33:39] ========== [PASSED] ttm_pool_alloc_basic_dma_addr ========== [23:33:39] [PASSED] ttm_pool_alloc_order_caching_match [23:33:39] [PASSED] ttm_pool_alloc_caching_mismatch [23:33:39] [PASSED] ttm_pool_alloc_order_mismatch [23:33:39] [PASSED] ttm_pool_free_dma_alloc [23:33:39] [PASSED] ttm_pool_free_no_dma_alloc [23:33:39] [PASSED] ttm_pool_fini_basic [23:33:39] ==================== [PASSED] ttm_pool ===================== [23:33:39] ================ ttm_resource (8 subtests) ================= [23:33:39] ================= ttm_resource_init_basic ================= [23:33:39] [PASSED] Init resource in TTM_PL_SYSTEM [23:33:39] [PASSED] Init resource in TTM_PL_VRAM [23:33:39] [PASSED] Init resource in a private placement [23:33:39] [PASSED] Init resource in TTM_PL_SYSTEM, set placement flags [23:33:39] ============= [PASSED] ttm_resource_init_basic ============= [23:33:39] [PASSED] ttm_resource_init_pinned [23:33:39] [PASSED] ttm_resource_fini_basic [23:33:39] [PASSED] ttm_resource_manager_init_basic [23:33:39] [PASSED] ttm_resource_manager_usage_basic [23:33:39] [PASSED] ttm_resource_manager_set_used_basic [23:33:39] [PASSED] ttm_sys_man_alloc_basic [23:33:39] [PASSED] ttm_sys_man_free_basic [23:33:39] ================== [PASSED] ttm_resource =================== [23:33:39] =================== ttm_tt (15 subtests) =================== [23:33:39] ==================== ttm_tt_init_basic ==================== [23:33:39] [PASSED] Page-aligned size [23:33:39] [PASSED] Extra pages requested [23:33:39] ================ [PASSED] ttm_tt_init_basic ================ [23:33:39] [PASSED] ttm_tt_init_misaligned [23:33:39] [PASSED] ttm_tt_fini_basic [23:33:39] [PASSED] ttm_tt_fini_sg [23:33:39] [PASSED] ttm_tt_fini_shmem [23:33:39] [PASSED] ttm_tt_create_basic [23:33:39] [PASSED] ttm_tt_create_invalid_bo_type [23:33:39] [PASSED] ttm_tt_create_ttm_exists [23:33:39] [PASSED] ttm_tt_create_failed [23:33:39] [PASSED] ttm_tt_destroy_basic [23:33:39] [PASSED] ttm_tt_populate_null_ttm [23:33:39] [PASSED] ttm_tt_populate_populated_ttm [23:33:39] [PASSED] ttm_tt_unpopulate_basic [23:33:39] [PASSED] ttm_tt_unpopulate_empty_ttm [23:33:39] [PASSED] ttm_tt_swapin_basic [23:33:39] ===================== [PASSED] ttm_tt ====================== [23:33:39] =================== ttm_bo (14 subtests) =================== [23:33:39] =========== ttm_bo_reserve_optimistic_no_ticket =========== [23:33:39] [PASSED] Cannot be interrupted and sleeps [23:33:39] [PASSED] Cannot be interrupted, locks straight away [23:33:39] [PASSED] Can be interrupted, sleeps [23:33:39] ======= [PASSED] ttm_bo_reserve_optimistic_no_ticket ======= [23:33:39] [PASSED] ttm_bo_reserve_locked_no_sleep [23:33:39] [PASSED] ttm_bo_reserve_no_wait_ticket [23:33:39] [PASSED] ttm_bo_reserve_double_resv [23:33:39] [PASSED] ttm_bo_reserve_interrupted [23:33:39] [PASSED] ttm_bo_reserve_deadlock [23:33:39] [PASSED] ttm_bo_unreserve_basic [23:33:39] [PASSED] ttm_bo_unreserve_pinned [23:33:39] [PASSED] ttm_bo_unreserve_bulk [23:33:39] [PASSED] ttm_bo_fini_basic [23:33:39] [PASSED] ttm_bo_fini_shared_resv [23:33:39] [PASSED] ttm_bo_pin_basic [23:33:39] [PASSED] ttm_bo_pin_unpin_resource [23:33:39] [PASSED] ttm_bo_multiple_pin_one_unpin [23:33:39] ===================== [PASSED] ttm_bo ====================== [23:33:39] ============== ttm_bo_validate (22 subtests) =============== [23:33:39] ============== ttm_bo_init_reserved_sys_man =============== [23:33:39] [PASSED] Buffer object for userspace [23:33:39] [PASSED] Kernel buffer object [23:33:39] [PASSED] Shared buffer object [23:33:39] ========== [PASSED] ttm_bo_init_reserved_sys_man =========== [23:33:39] ============== ttm_bo_init_reserved_mock_man ============== [23:33:39] [PASSED] Buffer object for userspace [23:33:39] [PASSED] Kernel buffer object [23:33:39] [PASSED] Shared buffer object [23:33:39] ========== [PASSED] ttm_bo_init_reserved_mock_man ========== [23:33:39] [PASSED] ttm_bo_init_reserved_resv [23:33:39] ================== ttm_bo_validate_basic ================== [23:33:39] [PASSED] Buffer object for userspace [23:33:39] [PASSED] Kernel buffer object [23:33:39] [PASSED] Shared buffer object [23:33:39] ============== [PASSED] ttm_bo_validate_basic ============== [23:33:39] [PASSED] ttm_bo_validate_invalid_placement [23:33:39] ============= ttm_bo_validate_same_placement ============== [23:33:39] [PASSED] System manager [23:33:39] [PASSED] VRAM manager [23:33:39] ========= [PASSED] ttm_bo_validate_same_placement ========== [23:33:39] [PASSED] ttm_bo_validate_failed_alloc [23:33:39] [PASSED] ttm_bo_validate_pinned [23:33:39] [PASSED] ttm_bo_validate_busy_placement [23:33:39] ================ ttm_bo_validate_multihop ================= [23:33:39] [PASSED] Buffer object for userspace [23:33:39] [PASSED] Kernel buffer object [23:33:39] [PASSED] Shared buffer object [23:33:39] ============ [PASSED] ttm_bo_validate_multihop ============= [23:33:39] ========== ttm_bo_validate_no_placement_signaled ========== [23:33:39] [PASSED] Buffer object in system domain, no page vector [23:33:39] [PASSED] Buffer object in system domain with an existing page vector [23:33:39] ====== [PASSED] ttm_bo_validate_no_placement_signaled ====== [23:33:39] ======== ttm_bo_validate_no_placement_not_signaled ======== [23:33:39] [PASSED] Buffer object for userspace [23:33:39] [PASSED] Kernel buffer object [23:33:39] [PASSED] Shared buffer object [23:33:39] ==== [PASSED] ttm_bo_validate_no_placement_not_signaled ==== [23:33:39] [PASSED] ttm_bo_validate_move_fence_signaled [23:33:39] ========= ttm_bo_validate_move_fence_not_signaled ========= [23:33:39] [PASSED] Waits for GPU [23:33:39] [PASSED] Tries to lock straight away [23:33:39] ===== [PASSED] ttm_bo_validate_move_fence_not_signaled ===== [23:33:39] [PASSED] ttm_bo_validate_swapout [23:33:39] [PASSED] ttm_bo_validate_happy_evict [23:33:39] [PASSED] ttm_bo_validate_all_pinned_evict [23:33:39] [PASSED] ttm_bo_validate_allowed_only_evict [23:33:39] [PASSED] ttm_bo_validate_deleted_evict [23:33:39] [PASSED] ttm_bo_validate_busy_domain_evict [23:33:39] [PASSED] ttm_bo_validate_evict_gutting [23:33:39] [PASSED] ttm_bo_validate_recrusive_evict [23:33:39] ================= [PASSED] ttm_bo_validate ================= [23:33:39] ============================================================ [23:33:39] Testing complete. Ran 102 tests: passed: 102 [23:33:39] Elapsed time: 11.800s total, 1.725s configuring, 9.860s building, 0.185s running
+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel
Patchwork
CI✓ Xe.CI.BAT: success for Fine grained fault locking, threaded prefetch, storm cache (rev8)
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Fine grained fault locking, threaded prefetch, storm cache (rev8)
URL : https://patchwork.freedesktop.org/series/162167/
State : success
== Summary ==
CI Bug Log - changes from xe-5481-03696c1dc9eabc0658baefcff8787dab48c89d93_BAT -> xe-pw-162167v8_BAT
====================================================
Summary
-------
**SUCCESS**
No regressions found.
Participating hosts (13 -> 13)
------------------------------
No changes in participating hosts
Changes
-------
No changes found
Build changes
-------------
* Linux: xe-5481-03696c1dc9eabc0658baefcff8787dab48c89d93 -> xe-pw-162167v8
IGT_9025: 852c13ddd448fd668c43584ca3ad1aec49c4633c @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
xe-5481-03696c1dc9eabc0658baefcff8787dab48c89d93: 03696c1dc9eabc0658baefcff8787dab48c89d93
xe-pw-162167v8: 162167v8
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-162167v8/index.html