drm/amdgpu: unify SVM notifier and VM eviction locking
Subsystems (3)
cover drm/amdgpu: unify SVM notifier and VM eviction locking
From: Honglei Huang <[email protected]>
This series follows Christian's suggestions and builds on his earlier fix work in the amdgpu VM critical section.
My understanding of all amdgpu VM locking details may be incomplete, so review and confirmation from the VM maintainers would be appreciated.
It is described that the ordering problem and why both paths need the same lock:
"Originally the notifier_lock only made sure that the CPU page table
updates were done in order and originally the eviction lock made sure that the GPU page table updates were done in order, but essentially we need the order for both.
The point is that the updates need to be serialized. In other words when one CPU is doing a mapping operation and another CPU is doing an unmap through an MMU notifier we somehow need to make sure that the corresponding GPU page table updates execute in the correct order."
Before this series, the update flow uses two locks:
map thread:
notifier_lock (lock A)
validate range
pt_alloc drops only eviction_lock (lock B)
allocation enters direct reclaim
MMU notifier tries notifier_lock (lock A again)
deadlock here
The allocation cannot finish because reclaim waits for a read lock that the same thread releases only after the allocation returns. The VM helper also cannot drop and revalidate the notifier lock because it only knows about the separate eviction lock.
After this series, all three paths use the same rwsem:
map thread:
unified notifier/eviction lock for read (lock A)
validate range
pt_alloc drops lock A
allocation can enter reclaim
MMU notifier can take lock A for write and finish
reacquire lock A for read
revalidate range
range valid -> update PTEs and PDEs -> publish the mapping range changed -> return -EAGAIN and retry
MMU notifier / eviction:
unified notifier/eviction lock for write (lock A)
clear PTEs or evict page tables
The allocation no longer holds lock A during reclaim. If an MMU notifier invalidates the range while lock A is dropped, the map thread detects the changed range after reacquiring lock A and retries instead of installing stale PTEs.
The three patches make the VM eviction lock read/write, support a caller held lock with post allocation range revalidation, and use the drm_gpusvm notifier lock for SVM VMs. Non SVM callers pass NULL and keep the existing internal locking path.
The patches apply after the AMDGPU SVM build and VM fault-path integration patch.
Testing:
MI60: KFD svm test passed, 2 known attribute failures (attribute get refactor). HIP catch_tests: 98% passed.
Honglei Huang (3):
drm/amdgpu: make VM eviction lock read/write to match notifier lock drm/amdgpu: recheck range validity after page table allocation drm/amdgpu: use drm_gpusvm notifier lock for VM eviction
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 37 ++++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 52 +++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 84 ++++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 15 ++-- .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 30 ++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 29 ++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 7 +- 11 files changed, 176 insertions(+), 86 deletions(-)
Signature · 1 line
2.34.1
RFC PATCH 1/3 drm/amdgpu: make VM eviction lock read/write to match notifier lock
From: Honglei Huang <[email protected]>
Make the VM eviction lock read/write to match the drm_gpusvm notifier lock. This allows VM updates, MMU notifier invalidation and eviction to share the same rwsem.
Replace the VM eviction mutex with a selectable rwsem. Updates made with the root BO reserved take the read side. Updates without the root BO reservation take the write side, as does eviction. A VM using drm_gpusvm can use the notifier lock as its eviction lock.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 9 +++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 8 ++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 12 ++++++++++-- 3 files changed, 19 insertions(+), 10 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +5 -4
@@ -617,7 +617,7 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm,* As soon as all page tables are in place we can start updating them * again. */ -scoped_guard (mutex, &vm->eviction_lock) +scoped_guard (rwsem_write, vm->eviction_lock) vm->evicting = false; list_for_each_entry_safe(bo_base, tmp, &vm->always_valid.evicted,
@@ -678,7 +678,7 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm)amdgpu_vm_assert_locked (vm); -scoped_guard (mutex, &vm->eviction_lock) +scoped_guard (rwsem_read, vm->eviction_lock) ret = !vm->evicting; ret &=list_empty (&vm->kernel.evicted);
@@ -2351,7 +2351,7 @@ bool amdgpu_vm_evictable(struct amdgpu_bo *bo)/* Try to block ongoing updates */ vm = bo_base->vm; - scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) { + scoped_cond_guard(rwsem_write_try, return false, vm->eviction_lock) {/* Don't evict VM page tables while they are updated */ if (!dma_fence_is_signaled (vm->last_unlocked))
@@ -2707,7 +2707,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, vm->last_tlb_flush =dma_fence_get_stub (); vm->generation =amdgpu_vm_generation (adev,NULL ); -mutex_init (&vm->eviction_lock); +init_rwsem (&vm->default_eviction_lock); + vm->eviction_lock = &vm->default_eviction_lock; vm->evicting = false; vm->tlb_fence_context =dma_fence_context_alloc (1 );
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +4 -4
@@ -27,6 +27,7 @@#include <linux/idr.h> #include <linux/kfifo.h> #include <linux/rbtree.h> +#include <linux/rwsem.h> #include <drm/gpu_scheduler.h> #include <drm/drm_file.h> #include <drm/ttm/ttm_bo.h>
@@ -283,10 +284,9 @@ struct amdgpu_vm {/* tree of virtual addresses mapped */ struct rb_root_cached va; -/* Lock to prevent eviction while we are updating page tables -* use vm_eviction_lock/unlock(vm) -*/ -struct mutex eviction_lock; +/* SVM can replace this pointer with its notifier lock. */ +struct rw_semaphore default_eviction_lock; +struct rw_semaphore *eviction_lock;bool evicting;/* Memory statistics for this vm, protected by stats_lock */
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h +10 -2
@@ -157,7 +157,12 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);*/ static inline int amdgpu_vm_begin_critical (struct amdgpu_vm_update_params *p) { -mutex_lock (&p->vm->eviction_lock); +/* Use the write lock to serialize updates when the root BO is unlocked. */ +if (p->unlocked) +down_write (p->vm->eviction_lock); +else +down_read (p->vm->eviction_lock); + p->saved_flags =memalloc_noreclaim_save ();if (p->vm->evicting)return -EBUSY ;
@@ -175,7 +180,10 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)static inline void amdgpu_vm_end_critical (struct amdgpu_vm_update_params *p) {memalloc_noreclaim_restore (p->saved_flags); -mutex_unlock (&p->vm->eviction_lock); +if (p->unlocked) +up_write (p->vm->eviction_lock); +else +up_read (p->vm->eviction_lock); }#endif
Signature · 1 line
2.34.1
RFC PATCH 3/3 drm/amdgpu: use drm_gpusvm notifier lock for VM eviction
In reply to Honglei HuangFrom: Honglei Huang <[email protected]>
SVM mapping updates, MMU notifier unmaps and VM eviction need to use the same lock. Separate locks cannot guarantee the order of GPU page table updates when CPU mappings are invalidated concurrently.
Point vm->eviction_lock at the drm_gpusvm notifier rwsem for SVM VMs. Hold the read side across range validation, PTE and PDE updates, and mapping publication. Use the write side for MMU notifier invalidation, attribute PTE zaps and VM eviction.
Publish the selected lock before making vm->svm visible. During teardown, reserve the root BO and remove the MMU notifiers while the alias is still valid, then restore the default lock before releasing the SVM context.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 37 +++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 53 ++++++++----------- 2 files changed, 50 insertions(+), 40 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c +28 -9
@@ -157,10 +157,13 @@ amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev, uint32_t pasid)amdgpu_pasid_lock (&irqflags); fpriv =amdgpu_pasid_get_fpriv_locked (pasid); vm = fpriv ? &fpriv->vm :NULL ; -if (vm && vm->svm) { - svm = vm->svm; +/* +* eviction_lock changes when SVM is enabled, so acquire vm->svm +* after the new lock alias is published. +*/ + svm = vm ?smp_load_acquire (&vm->svm) :NULL ; +if (svm)kref_get (&svm->refcount); - }amdgpu_pasid_unlock (irqflags);return svm;
@@ -488,6 +491,8 @@ static int amdgpu_svm_init_compute(struct amdgpu_device *adev,struct amdgpu_svm *svm;int ret; +dma_resv_assert_held (vm->root.bo->tbo.base.resv); +if (vm->svm)return 0 ;
@@ -522,7 +527,9 @@ static int amdgpu_svm_init_compute(struct amdgpu_device *adev,1UL << (svm->default_granularity +PAGE_SHIFT ), svm->xnack_enabled ? "enabled" : "disabled"); - vm->svm = svm; + vm->eviction_lock = &svm->gpusvm.notifier_lock; +/* Publish the context after its notifier lock alias. */ +smp_store_release (&vm->svm, svm);return 0 ; err_free:
@@ -598,31 +605,43 @@ void amdgpu_svm_close(struct amdgpu_vm *vm)* amdgpu_svm_fini() - Finalize and release a VM's SVM context * @vm: The VM whose SVM context is being torn down. * -* Close the context, tear down the embedded drm_gpusvm under the SVM lock, -* destroy the attribute tree and work queues, and drop the context -* reference. Safe to call on a VM without an SVM context. +* Tear down the embedded drm_gpusvm, attribute tree and work queues, and drop +* the context reference. */ void amdgpu_svm_fini (struct amdgpu_vm *vm) {struct amdgpu_svm *svm = vm->svm; +struct amdgpu_bo *root;if (!svm)return ;amdgpu_svm_close (vm);amdgpu_svm_lock (svm); + + root =amdgpu_bo_ref (vm->root.bo); +amdgpu_bo_reserve (root, true); +drm_gpusvm_fini (&svm->gpusvm); + + vm->eviction_lock = &vm->default_eviction_lock; +/* Publish the lock reset before making SVM unavailable. */ +smp_store_release (&vm->svm,NULL ); +amdgpu_svm_unlock (svm);amdgpu_svm_attr_tree_destroy (svm->attr_tree);amdgpu_svm_work_fini (svm); - vm->svm =NULL ;amdgpu_svm_put (svm); + +amdgpu_bo_unreserve (root); +amdgpu_bo_unref (&root); }bool amdgpu_svm_is_enabled (struct amdgpu_vm *vm) { -return vm->svm !=NULL ; +/* Pairs with publishing vm->svm after its lock alias is initialized. */ +return smp_load_acquire (&vm->svm) !=NULL ; } static int amdgpu_svm_copy_attrs(const struct drm_amdgpu_gem_svm *args,
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c +22 -31
105 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH 2/3 drm/amdgpu: recheck range validity after page table allocation
In reply to Honglei HuangFrom: Honglei Huang <[email protected]>
The notifier and eviction locks need to be the same lock. Page table allocation can enter memory reclaim, and reclaim can invoke an MMU notifier that takes the same lock. Release the lock before allocating page tables to avoid a deadlock.
Dropping the lock allows an unmap operation to invalidate the range concurrently. Recheck the HMM or GPU SVM range after reacquiring the lock and return -EAGAIN if its sequence number has changed.
Allow callers that already hold the eviction lock to pass it to the VM update helpers. Drop the lock around amdgpu_vm_pt_create() and reacquire it before continuing. Callers that pass NULL retain the existing internal locking path.
--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 5 +- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 75 +++++++++++++------ drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 9 ++- .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 28 +++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 29 ++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 7 +- 10 files changed, 116 insertions(+), 45 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1 -1
@@ -503,7 +503,7 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync)struct amdgpu_device *adev =amdgpu_ttm_adev (pd->tbo.bdev);int ret; - ret =amdgpu_vm_update_pdes (adev, vm, false); + ret =amdgpu_vm_update_pdes (adev, vm, false,NULL );if (ret)return ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
@@ -1184,7 +1184,7 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)if (r)return r; - r =amdgpu_vm_update_pdes (adev, vm, false); + r =amdgpu_vm_update_pdes (adev, vm, false,NULL );if (r)return r;
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1 -1
@@ -795,7 +795,7 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, }/* Always update PDEs after we touched the mappings. */ - r =amdgpu_vm_update_pdes (adev, vm, false); + r =amdgpu_vm_update_pdes (adev, vm, false,NULL );if (r) goto error;
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c +3 -2
@@ -73,7 +73,7 @@ amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm, flags =memalloc_noreclaim_save (); ret =amdgpu_vm_unmap_range (svm->adev, svm->vm,NULL , - start_page, last_page,0 , &fence); + start_page, last_page,0 ,NULL , &fence);memalloc_noreclaim_restore (flags);if (fence) {
@@ -222,6 +222,7 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm, flush_tlb && is_last_seg, true,NULL , start_page, last_page, pte_flags,0 , entry->addr,NULL ,NULL ,NULL , +NULL ,NULL , wait_fence && is_last_seg ? fence : NULL);if (ret)return ret;
@@ -330,7 +331,7 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,dma_fence_put (fence);if (!ret) - ret =amdgpu_vm_update_pdes (svm->adev, svm->vm, false); + ret =amdgpu_vm_update_pdes (svm->adev, svm->vm, false,NULL );if (!ret) {if (flush_tlb)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +1 -1
@@ -1151,7 +1151,7 @@ amdgpu_userq_vm_validate_and_restore_queue(struct amdgpu_userq_mgr *uq_mgr) goto retry_lock; } - ret =amdgpu_vm_update_pdes (adev, vm, false); + ret =amdgpu_vm_update_pdes (adev, vm, false,NULL );if (ret) goto unlock_all;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +51 -24
183 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +7 -2
@@ -39,6 +39,7 @@#include "amdgpu_ttm.h" struct drm_exec ; +struct drm_gpusvm_range ;struct amdgpu_bo_va ;struct amdgpu_job ;
@@ -284,7 +285,7 @@ struct amdgpu_vm {/* tree of virtual addresses mapped */ struct rb_root_cached va; -/* SVM can replace this pointer with its notifier lock. */ +/* SVM uses gpusvm.notifier_lock, other VMs use default_eviction_lock. */ struct rw_semaphore default_eviction_lock;struct rw_semaphore *eviction_lock;bool evicting;
@@ -447,7 +448,8 @@ void amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, bool *need_pipe_sync, bool *emit_spm_needed, bool *emit_gds_needed);int amdgpu_vm_update_pdes (struct amdgpu_device *adev, -struct amdgpu_vm *vm,bool immediate); +struct amdgpu_vm *vm,bool immediate, +struct rw_semaphore *eviction_lock);int amdgpu_vm_clear_freed (struct amdgpu_device *adev,struct amdgpu_vm *vm,struct dma_fence **fence);
@@ -467,10 +469,13 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm, uint64_t vram_base, struct ttm_resource *res, dma_addr_t *pages_addr, struct amdgpu_hmm_range *hmm_range, + struct rw_semaphore *eviction_lock, + struct drm_gpusvm_range *gpusvm_range, struct dma_fence **fence);int amdgpu_vm_unmap_range (struct amdgpu_device *adev,struct amdgpu_vm *vm,struct amdgpu_sync *sync,uint64_t start,uint64_t last,uint64_t flags, +struct rw_semaphore *eviction_lock,struct dma_fence **fence); int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h +22 -6
@@ -24,6 +24,7 @@#ifndef __AMDGPU_VM_INTERNAL_H__#define __AMDGPU_VM_INTERNAL_H__ +#include <drm/drm_gpusvm.h> #include <linux/types.h> #include <linux/list.h> #include "amdgpu_hmm.h"
@@ -80,6 +81,12 @@ struct amdgpu_vm_update_params {*/ struct amdgpu_hmm_range *hmm_range; +/** @eviction_lock: Optional caller-held eviction lock. */ +struct rw_semaphore *eviction_lock; + +/** @gpusvm_range: Optional SVM range to revalidate after taking the lock. */ +struct drm_gpusvm_range *gpusvm_range; +/** * @job: job to used for hw submission */
@@ -149,6 +156,7 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);* * Serialize all updates, check parameters and make sure that memory allocations * don't enter the reclaim path so that we don't deadlock with MMU notifiers. +* A non-NULL eviction_lock means the caller already holds the selected lock. * * Returns: *
@@ -158,7 +166,9 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);static inline int amdgpu_vm_begin_critical (struct amdgpu_vm_update_params *p) {/* Use the write lock to serialize updates when the root BO is unlocked. */ -if (p->unlocked) +if (p->eviction_lock) +lockdep_assert_held (p->eviction_lock); +else if (p->unlocked)down_write (p->vm->eviction_lock);else down_read (p->vm->eviction_lock);
@@ -168,6 +178,10 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)return -EBUSY ;if (p->hmm_range && !amdgpu_hmm_range_valid (p->hmm_range))return -EAGAIN ; +if (p->gpusvm_range && + !drm_gpusvm_range_pages_valid (p->gpusvm_range->gpusvm, + p->gpusvm_range)) +return -EAGAIN ;return 0 ; }
@@ -175,15 +189,17 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)* amdgpu_vm_end_critical - end the critical section of the update * @p: The update parameters * -* Restore the GFP flags and drop the lock. +* Restore the GFP flags. */ static inline void amdgpu_vm_end_critical (struct amdgpu_vm_update_params *p) {memalloc_noreclaim_restore (p->saved_flags); -if (p->unlocked) -up_write (p->vm->eviction_lock); -else -up_read (p->vm->eviction_lock); +if (!p->eviction_lock) { +if (p->unlocked) +up_write (p->vm->eviction_lock); +else +up_read (p->vm->eviction_lock); + } }#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c +25 -4
@@ -475,6 +475,23 @@ int amdgpu_vm_pt_create(struct amdgpu_device *adev, struct amdgpu_vm *vm,return amdgpu_bo_create_vm (adev, &bp, vmbo); } +static void amdgpu_vm_update_unlock (struct amdgpu_vm_update_params *p) +{ +amdgpu_vm_end_critical (p); +if (p->eviction_lock) { +lockdep_assert_held_read (p->eviction_lock); +up_read (p->eviction_lock); + } +} + +static int amdgpu_vm_update_relock (struct amdgpu_vm_update_params *p) +{ +if (p->eviction_lock) +down_read (p->eviction_lock); + +return amdgpu_vm_begin_critical (p); +} +/** * amdgpu_vm_pt_alloc - Allocate a specific page table *
@@ -498,19 +515,23 @@ static int amdgpu_vm_pt_alloc(struct amdgpu_vm_update_params *p,if (entry->bo)return 0 ; -amdgpu_vm_end_critical (p); +amdgpu_vm_update_unlock (p); r =amdgpu_vm_pt_create (p->adev, p->vm, cursor->level, &pt, p->vm->root.bo->xcp_id); - r2 =amdgpu_vm_begin_critical (p); + r2 =amdgpu_vm_update_relock (p);if (r)return r; -if (r2) + pt_bo = &pt->bo; +if (r2) { +amdgpu_vm_update_unlock (p); +amdgpu_bo_unref (&pt_bo); + (void )amdgpu_vm_update_relock (p);return r2; + }/* Keep a reference to the root directory to avoid * freeing them up in the wrong order. */ - pt_bo = &pt->bo; pt_bo->parent =amdgpu_bo_ref (cursor->parent->bo);amdgpu_vm_bo_base_init (entry, p->vm, pt_bo); r =amdgpu_vm_pt_clear (p->adev, p->vm, pt);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +4 -3
@@ -1407,7 +1407,7 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, }return amdgpu_vm_unmap_range (adev, vm,NULL , gpu_start, gpu_end, - init_pte_value, fence); + init_pte_value,NULL , fence); } static int
@@ -1522,7 +1522,8 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange, gpu_start, gpu_end, pte_flags, (last_start - prange->start) <<PAGE_SHIFT , bo_adev ? bo_adev->vm_manager.vram_base_offset :0 , - NULL, dma_addr, hmm_range, &vm->last_update); +NULL , dma_addr, hmm_range, + NULL, NULL, &vm->last_update);for (j = last_start - prange->start; j <= i; j++) dma_addr[j] |= last_domain;
@@ -1534,7 +1535,7 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange, last_start = prange->start + i +1 ; } - r =amdgpu_vm_update_pdes (adev, vm, false); + r =amdgpu_vm_update_pdes (adev, vm, false,NULL );if (r) {pr_debug ("failed %d to update directories 0x%lx\n" , r, prange->start);
Signature · 1 line
2.34.1
Quoted 6 lines · Honglei Huang
From: Honglei Huang <[email protected]>
This series follows Christian's suggestions and builds on his earlier fix work in the amdgpu VM critical section.
My understanding of all amdgpu VM locking details may be incomplete, so review and confirmation from the VM maintainers would be appreciated.
Hi Christian,
We are preparing the V10 of amdgpu drmsvm, cause your fix series (drm/amdgpu: fix the HMM range handling for KFD SVM) is absolutely necessary for the amdgpu drmsvm, so can we
- Include your fix series into the amdgpu drm svm series, cause it is easy to rebase / test / cherry pick.
- The hole series based on your locking unify design so will add suggested by you in entire amdgpu drm svm patches.
- Your fix series maybe need some base / change according to the latest amdgpu drm next, so can we add co-developed by in your fix series?
Regards,
Honglei
Quoted 81 lines
It is described that the ordering problem and why both paths need the same lock:
"Originally the
notifier_lockonly made sure that the CPU page table
updates were done in order and originally the eviction lock made sure that the GPU page table updates were done in order, but essentially we need the order for both.The point is that the updates need to be serialized. In other words when one CPU is doing a mapping operation and another CPU is doing an unmap through an MMU notifier we somehow need to make sure that the corresponding GPU page table updates execute in the correct order."
Before this series, the update flow uses two locks:
map thread:
notifier_lock(lock A)
validate range
pt_allocdrops onlyeviction_lock(lock B)
allocation enters direct reclaim
MMU notifier triesnotifier_lock(lock A again)
deadlock hereThe allocation cannot finish because reclaim waits for a read lock that the same thread releases only after the allocation returns. The VM helper also cannot drop and revalidate the notifier lock because it only knows about the separate eviction lock.
After this series, all three paths use the same rwsem:
map thread:
unified notifier/eviction lock for read (lock A)
validate range
pt_allocdrops lock A
allocation can enter reclaim
MMU notifier can take lock A for write and finish
reacquire lock A for read
revalidate range
range valid -> update PTEs and PDEs -> publish the mapping range changed -> return-EAGAINand retryMMU notifier / eviction:
unified notifier/eviction lock for write (lock A)
clear PTEs or evict page tablesThe allocation no longer holds lock A during reclaim. If an MMU notifier invalidates the range while lock A is dropped, the map thread detects the changed range after reacquiring lock A and retries instead of installing stale PTEs.
The three patches make the VM eviction lock read/write, support a caller held lock with post allocation range revalidation, and use the
drm_gpusvmnotifier lock for SVM VMs. Non SVM callers pass NULL and keep the existing internal locking path.The patches apply after the AMDGPU SVM build and VM fault-path integration patch.
Testing:
MI60: KFD svm test passed, 2 known attribute failures (attribute get refactor). HIPcatch_tests: 98% passed.Honglei Huang (3):
drm/amdgpu: make VM eviction lock read/write to match notifier lock drm/amdgpu: recheck range validity after page table allocation drm/amdgpu: usedrm_gpusvmnotifier lock for VM eviction.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 37 ++++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 52 +++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 84 ++++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 15 ++-- .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 30 ++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 29 ++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 7 +- 11 files changed, 176 insertions(+), 86 deletions(-)