Skip to contentSkip to conversation

drm/amdkfd: improve performance with XNACK enable

PATCH Superseded James Zhu 4 participants 11 messages May 8, 2025 – Jun 12, 2025

Subsystems (3)
lore
Share & tools
lore

James Zhu

author
#0 lore raw

PATCH 1/1 drm/amdkfd: improve performance with XNACK enable

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
index 72be6e152e88..5f6ed70559b7 100644
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
 {
 retry_flush_work:
 	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }
 
 	if (list_empty(&svms->deferred_range_list))
 		return;
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
 {
 retry_flush_work:
 	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
 	if (list_empty(&svms->deferred_range_list))
 		return;
#2 lore raw
Quoted 50 lines · Christian König
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until

    restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read
    until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's
    write after the above two mm->mmap_lock's read are released, and in the
    meantime which will block all following mm->mmap_lock's read request.

5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
  {
  retry_flush_work:
  	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

[JZ]  At 1st beginning, I am thinking adding sync with restoring pages done.

but the original restoring work design philosophy is blindly scheduled after certain delay.

the changes with sync may take more time and risk. I would like Felix and Philip give comments

if there is efficient and safe way to fix it. Thanks!

Quoted 7 lines

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
  	if (list_empty(&svms->deferred_range_list))
  		return;
#3 lore raw
Quoted 57 lines · James Zhu
Quoted 51 lines · Christian König
Quoted 40 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-

  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct 
 svm_range_list *svms,
   {
Quoted 8 lines

  retry_flush_work:
      flush_work(&svms->deferred_list_work);
-    mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock))) +                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

Quoted 7 lines

[JZ]  At 1st beginning, I am thinking adding sync with restoring pages done.

but the original restoring work design philosophy is blindly scheduled after certain delay.

the changes with sync may take more time and risk. I would like Felix and Philip give comments

if there is efficient and safe way to fix it. Thanks!

[JZ] BTW, in worse case, mmap_write_lock will fall into rwsem_down_write_slowpath(), schedule_preempt_disabled() and schedule();

Quoted 7 lines
Quoted 7 lines

Regards,
Christian.

Quoted 2 lines

        if (list_empty(&svms->deferred_range_list)) return;

maintainer
#4 lore raw
Quoted 58 lines · James Zhu
Quoted 56 lines · James Zhu
Quoted 53 lines · Christian König
Quoted 39 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-

  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
   {
Quoted 8 lines

  retry_flush_work:
      flush_work(&svms->deferred_list_work);
-    mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock))) +                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

Quoted 8 lines
Quoted 7 lines

[JZ]  At 1st beginning, I am thinking adding sync with restoring pages done.

but the original restoring work design philosophy is blindly scheduled after certain delay.

the changes with sync may take more time and risk. I would like Felix and Philip give comments

if there is efficient and safe way to fix it. Thanks!

[JZ] BTW, in worse case, mmap_write_lock will fall into rwsem_down_write_slowpath(), schedule_preempt_disabled() and schedule();

Yeah, but drivers are not allowed to re-implement or even bypass that logic.

Regards,
Christian.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Regards,
Christian.

Quoted 2 lines

        if (list_empty(&svms->deferred_range_list)) return;

#5 lore raw
Quoted 58 lines · Christian König
Quoted 57 lines · James Zhu
Quoted 56 lines · James Zhu
Quoted 50 lines · Christian König
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
   drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
   1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
   {
   retry_flush_work:
       flush_work(&svms->deferred_list_work);
-    mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

[JZ]  At 1st beginning, I am thinking adding sync with restoring pages done.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

but the original restoring work design philosophy is blindly scheduled after certain delay.

the changes with sync may take more time and risk. I would like Felix and Philip give comments

if there is efficient and safe way to fix it. Thanks!

[JZ] BTW, in worse case, mmap_write_lock will fall into rwsem_down_write_slowpath(), schedule_preempt_disabled() and schedule();

Yeah, but drivers are not allowed to re-implement or even bypass that logic.

[JZ] here  can take as a new version mmap_write_lock without blocking the following read request and competing for write

lock which means read has higher priority under this case. Logically sync is better way to replace it. In practice, under

current scenery, sync will increase other burdens an risk.

Quoted 9 lines

Regards,
Christian.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Regards,
Christian.

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
         if (list_empty(&svms->deferred_range_list))
           return;
maintainer
#6 lore raw
Quoted 65 lines · James Zhu
Quoted 63 lines · Christian König
Quoted 62 lines · James Zhu
Quoted 61 lines · James Zhu
Quoted 50 lines · Christian König
Quoted 39 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-

  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
   {
Quoted 8 lines

  retry_flush_work:
      flush_work(&svms->deferred_list_work);
-    mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock))) +                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

[JZ]  At 1st beginning, I am thinking adding sync with restoring pages done.

but the original restoring work design philosophy is blindly scheduled after certain delay.

the changes with sync may take more time and risk. I would like Felix and Philip give comments

if there is efficient and safe way to fix it. Thanks!

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

[JZ] BTW, in worse case, mmap_write_lock will fall into rwsem_down_write_slowpath(), schedule_preempt_disabled() and schedule();

Yeah, but drivers are not allowed to re-implement or even bypass that logic.

[JZ] here  can take as a new version mmap_write_lock without blocking the following read request and competing for write

lock which means read has higher priority under this case. Logically sync is better way to replace it. In practice, under

current scenery, sync will increase other burdens an risk.

Yeah and exactly that is messing with mutex internals and is an absolutely clear no-go.

Stuff like that is not allowed, not even remotely.

Regards,
Christian.

Quoted 9 lines
Quoted 9 lines

Regards,
Christian.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Regards,
Christian.

Quoted 2 lines

        if (list_empty(&svms->deferred_range_list)) return;

maintainer
#7 lore raw
In reply to Christian König
Quoted 50 lines · Christian König
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until

    restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read
    until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's
    write after the above two mm->mmap_lock's read are released, and in the
    meantime which will block all following mm->mmap_lock's read request.

5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
  {
  retry_flush_work:
  	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

I could think of a few other higher level mitigations to avoid getting into this situation in the first place.

1. Give BOs allocated with KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED a higher LRU priority, to make them less likely to be evicted
2. If that doesn't help, set bp.no_wait_gpu in svm_range_vram_node_new. This should avoid causing evictions when trying to migrate data to VRAM in the page fault handler. In this case, migration should fail and the page fault should map data in system memory

Regards,
  Felix

Quoted 7 lines

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
  	if (list_empty(&svms->deferred_range_list))
  		return;
#8 lore raw
Quoted 51 lines · Felix Kuehling
Quoted 50 lines · Christian König
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until

    restoring pages is completed
    step 2B, evictiion work queue process will hold one mm->mmap_lock's read
    until evicting bo is completed
    step 3, restoring work queue process is trying to acquire one mm->mmap_lock's
    write after the above two mm->mmap_lock's read are released, and in the
    meantime which will block all following mm->mmap_lock's read request.

5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
  {
  retry_flush_work:
  	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

Quoted 9 lines
Quoted 3 lines

As far as I know that is not something we can do so easily.

Would it be possible to wait for progress instead of calling schedule() here?

I could think of a few other higher level mitigations to avoid getting into this situation in the first place.

1. Give BOs allocated with KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED a higher LRU priority, to make them less likely to be evicted

[JZ] Less likely means still exist. we still see hang with this mmap_lock when VRAM under pressure.

Quoted 7 lines

1. If that doesn't help, set bp.no_wait_gpu in svm_range_vram_node_new. This should avoid causing evictions when trying to migrate data to VRAM in the page fault handler. In this case, migration should fail and the page fault should map data in system memory

[JZ] IIUC, It will sacrifice the performance in GPU computing stage.

Quoted 14 lines

1.

Regards,
  Felix

Quoted 6 lines

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
  	if (list_empty(&svms->deferred_range_list))
  		return;
maintainer
#9 lore raw
Quoted 56 lines · James Zhu
Quoted 53 lines · Felix Kuehling
Quoted 52 lines · Christian König
Quoted 38 lines · James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 72be6e152e88..5f6ed70559b7 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_list_lock_and_flush_work(struct svm_range_list *svms, · line 1801

Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
@@ -1794,7 +1794,11 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
 {
 retry_flush_work:
 	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+        while (true) {
+                if (down_write_trylock(&(mm->mmap_lock)))
+                        break;
+                schedule();
+        }

Oh, stuff like that is usually an absolutely clear NAK from upstream.

As far as I know that is not something we can do so easily.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Would it be possible to wait for progress instead of calling schedule() here?

I could think of a few other higher level mitigations to avoid getting into this situation in the first place.

1. Give BOs allocated with KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED a higher LRU priority, to make them less likely to be evicted

[JZ] Less likely means still exist. we still see hang with this mmap_lock when VRAM under pressure.

Is it a hang? The commit headline talks about a performance issue.

VRAM pressure would have be quite high. Basically, you'd only evict a KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED BO if there are no other SVM BOs left to evict. In that case you're probably in a pretty bad place already, in terms of performance.

Quoted 3 lines
Quoted 2 lines

1. If that doesn't help, set bp.no_wait_gpu in svm_range_vram_node_new. This should avoid causing evictions when trying to migrate data to VRAM in the page fault handler. In this case, migration should fail and the page fault should map data in system memory

[JZ] IIUC, It will sacrifice the performance in GPU computing stage.

Right. On the other hand, it would reduce the amount of time taken to handle page faults. It may reduce thrashing and actually improve performance.

Regards,
  Felix

Quoted 14 lines
Quoted 14 lines

1.

Regards,
  Felix

Quoted 6 lines

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
 	if (list_empty(&svms->deferred_range_list))
 		return;
author
#10 lore raw

PATCH v2 1/1 drm/amdkfd: improve performance with XNACK enable

In reply to James Zhu

When XNACK on, hang or low performance is observed with some test cases. The restoring page process has unexpected stuck during evicting/restoring if some bo's flag has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting:
1. when xnack on, retry pagefault will invoke restoring pages process
2. A. if there is enough VRAM space, simply migrating pages from ram to vram B. if there is no enough VRAM space left, searching resource LRU list, and scheduling a new eviction work queue to evict LRU bo from vram to ram first, then resume restoring pages process, or waiting for eviction timeout and try to schedule evicting next LRU bo
3. for case 2B, if bo has KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED setting, queue eviction will be triggered.So restoring work queue will be scheduled.
4. step 1, restoring pages process will hold one mm->mmap_lock's read until restoring pages is completed
step 2B, evictiion work queue process will hold one mm->mmap_lock's read until evicting bo is completed
step 3, restoring work queue process is trying to acquire one mm->mmap_lock's write after the above two mm->mmap_lock's read are released, and in the meantime which will block all following mm->mmap_lock's read request.
5. in step 2, if the first eviction bo's size is big enough for step 1 restoring pages request, everything is fine. if not, which means that the mm->mmap_lock's read step 1 won't be release right the way. In step 3, first eviction bo's restoring work queue will compete for mm->mmap_lock's write, the second and following LRU bo's evictiion work queue will be blocked by tring to acquire mm->mmap_lock's read until timeout. All restoring pages process will be stuck here.
Using down_write_trylock to replace mmap_write_lock will help not block the second and following evictiion work queue process.

-v2: just return if failed to get write lock, lets caller decides if retry.

---
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c |  4 +++-
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c     | 19 ++++++++++++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.h     |  2 +-
 3 files changed, 18 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +3 -1
index a2149afa5803..ba232cc13e9b 100644
@@ -1068,7 +1068,9 @@ static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
 	/* Flush pending deferred work to avoid racing with deferred actions
 	 * from previous memory map changes (e.g. munmap).
 	 */
-	svm_range_list_lock_and_flush_work(&p->svms, current->mm);
+	err = svm_range_list_lock_and_flush_work(&p->svms, current->mm);
+	if (err)
+		return err;
 	mutex_lock(&p->svms.lock);
 	mmap_write_unlock(current->mm);
 	if (interval_tree_iter_first(&p->svms.objects,
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +14 -5
index 865dca2547de..d4eccc6567ad 100644
@@ -1785,16 +1785,21 @@ static int svm_range_validate_and_map(struct mm_struct *mm,
  * @svms: the svm range list
  * @mm: the mm structure
  *
- * Context: Returns with mmap write lock held, pending deferred work flushed
+ * If gets write lock, returns with mmap write lock held and pending
+ * deferred work flushed, otherwise return -EAGAIN, lets caller
+ * decides if it needs retry to get write lock. Since busy wait for
+ * write lock here properly blocks restore pages process which held
+ * read lock in the beginning and asks for read lock during migration.
  *
  */
-void
+int
 svm_range_list_lock_and_flush_work(struct svm_range_list *svms,
 				   struct mm_struct *mm)
 {
 retry_flush_work:
 	flush_work(&svms->deferred_list_work);
-	mmap_write_lock(mm);
+	if (!down_write_trylock(&(mm->mmap_lock)))
+		return -EAGAIN;
 
 	if (list_empty(&svms->deferred_range_list))
 		return;
@@ -1833,7 +1838,8 @@ static void svm_range_restore_work(struct work_struct *work)
 	}
 
 	mutex_lock(&process_info->lock);
-	svm_range_list_lock_and_flush_work(svms, mm);
+	if (svm_range_list_lock_and_flush_work(svms, mm))
+		goto out_reschedule1;
 	mutex_lock(&svms->lock);
 
 	evicted_ranges = atomic_read(&svms->evicted_ranges);
@@ -1885,6 +1891,7 @@ static void svm_range_restore_work(struct work_struct *work)
 out_reschedule:
 	mutex_unlock(&svms->lock);
 	mmap_write_unlock(mm);
+out_reschedule1:
 	mutex_unlock(&process_info->lock);
 
 	/* If validation failed, reschedule another attempt */
@@ -3638,7 +3645,9 @@ svm_range_set_attr(struct kfd_process *p, struct mm_struct *mm,
 
 	mutex_lock(&process_info->lock);
 
-	svm_range_list_lock_and_flush_work(svms, mm);
+	r = svm_range_list_lock_and_flush_work(svms, mm);
+	if (r)
+		goto out;
 
 	r = svm_range_is_valid(p, start, size);
 	if (r) {
drivers/gpu/drm/amd/amdkfd/kfd_svm.h +1 -1
index 01c7a4877904..c8c9bc7eead9 100644
@@ -196,7 +196,7 @@ int kfd_criu_restore_svm(struct kfd_process *p,
 int kfd_criu_resume_svm(struct kfd_process *p);
 struct kfd_process_device *
 svm_range_get_pdd_by_node(struct svm_range *prange, struct kfd_node *node);
-void svm_range_list_lock_and_flush_work(struct svm_range_list *svms, struct mm_struct *mm);
+int svm_range_list_lock_and_flush_work(struct svm_range_list *svms, struct mm_struct *mm);
 
 /* SVM API and HMM page migration work together, device memory type
  * is initialized to not 0 when page migration register device memory.
Signature · 1 line
2.34.1