Skip to contentSkip to conversation

drm/amdgpu: Fix vm free pts race when process exiting

PATCH In review Philip Yang 4 participants 6 messages Aug 2, 2021 – Aug 4, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Fix vm free pts race when process exiting

Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
 ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct amdgpu_vm_bo_base *entry)
 		return;
 	shadow = amdgpu_bo_shadowed(entry->bo);
 	entry->bo->vm_bo = NULL;
+	spin_lock(&entry->vm->invalidated_lock);
 	list_del(&entry->vm_status);
+	spin_unlock(&entry->vm->invalidated_lock);
 	amdgpu_bo_unref(&shadow);
 	amdgpu_bo_unref(&entry->bo);
 }
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw

Am 2021-08-02 um 6:33 p.m. schrieb Philip Yang:

Quoted 25 lines

Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
 ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40
Quoted 1 line

The patch is

Quoted 18 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct amdgpu_vm_bo_base *entry)
 		return;
 	shadow = amdgpu_bo_shadowed(entry->bo);
 	entry->bo->vm_bo = NULL;
+	spin_lock(&entry->vm->invalidated_lock);
 	list_del(&entry->vm_status);
+	spin_unlock(&entry->vm->invalidated_lock);
 	amdgpu_bo_unref(&shadow);
 	amdgpu_bo_unref(&entry->bo);
 }
#2 lore raw
In reply to Philip Yang

Am 03.08.21 um 00:33 schrieb Philip Yang:

Quoted 16 lines

Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
  ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
Quoted 8 lines
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40

Wow, well this is a big NAK.

The page tables should never ever be on the invalidation list or otherwise we would try to point PTEs to them which is a huge security issue.

Taking the lock just workaround that. Can you investigate how it happens that a page table ends up on that list?

Thanks in advance,
Christian.

Quoted 20 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct amdgpu_vm_bo_base *entry)
  		return;
  	shadow = amdgpu_bo_shadowed(entry->bo);
  	entry->bo->vm_bo = NULL;
+	spin_lock(&entry->vm->invalidated_lock);
  	list_del(&entry->vm_status);
+	spin_unlock(&entry->vm->invalidated_lock);
  	amdgpu_bo_unref(&shadow);
  	amdgpu_bo_unref(&entry->bo);
  }
#3 lore raw

On 2021-08-04 5:01 a.m., Christian König wrote:
Am 03.08.21 um 00:33 schrieb Philip Yang:
Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
  ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40

Wow, well this is a big NAK.

The page tables should never ever be on the invalidation list or otherwise we would try to point PTEs to them which is a huge security issue.

Taking the lock just workaround that. Can you investigate how it happens that a page table ends up on that list?
I will be off 3 days, I can investigate this further next Monday. From debug, amdgpu_vm_bo_invalidate is called many times. The background: app has 8 processes, running on 8 GPUs, 64 VMs, VRAM usage is around 85% from rocm-info, 1/5 chance this happens (kernel BUG, server die) after application segmentation fault crash (another app issue). It is new issue on rocm release 4.3, never happened before on rocm 4.1 (no app crash on 4.1 either). kernel version is 4.18 on CentOS.

Regards,
Philip

Thanks in advance,
Christian.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++

  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct amdgpu_vm_bo_base *entry)

          return;
      shadow = amdgpu_bo_shadowed(entry->bo);
      entry->bo->vm_bo = NULL;
+    spin_lock(&entry->vm->invalidated_lock);
      list_del(&entry->vm_status);
+    spin_unlock(&entry->vm->invalidated_lock);
      amdgpu_bo_unref(&shadow);
      amdgpu_bo_unref(&entry->bo);
  }

#4 lore raw

Am 04.08.21 um 17:11 schrieb philip yang:

Quoted 29 lines · Christian König
Quoted 27 lines

Am 03.08.21 um 00:33 schrieb Philip Yang:

Quoted 24 lines

Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
  ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40
Quoted 8 lines
Quoted 5 lines

Wow, well this is a big NAK.

The page tables should never ever be on the invalidation list or otherwise we would try to point PTEs to them which is a huge security issue.

Taking the lock just workaround that. Can you investigate how it happens that a page table ends up on that list?

I will be off 3 days, I can investigate this further next Monday. From debug, amdgpu_vm_bo_invalidate is called many times. The background: app has 8 processes, running on 8 GPUs, 64 VMs, VRAM usage is around 85% from rocm-info, 1/5 chance this happens (kernel BUG, server die) after application segmentation fault crash (another app issue). It is new issue on rocm release 4.3, never happened before on rocm 4.1 (no app crash on 4.1 either). kernel version is 4.18 on CentOS.

It could be a bug in the VM code. I'm on vacation till 16th of August, but I will try to take a look when I have time.

Thanks for the notice,
Christian.

Quoted 30 lines

Regards,

Philip

Quoted 26 lines

Thanks in advance,
Christian.

Quoted 21 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++

  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct 
 amdgpu_vm_bo_base *entry)
           return;
       shadow = amdgpu_bo_shadowed(entry->bo);
       entry->bo->vm_bo = NULL;
+    spin_lock(&entry->vm->invalidated_lock);
       list_del(&entry->vm_status);
+    spin_unlock(&entry->vm->invalidated_lock);
       amdgpu_bo_unref(&shadow);
       amdgpu_bo_unref(&entry->bo);

  }

#5 lore raw
In reply to Christian König

Am 2021-08-04 um 5:01 a.m. schrieb Christian König:

Quoted 26 lines

Am 03.08.21 um 00:33 schrieb Philip Yang:

Quoted 16 lines

Take vm->invalidated_lock spinlock to remove vm pd and pt bos, to avoid link list corruption with crash backtrace:

[ 2290.505111] list_del corruption. next->prev should be
  ffff9b2514ec0018, but was 4e03280211010f04
[ 2290.505154] kernel BUG at lib/list_debug.c:56!
[ 2290.505176] invalid opcode: 0000 [#1] SMP NOPTI
[ 2290.505254] Workqueue: events delayed_fput
[ 2290.505271] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[ 2290.505513] Call Trace:
[ 2290.505623]  amdgpu_vm_free_table+0x26/0x80 [amdgpu]
[ 2290.505705]  amdgpu_vm_free_pts+0x7a/0xf0 [amdgpu]
[ 2290.505786]  amdgpu_vm_fini+0x1f0/0x440 [amdgpu]
[ 2290.505864]  amdgpu_driver_postclose_kms+0x172/0x290 [amdgpu]
[ 2290.505893]  drm_file_free.part.10+0x1d4/0x270 [drm]
[ 2290.505916]  drm_release+0xa9/0xe0 [drm]
Quoted 8 lines
[ 2290.505930]  __fput+0xb7/0x230
[ 2290.505942]  delayed_fput+0x1c/0x30
[ 2290.505957]  process_one_work+0x1a7/0x360
[ 2290.505971]  worker_thread+0x30/0x390
[ 2290.505985]  ? create_worker+0x1a0/0x1a0
[ 2290.505999]  kthread+0x112/0x130
[ 2290.506011]  ? kthread_flush_work_fn+0x10/0x10
[ 2290.506027]  ret_from_fork+0x22/0x40
Quoted 5 lines

Wow, well this is a big NAK.

The page tables should never ever be on the invalidation list or otherwise we would try to point PTEs to them which is a huge security issue.

entry->vm_status is used on other lists as well, and I think page tables can be added to those when they are evicted:

    static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
    {
            struct amdgpu_vm *vm = vm_bo->vm;
            struct amdgpu_bo *bo = vm_bo->bo;

            vm_bo->moved = true;
            if (bo->tbo.type == ttm_bo_type_kernel)
                    list_move(&vm_bo->vm_status, &vm->evicted);
            else
                    list_move_tail(&vm_bo->vm_status, &vm->evicted);
    }

But that never seems to involve the invalidated_lock. Maybe there is some other lock we should be holding somewhere (not necessarily in amdgpu_vm_free_table) to avoid this list corruption.

Regards,
  Felix

Quoted 30 lines

Taking the lock just workaround that. Can you investigate how it happens that a page table ends up on that list?

Thanks in advance,
Christian.

Quoted 22 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 ++

  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 2a88ed5d983b..5c4c355e7d6b 100644
@@ -1045,7 +1045,9 @@ static void amdgpu_vm_free_table(struct
 amdgpu_vm_bo_base *entry)
           return;
       shadow = amdgpu_bo_shadowed(entry->bo);
       entry->bo->vm_bo = NULL;
+    spin_lock(&entry->vm->invalidated_lock);
       list_del(&entry->vm_status);
+    spin_unlock(&entry->vm->invalidated_lock);
       amdgpu_bo_unref(&shadow);
       amdgpu_bo_unref(&entry->bo);

  }