Skip to contentSkip to conversation

nouveau: fix instmem race condition around ptr stores

PATCH In review Dave Airlie 3 participants 4 messages Apr 9, 2024

Subsystems (4)
lore
Share & tools
lore

Dave Airlie

author maintainer
#0 lore raw

PATCH 1/1 nouveau: fix instmem race condition around ptr stores

From: Dave Airlie <[email protected]>

Running a lot of VK CTS in parallel against nouveau, once every few hours you might see something like this crash.

BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS:  00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

...

 ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
 ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
 nvkm_vmm_iter+0x351/0xa20 [nouveau]
 ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 ? __lock_acquire+0x3ed/0x2170
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
 ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]

Adding any sort of useful debug usually makes it go away, so I hand wrote the function in a line, and debugged the asm.

Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in the nv50_instobj_acquire called from nvkm_kmap.

If Thread A and Thread B both get to nv50_instobj_acquire around the same time, and Thread A hits the refcount_set line, and in lockstep thread B succeeds at refcount_inc_not_zero, there is a chance the ptrs value won't have been stored since refcount_set is unordered. Force a memory barrier here, I picked smp_mb, since we want it on all CPUs and it's write followed by a read.

Cc linux-stable
---
 drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c +3 -0
index a7f3fc342d87..cbacc7b11f8c 100644
@@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
 			iobj->base.memory.ptrs = &nv50_instobj_fast;
 		else
 			iobj->base.memory.ptrs = &nv50_instobj_slow;
+		/* barrier to ensure ptrs is written before another thread
+		   does refcount_inc_not_zero successfully. */
+		smp_mb();
 		refcount_set(&iobj->maps, 1);
 	}
 
Signature · 1 line
2.43.2
#1 lore raw

Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:

Quoted 64 lines

From: Dave Airlie <[email protected]>

Running a lot of VK CTS in parallel against nouveau, once every few hours you might see something like this crash.

BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS:  00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

...

 ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
 ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
 nvkm_vmm_iter+0x351/0xa20 [nouveau]
 ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 ? __lock_acquire+0x3ed/0x2170
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
 ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
 ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
 nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]

Adding any sort of useful debug usually makes it go away, so I hand wrote the function in a line, and debugged the asm.

Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in the nv50_instobj_acquire called from nvkm_kmap.

If Thread A and Thread B both get to nv50_instobj_acquire around the same time, and Thread A hits the refcount_set line, and in lockstep thread B succeeds at refcount_inc_not_zero, there is a chance the ptrs value won't have been stored since refcount_set is unordered. Force a memory barrier here, I picked smp_mb, since we want it on all CPUs and it's write followed by a read.

Cc linux-stable
---
 drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c +0 -0
index a7f3fc342d87..cbacc7b11f8c 100644

drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c nv50_instobj_acquire(struct nvkm_memory *memory) · line 255

Quoted 6 lines
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c +3 -0
@@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
 			iobj->base.memory.ptrs = &nv50_instobj_fast;
 		else
 			iobj->base.memory.ptrs = &nv50_instobj_slow;
+		/* barrier to ensure ptrs is written before another thread
+		   does refcount_inc_not_zero successfully. */
+		smp_mb();

Doesn't this miss the corresponding smp_rmb after refcount_inc_not_zero? Without it a sufficiently speculating CPU might still hoist the NULL ptr load across the refcount increase.

Regards,
Lucas

#2 lore raw
Quoted 53 lines · Lucas Stach

Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:

Quoted 52 lines

From: Dave Airlie <[email protected]>

Running a lot of VK CTS in parallel against nouveau, once every few hours you might see something like this crash.

BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS:  00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

...

  ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
  ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
  nvkm_vmm_iter+0x351/0xa20 [nouveau]
  ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  ? __lock_acquire+0x3ed/0x2170
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
  ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]

Adding any sort of useful debug usually makes it go away, so I hand wrote the function in a line, and debugged the asm.

Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in the nv50_instobj_acquire called from nvkm_kmap.

If Thread A and Thread B both get to nv50_instobj_acquire around the same time, and Thread A hits the refcount_set line, and in lockstep thread B succeeds at refcount_inc_not_zero, there is a chance the ptrs value won't have been stored since refcount_set is unordered. Force a memory barrier here, I picked smp_mb, since we want it on all CPUs and it's write followed by a read.

Good catch!

Quoted 7 lines
Quoted 7 lines
Cc linux-stable
---
  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
  1 file changed, 3 insertions(+)
Quoted 15 lines

drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c nv50_instobj_acquire(struct nvkm_memory *memory) · line 255

Quoted 11 lines
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c +3 -0
index a7f3fc342d87..cbacc7b11f8c 100644
@@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
  			iobj->base.memory.ptrs = &nv50_instobj_fast;
  		else
  			iobj->base.memory.ptrs = &nv50_instobj_slow;
+		/* barrier to ensure ptrs is written before another thread
+		   does refcount_inc_not_zero successfully. */
+		smp_mb();

Doesn't this miss the corresponding smp_rmb after refcount_inc_not_zero? Without it a sufficiently speculating CPU might still hoist the NULL ptr load across the refcount increase.

Agree, also think this one could be smp_wmb() only.

I also think it's reasonable to keep "the fast path refcount_inc_not_zero that doesn't take the lock", since the scope for this being potentially racy is limited to this function only.

Quoted 3 lines

Regards,
Lucas

author maintainer
#3 lore raw
Quoted 81 lines · Danilo Krummrich
Quoted 53 lines · Lucas Stach

Am Dienstag, dem 09.04.2024 um 10:34 +1000 schrieb Dave Airlie:

Quoted 52 lines

From: Dave Airlie <[email protected]>

Running a lot of VK CTS in parallel against nouveau, once every few hours you might see something like this crash.

BUG: kernel NULL pointer dereference, address: 0000000000000008
PGD 8000000114e6e067 P4D 8000000114e6e067 PUD 109046067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 7 PID: 53891 Comm: deqp-vk Not tainted 6.8.0-rc6+ #27
Hardware name: Gigabyte Technology Co., Ltd. Z390 I AORUS PRO WIFI/Z390 I AORUS PRO WIFI-CF, BIOS F8 11/05/2021
RIP: 0010:gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
Code: c7 48 01 c8 49 89 45 58 85 d2 0f 84 95 00 00 00 41 0f b7 46 12 49 8b 7e 08 89 da 42 8d 2c f8 48 8b 47 08 41 83 c7 01 48 89 ee <48> 8b 40 08 ff d0 0f 1f 00 49 8b 7e 08 48 89 d9 48 8d 75 04 48 c1
RSP: 0000:ffffac20c5857838 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 00000000004d8001 RCX: 0000000000000001
RDX: 00000000004d8001 RSI: 00000000000006d8 RDI: ffffa07afe332180
RBP: 00000000000006d8 R08: ffffac20c5857ad0 R09: 0000000000ffff10
R10: 0000000000000001 R11: ffffa07af27e2de0 R12: 000000000000001c
R13: ffffac20c5857ad0 R14: ffffa07a96fe9040 R15: 000000000000001c
FS:  00007fe395eed7c0(0000) GS:ffffa07e2c980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000011febe001 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

...

  ? gp100_vmm_pgt_mem+0xe3/0x180 [nouveau]
  ? gp100_vmm_pgt_mem+0x37/0x180 [nouveau]
  nvkm_vmm_iter+0x351/0xa20 [nouveau]
  ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  ? __lock_acquire+0x3ed/0x2170
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  nvkm_vmm_ptes_get_map+0xc2/0x100 [nouveau]
  ? __pfx_nvkm_vmm_ref_ptes+0x10/0x10 [nouveau]
  ? __pfx_gp100_vmm_pgt_mem+0x10/0x10 [nouveau]
  nvkm_vmm_map_locked+0x224/0x3a0 [nouveau]

Adding any sort of useful debug usually makes it go away, so I hand wrote the function in a line, and debugged the asm.

Every so often pt->memory->ptrs is NULL. This ptrs ptr is set in the nv50_instobj_acquire called from nvkm_kmap.

If Thread A and Thread B both get to nv50_instobj_acquire around the same time, and Thread A hits the refcount_set line, and in lockstep thread B succeeds at refcount_inc_not_zero, there is a chance the ptrs value won't have been stored since refcount_set is unordered. Force a memory barrier here, I picked smp_mb, since we want it on all CPUs and it's write followed by a read.

Good catch!

Quoted 7 lines
Quoted 7 lines
Cc linux-stable
---
  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c | 3 +++
  1 file changed, 3 insertions(+)
Quoted 15 lines

drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c nv50_instobj_acquire(struct nvkm_memory *memory) · line 255

Quoted 11 lines
drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.c +3 -0
index a7f3fc342d87..cbacc7b11f8c 100644
@@ -250,6 +250,9 @@ nv50_instobj_acquire(struct nvkm_memory *memory)
                     iobj->base.memory.ptrs = &nv50_instobj_fast;
             else
                     iobj->base.memory.ptrs = &nv50_instobj_slow;
+            /* barrier to ensure ptrs is written before another thread
+               does refcount_inc_not_zero successfully. */
+            smp_mb();

Doesn't this miss the corresponding smp_rmb after refcount_inc_not_zero? Without it a sufficiently speculating CPU might still hoist the NULL ptr load across the refcount increase.

Quoted 5 lines

Agree, also think this one could be smp_wmb() only.

I also think it's reasonable to keep "the fast path refcount_inc_not_zero that doesn't take the lock", since the scope for this being potentially racy is limited to this function only.

I've been retesting with just barrier() here, since this seems at least to be compiler related, but probably the smp_rmb/smp_wmb combo is the safest answer across arches.

Dave.