Skip to contentSkip to conversation

nouveau/fence: handle cross device fences properly. (v3)

PATCH Applied Dave Airlie 4 participants 6 messages Jan 9, 2025 – Jan 13, 2025

Subsystems (3)
lore
Share & tools
lore

Dave Airlie

author maintainer
#0 lore raw

PATCH 1/1 nouveau/fence: handle cross device fences properly. (v3)

From: Dave Airlie <[email protected]>

This is the 3rd iteration of this after talking to Ben and Danilo, I think this makes sense now.

The fence sync logic doesn't handle a fence sync across devices as it tries to write to a channel offset from one device into the fence bo from a different device, which won't work so well.

This patch fixes that to avoid using the sync path in the case where the fences come from different nouveau drm devices.

This works fine on a single device as the fence bo is shared across the devices, and mapped into each channels vma space, the channel offsets are therefore okay to pass between sides, so one channel can sync on the seqnos from the other by using the offset into it's vma.

---
 drivers/gpu/drm/nouveau/nouveau_fence.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +4 -2
index ee5e9d40c166..a3eb1f447a29 100644
@@ -367,11 +367,13 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan,
 			if (f) {
 				struct nouveau_channel *prev;
 				bool must_wait = true;
+				bool local;
 
 				rcu_read_lock();
 				prev = rcu_dereference(f->channel);
-				if (prev && (prev == chan ||
-					     fctx->sync(f, prev, chan) == 0))
+				local = prev && prev->drm == chan->drm;
+				if (local && (prev == chan ||
+					      fctx->sync(f, prev, chan) == 0))
 					must_wait = false;
 				rcu_read_unlock();
 				if (!must_wait)
Signature · 1 line
2.43.0
#1 lore raw
Quoted 12 lines · Dave Airlie

From: Dave Airlie <[email protected]>

This is the 3rd iteration of this after talking to Ben and Danilo, I think this makes sense now.

The fence sync logic doesn't handle a fence sync across devices as it tries to write to a channel offset from one device into the fence bo from a different device, which won't work so well.

This patch fixes that to avoid using the sync path in the case where the fences come from different nouveau drm devices.

Quoted 7 lines

This works fine on a single device as the fence bo is shared across the devices, and mapped into each channels vma space, the channel offsets are therefore okay to pass between sides, so one channel can sync on the seqnos from the other by using the offset into it's vma.

Quoted 25 lines
---
  drivers/gpu/drm/nouveau/nouveau_fence.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +4 -2
index ee5e9d40c166..a3eb1f447a29 100644
@@ -367,11 +367,13 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan,
  			if (f) {
  				struct nouveau_channel *prev;
  				bool must_wait = true;
+				bool local;
 
  				rcu_read_lock();
  				prev = rcu_dereference(f->channel);
-				if (prev && (prev == chan ||
-					     fctx->sync(f, prev, chan) == 0))
+				local = prev && prev->drm == chan->drm;
+				if (local && (prev == chan ||
+					      fctx->sync(f, prev, chan) == 0))
  					must_wait = false;
  				rcu_read_unlock();
  				if (!must_wait)
Reviewed-by
#2 lore raw
Quoted 14 lines · Ben Skeggs
Quoted 12 lines · Dave Airlie

From: Dave Airlie <[email protected]>

This is the 3rd iteration of this after talking to Ben and Danilo, I think this makes sense now.

The fence sync logic doesn't handle a fence sync across devices as it tries to write to a channel offset from one device into the fence bo from a different device, which won't work so well.

This patch fixes that to avoid using the sync path in the case where the fences come from different nouveau drm devices.

Quoted 8 lines
Quoted 7 lines

This works fine on a single device as the fence bo is shared across the devices, and mapped into each channels vma space, the channel offsets are therefore okay to pass between sides, so one channel can sync on the seqnos from the other by using the offset into it's vma.

Err, not sure what my fingers did there ;)

Quoted 26 lines
Quoted 26 lines

---
  drivers/gpu/drm/nouveau/nouveau_fence.c | 6 ++++--

  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +4 -2
index ee5e9d40c166..a3eb1f447a29 100644
@@ -367,11 +367,13 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, 
 struct nouveau_channel *chan,
               if (f) {
                   struct nouveau_channel *prev;
                   bool must_wait = true;
+                bool local;
                     rcu_read_lock();
                   prev = rcu_dereference(f->channel);
-                if (prev && (prev == chan ||
-                         fctx->sync(f, prev, chan) == 0))
+                local = prev && prev->drm == chan->drm;
+                if (local && (prev == chan ||
+                          fctx->sync(f, prev, chan) == 0))
                       must_wait = false;
                   rcu_read_unlock();
                   if (!must_wait)
maintainer
#3 lore raw
In reply to Dave Airlie
Quoted 9 lines · Dave Airlie

From: Dave Airlie <[email protected]>

This is the 3rd iteration of this after talking to Ben and Danilo, I think this makes sense now.

The fence sync logic doesn't handle a fence sync across devices as it tries to write to a channel offset from one device into the fence bo from a different device, which won't work so well.

Quoted 8 lines

This patch fixes that to avoid using the sync path in the case where the fences come from different nouveau drm devices.

This works fine on a single device as the fence bo is shared across the devices, and mapped into each channels vma space, the channel offsets are therefore okay to pass between sides, so one channel can sync on the seqnos from the other by using the offset into it's vma.

Huh, they indeed all share and map drm->fence->bo, good catch.

Quoted 12 lines
---
 drivers/gpu/drm/nouveau/nouveau_fence.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +0 -0
index ee5e9d40c166..a3eb1f447a29 100644

drivers/gpu/drm/nouveau/nouveau_fence.c nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan, · line 374

Quoted 10 lines
drivers/gpu/drm/nouveau/nouveau_fence.c +2 -2
@@ -367,11 +367,13 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan,
 			if (f) {
 				struct nouveau_channel *prev;
 				bool must_wait = true;
+				bool local;
 
 				rcu_read_lock();
 				prev = rcu_dereference(f->channel);
-				if (prev && (prev == chan ||
-					     fctx->sync(f, prev, chan) == 0))
+				local = prev && prev->drm == chan->drm;

struct nouveau_channel has no pointer to a struct nouveau_drm, this should be prev->cli->drm and chan->cli->drm instead.

No need to resend, I can fix it when applying the patch if you want.

Quoted 8 lines
drivers/gpu/drm/nouveau/nouveau_fence.c +2 -0
+				if (local && (prev == chan ||
+					      fctx->sync(f, prev, chan) == 0))
 					must_wait = false;
 				rcu_read_unlock();
 				if (!must_wait)
Signature · 1 line
2.43.0
maintainer Applied
#4 lore raw
Quoted 47 lines · Danilo Krummrich
Quoted 9 lines · Dave Airlie

From: Dave Airlie <[email protected]>

This is the 3rd iteration of this after talking to Ben and Danilo, I think this makes sense now.

The fence sync logic doesn't handle a fence sync across devices as it tries to write to a channel offset from one device into the fence bo from a different device, which won't work so well.

Quoted 8 lines

This patch fixes that to avoid using the sync path in the case where the fences come from different nouveau drm devices.

This works fine on a single device as the fence bo is shared across the devices, and mapped into each channels vma space, the channel offsets are therefore okay to pass between sides, so one channel can sync on the seqnos from the other by using the offset into it's vma.

Huh, they indeed all share and map drm->fence->bo, good catch.

Quoted 12 lines
---
 drivers/gpu/drm/nouveau/nouveau_fence.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +0 -0
index ee5e9d40c166..a3eb1f447a29 100644

drivers/gpu/drm/nouveau/nouveau_fence.c nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan, · line 374

Quoted 10 lines
drivers/gpu/drm/nouveau/nouveau_fence.c +2 -2
@@ -367,11 +367,13 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan,
 			if (f) {
 				struct nouveau_channel *prev;
 				bool must_wait = true;
+				bool local;
 
 				rcu_read_lock();
 				prev = rcu_dereference(f->channel);
-				if (prev && (prev == chan ||
-					     fctx->sync(f, prev, chan) == 0))
+				local = prev && prev->drm == chan->drm;

struct nouveau_channel has no pointer to a struct nouveau_drm, this should be prev->cli->drm and chan->cli->drm instead.

Quoted 1 line

No need to resend, I can fix it when applying the patch if you want.

Applied to drm-misc-fixes.

Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/nouveau/nouveau_fence.c +2 -0
+				if (local && (prev == chan ||
+					      fctx->sync(f, prev, chan) == 0))
 					must_wait = false;
 				rcu_read_unlock();
 				if (!must_wait)
Signature · 1 line
2.43.0
#5 lore raw
In reply to Dave Airlie
kernel test robot noticed the following build errors:

Hi Dave,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on drm-misc/drm-misc-next drm-tip/drm-tip v6.13-rc6 next-20250109] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Dave-Airlie/nouveau-fence-handle-cross-device-fences-properly-v3/20250109-085805
base:   linus/master
patch link:    https://lore.kernel.org/r/20250109005553.623947-1-airlied%40gmail.com
patch subject: [PATCH] nouveau/fence: handle cross device fences properly. (v3)
config: loongarch-randconfig-002-20250110 (https://download.01.org/0day-ci/archive/20250110/[email protected]/config)
compiler: loongarch64-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250110/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

drivers/gpu/drm/nouveau/nouveau_fence.c: In function 'nouveau_fence_sync':

>> drivers/gpu/drm/nouveau/nouveau_fence.c:394:53: error: 'struct nouveau_channel' has no member named 'drm'
     394 |                                 local = prev && prev->drm == chan->drm;
         |                                                     ^~
   drivers/gpu/drm/nouveau/nouveau_fence.c:394:66: error: 'struct nouveau_channel' has no member named 'drm'
     394 |                                 local = prev && prev->drm == chan->drm;
         |                                                                  ^~

vim +394 drivers/gpu/drm/nouveau/nouveau_fence.c

356
357 int
358 nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan,
359 bool exclusive, bool intr)
360 {
361 struct nouveau_fence_chan *fctx = chan->fence;
362 struct dma_resv *resv = nvbo->bo.base.resv;
363 int i, ret;
364
365 ret = dma_resv_reserve_fences(resv, 1);
366 if (ret)
367 return ret;
368
369 /* Waiting for the writes first causes performance regressions
370 * under some circumstances. So manually wait for the reads first.
371 */
372 for (i = 0; i < 2; ++i) {
373 struct dma_resv_iter cursor;
374 struct dma_fence *fence;
375
376 dma_resv_for_each_fence(&cursor, resv,
377 dma_resv_usage_rw(exclusive),
378 fence) {
379 enum dma_resv_usage usage;
380 struct nouveau_fence *f;
381
382 usage = dma_resv_iter_usage(&cursor);
383 if (i == 0 && usage == DMA_RESV_USAGE_WRITE)
384 continue;
385
386 f = nouveau_local_fence(fence, chan->cli->drm);
387 if (f) {
388 struct nouveau_channel *prev;
389 bool must_wait = true;
390 bool local;
391
392 rcu_read_lock();
393 prev = rcu_dereference(f->channel);

 > 394					local = prev && prev->drm == chan->drm;

395 if (local && (prev == chan ||
396 fctx->sync(f, prev, chan) == 0))
397 must_wait = false;
398 rcu_read_unlock();
399 if (!must_wait)
400 continue;
401 }
402
403 ret = dma_fence_wait(fence, intr);
404 if (ret)
405 return ret;
406 }
407 }
408
409 return 0;
410 }
411

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki