drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence
Subsystems (4)
Maíra Canal
maintainerPATCH 1/1 drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence
v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL register and poll its status bits. The mutex cache_clean_lock exists to serialize them, but v3d_clean_caches() only took the lock around its final FLM_CLEAN write.
These functions run concurrently: v3d_flush_l2t() is issued from the BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the CACHE_CLEAN queue, and each queue's scheduler uses its own ordered workqueue, so their run_job callbacks execute in parallel.
Because clean locked only its final write, a concurrent flush can write L2TCACTL during clean's unlocked phase. Both use non read-modify-write writes to the one register, so whichever lands last wins: clean's TMUWCF write can land on the flush's in-flight L2TFLS invalidate, triggering the GFXH-1897 hazard of writing L2TCACTL while a flush is pending.
Hold cache_clean_lock across the entire L2TCACTL access sequence so it is fully mutually exclusive with v3d_flush_l2t(), which already takes the lock around its own write.
--- drivers/gpu/drm/v3d/v3d_gem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +2 -3
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)struct drm_device *dev = &v3d->drm;int core =0 ; +guard (mutex)(&v3d->cache_clean_lock); +trace_v3d_cache_clean_begin (dev);/* GFXH-1897: Ensure pending flushes complete before writing L2TCACTL */
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)drm_err (dev,"Timeout waiting for TMU write combiner flush\n" ); } -mutex_lock (&v3d->cache_clean_lock);V3D_CORE_WRITE (core,V3D_CTL_L2TCACTL ,V3D_L2TCACTL_L2TFLS |V3D_SET_FIELD (V3D_L2TCACTL_FLM_CLEAN ,V3D_L2TCACTL_FLM ));
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)drm_err (dev,"Timeout waiting for L2T clean\n" ); } -mutex_unlock (&v3d->cache_clean_lock); -trace_v3d_cache_clean_end (dev); }
Signature · 1 line
2.54.0
Hi Maíra,
has this change fixed anything?
Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if this change has any visible performance impact on Pi5?
Iago
Quoted 70 lines · Maíra Canal
v3d_clean_caches()andv3d_flush_l2t()both write the single L2TCACTL register and poll its status bits. The mutexcache_clean_lockexists to
serialize them, butv3d_clean_caches()only took the lock around its finalFLM_CLEANwrite.These functions run concurrently:
v3d_flush_l2t()is issued from the BIN/RENDER/CSD invalidate path whilev3d_clean_caches()runs from theCACHE_CLEANqueue, and each queue's scheduler uses its own ordered workqueue, so theirrun_jobcallbacks execute in parallel.Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.Hold
cache_clean_lockacross the entire L2TCACTL access sequence so it
is fully mutually exclusive withv3d_flush_l2t(), which already takes the
lock around its own write.Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush beforecleaning caches")
Signed-off-by Maíra Canal <[email protected]>---
drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---1 file changed, 2 insertions(+), 3 deletions(-)drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d) struct drm_device *dev = &v3d->drm; int core = 0;drivers/gpu/drm/v3d/v3d_gem.c +2
-0+guard (mutex)(&v3d->cache_clean_lock); +
trace_v3d_cache_clean_begin(dev);
/* GFXH-1897: Ensure pending flushes complete before writing L2TCACTL */drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
drm_err(dev, "Timeout waiting for TMU write combiner flush\n");
}drivers/gpu/drm/v3d/v3d_gem.c +0
-1-mutex_lock (&v3d->cache_clean_lock);V3D_CORE_WRITE(core,
V3D_CTL_L2TCACTL,V3D_L2TCACTL_L2TFLS|V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN, V3D_L2TCACTL_FLM));drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for L2T clean\n"); }drivers/gpu/drm/v3d/v3d_gem.c +0
-2-mutex_unlock (&v3d->cache_clean_lock); -
trace_v3d_cache_clean_end(dev);
}
Maíra Canal
maintainerHi Iago,
Quoted 3 lines · Iago Toral
Hi Maíra,
has this change fixed anything?
Not real bug, just the theoretical race-condition between v3d_clean_caches() and v3d_flush_l2t().
Quoted 3 lines
Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if this change has any visible performance impact on Pi5?
I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a CACHE_CLEAN job running simultaneously to v3d_invalidate_caches().
One thing I believe it might be possible is to skip a flush if a flush is already happening, but it would be a micro-optimization.
Best regards,
- Maíra
Quoted 76 lines
Iago
Quoted 70 lines · Maíra Canal
v3d_clean_caches()andv3d_flush_l2t()both write the single L2TCACTL register and poll its status bits. The mutexcache_clean_lockexists to
serialize them, butv3d_clean_caches()only took the lock around its finalFLM_CLEANwrite.These functions run concurrently:
v3d_flush_l2t()is issued from the BIN/RENDER/CSD invalidate path whilev3d_clean_caches()runs from theCACHE_CLEANqueue, and each queue's scheduler uses its own ordered workqueue, so theirrun_jobcallbacks execute in parallel.Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.Hold
cache_clean_lockacross the entire L2TCACTL access sequence so it
is fully mutually exclusive withv3d_flush_l2t(), which already takes the
lock around its own write.Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush beforecleaning caches")
Signed-off-by Maíra Canal <[email protected]>--- drivers/gpu/drm/v3d/v3d_gem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)drivers/gpu/drm/v3d/v3d_gem.c +2
-0@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d) struct drm_device *dev = &v3d->drm; int core = 0; +guard (mutex)(&v3d->cache_clean_lock); + trace_v3d_cache_clean_begin(dev); /* GFXH-1897: Ensure pending flushes complete before writingL2TCACTL */
drivers/gpu/drm/v3d/v3d_gem.c +0
-1@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for TMU write combiner flush\n"); } -mutex_lock (&v3d->cache_clean_lock); V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL, V3D_L2TCACTL_L2TFLS |V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN,
V3D_L2TCACTL_FLM));drivers/gpu/drm/v3d/v3d_gem.c +0
-2@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for L2T clean\n"); } -mutex_unlock (&v3d->cache_clean_lock); - trace_v3d_cache_clean_end(dev); }
Iago Toral
Reviewed-byThanks, if we don't see any impact on performance on Pi5, this is:
Quoted 115 lines · Maíra Canal
Hi Iago,
Quoted 3 lines · Iago Toral
Hi Maíra,
has this change fixed anything?
Not real bug, just the theoretical race-condition between
v3d_clean_caches()andv3d_flush_l2t().Quoted 4 lines
Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if
this change has any visible performance impact on Pi5?I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a
CACHE_CLEANjob running simultaneously tov3d_invalidate_caches().One thing I believe it might be possible is to skip a flush if a flush
is already happening, but it would be a micro-optimization.Best regards,
- MaíraQuoted 87 lines
Iago
Quoted 81 lines · Maíra Canal
v3d_clean_caches()andv3d_flush_l2t()both write the single L2TCACTL
register and poll its status bits. The mutexcache_clean_lockexists
to
serialize them, butv3d_clean_caches()only took the lock around its
finalFLM_CLEANwrite.These functions run concurrently:
v3d_flush_l2t()is issued from the
BIN/RENDER/CSD invalidate path whilev3d_clean_caches()runs from theCACHE_CLEANqueue, and each queue's scheduler uses its own ordered
workqueue, so theirrun_jobcallbacks execute in parallel.Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering
the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.Hold
cache_clean_lockacross the entire L2TCACTL access sequence so
it
is fully mutually exclusive withv3d_flush_l2t(), which already takes
the
lock around its own write.Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush beforecleaning caches")
Signed-off-by Maíra Canal <[email protected]>---
drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---1 file changed, 2 insertions(+), 3 deletions(-)drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d) struct drm_device *dev = &v3d->drm; int core = 0;drivers/gpu/drm/v3d/v3d_gem.c +2
-0+guard (mutex)(&v3d->cache_clean_lock); +
trace_v3d_cache_clean_begin(dev);
/* GFXH-1897: Ensure pending flushes complete before writing
L2TCACTL */drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)drm_err(dev, "Timeout waiting for TMU write combiner
flush\n");
}drivers/gpu/drm/v3d/v3d_gem.c +0
-1-mutex_lock (&v3d->cache_clean_lock);V3D_CORE_WRITE(core,
V3D_CTL_L2TCACTL,V3D_L2TCACTL_L2TFLS|V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN, V3D_L2TCACTL_FLM));drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for L2T clean\n"); }drivers/gpu/drm/v3d/v3d_gem.c +0
-2-mutex_unlock (&v3d->cache_clean_lock); -
trace_v3d_cache_clean_end(dev);
}
Chema Casanova
Reviewed-byHi Maíra, Iago,
Finally, this is not only theoretical. On Raspberry Pi 4 the race causes order-dependent dEQP-VK failures in robustness *.vert andbuffer_device_address: TMU writes from a job are still in flight when its fence signals. I reached an equivalent soution to address this issue:
- 51-case robustness list: 41/1000 runs failing → 0/1000
- full robustness vertex list: 7/8 → 0/8
- full buffer_device_address: 3/8 → 0/8
The impact on Pi 5 on GfxBench GL-ES and Vulkan (Aztec, Manhattan, T-Rex, Egypt) stay within ±0.1%, which is run-to-run noise.
Quoted 115 lines · Iago Toral
Thanks, if we don't see any impact on performance on Pi5, this is:
Reviewed-by Iago Toral Quiroga <[email protected]>Quoted 109 lines · Maíra Canal
Hi Iago,
Quoted 3 lines · Iago Toral
Hi Maíra,
has this change fixed anything?
Not real bug, just the theoretical race-condition between
v3d_clean_caches()andv3d_flush_l2t().Quoted 3 lines
Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if
this change has any visible performance impact on Pi5?I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a
CACHE_CLEANjob running simultaneously tov3d_invalidate_caches().One thing I believe it might be possible is to skip a flush if a flush
is already happening, but it would be a micro-optimization.Best regards,
- MaíraQuoted 85 lines
Iago
Quoted 81 lines · Maíra Canal
v3d_clean_caches()andv3d_flush_l2t()both write the single L2TCACTL
register and poll its status bits. The mutexcache_clean_lockexists
to
serialize them, butv3d_clean_caches()only took the lock around its
finalFLM_CLEANwrite.These functions run concurrently:
v3d_flush_l2t()is issued from the
BIN/RENDER/CSD invalidate path whilev3d_clean_caches()runs from theCACHE_CLEANqueue, and each queue's scheduler uses its own ordered
workqueue, so theirrun_jobcallbacks execute in parallel.Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering
the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.Hold
cache_clean_lockacross the entire L2TCACTL access sequence so
it
is fully mutually exclusive withv3d_flush_l2t(), which already takes
the
lock around its own write.Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush beforecleaning caches")
Signed-off-by Maíra Canal <[email protected]>--- drivers/gpu/drm/v3d/v3d_gem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)drivers/gpu/drm/v3d/v3d_gem.c +2
-0@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d) struct drm_device *dev = &v3d->drm; int core = 0; +guard (mutex)(&v3d->cache_clean_lock); + trace_v3d_cache_clean_begin(dev); /* GFXH-1897: Ensure pending flushes complete beforewriting
L2TCACTL */drivers/gpu/drm/v3d/v3d_gem.c +0
-0@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for TMU writecombiner
flush\n");
}drivers/gpu/drm/v3d/v3d_gem.c +0
-1-mutex_lock (&v3d->cache_clean_lock); V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL, V3D_L2TCACTL_L2TFLS | V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN,
V3D_L2TCACTL_FLM));drivers/gpu/drm/v3d/v3d_gem.c +0
-2@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d) drm_err(dev, "Timeout waiting for L2T clean\n"); } -mutex_unlock (&v3d->cache_clean_lock); - trace_v3d_cache_clean_end(dev); }
Quoted 11 lines · Maíra Canal
v3d_clean_caches()andv3d_flush_l2t()both write the single L2TCACTL register and poll its status bits. The mutexcache_clean_lockexists to serialize them, butv3d_clean_caches()only took the lock around its finalFLM_CLEANwrite.These functions run concurrently:
v3d_flush_l2t()is issued from the BIN/RENDER/CSD invalidate path whilev3d_clean_caches()runs from theCACHE_CLEANqueue, and each queue's scheduler uses its own ordered workqueue, so theirrun_jobcallbacks execute in parallel.[...]
Applied, thanks!
[1/1] drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence
commit: 994ba6e5c87d6a3d471ea987fe8622168f092334
Best regards,
- Maíra