Skip to contentSkip to conversation

drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence

PATCH Applied Maíra Canal 3 participants 6 messages Jul 7 – Nov 2

Subsystems (4)
lore
Share & tools
lore

Maíra Canal

author maintainer
#0 lore raw

PATCH 1/1 drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL register and poll its status bits. The mutex cache_clean_lock exists to serialize them, but v3d_clean_caches() only took the lock around its final FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the CACHE_CLEAN queue, and each queue's scheduler uses its own ordered workqueue, so their run_job callbacks execute in parallel.

Because clean locked only its final write, a concurrent flush can write L2TCACTL during clean's unlocked phase. Both use non read-modify-write writes to the one register, so whichever lands last wins: clean's TMUWCF write can land on the flush's in-flight L2TFLS invalidate, triggering the GFXH-1897 hazard of writing L2TCACTL while a flush is pending.

Hold cache_clean_lock across the entire L2TCACTL access sequence so it is fully mutually exclusive with v3d_flush_l2t(), which already takes the lock around its own write.

Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush before cleaning caches")
---
 drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +2 -3
index c43d9af41374..e597b6fd47c4 100644
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)
 	struct drm_device *dev = &v3d->drm;
 	int core = 0;
 
+	guard(mutex)(&v3d->cache_clean_lock);
+
 	trace_v3d_cache_clean_begin(dev);
 
 	/* GFXH-1897: Ensure pending flushes complete before writing L2TCACTL */
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
 		drm_err(dev, "Timeout waiting for TMU write combiner flush\n");
 	}
 
-	mutex_lock(&v3d->cache_clean_lock);
 	V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL,
 		       V3D_L2TCACTL_L2TFLS |
 		       V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN, V3D_L2TCACTL_FLM));
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
 		drm_err(dev, "Timeout waiting for L2T clean\n");
 	}
 
-	mutex_unlock(&v3d->cache_clean_lock);
-
 	trace_v3d_cache_clean_end(dev);
 }
 
Signature · 1 line
2.54.0
#1 lore raw

Hi Maíra,

has this change fixed anything?

Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if this change has any visible performance impact on Pi5?

Iago

Quoted 70 lines · Maíra Canal

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL register and poll its status bits. The mutex cache_clean_lock exists to
serialize them, but v3d_clean_caches() only took the lock around its final
FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the CACHE_CLEAN queue, and each queue's scheduler uses its own ordered workqueue, so their run_job callbacks execute in parallel.

Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.

Hold cache_clean_lock across the entire L2TCACTL access sequence so it
is fully mutually exclusive with v3d_flush_l2t(), which already takes the
lock around its own write.

Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush before

cleaning caches")

---
 drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---

 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +0 -0
index c43d9af41374..e597b6fd47c4 100644
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)
  	struct drm_device *dev = &v3d->drm;
  	int core = 0;

drivers/gpu/drm/v3d/v3d_gem.c +2 -0
+	guard(mutex)(&v3d->cache_clean_lock);
+

  trace_v3d_cache_clean_begin(dev);

  /* GFXH-1897: Ensure pending flushes complete before writing L2TCACTL */

drivers/gpu/drm/v3d/v3d_gem.c +0 -0
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)

  drm_err(dev, "Timeout waiting for TMU write combiner flush\n");
  }

drivers/gpu/drm/v3d/v3d_gem.c +0 -1
-	mutex_lock(&v3d->cache_clean_lock);

  V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL, V3D_L2TCACTL_L2TFLS | V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN, V3D_L2TCACTL_FLM));

drivers/gpu/drm/v3d/v3d_gem.c +0 -0
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
  		drm_err(dev, "Timeout waiting for L2T clean\n");
  	}

drivers/gpu/drm/v3d/v3d_gem.c +0 -2
-	mutex_unlock(&v3d->cache_clean_lock);
-

  trace_v3d_cache_clean_end(dev);
 }

author maintainer
#2 lore raw

Hi Iago,

Quoted 3 lines · Iago Toral

Hi Maíra,

has this change fixed anything?

Not real bug, just the theoretical race-condition between v3d_clean_caches() and v3d_flush_l2t().

Quoted 3 lines

Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if this change has any visible performance impact on Pi5?

I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a CACHE_CLEAN job running simultaneously to v3d_invalidate_caches().

One thing I believe it might be possible is to skip a flush if a flush is already happening, but it would be a micro-optimization.

Best regards,
- Maíra

Quoted 76 lines

Iago

Quoted 70 lines · Maíra Canal

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL register and poll its status bits. The mutex cache_clean_lock exists to
serialize them, but v3d_clean_caches() only took the lock around its final
FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the CACHE_CLEAN queue, and each queue's scheduler uses its own ordered workqueue, so their run_job callbacks execute in parallel.

Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.

Hold cache_clean_lock across the entire L2TCACTL access sequence so it
is fully mutually exclusive with v3d_flush_l2t(), which already takes the
lock around its own write.

Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush before

cleaning caches")

---
  drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---
  1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +2 -0
index c43d9af41374..e597b6fd47c4 100644
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)
  	struct drm_device *dev = &v3d->drm;
  	int core = 0;
 
+	guard(mutex)(&v3d->cache_clean_lock);
+
  	trace_v3d_cache_clean_begin(dev);
 
  	/* GFXH-1897: Ensure pending flushes complete before writing

L2TCACTL */

drivers/gpu/drm/v3d/v3d_gem.c +0 -1
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
  		drm_err(dev, "Timeout waiting for TMU write combiner
 flush\n");
  	}
 
-	mutex_lock(&v3d->cache_clean_lock);
  	V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL,
  		       V3D_L2TCACTL_L2TFLS |

         V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN,
V3D_L2TCACTL_FLM));

drivers/gpu/drm/v3d/v3d_gem.c +0 -2
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
  		drm_err(dev, "Timeout waiting for L2T clean\n");
  	}
 
-	mutex_unlock(&v3d->cache_clean_lock);
-
  	trace_v3d_cache_clean_end(dev);
  }
 
Reviewed-by
#3 lore raw

Thanks, if we don't see any impact on performance on Pi5, this is:

Quoted 115 lines · Maíra Canal

Hi Iago,

Quoted 3 lines · Iago Toral

Hi Maíra,

has this change fixed anything?

Not real bug, just the theoretical race-condition between v3d_clean_caches() and v3d_flush_l2t().

Quoted 4 lines

Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if
this change has any visible performance impact on Pi5?

I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a CACHE_CLEAN job running simultaneously to v3d_invalidate_caches().

One thing I believe it might be possible is to skip a flush if a flush
is already happening, but it would be a micro-optimization.

Best regards,
- Maíra

Quoted 87 lines

Iago

Quoted 81 lines · Maíra Canal

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL
register and poll its status bits. The mutex cache_clean_lock exists
to
serialize them, but v3d_clean_caches() only took the lock around its
final
FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the
BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the
CACHE_CLEAN queue, and each queue's scheduler uses its own ordered
workqueue, so their run_job callbacks execute in parallel.

Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering
the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.

Hold cache_clean_lock across the entire L2TCACTL access sequence so
it
is fully mutually exclusive with v3d_flush_l2t(), which already takes
the
lock around its own write.

Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush before

cleaning caches")

---
  drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---

  1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +0 -0
index c43d9af41374..e597b6fd47c4 100644
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)
   	struct drm_device *dev = &v3d->drm;
   	int core = 0;

drivers/gpu/drm/v3d/v3d_gem.c +2 -0
+	guard(mutex)(&v3d->cache_clean_lock);
+

   trace_v3d_cache_clean_begin(dev);

   /* GFXH-1897: Ensure pending flushes complete before writing
L2TCACTL */

drivers/gpu/drm/v3d/v3d_gem.c +0 -0
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)

   drm_err(dev, "Timeout waiting for TMU write combiner
flush\n");
   }

drivers/gpu/drm/v3d/v3d_gem.c +0 -1
-	mutex_lock(&v3d->cache_clean_lock);

   V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL, V3D_L2TCACTL_L2TFLS | V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN, V3D_L2TCACTL_FLM));

drivers/gpu/drm/v3d/v3d_gem.c +0 -0
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
   		drm_err(dev, "Timeout waiting for L2T clean\n");
   	}

drivers/gpu/drm/v3d/v3d_gem.c +0 -2
-	mutex_unlock(&v3d->cache_clean_lock);
-

   trace_v3d_cache_clean_end(dev);
  }

Reviewed-by
#4 lore raw

Hi Maíra, Iago,

Finally, this is not only theoretical. On Raspberry Pi 4 the race causes order-dependent dEQP-VK failures in robustness *.vert and
buffer_device_address: TMU writes from a job are still in flight when its fence signals. I reached an equivalent soution to address this issue:

- 51-case robustness list: 41/1000 runs failing → 0/1000
- full robustness vertex list: 7/8 → 0/8
- full buffer_device_address: 3/8 → 0/8

The impact on Pi 5 on GfxBench GL-ES and Vulkan (Aztec, Manhattan, T-Rex, Egypt) stay within ±0.1%, which is run-to-run noise.

Quoted 115 lines · Iago Toral

Thanks, if we don't see any impact on performance on Pi5, this is:

Quoted 109 lines · Maíra Canal

Hi Iago,

Quoted 3 lines · Iago Toral

Hi Maíra,

has this change fixed anything?

Not real bug, just the theoretical race-condition between v3d_clean_caches() and v3d_flush_l2t().

Quoted 3 lines

Also, I believe that GFXH-1897 is exclusive to Pi4. Have you tested if
this change has any visible performance impact on Pi5?

I'll test it, however I don't imagine it would have a impact, as the race condition should be rare enough. We would need a CACHE_CLEAN job running simultaneously to v3d_invalidate_caches().

One thing I believe it might be possible is to skip a flush if a flush
is already happening, but it would be a micro-optimization.

Best regards,
- Maíra

Quoted 85 lines

Iago

Quoted 81 lines · Maíra Canal

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL
register and poll its status bits. The mutex cache_clean_lock exists
to
serialize them, but v3d_clean_caches() only took the lock around its
final
FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the
BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the
CACHE_CLEAN queue, and each queue's scheduler uses its own ordered
workqueue, so their run_job callbacks execute in parallel.

Because clean locked only its final write, a concurrent flush can write
L2TCACTL during clean's unlocked phase. Both use non read-modify- write
writes to the one register, so whichever lands last wins: clean's TMUWCF
write can land on the flush's in-flight L2TFLS invalidate, triggering
the
GFXH-1897 hazard of writing L2TCACTL while a flush is pending.

Hold cache_clean_lock across the entire L2TCACTL access sequence so
it
is fully mutually exclusive with v3d_flush_l2t(), which already takes
the
lock around its own write.

Fixes abf888b03a98 ("drm/v3d: Wait for pending L2T flush before

cleaning caches")

---
   drivers/gpu/drm/v3d/v3d_gem.c | 5 ++---
   1 file changed, 2 insertions(+), 3 deletions(-)
drivers/gpu/drm/v3d/v3d_gem.c +2 -0
index c43d9af41374..e597b6fd47c4 100644
@@ -204,6 +204,8 @@ v3d_clean_caches(struct v3d_dev *v3d)
   	struct drm_device *dev = &v3d->drm;
   	int core = 0;
 
+	guard(mutex)(&v3d->cache_clean_lock);
+
   	trace_v3d_cache_clean_begin(dev);
 
   	/* GFXH-1897: Ensure pending flushes complete before

writing
L2TCACTL */

drivers/gpu/drm/v3d/v3d_gem.c +0 -0
@@ -220,7 +222,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
   		drm_err(dev, "Timeout waiting for TMU write

combiner
flush\n");
   }

drivers/gpu/drm/v3d/v3d_gem.c +0 -1
-	mutex_lock(&v3d->cache_clean_lock);
   	V3D_CORE_WRITE(core, V3D_CTL_L2TCACTL,
   		       V3D_L2TCACTL_L2TFLS |
   		       V3D_SET_FIELD(V3D_L2TCACTL_FLM_CLEAN,

V3D_L2TCACTL_FLM));

drivers/gpu/drm/v3d/v3d_gem.c +0 -2
@@ -230,8 +231,6 @@ v3d_clean_caches(struct v3d_dev *v3d)
   		drm_err(dev, "Timeout waiting for L2T clean\n");
   	}
 
-	mutex_unlock(&v3d->cache_clean_lock);
-
   	trace_v3d_cache_clean_end(dev);
   }
 
author maintainer Applied
#5 lore raw
In reply to Maíra Canal
Quoted 11 lines · Maíra Canal

v3d_clean_caches() and v3d_flush_l2t() both write the single L2TCACTL register and poll its status bits. The mutex cache_clean_lock exists to serialize them, but v3d_clean_caches() only took the lock around its final FLM_CLEAN write.

These functions run concurrently: v3d_flush_l2t() is issued from the BIN/RENDER/CSD invalidate path while v3d_clean_caches() runs from the CACHE_CLEAN queue, and each queue's scheduler uses its own ordered workqueue, so their run_job callbacks execute in parallel.

[...]

Applied, thanks!

[1/1] drm/v3d: Widen cache_clean_lock over the whole L2TCACTL sequence

      commit: 994ba6e5c87d6a3d471ea987fe8622168f092334

Best regards,
- Maíra