Skip to contentSkip to conversation

drm/xe: Use separate rpm lockdep map for non-d3cold-capable devices

PATCH Thomas Hellström 3 participants 6 messages Aug 23, 2024 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hi, Rodrigo.

Quoted 4 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 48 lines

but a few more comments below...

Quoted 8 lines

For non-d3cold-capable devices we'd like to be able to wake up the device from reclaim. In particular, for Lunar Lake we'd like to be able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Therefore use a separate lockdep map for such devices and prime it reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 25 lines

---
 drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++---
----

 1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +0 -0
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
   */


 #ifdef CONFIG_LOCKDEP

drivers/gpu/drm/xe/xe_pm.c +6 -2
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"
+static struct lockdep_map xe_pm_runtime_d3cold_map = {
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
Quoted 4 lines

 };
 #endif

+static void xe_pm_runtime_acquire(const struct xe_device *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 77 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 70 lines
drivers/gpu/drm/xe/xe_pm.c +13 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device *xe)
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+

 /**
  * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0-

Quoted 1 line

S2idle

  * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)
  	 * annotation here and in xe_pm_runtime_get() lockdep will

see
  * the potential lock inversion and give us a nice splat. */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  /*
  * Applying lock for entire list op as xe_ttm_bo_destroy and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)

 out:
  if (err)
  xe_display_pm_resume(xe, true);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe)
  	/* Disable access_ongoing asserts and prevent recursive pm

calls */
  xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  if (xe->d3cold.allowed) {
  err = xe_pcode_ready(xe, true);

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe)
  			goto out;
  	}

 out:

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +8 -2
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device *xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under reclaim. */
Quoted 8 lines

drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device *xe) · line 491

Quoted 5 lines
drivers/gpu/drm/xe/xe_pm.c +5 -0
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?

Hm. I forgot to check when this function was called. I was assuming it was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but it seems to be called on each runtime_get(). Is that intentional?

/Thomas

Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0 -0
  }


 /**

Signature · 1 line
2.44.0
#3 lore raw
Quoted 164 lines · Thomas Hellström

Hi, Rodrigo.

Quoted 4 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 53 lines

but a few more comments below...

Quoted 11 lines

For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 27 lines

Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
 drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- --
----

 1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +0 -0
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
   */


 #ifdef CONFIG_LOCKDEP

drivers/gpu/drm/xe/xe_pm.c +6 -2
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"
+static struct lockdep_map xe_pm_runtime_d3cold_map = {
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
Quoted 4 lines

 };
 #endif

+static void xe_pm_runtime_acquire(const struct xe_device *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 86 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 79 lines
drivers/gpu/drm/xe/xe_pm.c +13 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device *xe)
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+

 /**
  * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0-

Quoted 1 line

S2idle

  * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)
  	 * annotation here and in xe_pm_runtime_get() lockdep

will
see
  * the potential lock inversion and give us a nice splat.
  */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  /*
  * Applying lock for entire list op as xe_ttm_bo_destroy and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)

 out:
  if (err)
  xe_display_pm_resume(xe, true);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  	/* Disable access_ongoing asserts and prevent recursive

pm
calls */
  xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  if (xe->d3cold.allowed) {
  err = xe_pcode_ready(xe, true);

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  			goto out;
  	}

 out:

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +8 -2
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under reclaim.
 */
Quoted 8 lines

drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493

Quoted 5 lines
drivers/gpu/drm/xe/xe_pm.c +5 -0
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?

Quoted 6 lines

Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on each runtime_get(). Is that intentional?

Otherwise I'd like to change that to be called at module init?

/Thomas

Quoted 12 lines

/Thomas

Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0 -0
  }


 /**

Signature · 1 line
2.44.0
#4 lore raw
Quoted 165 lines · Thomas Hellström
Quoted 164 lines · Thomas Hellström

Hi, Rodrigo.

Quoted 4 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 53 lines

but a few more comments below...

Quoted 11 lines

For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 27 lines

Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
 drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- --
----

 1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +0 -0
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
   */


 #ifdef CONFIG_LOCKDEP

drivers/gpu/drm/xe/xe_pm.c +6 -2
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"
+static struct lockdep_map xe_pm_runtime_d3cold_map = {
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
Quoted 4 lines

 };
 #endif

+static void xe_pm_runtime_acquire(const struct xe_device *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 94 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 74 lines
drivers/gpu/drm/xe/xe_pm.c +13 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device *xe)
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+

 /**
  * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0-

Quoted 1 line

S2idle

  * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)
  	 * annotation here and in xe_pm_runtime_get() lockdep

will
see
  * the potential lock inversion and give us a nice splat.
  */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  /*
  * Applying lock for entire list op as xe_ttm_bo_destroy and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)

 out:
  if (err)
  xe_display_pm_resume(xe, true);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  	/* Disable access_ongoing asserts and prevent recursive

pm
calls */
  xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


  if (xe->d3cold.allowed) {
  err = xe_pcode_ready(xe, true);

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  			goto out;
  	}

 out:

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

  xe_pm_write_callback_task(xe, NULL);
  return err;
 }

drivers/gpu/drm/xe/xe_pm.c +4 -2
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);

drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493

Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +9 -0
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under reclaim.
 */
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?

Quoted 8 lines
Quoted 6 lines

Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on each runtime_get(). Is that intentional?

Otherwise I'd like to change that to be called at module init?

Probably my mistake when refactoring it. I believe it makes sense to move that to the init function and then keep as it is now here.

Quoted 18 lines

/Thomas

Quoted 12 lines

/Thomas

Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0 -0
  }


 /**

Signature · 1 line
2.44.0
#5 lore raw
In reply to Thomas Hellström
Quoted 165 lines · Thomas Hellström
Quoted 164 lines · Thomas Hellström

Hi, Rodrigo.

Quoted 4 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 53 lines

but a few more comments below...

Quoted 11 lines

For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 21 lines

Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>

---
  drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++-
--
----
  1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +0 -2
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
   */
 
  #ifdef CONFIG_LOCKDEP
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"

drivers/gpu/drm/xe/xe_pm.c line 82

Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +7 -0
+static struct lockdep_map xe_pm_runtime_d3cold_map = {
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
  };
  #endif
 
+static void xe_pm_runtime_acquire(const struct xe_device *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 94 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 74 lines
drivers/gpu/drm/xe/xe_pm.c +13 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device *xe)
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
  /**
   * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0-
Quoted 1 line

S2idle

  * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)
  	 * annotation here and in xe_pm_runtime_get() lockdep

will
see
  * the potential lock inversion and give us a nice
splat.
  */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);
 
  	/*
  	 * Applying lock for entire list op as xe_ttm_bo_destroy

and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +1 -1
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device
 *xe)
  out:
  	if (err)
  		xe_display_pm_resume(xe, true);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);
  	xe_pm_write_callback_task(xe, NULL);
  	return err;

 }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  	/* Disable access_ongoing asserts and prevent recursive

pm
calls */
  xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +2 -2
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);
 
  	if (xe->d3cold.allowed) {
  		err = xe_pcode_ready(xe, true);
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
  			goto out;
  	}
  out:
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);
  	xe_pm_write_callback_task(xe, NULL);
  	return err;

 }

drivers/gpu/drm/xe/xe_pm.c +4 -2
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);

drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493

Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +9 -0
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under reclaim.
 */
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?

Quoted 8 lines
Quoted 6 lines

Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on each runtime_get(). Is that intentional?

Otherwise I'd like to change that to be called at module init?

hmm, don't we want to record the locks that are held when calling into the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.

Quoted 18 lines

/Thomas

Quoted 12 lines

/Thomas

Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0 -0
  }

 /**

Signature · 1 line
2.44.0
#6 lore raw
Quoted 193 lines · Matthew Auld
Quoted 192 lines · Thomas Hellström
Quoted 190 lines · Thomas Hellström

Hi, Rodrigo.

Quoted 5 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 59 lines

but a few more comments below...

Quoted 11 lines

For non-d3cold-capable devices we'd like to be able to wake up
the
device from reclaim. In particular, for Lunar Lake we'd like to
be
able to blit CCS metadata to system at shrink time; at least from
kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Quoted 4 lines

Therefore use a separate lockdep map for such devices and prime
it
reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 28 lines

Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>
---
  drivers/gpu/drm/xe/xe_pm.c | 45
+++++++++++++++++++++++++++++++-
--
----

  1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +0 -0
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
    */


  #ifdef CONFIG_LOCKDEP

drivers/gpu/drm/xe/xe_pm.c +6 -2
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"
+static struct lockdep_map xe_pm_runtime_d3cold_map = {
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
Quoted 5 lines

  };
  #endif

+static void xe_pm_runtime_acquire(const struct xe_device *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 96 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 89 lines
drivers/gpu/drm/xe/xe_pm.c +7 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device

*xe)

drivers/gpu/drm/xe/xe_pm.c +6 -0
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+

  /**
   * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0-

Quoted 1 line

S2idle

   * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct

xe_device
*xe)
   * annotation here and in xe_pm_runtime_get() lockdep
will
see
   * the potential lock inversion and give us a nice splat.
   */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


   /*
   * Applying lock for entire list op as
xe_ttm_bo_destroy
and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct

xe_device
*xe)
  out:
   if (err)
   xe_display_pm_resume(xe, true);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

   xe_pm_write_callback_task(xe, NULL);
   return err;
  }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   	/* Disable access_ongoing asserts and prevent

recursive
pm
calls */
   xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);


   if (xe->d3cold.allowed) {
   err = xe_pcode_ready(xe, true);

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   			goto out;
   	}

  out:

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);

   xe_pm_write_callback_task(xe, NULL);
   return err;
  }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct

xe_device
*xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {

drivers/gpu/drm/xe/xe_pm.c +8 -2
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under

reclaim.
*/

drivers/gpu/drm/xe/xe_pm.c +1 -0
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
Quoted 8 lines

drivers/gpu/drm/xe/xe_pm.c

Quoted 4 lines
drivers/gpu/drm/xe/xe_pm.c +4 -0
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable
here could have 2 different smaller sequences?

Hm. I forgot to check when this function was called. I was assuming
it
was called only once per driver instance and in that case we need it
all since we can have multiple devices with different capabilities,
but
it seems to be called on each runtime_get(). Is that intentional?

Quoted 6 lines
Quoted 1 line

Otherwise I'd like to change that to be called at module init?

hmm, don't we want to record the locks that are held when calling into
the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.

Ah, yes, so it's more of a might_lock() type of function rather than priming? I should have noticed since it really didn't explicitly order any locks....

Would it be ok then If I added a new module_init prime function that looks like the prime function in the patch, and rename the existing prime to something like

xe_might_enter_rpm_callback()?

/Thomas

Quoted 19 lines
Quoted 19 lines

/Thomas

Quoted 12 lines

/Thomas

Quoted 7 lines
Quoted 7 lines

  }

  /**

Signature · 1 line
2.44.0
#7 lore raw
Quoted 205 lines · Thomas Hellström
Quoted 193 lines · Matthew Auld
Quoted 192 lines · Thomas Hellström
Quoted 190 lines · Thomas Hellström

Hi, Rodrigo.

Quoted 5 lines · Rodrigo Vivi

On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:

Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)

You're right. Well it's friday afternoon here...

Quoted 59 lines

but a few more comments below...

Quoted 11 lines

For non-d3cold-capable devices we'd like to be able to wake up
the
device from reclaim. In particular, for Lunar Lake we'd like to
be
able to blit CCS metadata to system at shrink time; at least from
kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.

Quoted 4 lines

Therefore use a separate lockdep map for such devices and prime
it
reclaim-tainted.

Cc Matthew Auld <[email protected]>
Quoted 23 lines

Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>

---
   drivers/gpu/drm/xe/xe_pm.c | 45
+++++++++++++++++++++++++++++++-
--
----
   1 file changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_pm.c +1 -2
index 9f3c14fd9f33..2e9fdb5da8bb 100644
@@ -70,11 +70,29 @@
    */
 
   #ifdef CONFIG_LOCKDEP
-static struct lockdep_map xe_pm_runtime_lockdep_map = {
-	.name = "xe_pm_runtime_lockdep_map"
+static struct lockdep_map xe_pm_runtime_d3cold_map = {

drivers/gpu/drm/xe/xe_pm.c line 83

Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +6 -0
+	.name = "xe_rpm_d3cold_map"
+};
+
+static struct lockdep_map xe_pm_runtime_nod3cold_map = {
+	.name = "xe_rpm_nod3cold_map"
   };
   #endif
 
+static void xe_pm_runtime_acquire(const struct xe_device
 *xe)

I believe this should have a different name. runtime_acquire and runtime_release sounds like runtime_get and runtime_put...

Quoted 3 lines

since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
and xe_rpm_lockmap_release

Sure, I'll fix

Quoted 96 lines

or
d3_lockmap_acquire
d3_lockmap_release ?

or something like that...

Quoted 89 lines
drivers/gpu/drm/xe/xe_pm.c +13 -0
+{
+	lock_map_acquire(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
+static void xe_pm_runtime_release(const struct xe_device
 *xe)
+{
+	lock_map_release(xe->d3cold.capable ?
+			 &xe_pm_runtime_d3cold_map :
+			 &xe_pm_runtime_nod3cold_map);
+}
+
   /**

   * xe_pm_suspend - Helper for System suspend, i.e. S0->S3 /
S0-

Quoted 1 line

S2idle

   * @xe: xe device instance

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct

xe_device
*xe)
   * annotation here and in xe_pm_runtime_get()
lockdep
will
see
   * the potential lock inversion and give us a nice
splat.
   */

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);
 
   	/*
   	 * Applying lock for entire list op as

xe_ttm_bo_destroy
and xe_bo_move_notify

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct

xe_device
*xe)
  out:
   if (err)
   xe_display_pm_resume(xe, true);

drivers/gpu/drm/xe/xe_pm.c +1 -1
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);
   	xe_pm_write_callback_task(xe, NULL);
   	return err;
   }
@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   	/* Disable access_ongoing asserts and prevent

recursive
pm
calls */
   xe_pm_write_callback_task(xe, current);

drivers/gpu/drm/xe/xe_pm.c +2 -2
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_acquire(xe);
 
   	if (xe->d3cold.allowed) {
   		err = xe_pcode_ready(xe, true);
@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device
 *xe)
   			goto out;
   	}
   out:
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_runtime_release(xe);
   	xe_pm_write_callback_task(xe, NULL);
   	return err;

  }

drivers/gpu/drm/xe/xe_pm.c +0 -0
@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct

xe_device
*xe)
   */
  static void pm_runtime_lockdep_prime(void)
  {

drivers/gpu/drm/xe/xe_pm.c +8 -2
-	lock_map_acquire(&xe_pm_runtime_lockdep_map);
-	lock_map_release(&xe_pm_runtime_lockdep_map);
+	struct dma_resv lockdep_resv;
+
+	dma_resv_init(&lockdep_resv);
+	lock_map_acquire(&xe_pm_runtime_d3cold_map);
+	/* D3Cold takes the dma_resv locks to evict bos */
+	dma_resv_lock(&lockdep_resv, NULL);
+	fs_reclaim_acquire(GFP_KERNEL);
+	/* Shrinkers like to wake up the device under

reclaim.
*/

drivers/gpu/drm/xe/xe_pm.c +1 -0
+	lock_map_acquire(&xe_pm_runtime_nod3cold_map);
Quoted 8 lines

drivers/gpu/drm/xe/xe_pm.c

Quoted 4 lines
drivers/gpu/drm/xe/xe_pm.c +4 -0
+	lock_map_release(&xe_pm_runtime_nod3cold_map);
+	fs_reclaim_release(GFP_KERNEL);
+	dma_resv_unlock(&lockdep_resv);
+	lock_map_release(&xe_pm_runtime_d3cold_map);

do we really need this entire sequence? or checking for d3capable
here could have 2 different smaller sequences?

Hm. I forgot to check when this function was called. I was assuming
it
was called only once per driver instance and in that case we need it
all since we can have multiple devices with different capabilities,
but
it seems to be called on each runtime_get(). Is that intentional?

Quoted 6 lines
Quoted 1 line

Otherwise I'd like to change that to be called at module init?

hmm, don't we want to record the locks that are held when calling into
the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.

Ah, yes, so it's more of a might_lock() type of function rather than priming? I should have noticed since it really didn't explicitly order any locks....

Quoted 5 lines

Would it be ok then If I added a new module_init prime function that looks like the prime function in the patch, and rename the existing prime to something like

xe_might_enter_rpm_callback()?

Sure, that sounds better than prime.

Quoted 27 lines

/Thomas

Quoted 19 lines
Quoted 19 lines

/Thomas

Quoted 12 lines

/Thomas

Quoted 7 lines
Quoted 7 lines

  }

  /**

Signature · 1 line
2.44.0