drm/xe: Use separate rpm lockdep map for non-d3cold-capable devices
Hi, Rodrigo.
Quoted 4 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 48 lines
but a few more comments below...
Quoted 8 lines
For non-d3cold-capable devices we'd like to be able to wake up the device from reclaim. In particular, for Lunar Lake we'd like to be able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.
Therefore use a separate lockdep map for such devices and prime it reclaim-tainted.
Cc Matthew Auld <[email protected]>Quoted 25 lines
Cc "Vivi, Rodrigo" <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++---
----1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -70,11 +70,29 @@*/
#ifdefCONFIG_LOCKDEPdrivers/gpu/drm/xe/xe_pm.c +6
-2-static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name ="xe_pm_runtime_lockdep_map" +static struct lockdep_map xe_pm_runtime_d3cold_map = { + .name ="xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" Quoted 4 lines
};
#endif
+static voidxe_pm_runtime_acquire(const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...
Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_release
Sure, I'll fix
Quoted 77 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 70 lines
drivers/gpu/drm/xe/xe_pm.c +13
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release (const struct xe_device *xe) +{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} +/**
*xe_pm_suspend- Helper for System suspend, i.e.S0->S3/ S0-Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) * annotation here and in xe_pm_runtime_get() lockdep willsee
* the potential lock inversion and give us a nice splat. */drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
/*
* Applying lock for entire list op asxe_ttm_bo_destroyandxe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +0
-0@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)out:
if (err)
xe_display_pm_resume(xe, true);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and prevent recursive pmcalls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
if (xe->d3cold.allowed) {
err =xe_pcode_ready(xe, true);drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; }out:
drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +8
-2@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device *xe)*/ static void pm_runtime_lockdep_prime(void) { -lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map); +/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim. */
Quoted 8 lines
drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device *xe) · line 491
Quoted 5 lines
drivers/gpu/drm/xe/xe_pm.c +5
-0+lock_map_acquire (&xe_pm_runtime_nod3cold_map); +lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?
Hm. I forgot to check when this function was called. I was assuming it was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but it seems to be called on each runtime_get(). Is that intentional?
/Thomas
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0
-0}
/**Signature · 1 line
2.44.0
Quoted 164 lines · Thomas Hellström
Hi, Rodrigo.
Quoted 4 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 53 lines
but a few more comments below...
Quoted 11 lines
For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.Cc Matthew Auld <[email protected]>Quoted 27 lines
Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- --
----1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -70,11 +70,29 @@*/
#ifdefCONFIG_LOCKDEPdrivers/gpu/drm/xe/xe_pm.c +6
-2-static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name ="xe_pm_runtime_lockdep_map" +static struct lockdep_map xe_pm_runtime_d3cold_map = { + .name ="xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" Quoted 4 lines
};
#endif
+static voidxe_pm_runtime_acquire(const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_releaseSure, I'll fix
Quoted 86 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 79 lines
drivers/gpu/drm/xe/xe_pm.c +13
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release (const struct xe_device *xe) +{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} +/**
*xe_pm_suspend- Helper for System suspend, i.e.S0->S3/ S0-Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) * annotation here and in xe_pm_runtime_get() lockdepwill
see
* the potential lock inversion and give us a nice splat.
*/drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
/*
* Applying lock for entire list op asxe_ttm_bo_destroyandxe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +0
-0@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)out:
if (err)
xe_display_pm_resume(xe, true);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and prevent recursivepm
calls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
if (xe->d3cold.allowed) {
err =xe_pcode_ready(xe, true);drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; }out:
drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +8
-2@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device*xe) */ static void pm_runtime_lockdep_prime(void) { -lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map); +/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim. */ Quoted 8 lines
drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493
Quoted 5 lines
drivers/gpu/drm/xe/xe_pm.c +5
-0+lock_map_acquire (&xe_pm_runtime_nod3cold_map); +lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?
Quoted 6 lines
Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on eachruntime_get(). Is that intentional?
Otherwise I'd like to change that to be called at module init?
/Thomas
Quoted 12 lines
/Thomas
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0
-0}
/**Signature · 1 line
2.44.0
Quoted 165 lines · Thomas Hellström
Quoted 164 lines · Thomas Hellström
Hi, Rodrigo.
Quoted 4 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 53 lines
but a few more comments below...
Quoted 11 lines
For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.Cc Matthew Auld <[email protected]>Quoted 27 lines
Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- --
----1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -70,11 +70,29 @@*/
#ifdefCONFIG_LOCKDEPdrivers/gpu/drm/xe/xe_pm.c +6
-2-static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name ="xe_pm_runtime_lockdep_map" +static struct lockdep_map xe_pm_runtime_d3cold_map = { + .name ="xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" Quoted 4 lines
};
#endif
+static voidxe_pm_runtime_acquire(const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_releaseSure, I'll fix
Quoted 94 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 74 lines
drivers/gpu/drm/xe/xe_pm.c +13
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release (const struct xe_device *xe) +{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} +/**
*xe_pm_suspend- Helper for System suspend, i.e.S0->S3/ S0-Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) * annotation here and in xe_pm_runtime_get() lockdepwill
see
* the potential lock inversion and give us a nice splat.
*/drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
/*
* Applying lock for entire list op asxe_ttm_bo_destroyandxe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +0
-0@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)out:
if (err)
xe_display_pm_resume(xe, true);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and prevent recursivepm
calls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
if (xe->d3cold.allowed) {
err =xe_pcode_ready(xe, true);drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; }out:
drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +4
-2@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device*xe) */ static void pm_runtime_lockdep_prime(void) { -lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map);drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493
Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +9
-0+/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim. */ +lock_map_acquire (&xe_pm_runtime_nod3cold_map); +lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?
Quoted 8 lines
Quoted 6 lines
Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on eachruntime_get(). Is that intentional?Otherwise I'd like to change that to be called at module init?
Probably my mistake when refactoring it. I believe it makes sense to move that to the init function and then keep as it is now here.
Quoted 18 lines
/Thomas
Quoted 12 lines
/Thomas
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0
-0}
/**Signature · 1 line
2.44.0
Quoted 165 lines · Thomas Hellström
Quoted 164 lines · Thomas Hellström
Hi, Rodrigo.
Quoted 4 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 53 lines
but a few more comments below...
Quoted 11 lines
For non-d3cold-capable devices we'd like to be able to wake up the
device from reclaim. In particular, for Lunar Lake we'd like to be
able to blit CCS metadata to system at shrink time; at least from kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.Therefore use a separate lockdep map for such devices and prime it
reclaim-tainted.Cc Matthew Auld <[email protected]>Quoted 21 lines
Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>--- drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- -- ---- 1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +0
-2@@ -70,11 +70,29 @@*/ #ifdef CONFIG_LOCKDEP -static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name = "xe_pm_runtime_lockdep_map"drivers/gpu/drm/xe/xe_pm.c line 82
Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +7
-0+static struct lockdep_map xe_pm_runtime_d3cold_map = { + .name ="xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" }; #endif +static void xe_pm_runtime_acquire (const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_releaseSure, I'll fix
Quoted 94 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 74 lines
drivers/gpu/drm/xe/xe_pm.c +13
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release (const struct xe_device *xe) +{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + /*** xe_pm_suspend - Helper for System suspend, i.e. S0->S3 / S0- Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) * annotation here and in xe_pm_runtime_get() lockdepwill
see
* the potential lock inversion and give us a nice
splat.
*/drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe); /** Applying lock for entire list op as xe_ttm_bo_destroy and
xe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +1
-1@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) out: if (err) xe_display_pm_resume(xe, true); -lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe); xe_pm_write_callback_task(xe, NULL); return err;}
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and prevent recursivepm
calls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +2
-2-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe); if (xe->d3cold.allowed) { err = xe_pcode_ready(xe, true);@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; } out: -lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe); xe_pm_write_callback_task(xe, NULL); return err;}
drivers/gpu/drm/xe/xe_pm.c +4
-2@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct xe_device*xe) */ static void pm_runtime_lockdep_prime(void) { -lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map);drivers/gpu/drm/xe/xe_pm.c int xe_pm_runtime_resume(struct xe_device · line 493
Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +9
-0+/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim. */ +lock_map_acquire (&xe_pm_runtime_nod3cold_map); +lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable here could have 2 different smaller sequences?
Quoted 8 lines
Quoted 6 lines
Hm. I forgot to check when this function was called. I was assuming it
was called only once per driver instance and in that case we need it all since we can have multiple devices with different capabilities, but
it seems to be called on eachruntime_get(). Is that intentional?Otherwise I'd like to change that to be called at module init?
hmm, don't we want to record the locks that are held when calling into the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.
Quoted 18 lines
/Thomas
Quoted 12 lines
/Thomas
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pm.c +0
-0}/**
Signature · 1 line
2.44.0
Quoted 193 lines · Matthew Auld
Quoted 192 lines · Thomas Hellström
Quoted 190 lines · Thomas Hellström
Hi, Rodrigo.
Quoted 5 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 59 lines
but a few more comments below...
Quoted 11 lines
For non-d3cold-capable devices we'd like to be able to wake up
the
device from reclaim. In particular, for Lunar Lake we'd like to
be
able to blit CCS metadata to system at shrink time; at least from
kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.Quoted 4 lines
Therefore use a separate lockdep map for such devices and prime
it
reclaim-tainted.Cc Matthew Auld <[email protected]>Quoted 28 lines
Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>
---
drivers/gpu/drm/xe/xe_pm.c | 45
+++++++++++++++++++++++++++++++-
--
----1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -70,11 +70,29 @@*/
#ifdefCONFIG_LOCKDEPdrivers/gpu/drm/xe/xe_pm.c +6
-2-static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name ="xe_pm_runtime_lockdep_map" +static struct lockdep_map xe_pm_runtime_d3cold_map = { + .name ="xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" Quoted 5 lines
};
#endif
+static voidxe_pm_runtime_acquire(const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_releaseSure, I'll fix
Quoted 96 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 89 lines
drivers/gpu/drm/xe/xe_pm.c +7
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release(const struct xe_device*xe)
drivers/gpu/drm/xe/xe_pm.c +6
-0+{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} +/**
*xe_pm_suspend- Helper for System suspend, i.e.S0->S3/ S0-Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct
xe_device
*xe)
* annotation here and inxe_pm_runtime_get()lockdep
will
see
* the potential lock inversion and give us a nice splat.
*/drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
/*
* Applying lock for entire list op asxe_ttm_bo_destroy
andxe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +0
-0@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct
xe_device
*xe)
out:
if (err)
xe_display_pm_resume(xe, true);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and preventrecursive
pm
calls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe);
if (xe->d3cold.allowed) {
err =xe_pcode_ready(xe, true);drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; }out:
drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe);
xe_pm_write_callback_task(xe, NULL);
return err;
}drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct
xe_device
*xe)
*/
static voidpm_runtime_lockdep_prime(void)
{drivers/gpu/drm/xe/xe_pm.c +8
-2-lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map); +/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim.
*/drivers/gpu/drm/xe/xe_pm.c +1
-0+lock_map_acquire (&xe_pm_runtime_nod3cold_map);Quoted 8 lines
drivers/gpu/drm/xe/xe_pm.c
Quoted 4 lines
drivers/gpu/drm/xe/xe_pm.c +4
-0+lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable
here could have 2 different smaller sequences?Hm. I forgot to check when this function was called. I was assuming
it
was called only once per driver instance and in that case we need it
all since we can have multiple devices with different capabilities,
but
it seems to be called on eachruntime_get(). Is that intentional?
Quoted 6 lines
Quoted 1 line
Otherwise I'd like to change that to be called at module init?
hmm, don't we want to record the locks that are held when calling into
the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.
Ah, yes, so it's more of a might_lock() type of function rather than priming? I should have noticed since it really didn't explicitly order any locks....
Would it be ok then If I added a new module_init prime function that looks like the prime function in the patch, and rename the existing prime to something like
xe_might_enter_rpm_callback()?
/Thomas
Quoted 19 lines
Quoted 19 lines
/Thomas
Quoted 12 lines
/Thomas
Quoted 7 lines
Quoted 7 lines
}
/**Signature · 1 line
2.44.0
Quoted 205 lines · Thomas Hellström
Quoted 193 lines · Matthew Auld
Quoted 192 lines · Thomas Hellström
Quoted 190 lines · Thomas Hellström
Hi, Rodrigo.
Quoted 5 lines · Rodrigo Vivi
On Fri, Aug 23, 2024 at 03:59:06PM +0200, Thomas Hellström wrote:
Hi Thomas, first of all, please notice that you used the wrong address from our mailing list ;)
You're right. Well it's friday afternoon here...
Quoted 59 lines
but a few more comments below...
Quoted 11 lines
For non-d3cold-capable devices we'd like to be able to wake up
the
device from reclaim. In particular, for Lunar Lake we'd like to
be
able to blit CCS metadata to system at shrink time; at least from
kswapd where it's reasonable OK to wait for rpm resume and a preceding rpm suspend.Quoted 4 lines
Therefore use a separate lockdep map for such devices and prime
it
reclaim-tainted.Cc Matthew Auld <[email protected]>Quoted 23 lines
Cc: "Vivi, Rodrigo" <[email protected]>
Signed-off-by: Thomas Hellström
<[email protected]>--- drivers/gpu/drm/xe/xe_pm.c | 45 +++++++++++++++++++++++++++++++- -- ---- 1 file changed, 37 insertions(+), 8 deletions(-)drivers/gpu/drm/xe/xe_pm.c +1
-2@@ -70,11 +70,29 @@*/ #ifdef CONFIG_LOCKDEP -static struct lockdep_map xe_pm_runtime_lockdep_map = { - .name = "xe_pm_runtime_lockdep_map" +static struct lockdep_map xe_pm_runtime_d3cold_map = {drivers/gpu/drm/xe/xe_pm.c line 83
Quoted 10 lines
drivers/gpu/drm/xe/xe_pm.c +6
-0+ .name = "xe_rpm_d3cold_map" +}; + +static struct lockdep_map xe_pm_runtime_nod3cold_map = { + .name ="xe_rpm_nod3cold_map" }; #endif +static void xe_pm_runtime_acquire (const struct xe_device *xe)I believe this should have a different name.
runtime_acquireandruntime_releasesounds likeruntime_getandruntime_put...Quoted 3 lines
since it is static perhaps we should only call it
xe_rpm_lockmap_acquire
andxe_rpm_lockmap_releaseSure, I'll fix
Quoted 96 lines
or
d3_lockmap_acquired3_lockmap_release?or something like that...
Quoted 89 lines
drivers/gpu/drm/xe/xe_pm.c +13
-0+{ +lock_map_acquire (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + +static void xe_pm_runtime_release (const struct xe_device *xe) +{ +lock_map_release (xe->d3cold.capable ? + &xe_pm_runtime_d3cold_map : + &xe_pm_runtime_nod3cold_map); +} + /***
xe_pm_suspend- Helper for System suspend, i.e.S0->S3/
S0-Quoted 1 line
S2idle
* @xe: xe device instance
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -354,7 +372,7 @@ int xe_pm_runtime_suspend(struct
xe_device
*xe)
* annotation here and inxe_pm_runtime_get()
lockdep
will
see
* the potential lock inversion and give us a nice
splat.
*/drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe); /** Applying lock for entire list op as
xe_ttm_bo_destroy
andxe_bo_move_notifydrivers/gpu/drm/xe/xe_pm.c +0
-0@@ -386,7 +404,7 @@ int xe_pm_runtime_suspend(struct
xe_device
*xe)
out:
if (err)
xe_display_pm_resume(xe, true);drivers/gpu/drm/xe/xe_pm.c +1
-1-lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe); xe_pm_write_callback_task(xe, NULL); return err; }@@ -407,7 +425,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) /* Disable access_ongoing asserts and preventrecursive
pm
calls */
xe_pm_write_callback_task(xe, current);drivers/gpu/drm/xe/xe_pm.c +2
-2-lock_map_acquire (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_acquire (xe); if (xe->d3cold.allowed) { err = xe_pcode_ready(xe, true);@@ -437,7 +455,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) goto out; } out: -lock_map_release (&xe_pm_runtime_lockdep_map); +xe_pm_runtime_release (xe); xe_pm_write_callback_task(xe, NULL); return err;}
drivers/gpu/drm/xe/xe_pm.c +0
-0@@ -458,8 +476,19 @@ int xe_pm_runtime_resume(struct
xe_device
*xe)
*/
static voidpm_runtime_lockdep_prime(void)
{drivers/gpu/drm/xe/xe_pm.c +8
-2-lock_map_acquire (&xe_pm_runtime_lockdep_map); -lock_map_release (&xe_pm_runtime_lockdep_map); +struct dma_resv lockdep_resv; + +dma_resv_init (&lockdep_resv); +lock_map_acquire (&xe_pm_runtime_d3cold_map); +/* D3Cold takes the dma_resv locks to evict bos */ +dma_resv_lock (&lockdep_resv,NULL ); +fs_reclaim_acquire (GFP_KERNEL ); +/* Shrinkers like to wake up the device under reclaim.
*/drivers/gpu/drm/xe/xe_pm.c +1
-0+lock_map_acquire (&xe_pm_runtime_nod3cold_map);Quoted 8 lines
drivers/gpu/drm/xe/xe_pm.c
Quoted 4 lines
drivers/gpu/drm/xe/xe_pm.c +4
-0+lock_map_release (&xe_pm_runtime_nod3cold_map); +fs_reclaim_release (GFP_KERNEL ); +dma_resv_unlock (&lockdep_resv); +lock_map_release (&xe_pm_runtime_d3cold_map);do we really need this entire sequence? or checking for d3capable
here could have 2 different smaller sequences?Hm. I forgot to check when this function was called. I was assuming
it
was called only once per driver instance and in that case we need it
all since we can have multiple devices with different capabilities,
but
it seems to be called on eachruntime_get(). Is that intentional?Quoted 6 lines
Quoted 1 line
Otherwise I'd like to change that to be called at module init?
hmm, don't we want to record the locks that are held when calling into
the sync rpm get? We then make sure those locks are never grabbed in either of the callbacks. At least that was the original intention.Ah, yes, so it's more of a
might_lock()type of function rather than priming? I should have noticed since it really didn't explicitly order any locks....
Quoted 5 lines
Would it be ok then If I added a new
module_initprime function that looks like the prime function in the patch, and rename the existing prime to something like
xe_might_enter_rpm_callback()?
Sure, that sounds better than prime.
Quoted 27 lines
/Thomas
Quoted 19 lines
Quoted 19 lines
/Thomas
Quoted 12 lines
/Thomas
Quoted 7 lines
Quoted 7 lines
}
/**
Signature · 1 line
2.44.0