Skip to contentSkip to conversation

drm/xe/display: Fixes to make kexec work.

PATCH In review Maarten Lankhorst 4 participants 10 messages Aug 19, 2024 – Aug 20, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover drm/xe/display: Fixes to make kexec work.

Lets see if we can get away with performing a FLR as shutdown handler. It should work. These are all the patches I have used when testing if I could kexec from Xe with a game running to a new kernel.

Tested with:
kexec --reuse-cmdline --initrd=initrd.img -f vmlinuz

I have tested about 4-5 times successive kexecs on my ADL-P, while having a game actively running to keep the GPU loaded.

With this series xe kept running succesfully. After that I accidentally rebooted, but I see no reason this wouldn't work even more times.

Changes since previous version:
- s/r patches have been merged.

Maarten Lankhorst (3):
drm/xe: Read out rawclk_freq for display
drm/xe: Wire up device shutdown handler
CI-only: Test if FLR is disabled?

 drivers/gpu/drm/xe/display/xe_display.c | 22 +++++++++++--
 drivers/gpu/drm/xe/display/xe_display.h |  3 +-
 drivers/gpu/drm/xe/xe_device.c          | 43 +++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_gt.c              |  7 ++++
 drivers/gpu/drm/xe/xe_gt.h              |  1 +
 drivers/gpu/drm/xe/xe_pm.c              |  4 +--
 6 files changed, 69 insertions(+), 11 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

PATCH v2 1/3 drm/xe: Read out rawclk_freq for display

Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.

Fixes 44e694958b95 ("drm/xe/display: Implement display support")
Cc <[email protected]> # v6.8+
---
 drivers/gpu/drm/xe/display/xe_display.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/display/xe_display.c +3 -0
index 30dfdac9f6fa9..79add15c6c4c7 100644
@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)
 
 	intel_display_device_info_runtime_init(xe);
 
+	RUNTIME_INFO(xe)->rawclk_freq = intel_read_rawclk(xe);
+	drm_dbg(&xe->drm, "rawclk rate: %d kHz\n", RUNTIME_INFO(xe)->rawclk_freq);
+
 	err = intel_display_driver_probe_noirq(xe);
 	if (err) {
 		intel_opregion_cleanup(display);
Signature · 1 line
2.45.2
maintainer
#2 lore raw

Re: [PATCH v2 1/3] drm/xe: Read out rawclk_freq for display

Quoted 7 lines · Maarten Lankhorst

Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.

Fixes 44e694958b95 ("drm/xe/display: Implement display support")
Cc <[email protected]> # v6.8+

Please find another way. See [1]. I'm trying to clean up the whole RUNTIME_INFO() and rawclk_freq thing, and this makes it harder.

BR,
Jani.

[1] https://lore.kernel.org/r/ddd05f84ca4a6597133bee55ddf4ab593a16e99d.1717672515.git.jani.nikula@intel.com

Quoted 18 lines
---
 drivers/gpu/drm/xe/display/xe_display.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/display/xe_display.c +3 -0
index 30dfdac9f6fa9..79add15c6c4c7 100644
@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)
 
 	intel_display_device_info_runtime_init(xe);
 
+	RUNTIME_INFO(xe)->rawclk_freq = intel_read_rawclk(xe);
+	drm_dbg(&xe->drm, "rawclk rate: %d kHz\n", RUNTIME_INFO(xe)->rawclk_freq);
+
 	err = intel_display_driver_probe_noirq(xe);
 	if (err) {
 		intel_opregion_cleanup(display);
Signature · 1 line
Jani Nikula, Intel
maintainer
#3 lore raw

Re: [PATCH v2 1/3] drm/xe: Read out rawclk_freq for display

Quoted 11 lines · Jani Nikula
Quoted 7 lines · Maarten Lankhorst

Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.

Fixes 44e694958b95 ("drm/xe/display: Implement display support")
Cc <[email protected]> # v6.8+

Please find another way. See [1]. I'm trying to clean up the whole RUNTIME_INFO() and rawclk_freq thing, and this makes it harder.

Had another look, and brushed up my old patches, new version at [1].

BR,
Jani.

[1] https://lore.kernel.org/r/[email protected]

Quoted 25 lines

BR,
Jani.

[1] https://lore.kernel.org/r/ddd05f84ca4a6597133bee55ddf4ab593a16e99d.1717672515.git.jani.nikula@intel.com

Quoted 18 lines
---
 drivers/gpu/drm/xe/display/xe_display.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/display/xe_display.c +3 -0
index 30dfdac9f6fa9..79add15c6c4c7 100644
@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)
 
 	intel_display_device_info_runtime_init(xe);
 
+	RUNTIME_INFO(xe)->rawclk_freq = intel_read_rawclk(xe);
+	drm_dbg(&xe->drm, "rawclk rate: %d kHz\n", RUNTIME_INFO(xe)->rawclk_freq);
+
 	err = intel_display_driver_probe_noirq(xe);
 	if (err) {
 		intel_opregion_cleanup(display);
Signature · 1 line
Jani Nikula, Intel
#4 lore raw

PATCH v2 2/3 drm/xe: Wire up device shutdown handler

In reply to Maarten Lankhorst

The system is turning off, and we should probably put the device in a safe power state. We don't need to evict VRAM or suspend running jobs to a safe state, as the device is rebooted anyway.

This does not imply the system is necessarily reset, as we can kexec into a new kernel. Without shutting down, things like USB Type-C may mysteriously start failing.

References: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3500

---
 drivers/gpu/drm/xe/display/xe_display.c | 19 ++++++++++--
 drivers/gpu/drm/xe/display/xe_display.h |  3 +-
 drivers/gpu/drm/xe/xe_device.c          | 40 +++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_gt.c              |  7 +++++
 drivers/gpu/drm/xe/xe_gt.h              |  1 +
 drivers/gpu/drm/xe/xe_pm.c              |  4 +--
 6 files changed, 63 insertions(+), 11 deletions(-)
drivers/gpu/drm/xe/display/xe_display.c +17 -2
index 79add15c6c4c7..f549aee8ec261 100644
@@ -311,10 +311,10 @@ static void xe_display_flush_cleanup_work(struct xe_device *xe)
 	}
 }
 
-void xe_display_pm_suspend(struct xe_device *xe, bool runtime)
+void xe_display_pm_suspend(struct xe_device *xe, bool runtime, bool shutdown)
 {
 	struct intel_display *display = &xe->display;
-	bool s2idle = suspend_to_idle();
+	bool s2idle = suspend_to_idle() || shutdown;
 	if (!xe->info.probe_display)
 		return;
 
@@ -342,12 +342,27 @@ void xe_display_pm_suspend(struct xe_device *xe, bool runtime)
 		intel_display_driver_suspend_access(xe);
 
 	intel_encoder_suspend_all(&xe->display);
+	if (shutdown)
+		intel_encoder_shutdown_all(&xe->display);
 
 	intel_opregion_suspend(display, s2idle ? PCI_D1 : PCI_D3cold);
 
 	intel_dmc_suspend(xe);
 }
 
+void xe_display_pm_shutdown_late(struct xe_device *xe)
+{
+	if (!xe->info.enable_display)
+		return;
+
+	/*
+	 * The only requirement is to reboot with display DC states disabled,
+	 * for now leaving all display power wells in the INIT power domain
+	 * enabled.
+	 */
+	intel_power_domains_driver_remove(xe);
+}
+
 void xe_display_pm_suspend_late(struct xe_device *xe)
 {
 	bool s2idle = suspend_to_idle();
drivers/gpu/drm/xe/display/xe_display.h +2 -1
index 000fb5799df54..64229565f0d0d 100644
@@ -34,8 +34,9 @@ void xe_display_irq_enable(struct xe_device *xe, u32 gu_misc_iir);
 void xe_display_irq_reset(struct xe_device *xe);
 void xe_display_irq_postinstall(struct xe_device *xe, struct xe_gt *gt);
 
-void xe_display_pm_suspend(struct xe_device *xe, bool runtime);
+void xe_display_pm_suspend(struct xe_device *xe, bool runtime, bool shutdown);
 void xe_display_pm_suspend_late(struct xe_device *xe);
+void xe_display_pm_shutdown_late(struct xe_device *xe);
 void xe_display_pm_resume_early(struct xe_device *xe);
 void xe_display_pm_resume(struct xe_device *xe, bool runtime);
 
drivers/gpu/drm/xe/xe_device.c +34 -6
index b6db7e082d887..18c5a2c8530eb 100644
@@ -383,6 +383,11 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 	return ERR_PTR(err);
 }
 
+static bool xe_driver_flr_disabled(struct xe_device *xe)
+{
+	return xe_mmio_read32(xe_root_mmio_gt(xe), GU_CNTL_PROTECTED) & DRIVERINT_FLR_DIS;
+}
+
 /*
  * The driver-initiated FLR is the highest level of reset that we can trigger
  * from within the driver. It is different from the PCI FLR in that it doesn't
@@ -396,17 +401,12 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
  * if/when a new instance of i915 is bound to the device it will do a full
  * re-init anyway.
  */
-static void xe_driver_flr(struct xe_device *xe)
+static void __xe_driver_flr(struct xe_device *xe)
 {
 	const unsigned int flr_timeout = 3 * MICRO; /* specs recommend a 3s wait */
 	struct xe_gt *gt = xe_root_mmio_gt(xe);
 	int ret;
 
-	if (xe_mmio_read32(gt, GU_CNTL_PROTECTED) & DRIVERINT_FLR_DIS) {
-		drm_info_once(&xe->drm, "BIOS Disabled Driver-FLR\n");
-		return;
-	}
-
 	drm_dbg(&xe->drm, "Triggering Driver-FLR\n");
 
 	/*
@@ -447,6 +447,16 @@ static void xe_driver_flr(struct xe_device *xe)
 	xe_mmio_write32(gt, GU_DEBUG, DRIVERFLR_STATUS);
 }
 
+static void xe_driver_flr(struct xe_device *xe)
+{
+	if (xe_driver_flr_disabled(xe)) {
+		drm_info_once(&xe->drm, "BIOS Disabled Driver-FLR\n");
+		return;
+	}
+
+	__xe_driver_flr(xe);
+}
+
 static void xe_driver_flr_fini(void *arg)
 {
 	struct xe_device *xe = arg;
@@ -798,6 +808,24 @@ void xe_device_remove(struct xe_device *xe)
 
 void xe_device_shutdown(struct xe_device *xe)
 {
+	struct xe_gt *gt;
+	u8 id;
+
+	drm_dbg(&xe->drm, "Shutting down device\n");
+
+	if (xe_driver_flr_disabled(xe)) {
+		xe_display_pm_suspend(xe, false, true);
+
+		xe_irq_suspend(xe);
+
+		for_each_gt(gt, xe, id)
+			xe_gt_shutdown(gt);
+
+		xe_display_pm_shutdown_late(xe);
+	} else {
+		/* BOOM! */
+		__xe_driver_flr(xe);
+	}
 }
 
 /**
drivers/gpu/drm/xe/xe_gt.c +7 -0
index 224c137967c3c..9682d32e0d87d 100644
@@ -863,6 +863,13 @@ int xe_gt_suspend(struct xe_gt *gt)
 	return err;
 }
 
+void xe_gt_shutdown(struct xe_gt *gt)
+{
+	xe_force_wake_get(gt_to_fw(gt), XE_FORCEWAKE_ALL);
+	do_gt_reset(gt);
+	xe_force_wake_put(gt_to_fw(gt), XE_FORCEWAKE_ALL);
+}
+
 /**
  * xe_gt_sanitize_freq() - Restore saved frequencies if necessary.
  * @gt: the GT object
drivers/gpu/drm/xe/xe_gt.h +1 -0
index 8b1a5027dcf27..97def44afa4c8 100644
@@ -54,6 +54,7 @@ void xe_gt_record_user_engines(struct xe_gt *gt);
 
 void xe_gt_suspend_prepare(struct xe_gt *gt);
 int xe_gt_suspend(struct xe_gt *gt);
+void xe_gt_shutdown(struct xe_gt *gt);
 int xe_gt_resume(struct xe_gt *gt);
 void xe_gt_reset_async(struct xe_gt *gt);
 void xe_gt_sanitize(struct xe_gt *gt);
drivers/gpu/drm/xe/xe_pm.c +2 -2
index fcfb49af8c891..0ca135d4b9214 100644
@@ -93,7 +93,7 @@ int xe_pm_suspend(struct xe_device *xe)
 	for_each_gt(gt, xe, id)
 		xe_gt_suspend_prepare(gt);
 
-	xe_display_pm_suspend(xe, false);
+	xe_display_pm_suspend(xe, false, false);
 
 	/* FIXME: Super racey... */
 	err = xe_bo_evict_all(xe);
@@ -367,7 +367,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
 	if (xe->d3cold.allowed) {
-		xe_display_pm_suspend(xe, true);
+		xe_display_pm_suspend(xe, true, false);
 
 		err = xe_bo_evict_all(xe);
 		if (err)
Signature · 1 line
2.45.2
#5 lore raw

Re: [PATCH v2 2/3] drm/xe: Wire up device shutdown handler

drivers/gpu/drm/xe/display/xe_display.c static void xe_display_flush_cleanup_work(struct xe_device *xe) · line 314

Quoted 2 lines · Maarten Lankhorst
drivers/gpu/drm/xe/display/xe_display.c +1 -1
-void xe_display_pm_suspend(struct xe_device *xe, bool runtime)
+void xe_display_pm_suspend(struct xe_device *xe, bool runtime, bool shutdown)

https://lore.kernel.org/intel-xe/c4hemi6eshos64d6qxfux6wosyktw76thvzbwgvi26rbhbvvgr@3jkem2ozvclg/

as I said we keep adding these bools, making for a horrible interface. Please, don´t.

Lucas De Marchi

#6 lore raw

PATCH v2 3/3 CI-only: Test if FLR is disabled?

In reply to Maarten Lankhorst
---
 drivers/gpu/drm/xe/xe_device.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/xe_device.c +3 -0
index 18c5a2c8530eb..81009b78a2ca7 100644
@@ -593,6 +593,9 @@ int xe_device_probe_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	if (xe_driver_flr_disabled(xe))
+		drm_err(&xe->drm, "FLR disabled by firmware\n");
+
 	xe->wedged.mode = xe_modparam.wedged_mode;
 
 	return 0;
Signature · 1 line
2.45.2
#7 lore raw

✓ CI.Patch_applied: success for drm/xe/display: Fixes to make kexec work. (rev2)

In reply to Maarten Lankhorst
== Series Details ==

== Series Details ==

Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : success

== Summary ==

=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: 997f91bcdf12 drm-tip: 2024y-08m-19d-13h-28m-26s UTC integration manifest === git am output follows ===
Applying: drm/xe: Read out rawclk_freq for display
Applying: drm/xe: Wire up device shutdown handler
Applying: CI-only: Test if FLR is disabled?

#8 lore raw

✗ CI.checkpatch: warning for drm/xe/display: Fixes to make kexec work. (rev2)

In reply to Maarten Lankhorst
== Series Details ==

== Series Details ==

Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : warning

== Summary ==

+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 9fe5037901cabbcdf27a6fe0dfb047ca1474d363
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 4b5e3e0aeed53db4e36c2007845ebbcd2866d0b2

Author: Maarten Lankhorst <[email protected]>
Date:   Mon Aug 19 15:31:37 2024 +0200

CI-only: Test if FLR is disabled?

+ /mt/dim checkpatch 997f91bcdf1249b248ff040e89472779e3d8488b drm-intel
88dd43fec0d5 drm/xe: Read out rawclk_freq for display
54cc37795e01 drm/xe: Wire up device shutdown handler
-:14: WARNING:COMMIT_LOG_USE_LINK: Unknown link reference 'References:', use 'Link:' or 'Closes:' instead #14:
References: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3500

total: 0 errors, 1 warnings, 0 checks, 154 lines checked 4b5e3e0aeed5 CI-only: Test if FLR is disabled?
-:7: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one

total: 0 errors, 1 warnings, 0 checks, 9 lines checked

#9 lore raw

✗ CI.KUnit: failure for drm/xe/display: Fixes to make kexec work. (rev2)

In reply to Maarten Lankhorst
== Series Details ==

== Series Details ==

Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : failure

== Summary ==

+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_device.c: In function ‘xe_device_shutdown’:
../drivers/gpu/drm/xe/xe_device.c:820:3: error: too many arguments to function ‘xe_display_pm_suspend’

  820 |   xe_display_pm_suspend(xe, false, true);
      |   ^~~~~~~~~~~~~~~~~~~~~
In file included from ../drivers/gpu/drm/xe/xe_device.c:20:
../drivers/gpu/drm/xe/display/xe_display.h:67:20: note: declared here
   67 | static inline void xe_display_pm_suspend(struct xe_device *xe, bool runtime) {}
      |                    ^~~~~~~~~~~~~~~~~~~~~
../drivers/gpu/drm/xe/xe_device.c:827:3: error: implicit declaration of function ‘xe_display_pm_shutdown_late’; did you mean ‘xe_display_pm_suspend_late’? [-Werror=implicit-function-declaration]
  827 |   xe_display_pm_shutdown_late(xe);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |   xe_display_pm_suspend_late

cc1: some warnings being treated as errors
make[7]: *** [../scripts/Makefile.build:244: drivers/gpu/drm/xe/xe_device.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:485: drivers/gpu/drm/xe] Error 2 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:485: drivers/gpu/drm] Error 2
make[4]: *** [../scripts/Makefile.build:485: drivers/gpu] Error 2
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [../scripts/Makefile.build:485: drivers] Error 2
make[3]: *** Waiting for unfinished jobs....
../lib/iomap.c:156:5: warning: no previous prototype for ‘ioread64_lo_hi’ [-Wmissing-prototypes]
156 | u64 ioread64_lo_hi(const void __iomem *addr)

      |     ^~~~~~~~~~~~~~
../lib/iomap.c:163:5: warning: no previous prototype for ‘ioread64_hi_lo’ [-Wmissing-prototypes]
  163 | u64 ioread64_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~
../lib/iomap.c:170:5: warning: no previous prototype for ‘ioread64be_lo_hi’ [-Wmissing-prototypes]
  170 | u64 ioread64be_lo_hi(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:178:5: warning: no previous prototype for ‘ioread64be_hi_lo’ [-Wmissing-prototypes]
  178 | u64 ioread64be_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:264:6: warning: no previous prototype for ‘iowrite64_lo_hi’ [-Wmissing-prototypes]
  264 | void iowrite64_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:272:6: warning: no previous prototype for ‘iowrite64_hi_lo’ [-Wmissing-prototypes]
  272 | void iowrite64_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:280:6: warning: no previous prototype for ‘iowrite64be_lo_hi’ [-Wmissing-prototypes]
  280 | void iowrite64be_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~
../lib/iomap.c:288:6: warning: no previous prototype for ‘iowrite64be_hi_lo’ [-Wmissing-prototypes]
  288 | void iowrite64be_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~

make[2]: *** [/kernel/Makefile:1925: .] Error 2 make[1]: *** [/kernel/Makefile:224: __sub-make] Error 2
make: *** [Makefile:224: __sub-make] Error 2

[13:37:25] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[13:37:29] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel