drm/xe/display: Fixes to make kexec work.
v2 cover drm/xe/display: Fixes to make kexec work.
Lets see if we can get away with performing a FLR as shutdown handler. It should work. These are all the patches I have used when testing if I could kexec from Xe with a game running to a new kernel.
Tested with:
kexec --reuse-cmdline --initrd=initrd.img -f vmlinuz
I have tested about 4-5 times successive kexecs on my ADL-P, while having a game actively running to keep the GPU loaded.
With this series xe kept running succesfully. After that I accidentally rebooted, but I see no reason this wouldn't work even more times.
Changes since previous version:
- s/r patches have been merged.
Maarten Lankhorst (3):
drm/xe: Read out rawclk_freq for display
drm/xe: Wire up device shutdown handler
CI-only: Test if FLR is disabled?
drivers/gpu/drm/xe/display/xe_display.c | 22 +++++++++++-- drivers/gpu/drm/xe/display/xe_display.h | 3 +- drivers/gpu/drm/xe/xe_device.c | 43 +++++++++++++++++++++---- drivers/gpu/drm/xe/xe_gt.c | 7 ++++ drivers/gpu/drm/xe/xe_gt.h | 1 + drivers/gpu/drm/xe/xe_pm.c | 4 +-- 6 files changed, 69 insertions(+), 11 deletions(-)
Signature · 1 line
2.45.2
PATCH v2 1/3 drm/xe: Read out rawclk_freq for display
Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.
--- drivers/gpu/drm/xe/display/xe_display.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/display/xe_display.c +3 -0
@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)intel_display_device_info_runtime_init (xe); +RUNTIME_INFO (xe)->rawclk_freq =intel_read_rawclk (xe); +drm_dbg (&xe->drm,"rawclk rate: %d kHz\n" ,RUNTIME_INFO (xe)->rawclk_freq); + err =intel_display_driver_probe_noirq (xe);if (err) {intel_opregion_cleanup (display);
Signature · 1 line
2.45.2
Jani Nikula
maintainerRe: [PATCH v2 1/3] drm/xe: Read out rawclk_freq for display
Quoted 7 lines · Maarten Lankhorst
Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.
Signed-off-by Maarten Lankhorst <[email protected]>Fixes 44e694958b95 ("drm/xe/display: Implement display support")Cc <[email protected]> # v6.8+
Please find another way. See [1]. I'm trying to clean up the whole RUNTIME_INFO() and rawclk_freq thing, and this makes it harder.
BR,
Jani.
Quoted 18 lines
--- drivers/gpu/drm/xe/display/xe_display.c | 3 +++ 1 file changed, 3 insertions(+)drivers/gpu/drm/xe/display/xe_display.c +3
-0@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)intel_display_device_info_runtime_init (xe); +RUNTIME_INFO (xe)->rawclk_freq =intel_read_rawclk (xe); +drm_dbg (&xe->drm,"rawclk rate: %d kHz\n" ,RUNTIME_INFO (xe)->rawclk_freq); + err =intel_display_driver_probe_noirq (xe);if (err) {intel_opregion_cleanup (display);
Signature · 1 line
Jani Nikula, Intel
Jani Nikula
maintainerRe: [PATCH v2 1/3] drm/xe: Read out rawclk_freq for display
Quoted 11 lines · Jani Nikula
Quoted 7 lines · Maarten Lankhorst
Failing to read out rawclk makes it impossible to read out backlight, which results in backlight not working when the backlight is off during boot, or when reloading the module.
Signed-off-by Maarten Lankhorst <[email protected]>Fixes 44e694958b95 ("drm/xe/display: Implement display support")Cc <[email protected]> # v6.8+Please find another way. See [1]. I'm trying to clean up the whole
RUNTIME_INFO()andrawclk_freqthing, and this makes it harder.
Had another look, and brushed up my old patches, new version at [1].
BR,
Jani.
[1] https://lore.kernel.org/r/[email protected]
Quoted 25 lines
BR,
Jani.Quoted 18 lines
--- drivers/gpu/drm/xe/display/xe_display.c | 3 +++ 1 file changed, 3 insertions(+)drivers/gpu/drm/xe/display/xe_display.c +3
-0@@ -159,6 +159,9 @@ int xe_display_init_noirq(struct xe_device *xe)intel_display_device_info_runtime_init (xe); +RUNTIME_INFO (xe)->rawclk_freq =intel_read_rawclk (xe); +drm_dbg (&xe->drm,"rawclk rate: %d kHz\n" ,RUNTIME_INFO (xe)->rawclk_freq); + err =intel_display_driver_probe_noirq (xe);if (err) {intel_opregion_cleanup (display);
Signature · 1 line
Jani Nikula, Intel
PATCH v2 2/3 drm/xe: Wire up device shutdown handler
In reply to Maarten LankhorstThe system is turning off, and we should probably put the device in a safe power state. We don't need to evict VRAM or suspend running jobs to a safe state, as the device is rebooted anyway.
This does not imply the system is necessarily reset, as we can kexec into a new kernel. Without shutting down, things like USB Type-C may mysteriously start failing.
References: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3500
--- drivers/gpu/drm/xe/display/xe_display.c | 19 ++++++++++-- drivers/gpu/drm/xe/display/xe_display.h | 3 +- drivers/gpu/drm/xe/xe_device.c | 40 +++++++++++++++++++++---- drivers/gpu/drm/xe/xe_gt.c | 7 +++++ drivers/gpu/drm/xe/xe_gt.h | 1 + drivers/gpu/drm/xe/xe_pm.c | 4 +-- 6 files changed, 63 insertions(+), 11 deletions(-)
drivers/gpu/drm/xe/display/xe_display.c +17 -2
@@ -311,10 +311,10 @@ static void xe_display_flush_cleanup_work(struct xe_device *xe) } } -void xe_display_pm_suspend (struct xe_device *xe,bool runtime) +void xe_display_pm_suspend (struct xe_device *xe,bool runtime,bool shutdown) {struct intel_display *display = &xe->display; -bool s2idle =suspend_to_idle (); +bool s2idle =suspend_to_idle () || shutdown;if (!xe->info.probe_display)return ;
@@ -342,12 +342,27 @@ void xe_display_pm_suspend(struct xe_device *xe, bool runtime)intel_display_driver_suspend_access (xe);intel_encoder_suspend_all (&xe->display); +if (shutdown) +intel_encoder_shutdown_all (&xe->display);intel_opregion_suspend (display, s2idle ?PCI_D1 : PCI_D3cold);intel_dmc_suspend (xe); } +void xe_display_pm_shutdown_late (struct xe_device *xe) +{ +if (!xe->info.enable_display) +return ; + +/* +* The only requirement is to reboot with display DC states disabled, +* for now leaving all display power wells in the INIT power domain +* enabled. +*/ +intel_power_domains_driver_remove (xe); +} +void xe_display_pm_suspend_late (struct xe_device *xe) {bool s2idle =suspend_to_idle ();
drivers/gpu/drm/xe/display/xe_display.h +2 -1
@@ -34,8 +34,9 @@ void xe_display_irq_enable(struct xe_device *xe, u32 gu_misc_iir);void xe_display_irq_reset (struct xe_device *xe);void xe_display_irq_postinstall (struct xe_device *xe,struct xe_gt *gt); -void xe_display_pm_suspend (struct xe_device *xe,bool runtime); +void xe_display_pm_suspend (struct xe_device *xe,bool runtime,bool shutdown);void xe_display_pm_suspend_late (struct xe_device *xe); +void xe_display_pm_shutdown_late (struct xe_device *xe);void xe_display_pm_resume_early (struct xe_device *xe);void xe_display_pm_resume (struct xe_device *xe,bool runtime);
drivers/gpu/drm/xe/xe_device.c +34 -6
@@ -383,6 +383,11 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,return ERR_PTR (err); } +static bool xe_driver_flr_disabled (struct xe_device *xe) +{ +return xe_mmio_read32 (xe_root_mmio_gt (xe),GU_CNTL_PROTECTED ) &DRIVERINT_FLR_DIS ; +} +/* * The driver-initiated FLR is the highest level of reset that we can trigger * from within the driver. It is different from the PCI FLR in that it doesn't
@@ -396,17 +401,12 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,* if/when a new instance of i915 is bound to the device it will do a full * re-init anyway. */ -static void xe_driver_flr (struct xe_device *xe) +static void __xe_driver_flr (struct xe_device *xe) {const unsigned int flr_timeout =3 *MICRO ;/* specs recommend a 3s wait */ struct xe_gt *gt =xe_root_mmio_gt (xe);int ret; -if (xe_mmio_read32 (gt,GU_CNTL_PROTECTED ) &DRIVERINT_FLR_DIS ) { -drm_info_once (&xe->drm,"BIOS Disabled Driver-FLR\n" ); -return ; - } -drm_dbg (&xe->drm,"Triggering Driver-FLR\n" );/*
@@ -447,6 +447,16 @@ static void xe_driver_flr(struct xe_device *xe)xe_mmio_write32 (gt,GU_DEBUG ,DRIVERFLR_STATUS ); } +static void xe_driver_flr (struct xe_device *xe) +{ +if (xe_driver_flr_disabled (xe)) { +drm_info_once (&xe->drm,"BIOS Disabled Driver-FLR\n" ); +return ; + } + +__xe_driver_flr (xe); +} +static void xe_driver_flr_fini (void *arg) {struct xe_device *xe = arg;
@@ -798,6 +808,24 @@ void xe_device_remove(struct xe_device *xe)void xe_device_shutdown (struct xe_device *xe) { +struct xe_gt *gt; +u8 id; + +drm_dbg (&xe->drm,"Shutting down device\n" ); + +if (xe_driver_flr_disabled (xe)) { +xe_display_pm_suspend (xe, false, true); + +xe_irq_suspend (xe); + +for_each_gt (gt, xe, id) +xe_gt_shutdown (gt); + +xe_display_pm_shutdown_late (xe); + }else { +/* BOOM! */ +__xe_driver_flr (xe); + } }/**
drivers/gpu/drm/xe/xe_gt.c +7 -0
@@ -863,6 +863,13 @@ int xe_gt_suspend(struct xe_gt *gt)return err; } +void xe_gt_shutdown (struct xe_gt *gt) +{ +xe_force_wake_get (gt_to_fw (gt),XE_FORCEWAKE_ALL ); +do_gt_reset (gt); +xe_force_wake_put (gt_to_fw (gt),XE_FORCEWAKE_ALL ); +} +/** * xe_gt_sanitize_freq() - Restore saved frequencies if necessary. * @gt: the GT object
drivers/gpu/drm/xe/xe_gt.h +1 -0
@@ -54,6 +54,7 @@ void xe_gt_record_user_engines(struct xe_gt *gt);void xe_gt_suspend_prepare (struct xe_gt *gt);int xe_gt_suspend (struct xe_gt *gt); +void xe_gt_shutdown (struct xe_gt *gt);int xe_gt_resume (struct xe_gt *gt);void xe_gt_reset_async (struct xe_gt *gt);void xe_gt_sanitize (struct xe_gt *gt);
drivers/gpu/drm/xe/xe_pm.c +2 -2
@@ -93,7 +93,7 @@ int xe_pm_suspend(struct xe_device *xe)for_each_gt (gt, xe, id)xe_gt_suspend_prepare (gt); -xe_display_pm_suspend (xe, false); +xe_display_pm_suspend (xe, false, false);/* FIXME: Super racey... */ err =xe_bo_evict_all (xe);
@@ -367,7 +367,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe)mutex_unlock (&xe->mem_access.vram_userfault.lock);if (xe->d3cold.allowed) { -xe_display_pm_suspend (xe, true); +xe_display_pm_suspend (xe, true, false); err =xe_bo_evict_all (xe);if (err)
Signature · 1 line
2.45.2
Re: [PATCH v2 2/3] drm/xe: Wire up device shutdown handler
drivers/gpu/drm/xe/display/xe_display.c static void xe_display_flush_cleanup_work(struct xe_device *xe) · line 314
Quoted 2 lines · Maarten Lankhorst
drivers/gpu/drm/xe/display/xe_display.c +1
-1-void xe_display_pm_suspend (struct xe_device *xe,bool runtime) +void xe_display_pm_suspend (struct xe_device *xe,bool runtime,bool shutdown)
https://lore.kernel.org/intel-xe/c4hemi6eshos64d6qxfux6wosyktw76thvzbwgvi26rbhbvvgr@3jkem2ozvclg/
as I said we keep adding these bools, making for a horrible interface. Please, don´t.
Lucas De Marchi
PATCH v2 3/3 CI-only: Test if FLR is disabled?
In reply to Maarten Lankhorst--- drivers/gpu/drm/xe/xe_device.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/gpu/drm/xe/xe_device.c +3 -0
@@ -593,6 +593,9 @@ int xe_device_probe_early(struct xe_device *xe)if (err)return err; +if (xe_driver_flr_disabled (xe)) +drm_err (&xe->drm,"FLR disabled by firmware\n" ); + xe->wedged.mode = xe_modparam.wedged_mode;return 0 ;
Signature · 1 line
2.45.2
Patchwork
CI✓ CI.Patch_applied: success for drm/xe/display: Fixes to make kexec work. (rev2)
In reply to Maarten Lankhorst== Series Details ==
== Series Details ==
Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : success
== Summary ==
=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: 997f91bcdf12 drm-tip: 2024y-08m-19d-13h-28m-26s UTC integration manifest === git am output follows ===
Applying: drm/xe: Read out rawclk_freq for display
Applying: drm/xe: Wire up device shutdown handler
Applying: CI-only: Test if FLR is disabled?
Patchwork
CI✗ CI.checkpatch: warning for drm/xe/display: Fixes to make kexec work. (rev2)
In reply to Maarten Lankhorst== Series Details ==
== Series Details ==
Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : warning
== Summary ==
+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 9fe5037901cabbcdf27a6fe0dfb047ca1474d363
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 4b5e3e0aeed53db4e36c2007845ebbcd2866d0b2
Author: Maarten Lankhorst <[email protected]> Date: Mon Aug 19 15:31:37 2024 +0200
CI-only: Test if FLR is disabled?
+ /mt/dim checkpatch 997f91bcdf1249b248ff040e89472779e3d8488b drm-intel88dd43fec0d5 drm/xe: Read out rawclk_freq for display54cc37795e01 drm/xe: Wire up device shutdown handler
-:14: WARNING:COMMIT_LOG_USE_LINK: Unknown link reference 'References:', use 'Link:' or 'Closes:' instead #14:
References: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3500
total: 0 errors, 1 warnings, 0 checks, 154 lines checked 4b5e3e0aeed5 CI-only: Test if FLR is disabled?
-:7: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one
total: 0 errors, 1 warnings, 0 checks, 9 lines checked
Patchwork
CI✗ CI.KUnit: failure for drm/xe/display: Fixes to make kexec work. (rev2)
In reply to Maarten Lankhorst== Series Details ==
== Series Details ==
Series: drm/xe/display: Fixes to make kexec work. (rev2)
URL : https://patchwork.freedesktop.org/series/137453/
State : failure
== Summary ==
+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_device.c: In function ‘xe_device_shutdown’:../drivers/gpu/drm/xe/xe_device.c:820:3: error: too many arguments to function ‘xe_display_pm_suspend’
820 | xe_display_pm_suspend(xe, false, true);
| ^~~~~~~~~~~~~~~~~~~~~
In file included from ../drivers/gpu/drm/xe/xe_device.c:20:
../drivers/gpu/drm/xe/display/xe_display.h:67:20: note: declared here
67 | static inline void xe_display_pm_suspend(struct xe_device *xe, bool runtime) {}
| ^~~~~~~~~~~~~~~~~~~~~
../drivers/gpu/drm/xe/xe_device.c:827:3: error: implicit declaration of function ‘xe_display_pm_shutdown_late’; did you mean ‘xe_display_pm_suspend_late’? [-Werror=implicit-function-declaration]
827 | xe_display_pm_shutdown_late(xe);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
| xe_display_pm_suspend_latecc1: some warnings being treated as errors
make[7]: *** [../scripts/Makefile.build:244: drivers/gpu/drm/xe/xe_device.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:485: drivers/gpu/drm/xe] Error 2 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:485: drivers/gpu/drm] Error 2
make[4]: *** [../scripts/Makefile.build:485: drivers/gpu] Error 2
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [../scripts/Makefile.build:485: drivers] Error 2
make[3]: *** Waiting for unfinished jobs....../lib/iomap.c:156:5: warning: no previous prototype for ‘ioread64_lo_hi’ [-Wmissing-prototypes]
156 | u64 ioread64_lo_hi(const void __iomem *addr)
| ^~~~~~~~~~~~~~
../lib/iomap.c:163:5: warning: no previous prototype for ‘ioread64_hi_lo’ [-Wmissing-prototypes]
163 | u64 ioread64_hi_lo(const void __iomem *addr)
| ^~~~~~~~~~~~~~
../lib/iomap.c:170:5: warning: no previous prototype for ‘ioread64be_lo_hi’ [-Wmissing-prototypes]
170 | u64 ioread64be_lo_hi(const void __iomem *addr)
| ^~~~~~~~~~~~~~~~
../lib/iomap.c:178:5: warning: no previous prototype for ‘ioread64be_hi_lo’ [-Wmissing-prototypes]
178 | u64 ioread64be_hi_lo(const void __iomem *addr)
| ^~~~~~~~~~~~~~~~
../lib/iomap.c:264:6: warning: no previous prototype for ‘iowrite64_lo_hi’ [-Wmissing-prototypes]
264 | void iowrite64_lo_hi(u64 val, void __iomem *addr)
| ^~~~~~~~~~~~~~~
../lib/iomap.c:272:6: warning: no previous prototype for ‘iowrite64_hi_lo’ [-Wmissing-prototypes]
272 | void iowrite64_hi_lo(u64 val, void __iomem *addr)
| ^~~~~~~~~~~~~~~
../lib/iomap.c:280:6: warning: no previous prototype for ‘iowrite64be_lo_hi’ [-Wmissing-prototypes]
280 | void iowrite64be_lo_hi(u64 val, void __iomem *addr)
| ^~~~~~~~~~~~~~~~~
../lib/iomap.c:288:6: warning: no previous prototype for ‘iowrite64be_hi_lo’ [-Wmissing-prototypes]
288 | void iowrite64be_hi_lo(u64 val, void __iomem *addr)
| ^~~~~~~~~~~~~~~~~make[2]: *** [/kernel/Makefile:1925: .] Error 2 make[1]: *** [/kernel/Makefile:224: __sub-make] Error 2
make: *** [Makefile:224: __sub-make] Error 2
[13:37:25] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [13:37:29] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make ARCH=um O=.kunit --jobs=48 + cleanup ++ stat -c %u:%g /kernel + chown -R 1003:1003 /kernel