drm/i915/guc: Reset GuC and retry on firmware load failure
Subsystems (2)
PATCH v3 1/2 drm/i915/guc: Reset GuC and retry on firmware load failure
From: Arun Siluvery <[email protected]>
Due to timing issues in the HW some of the status bits required for GuC authentication doesn't get set occassionally, when that happens, GuC cannot be initialized and we will be left with a wedged GPU. The WA suggested is to perform a soft reset of GuC and attempt to reload the fw again for few times before giving up.
As the failure is dependent on timing, tests performed by triggering manual full gpu reset (i915_wedged) showed that we could sometimes hit this after several thousand iterations but sometimes tests ran even longer without any issues. Reset and reload mechanism proved helpful when we indeed hit fw load failure so it is better to include this to improve driver stability.
This change implements the following WA,
WaEnableuKernelHeaderValidFix:skl,bxt
WaEnableGuCBootHashCheckNotSet:skl,bxt
--- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/i915_guc_reg.h | 1 + drivers/gpu/drm/i915/i915_reg.h | 1 + drivers/gpu/drm/i915/intel_guc_loader.c | 49 +++++++++++++++++++++++++++++++-- drivers/gpu/drm/i915/intel_uncore.c | 19 +++++++++++++ 5 files changed, 69 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -2736,6 +2736,7 @@ extern long i915_compat_ioctl(struct file *filp, unsigned int cmd,extern int intel_gpu_reset (struct drm_device *dev,u32 engine_mask);extern bool intel_has_gpu_reset (struct drm_device *dev);extern int i915_reset (struct drm_device *dev); +extern int intel_guc_reset (struct drm_i915_private *dev_priv);extern void intel_engine_init_hangcheck (struct intel_engine_cs *engine);extern unsigned long i915_chipset_val (struct drm_i915_private *dev_priv);extern unsigned long i915_mch_val (struct drm_i915_private *dev_priv);
drivers/gpu/drm/i915/i915_guc_reg.h +1 -0
@@ -27,6 +27,7 @@/* Definitions of GuC H/W registers, bits, etc */ #define GUC_STATUS _MMIO(0xc000) +#define GS_MIA_IN_RESET (1 << 0)#define GS_BOOTROM_SHIFT 1#define GS_BOOTROM_MASK (0x7F << GS_BOOTROM_SHIFT)#define GS_BOOTROM_RSA_FAILED (0x50 << GS_BOOTROM_SHIFT)
drivers/gpu/drm/i915/i915_reg.h +1 -0
@@ -165,6 +165,7 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg)#define GEN6_GRDOM_MEDIA (1 << 2)#define GEN6_GRDOM_BLT (1 << 3)#define GEN6_GRDOM_VECS (1 << 4) +#define GEN9_GRDOM_GUC (1 << 5)#define GEN8_GRDOM_MEDIA2 (1 << 7)#define RING_PP_DIR_BASE (ring) _MMIO((ring)->mmio_base+0x228)
drivers/gpu/drm/i915/intel_guc_loader.c +47 -2
62 lines not on this page · raw message
drivers/gpu/drm/i915/intel_uncore.c +19 -0
@@ -1673,6 +1673,25 @@ bool intel_has_gpu_reset(struct drm_device *dev)return intel_get_gpu_reset (dev) !=NULL ; } +int intel_guc_reset (struct drm_i915_private *dev_priv) +{ +int ret; +unsigned long irqflags; + +if (!i915.enable_guc_submission) +return -EINVAL ; + +intel_uncore_forcewake_get (dev_priv,FORCEWAKE_ALL ); +spin_lock_irqsave (&dev_priv->uncore.lock, irqflags); + + ret =gen6_hw_domain_reset (dev_priv,GEN9_GRDOM_GUC ); + +spin_unlock_irqrestore (&dev_priv->uncore.lock, irqflags); +intel_uncore_forcewake_put (dev_priv,FORCEWAKE_ALL ); + +return ret; +} +bool intel_uncore_unclaimed_mmio (struct drm_i915_private *dev_priv) {return check_for_unclaimed_mmio (dev_priv);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 2/2 drm/i915/guc: always reset GuC before loading firmware
After a suspend-resume cycle, the resumed kernel has no idea what the booted kernel may have done to the GuC before replacing itself with the resumed image. In particular, it may have already loaded the GuC with firmware, which will then cause this kernel's attempt to (re)load the firmware to fail (GuC program memory is write-once!). The symptoms (GuC firmware reload fails after hibernation) are further described in the Bugzilla reference below.
So let's *always* reset the GuC just before (re)loading the firmware; then the hardware should then be in a well-known state, and we may even avoid some of the issues arising from unpredictable timing.
Also added some more fields & values to the definition of the GUC_STATUS register, which is the key diagnostic indicator if the GuC load fails.
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94390
--- drivers/gpu/drm/i915/i915_guc_reg.h | 12 ++++++++-- drivers/gpu/drm/i915/intel_guc_loader.c | 40 ++++++++++++++++----------------- 2 files changed, 29 insertions(+), 23 deletions(-)
drivers/gpu/drm/i915/i915_guc_reg.h +10 -2
@@ -27,10 +27,12 @@/* Definitions of GuC H/W registers, bits, etc */ #define GUC_STATUS _MMIO(0xc000) -#define GS_MIA_IN_RESET (1 << 0) +#define GS_RESET_SHIFT 0 +#define GS_MIA_IN_RESET (0x01 << GS_RESET_SHIFT)#define GS_BOOTROM_SHIFT 1#define GS_BOOTROM_MASK (0x7F << GS_BOOTROM_SHIFT)#define GS_BOOTROM_RSA_FAILED (0x50 << GS_BOOTROM_SHIFT) +#define GS_BOOTROM_JUMP_PASSED (0x76 << GS_BOOTROM_SHIFT)#define GS_UKERNEL_SHIFT 8#define GS_UKERNEL_MASK (0xFF << GS_UKERNEL_SHIFT)#define GS_UKERNEL_LAPIC_DONE (0x30 << GS_UKERNEL_SHIFT)
@@ -38,7 +40,13 @@#define GS_UKERNEL_READY (0xF0 << GS_UKERNEL_SHIFT)#define GS_MIA_SHIFT 16#define GS_MIA_MASK (0x07 << GS_MIA_SHIFT) -#define GS_MIA_CORE_STATE (1 << GS_MIA_SHIFT) +#define GS_MIA_CORE_STATE (0x01 << GS_MIA_SHIFT) +#define GS_MIA_HALT_REQUESTED (0x02 << GS_MIA_SHIFT) +#define GS_MIA_ISR_ENTRY (0x04 << GS_MIA_SHIFT) +#define GS_AUTH_STATUS_SHIFT 30 +#define GS_AUTH_STATUS_MASK (0x03 << GS_AUTH_STATUS_SHIFT) +#define GS_AUTH_STATUS_BAD (0x01 << GS_AUTH_STATUS_SHIFT) +#define GS_AUTH_STATUS_GOOD (0x02 << GS_AUTH_STATUS_SHIFT)#define SOFT_SCRATCH (n) _MMIO(0xc180 + (n) * 4)#define SOFT_SCRATCH_COUNT 16
drivers/gpu/drm/i915/intel_guc_loader.c +19 -21
@@ -387,7 +387,7 @@ int intel_guc_ucode_load(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;struct intel_guc_fw *guc_fw = &dev_priv->guc.guc_fw; -int err =0 ; +int retries, err =0 ;if (!i915.enable_guc_submission)return 0 ;
@@ -441,29 +441,26 @@ int intel_guc_ucode_load(struct drm_device *dev)* steppings also so this is extended as well. */ /* WaEnableGuCBootHashCheckNotSet:skl,bxt */ - err =guc_ucode_xfer (dev_priv); -if (err) { -int retries =3 ; - -DRM_ERROR ("GuC fw load failed, err=%d, attempting reset and retry\n" , err); - -while (retries--) { - err =i915_reset_guc (dev_priv); -if (err) -break ; - - err =guc_ucode_xfer (dev_priv); -if (!err) { -DRM_DEBUG_DRIVER ("GuC fw reload succeeded after reset\n" ); -break ; - } -DRM_DEBUG_DRIVER ("GuC fw reload retries left: %d\n" , retries); - } - +for (retries =3 ; ; ) { +/* +* Always reset the GuC just before (re)loading, so +* that the state and timing are fairly predictable +*/ + err =i915_reset_guc (dev_priv);if (err) { -DRM_ERROR ("GuC fw reload attempt failed, ret=%d\n" , err); +DRM_ERROR ("GuC reset failed, err %d\n" , err); goto fail; } + + err =guc_ucode_xfer (dev_priv); +if (!err) +break ; + +if (--retries ==0 ) + goto fail; + +DRM_INFO ("GuC fw load failed, err %d; will reset and " +"retry %d more time(s)\n" , err, retries); } guc_fw->guc_fw_load_status =GUC_FIRMWARE_SUCCESS ;
@@ -485,6 +482,7 @@ int intel_guc_ucode_load(struct drm_device *dev)return 0 ; fail: +DRM_ERROR ("GuC firmware load failed, err %d\n" , err);if (guc_fw->guc_fw_load_status ==GUC_FIRMWARE_PENDING ) guc_fw->guc_fw_load_status =GUC_FIRMWARE_FAIL ;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/2] drm/i915/guc: always reset GuC before loading firmware
Quoted 12 lines · Dave Gordon
After a suspend-resume cycle, the resumed kernel has no idea what the booted kernel may have done to the GuC before replacing itself with the resumed image. In particular, it may have already loaded the GuC with firmware, which will then cause this kernel's attempt to (re)load the firmware to fail (GuC program memory is write-once!). The symptoms (GuC firmware reload fails after hibernation) are further described in the Bugzilla reference below.
So let's *always* reset the GuC just before (re)loading the firmware; then the hardware should then be in a well-known state, and we may even avoid some of the issues arising from unpredictable timing.
Quoted 7 lines
Also added some more fields & values to the definition of the
GUC_STATUSregister, which is the key diagnostic indicator if the GuC load fails.Signed-off-by Dave Gordon <[email protected]>Reviewed-by Arun Siluvery <[email protected]>Cc Alex Dai <[email protected]>Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94390
I guess both of these should be labelled Cc: [email protected] when merging.
-Daniel
Quoted 114 lines
--- drivers/gpu/drm/i915/i915_guc_reg.h | 12 ++++++++-- drivers/gpu/drm/i915/intel_guc_loader.c | 40 ++++++++++++++++----------------- 2 files changed, 29 insertions(+), 23 deletions(-)drivers/gpu/drm/i915/i915_guc_reg.h +10
-2@@ -27,10 +27,12 @@/* Definitions of GuC H/W registers, bits, etc */ #define GUC_STATUS _MMIO(0xc000) -#define GS_MIA_IN_RESET (1 << 0) +#define GS_RESET_SHIFT 0 +#define GS_MIA_IN_RESET (0x01 << GS_RESET_SHIFT)#define GS_BOOTROM_SHIFT 1#define GS_BOOTROM_MASK (0x7F << GS_BOOTROM_SHIFT)#define GS_BOOTROM_RSA_FAILED (0x50 << GS_BOOTROM_SHIFT) +#define GS_BOOTROM_JUMP_PASSED (0x76 << GS_BOOTROM_SHIFT)#define GS_UKERNEL_SHIFT 8#define GS_UKERNEL_MASK (0xFF << GS_UKERNEL_SHIFT)#define GS_UKERNEL_LAPIC_DONE (0x30 << GS_UKERNEL_SHIFT)@@ -38,7 +40,13 @@#define GS_UKERNEL_READY (0xF0 << GS_UKERNEL_SHIFT)#define GS_MIA_SHIFT 16#define GS_MIA_MASK (0x07 << GS_MIA_SHIFT) -#define GS_MIA_CORE_STATE (1 << GS_MIA_SHIFT) +#define GS_MIA_CORE_STATE (0x01 << GS_MIA_SHIFT) +#define GS_MIA_HALT_REQUESTED (0x02 << GS_MIA_SHIFT) +#define GS_MIA_ISR_ENTRY (0x04 << GS_MIA_SHIFT) +#define GS_AUTH_STATUS_SHIFT 30 +#define GS_AUTH_STATUS_MASK (0x03 << GS_AUTH_STATUS_SHIFT) +#define GS_AUTH_STATUS_BAD (0x01 << GS_AUTH_STATUS_SHIFT) +#define GS_AUTH_STATUS_GOOD (0x02 << GS_AUTH_STATUS_SHIFT)#define SOFT_SCRATCH (n) _MMIO(0xc180 + (n) * 4)#define SOFT_SCRATCH_COUNT 16drivers/gpu/drm/i915/intel_guc_loader.c +19
-21@@ -387,7 +387,7 @@ int intel_guc_ucode_load(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;struct intel_guc_fw *guc_fw = &dev_priv->guc.guc_fw; -int err =0 ; +int retries, err =0 ;if (!i915.enable_guc_submission)return 0 ;@@ -441,29 +441,26 @@ int intel_guc_ucode_load(struct drm_device *dev)* steppings also so this is extended as well. */ /* WaEnableGuCBootHashCheckNotSet:skl,bxt */ - err =guc_ucode_xfer (dev_priv); -if (err) { -int retries =3 ; - -DRM_ERROR ("GuC fw load failed, err=%d, attempting reset and retry\n" , err); - -while (retries--) { - err =i915_reset_guc (dev_priv); -if (err) -break ; - - err =guc_ucode_xfer (dev_priv); -if (!err) { -DRM_DEBUG_DRIVER ("GuC fw reload succeeded after reset\n" ); -break ; - } -DRM_DEBUG_DRIVER ("GuC fw reload retries left: %d\n" , retries); - } - +for (retries =3 ; ; ) { +/* +* Always reset the GuC just before (re)loading, so +* that the state and timing are fairly predictable +*/ + err =i915_reset_guc (dev_priv);if (err) { -DRM_ERROR ("GuC fw reload attempt failed, ret=%d\n" , err); +DRM_ERROR ("GuC reset failed, err %d\n" , err); goto fail; } + + err =guc_ucode_xfer (dev_priv); +if (!err) +break ; + +if (--retries ==0 ) + goto fail; + +DRM_INFO ("GuC fw load failed, err %d; will reset and " +"retry %d more time(s)\n" , err, retries); } guc_fw->guc_fw_load_status =GUC_FIRMWARE_SUCCESS ;@@ -485,6 +482,7 @@ int intel_guc_ucode_load(struct drm_device *dev)return 0 ; fail: +DRM_ERROR ("GuC firmware load failed, err %d\n" , err);if (guc_fw->guc_fw_load_status ==GUC_FIRMWARE_PENDING ) guc_fw->guc_fw_load_status =GUC_FIRMWARE_FAIL ;Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/2] drm/i915/guc: always reset GuC before loading firmware
Quoted 16 lines · Daniel Vetter
Quoted 15 lines · Dave Gordon
After a suspend-resume cycle, the resumed kernel has no idea what the booted kernel may have done to the GuC before replacing itself with the resumed image. In particular, it may have already loaded the GuC with firmware, which will then cause this kernel's attempt to (re)load the firmware to fail (GuC program memory is write-once!). The symptoms (GuC firmware reload fails after hibernation) are further described in the Bugzilla reference below.
So let's *always* reset the GuC just before (re)loading the firmware; then the hardware should then be in a well-known state, and we may even avoid some of the issues arising from unpredictable timing.
Also added some more fields & values to the definition of the
GUC_STATUSregister, which is the key diagnostic indicator if the GuC load fails.
Quoted 8 lines
Quoted 4 lines
Signed-off-by Dave Gordon <[email protected]>Reviewed-by Arun Siluvery <[email protected]>Cc Alex Dai <[email protected]>Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94390
I guess both of these should be labelled Cc: [email protected] when merging.
-Daniel
Probably not necessary, as the patch to enable GuC loading (and fallback) by default hasn't yet been merged, so this code isn't reached unless the (unsafe) kernel parameter is overridden.
.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
BTW, this two-item patchset was just a repost to get CI to pick it up, without getting confused by it previously being embedded in a longer set
.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Patchwork
CI✗ Fi.CI.BAT: warning for series starting with [v3,1/2] drm/i915/guc: Reset GuC and retry on firmware load failure
In reply to Dave Gordon== Series Details ==
== Series Details ==
Series: series starting with [v3,1/2] drm/i915/guc: Reset GuC and retry on firmware load failure
URL : https://patchwork.freedesktop.org/series/4876/
State : warning
== Summary ==
Series 4876v1 Series without cover letter
http://patchwork.freedesktop.org/api/1.0/series/4876/revisions/1/mbox/
Test kms_pipe_crc_basic:
Subgroup suspend-read-crc-pipe-c:
pass -> DMESG-WARN (bsw-nuc-2)
Test pm_rpm:
Subgroup basic-pci-d3-state:
pass -> DMESG-WARN (bsw-nuc-2)
Subgroup basic-rte:
dmesg-warn -> PASS (byt-nuc) UNSTABLEbdw-nuci7 total:192 pass:179 dwarn:0 dfail:0 fail:1 skip:12 bdw-ultra total:192 pass:170 dwarn:0 dfail:0 fail:1 skip:21 bsw-nuc-2 total:192 pass:153 dwarn:2 dfail:0 fail:0 skip:37 byt-nuc total:192 pass:157 dwarn:0 dfail:0 fail:0 skip:35 hsw-brixbox total:192 pass:170 dwarn:0 dfail:0 fail:0 skip:22 hsw-gt2 total:192 pass:175 dwarn:0 dfail:0 fail:0 skip:17 ivb-t430s total:192 pass:167 dwarn:0 dfail:0 fail:0 skip:25 skl-i7k-2 total:192 pass:169 dwarn:0 dfail:0 fail:0 skip:23 snb-dellxps total:192 pass:158 dwarn:0 dfail:0 fail:0 skip:34 snb-x220t total:192 pass:158 dwarn:0 dfail:0 fail:1 skip:33
Results at /archive/results/CI_IGT_test/Patchwork_1717/
f5d413cccefa1f93d64c34f357151d42add63a84 drm-intel-nightly: 2016y-03m-24d-14h-34m-29s UTC integration manifest cff00fb37026cf1e75d9317e8570938e89197515 drm/i915/guc: always reset GuC before loading firmware 6c406232a0289a937e4478901e3edc57c8f9f01a drm/i915/guc: Reset GuC and retry on firmware load failure
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: ✗ Fi.CI.BAT: warning for series starting with [v3,1/2] drm/i915/guc: Reset GuC and retry on firmware load failure
Quoted 6 lines · Patchwork
== Series Details ==
Series: series starting with [v3,1/2] drm/i915/guc: Reset GuC and retry on firmware load failure
URL : https://patchwork.freedesktop.org/series/4876/
State : warning
Quoted 8 lines
== Summary ==
Series 4876v1 Series without cover letter
http://patchwork.freedesktop.org/api/1.0/series/4876/revisions/1/mbox/Test
kms_pipe_crc_basic:Subgroup suspend-read-crc-pipe-c: pass -> DMESG-WARN (bsw-nuc-2)
Unrelated, https://bugs.freedesktop.org/show_bug.cgi?id=94350
lockdep splat due to stop_machine() in ggtt pte programming
Quoted 3 lines
Test
pm_rpm:Subgroup basic-pci-d3-state: pass -> DMESG-WARN (bsw-nuc-2)
Unrelated, https://bugs.freedesktop.org/show_bug.cgi?id=94164
Unclaimed access detected prior to suspending
Quoted 14 lines
Subgroup basic-rte: dmesg-warn -> PASS (byt-nuc) UNSTABLEbdw-nuci7 total:192 pass:179 dwarn:0 dfail:0 fail:1 skip:12 bdw-ultra total:192 pass:170 dwarn:0 dfail:0 fail:1 skip:21 bsw-nuc-2 total:192 pass:153 dwarn:2 dfail:0 fail:0 skip:37 byt-nuc total:192 pass:157 dwarn:0 dfail:0 fail:0 skip:35 hsw-brixbox total:192 pass:170 dwarn:0 dfail:0 fail:0 skip:22 hsw-gt2 total:192 pass:175 dwarn:0 dfail:0 fail:0 skip:17 ivb-t430s total:192 pass:167 dwarn:0 dfail:0 fail:0 skip:25 skl-i7k-2 total:192 pass:169 dwarn:0 dfail:0 fail:0 skip:23 snb-dellxps total:192 pass:158 dwarn:0 dfail:0 fail:0 skip:34 snb-x220t total:192 pass:158 dwarn:0 dfail:0 fail:1 skip:33
Quoted 6 lines
Results at /archive/results/CI_IGT_test/Patchwork_1717/
f5d413cccefa1f93d64c34f357151d42add63a84drm-intel-nightly: 2016y-03m-24d-14h-34m-29s UTC integration manifestcff00fb37026cf1e75d9317e8570938e89197515drm/i915/guc: always reset GuC before loading firmware6c406232a0289a937e4478901e3edc57c8f9f01adrm/i915/guc: Reset GuC and retry on firmware load failure
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx