drm/amdgpu/vpe: increase VPE_IDLE_TIMEOUT to fix hang on Strix Halo
Subsystems (3)
PATCH 1/2 drm/amdgpu/vpe: increase VPE_IDLE_TIMEOUT to fix hang on Strix Halo
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then, amdgpu_device_delayed_init_work_handler with a delay of 2s is scheduled to run tests, one of which is testing VPE in vpe_ring_test_ib. This causes an ungate, After that test, vpe_idle_work_handler is scheduled with VPE_IDLE_TIMEOUT (1s).
When vpe_idle_work_handler runs and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.
Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel 0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd 0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0] flip_done timed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from 0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from 0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1] flip_done timed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!
In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally. 0x00000032 is the PowerDownVpe(50) command which is the common failure point in all failed resumes.
On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param: 0x00000000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param: 0x00000000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param: 0x00010000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param: 0x00010000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param: 0x00000000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param: 0x00000000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param: 0x00010000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param: 0x00000000, resp: 0x00000001 amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param: 0x00010000, resp: 0x00000001
To fix this, increase VPE_IDLE_TIMEOUT to 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2 -2
@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8
base-commit: c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9
Signature · 1 line
2.50.1
PATCH 2/2 drm/amd/display: Adjust AUX brightness to be a granularity of 100
Certain OLED devices malfunction on specific brightness levels. Specifically, when DP_SOURCE_BACKLIGHT_LEVEL is written to with the minor byte being 0x00 and sometimes 0x01, the panel forcibly turns off until the device sleeps again. This is an issue on multiple handhelds, including OneXPlayer F1 Pro and Ayaneo 3 (the panel is suspected to be the same-1080p 7in OLED).
Below are some examples. This was found by iterating over brighness ranges while printing DP_SOURCE_BACKLIGHT_LEVEL. It was found that the screen would malfunction on specific values, and some of them were collected.
Broken:
86016: 10101000000000000 86272: 10101000100000000 87808: 10101011100000000
251648: 111101011100000000
251649: 111101011100000001
Working:
86144: 10101000010000000
87809: 10101011100000001
251650: 111101011100000010
The reason for this is that the range manipulation is too granular. AUX is currently written to with a granularity of 1. Forcing 100, which on the Ayaneo 3 OLED yields 400*10=4000 values, is plenty of granularity and fixes this issue. Iterating over the values through Python shows that the final byte is never 0x00, and testing over the entire range with a cadence of 0.2s/it and 73 increments (to saturate the range) shows no issues. Windows likewise shows no issues.
--- .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +17 -11
@@ -4739,7 +4739,8 @@ static void amdgpu_dm_update_backlight_caps(struct amdgpu_display_manager *dm, }static int get_brightness_range (const struct amdgpu_dm_backlight_caps *caps, -unsigned int *min,unsigned int *max) +unsigned int *min,unsigned int *max, +unsigned int *multiple) {if (!caps)return 0 ;
@@ -4748,10 +4749,12 @@ static int get_brightness_range(const struct amdgpu_dm_backlight_caps *caps,// Firmware limits are in nits, DC API wants millinits. *max =1000 * caps->aux_max_input_signal; *min =1000 * caps->aux_min_input_signal; + *multiple =100 ; }else {// Firmware limits are 8-bit, PWM control is 16-bit. *max =0x101 * caps->max_input_signal; *min =0x101 * caps->min_input_signal; + *multiple =1 ; }return 1 ; }
@@ -4813,23 +4816,25 @@ static void convert_custom_brightness(const struct amdgpu_dm_backlight_caps *capstatic u32 convert_brightness_from_user (const struct amdgpu_dm_backlight_caps *caps,uint32_t brightness) { -unsigned int min, max; +unsigned int min, max, multiple; -if (!get_brightness_range (caps, &min, &max)) +if (!get_brightness_range (caps, &min, &max, &multiple))return brightness;convert_custom_brightness (caps, min, max, &brightness); -// Rescale 0..max to min..max -return min +DIV_ROUND_CLOSEST_ULL ((u64)(max - min) * brightness, max); +// Rescale 0..max to min..max rounding to nearest multiple +return rounddown ( + min +DIV_ROUND_CLOSEST_ULL ((u64)(max - min) * brightness, max), + multiple); }static u32 convert_brightness_to_user (const struct amdgpu_dm_backlight_caps *caps,uint32_t brightness) { -unsigned int min, max; +unsigned int min, max, multiple; -if (!get_brightness_range (caps, &min, &max)) +if (!get_brightness_range (caps, &min, &max, &multiple))return brightness;if (brightness < min)
@@ -4970,7 +4975,7 @@ amdgpu_dm_register_backlight_device(struct amdgpu_dm_connector *aconnector)struct backlight_properties props = {0 };struct amdgpu_dm_backlight_caps *caps;char bl_name[16 ]; -int min, max; +int min, max, multiple;if (aconnector->bl_idx ==-1 )return ;
@@ -4983,15 +4988,16 @@ amdgpu_dm_register_backlight_device(struct amdgpu_dm_connector *aconnector) } caps = &dm->backlight_caps[aconnector->bl_idx]; -if (get_brightness_range (caps, &min, &max)) { +if (get_brightness_range (caps, &min, &max, &multiple)) {if (power_supply_is_system_supplied () >0 ) props.brightness =DIV_ROUND_CLOSEST ((max - min) * caps->ac_level,100 );else props.brightness =DIV_ROUND_CLOSEST ((max - min) * caps->dc_level,100 );/* min is zero, so max needs to be adjusted */ props.max_brightness = max - min; -drm_dbg (drm,"Backlight caps: min: %d, max: %d, ac %d, dc %d\n" , min, max, - caps->ac_level, caps->dc_level); +drm_dbg (drm, +"Backlight caps: min: %d, max: %d, ac %d, dc %d, multiple: %d\n" , + min, max, caps->ac_level, caps->dc_level, multiple); }else props.brightness = props.max_brightness =MAX_BACKLIGHT_LEVEL ;
Signature · 1 line
2.50.1
kernel test robot
build botRe: [PATCH v1 2/2] drm/amd/display: Adjust AUX brightness to be a granularity of 100
kernel test robot noticed the following build errors:
Hi Antheas,
kernel test robot noticed the following build errors:
[auto build test ERROR on c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9]
url: https://github.com/intel-lab-lkp/linux/commits/Antheas-Kapenekakis/drm-amd-display-Adjust-AUX-brightness-to-be-a-granularity-of-100/20250824-165633 base: c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9 patch link: https://lore.kernel.org/r/20250824085351.454619-2-lkml%40antheas.dev patch subject: [PATCH v1 2/2] drm/amd/display: Adjust AUX brightness to be a granularity of 100 config: i386-buildonly-randconfig-002-20250824 (https://download.01.org/0day-ci/archive/20250824/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14+deb12u1) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250824/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
ld: drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.o: in function `amdgpu_dm_backlight_set_level':
>> amdgpu_dm.c:(.text+0x8b89): undefined reference to `__umoddi3'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v1 2/2] drm/amd/display: Adjust AUX brightness to be a granularity of 100
In reply to Antheas KapenekakisQuoted 25 lines · Antheas Kapenekakis
Certain OLED devices malfunction on specific brightness levels. Specifically, when
DP_SOURCE_BACKLIGHT_LEVELis written to with the minor byte being0x00and sometimes0x01, the panel forcibly turns off until the device sleeps again. This is an issue on multiple handhelds, including OneXPlayer F1 Pro and Ayaneo 3 (the panel is suspected to be the same-1080p 7in OLED).Below are some examples. This was found by iterating over brighness ranges while printing
DP_SOURCE_BACKLIGHT_LEVEL. It was found that the screen would malfunction on specific values, and some of them were collected.Broken:
86016: 10101000000000000 86272: 10101000100000000 87808: 10101011100000000251648: 111101011100000000
251649: 111101011100000001Working:
86144: 10101000010000000
87809: 10101011100000001
251650: 111101011100000010
Quoted 7 lines
The reason for this is that the range manipulation is too granular. AUX is currently written to with a granularity of 1. Forcing 100, which on the Ayaneo 3 OLED yields 400*10=4000 values, is plenty of granularity and fixes this issue. Iterating over the values through Python shows that the final byte is never
0x00, and testing over the entire range with a cadence of 0.2s/it and 73 increments (to saturate the range) shows no issues. Windows likewise shows no issues.
Well Phil managed to fall into the value 332800, which has a 0 minor bit. Unfortunate. In hindsight, every 256 hundreds there would be a zero anyway.
Before I made this patch I made a partial refactor of panel-quirks where a quirk like this could go to. But I would really prefer not to do quirks. Ill send that too.
Antheas
Quoted 98 lines
Signed-off-by Antheas Kapenekakis <[email protected]>--- .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-)drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +17
-12@@ -4739,7 +4739,8 @@ static void amdgpu_dm_update_backlight_caps(struct amdgpu_display_manager *dm, }static int get_brightness_range (const struct amdgpu_dm_backlight_caps *caps, -unsigned int *min,unsigned int *max) +unsigned int *min,unsigned int *max, +unsigned int *multiple) {if (!caps)return 0 ;@@ -4748,10 +4749,12 @@ static int get_brightness_range(const struct amdgpu_dm_backlight_caps *caps,// Firmware limits are in nits, DC API wants millinits. *max =1000 * caps->aux_max_input_signal; *min =1000 * caps->aux_min_input_signal; + *multiple =100 ; }else {// Firmware limits are 8-bit, PWM control is 16-bit. *max =0x101 * caps->max_input_signal; *min =0x101 * caps->min_input_signal; + *multiple =1 ; }return 1 ; }@@ -4813,23 +4816,25 @@ static void convert_custom_brightness(const struct amdgpu_dm_backlight_caps *capstatic u32 convert_brightness_from_user (const struct amdgpu_dm_backlight_caps *caps,uint32_t brightness) { -unsigned int min, max; +unsigned int min, max, multiple; -if (!get_brightness_range (caps, &min, &max)) +if (!get_brightness_range (caps, &min, &max, &multiple))return brightness;convert_custom_brightness (caps, min, max, &brightness); -// Rescale 0..max to min..max -return min +DIV_ROUND_CLOSEST_ULL ((u64)(max - min) * brightness, max); +// Rescale 0..max to min..max rounding to nearest multiple +return rounddown ( + min +DIV_ROUND_CLOSEST_ULL ((u64)(max - min) * brightness, max), + multiple); }static u32 convert_brightness_to_user (const struct amdgpu_dm_backlight_caps *caps,uint32_t brightness) { -unsigned int min, max; +unsigned int min, max, multiple; -if (!get_brightness_range (caps, &min, &max)) +if (!get_brightness_range (caps, &min, &max, &multiple))return brightness;if (brightness < min)@@ -4970,7 +4975,7 @@ amdgpu_dm_register_backlight_device(struct amdgpu_dm_connector *aconnector)struct backlight_properties props = {0 };struct amdgpu_dm_backlight_caps *caps;char bl_name[16 ]; -int min, max; +int min, max, multiple;if (aconnector->bl_idx ==-1 )return ;@@ -4983,15 +4988,16 @@ amdgpu_dm_register_backlight_device(struct amdgpu_dm_connector *aconnector) } caps = &dm->backlight_caps[aconnector->bl_idx]; -if (get_brightness_range (caps, &min, &max)) { +if (get_brightness_range (caps, &min, &max, &multiple)) {if (power_supply_is_system_supplied () >0 ) props.brightness =DIV_ROUND_CLOSEST ((max - min) * caps->ac_level,100 );else props.brightness =DIV_ROUND_CLOSEST ((max - min) * caps->dc_level,100 );/* min is zero, so max needs to be adjusted */ props.max_brightness = max - min; -drm_dbg (drm,"Backlight caps: min: %d, max: %d, ac %d, dc %d\n" , min, max, - caps->ac_level, caps->dc_level); +drm_dbg (drm, +"Backlight caps: min: %d, max: %d, ac %d, dc %d, multiple: %d\n" , + min, max, caps->ac_level, caps->dc_level, multiple); }else props.brightness = props.max_brightness =MAX_BACKLIGHT_LEVEL ; --2.50.1
Re: [PATCH v1 2/2] drm/amd/display: Adjust AUX brightness to be a granularity of 100
Quoted 9 lines · Antheas Kapenekakis
Well Phil managed to fall into the value 332800, which has a 0 minor bit. Unfortunate. In hindsight, every 256 hundreds there would be a zero anyway.
Before I made this patch I made a partial refactor of panel-quirks where a quirk like this could go to. But I would really prefer not to do quirks. Ill send that too.
Antheas
I was already looking into that OLED issue for several weeks. Changing granularity might hid the root cause and you might hit the issue less frequently.
Currently checking [1] which changes the first byte to 3 since when DP_SOURCE_BACKLIGHT_LEVEL is written to with
the first byte being 0x00 and sometimes 0x01, the panel forcibly turns off until the device sleeps again.
In the end the panel vendor has to fix it in firmware. If not a quirk might be better specific for each panel vendor.
I'm still not sure if that refactoring is needed, or a separate quirk function is more logical to be accepted upstream.
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.
When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
Quoted 28 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8
Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9
1s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
You can get SMU firmware version from this:
❯ grep . /sys/bus/platform/drivers/amd_pmc/*/smu_*
Are you on the most up to date firmware for your system from the manufacturer?
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
IE is it possible the delayed work handler amdgpu_device_delayed_init_work_handler() is causing a race with vpe_ring_begin_use()?
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001
Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echo date '+%s' -d '+ 60 seconds' | sudo tee /sys/class/rtc/rtc0/wakealarm sudo sh -c 'echo mem > /sys/power/state'
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done
Quoted 33 lines
Quoted 28 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*
grep . /sys/bus/platform/drivers/amd_pmc/*/smu_* /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 2 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?
I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'
Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done
👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0
Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l amdgpu_device_delayed_init_work_handler -l vpe_idle_work_handler -l vpe_ring_begin_use -l vpe_ring_end_use -l amdgpu_pmops_suspend -l amdgpu_pmops_resume
Here's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend <-- pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume <-- dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use <-- vpe_ring_test_ring
kworker/15:1-12285 [015] ..... 13666.960191: function:
amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/15:1-12285 [015] ..... 13666.963970: function:
vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/15:1-12285 [015] ..... 13666.965481: function:
vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/15:4-16354 [015] ..... 13667.981394: function: vpe_idle_work_handler <-- process_one_work
I did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the function_graph tracer.
Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
1) You can use CONFIG_PSTORE_FTRACE
2) If you add "tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.
Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/
Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACE
I can look into that
Quoted 9 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 5 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Quoted 7 lines
Quoted 2 lines
Antheas
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
Antheas
Quoted 1 line
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Alex
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24kworker/u97:37-18051 [001] ..... 13655.970108: function: amdgpu_pmops_suspend <-- pci_pm_suspend kworker/u97:21-18036 [002] ..... 13666.290715: function: amdgpu_pmops_resume <-- dpm_run_callback kworker/u97:21-18036 [015] ..... 13666.308295: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/u97:21-18036 [015] ..... 13666.308298: function: vpe_ring_end_use <-- vpe_ring_test_ring kworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 294 lines · Mario Limonciello
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24kworker/u97:37-18051 [001] ..... 13655.970108: function: amdgpu_pmops_suspend <-- pci_pm_suspend kworker/u97:21-18036 [002] ..... 13666.290715: function: amdgpu_pmops_resume <-- dpm_run_callback kworker/u97:21-18036 [015] ..... 13666.308295: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/u97:21-18036 [015] ..... 13666.308298: function: vpe_ring_end_use <-- vpe_ring_test_ring kworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 6 lines
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
Quoted 6 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 301 lines · Antheas Kapenekakis
Quoted 296 lines · Mario Limonciello
Quoted 293 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Alex
Quoted 7 lines
Quoted 4 lines
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
I can reproduce the hang on a stock 6.17-rc3 kernel on my own Flow Z13, froze within 10 cycles with Antheas’ script. I will setup pstore to get logs from it since nothing appears in my journal after force rebooting.
Matt
Quoted 9 lines
Quoted 7 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 307 lines · Matthew Schwartz
Quoted 307 lines · Antheas Kapenekakis
Quoted 300 lines · Mario Limonciello
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
Quoted 6 lines
Quoted 2 lines
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
I can reproduce the hang on a stock 6.17-rc3 kernel on my own Flow Z13, froze within 10 cycles with Antheas’ script. I will setup pstore to get logs from it since nothing appears in my journal after force rebooting.
Matt
Mine does not want to get reproduced right now. I will have to try later.
You will need these kernel arguments:
efi_pstore.pstore_disable=0 pstore.kmsg_bytes=200000
Here are some logging commands before the for loop
# clear pstore sudo bash -c "rm -rf /sys/fs/pstore/*"
# https://www.ais.com/understanding-pstore-linux-kernel-persistent-storage-file-system/
# Runtime logs
# echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_suspend/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_resume/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/tracing_on
# Enable panics on lockups
echo 255 | sudo tee /proc/sys/kernel/sysrq
echo 1 | sudo tee /proc/sys/kernel/softlockup_panic echo 1 | sudo tee /proc/sys/kernel/hardlockup_panic echo 1 | sudo tee /proc/sys/kernel/panic_on_oops echo 5 | sudo tee /proc/sys/kernel/panic # echo 64 | sudo tee /proc/sys/kernel/panic_print
# Enable these for hangs, shows Thread on hangs
# echo 1 | sudo tee /proc/sys/kernel/softlockup_all_cpu_backtrace # echo 1 | sudo tee /proc/sys/kernel/hardlockup_all_cpu_backtrace
# Enable pstore logging on panics
# Needs kernel param:
# efi_pstore.pstore_disable=0 pstore.kmsg_bytes=100000
# First enables, second sets the size to fit all cpus in case of a panic echo Y | sudo tee /sys/module/kernel/parameters/crash_kexec_post_notifiers echo Y | sudo tee /sys/module/printk/parameters/always_kmsg_dump
# Enable dynamic debug for various kernel components sudo bash -c "cat > /sys/kernel/debug/dynamic_debug/control" << EOF file drivers/acpi/x86/s2idle.c +p
file drivers/pinctrl/pinctrl-amd.c +p
file drivers/platform/x86/amd/pmc.c +p
file drivers/pci/pci-driver.c +p
file drivers/input/serio/* +p
file drivers/gpu/drm/amd/pm/* +p
file drivers/gpu/drm/amd/pm/swsmu/* +p
EOF
# file drivers/acpi/ec.c +p # file drivers/gpu/drm/amd/* +p # file drivers/gpu/drm/amd/display/dc/core/* -p
# Additional debugging for suspend/resume
echo 1 | sudo tee /sys/power/pm_debug_messages
Here is how to reconstruct the log:
rm -rf crash && mkdir crash
sudo bash -c "cp /sys/fs/pstore/dmesg-efi_pstore-* crash" sudo bash -c "rm -rf /sys/fs/pstore/*"
cat $(find crash/ -name "dmesg-*" | tac) > crash.txt
Antheas
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 363 lines · Antheas Kapenekakis
Quoted 307 lines · Matthew Schwartz
Quoted 307 lines · Antheas Kapenekakis
Quoted 300 lines · Mario Limonciello
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
Quoted 6 lines
Quoted 2 lines
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
I can reproduce the hang on a stock 6.17-rc3 kernel on my own Flow Z13, froze within 10 cycles with Antheas’ script. I will setup pstore to get logs from it since nothing appears in my journal after force rebooting.
Matt
Mine does not want to get reproduced right now. I will have to try later.
You will need these kernel arguments:
efi_pstore.pstore_disable=0 pstore.kmsg_bytes=200000Here are some logging commands before the for loop
# clear pstore sudo bash -c "rm -rf /sys/fs/pstore/*"# https://www.ais.com/understanding-pstore-linux-kernel-persistent-storage-file-system/
# Runtime logs
# echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_suspend/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_resume/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/tracing_on# Enable panics on lockups
echo 255 | sudo tee/proc/sys/kernel/sysrqecho 1 | sudo tee /proc/sys/kernel/softlockup_panic echo 1 | sudo tee /proc/sys/kernel/hardlockup_panic echo 1 | sudo tee /proc/sys/kernel/panic_on_oops echo 5 | sudo tee /proc/sys/kernel/panic # echo 64 | sudo tee /proc/sys/kernel/panic_print# Enable these for hangs, shows Thread on hangs
# echo 1 | sudo tee /proc/sys/kernel/softlockup_all_cpu_backtrace # echo 1 | sudo tee /proc/sys/kernel/hardlockup_all_cpu_backtrace# Enable pstore logging on panics
# Needs kernel param:
# efi_pstore.pstore_disable=0 pstore.kmsg_bytes=100000
# First enables, second sets the size to fit all cpus in case of a panic echo Y | sudo tee/sys/module/kernel/parameters/crash_kexec_post_notifiersecho Y | sudo tee/sys/module/printk/parameters/always_kmsg_dump# Enable dynamic debug for various kernel components sudo bash -c "cat >
/sys/kernel/debug/dynamic_debug/control" << EOF filedrivers/acpi/x86/s2idle.c+p
filedrivers/pinctrl/pinctrl-amd.c+p
filedrivers/platform/x86/amd/pmc.c+p
filedrivers/pci/pci-driver.c+p
filedrivers/input/serio/*+p
filedrivers/gpu/drm/amd/pm/*+p
filedrivers/gpu/drm/amd/pm/swsmu/*+p
EOF
Quoted 6 lines
# file drivers/acpi/ec.c +p # file drivers/gpu/drm/amd/* +p # file drivers/gpu/drm/amd/display/dc/core/* -p# Additional debugging for suspend/resume
echo 1 | sudo tee/sys/power/pm_debug_messages
So I ran the commands that you gave above while connected over ssh, and I could actually still interact with the system after the amdgpu failures started. Your suspend script also kept running for a while because of this, and pstore was not necessary.
My dmesg looks very similar to the snippet you posted in the patch contents. Full dmesg is here: https://gist.github.com/matte-schwartz/9ad4b925866d9228923e909618d045d9
I was able to run trace-cmd as Mario suggested, but nothing seemed out of order:
❯ trace-cmd report
kworker/22:6-9326 [022] ..... 4003.204988: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/22:6-9326 [022] ..... 4003.209383: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/22:6-9326 [022] ..... 4003.210152: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/22:6-9326 [022] ..... 4004.263841: function: vpe_idle_work_handler <-- process_one_work
kworker/u129:6-530 [001] ..... 4053.545634: function: amdgpu_pmops_suspend <-- pci_pm_suspend
kworker/u129:18-4060 [002] ..... 4114.908515: function: amdgpu_pmops_resume <-- dpm_run_callback
kworker/u129:18-4060 [023] ..... 4114.931055: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/u129:18-4060 [023] ..... 4114.931057: function: vpe_ring_end_use <-- vpe_ring_test_ring
kworker/7:5-5733 [007] ..... 4115.198936: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/7:5-5733 [007] ..... 4115.203185: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/7:5-5733 [007] ..... 4115.204141: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/7:0-7950 [007] ..... 4116.253971: function: vpe_idle_work_handler <-- process_one_work
kworker/u129:41-4083 [001] ..... 4165.539388: function: amdgpu_pmops_suspend <-- pci_pm_suspend
kworker/u129:58-4100 [001] ..... 4226.906561: function: amdgpu_pmops_resume <-- dpm_run_callback
kworker/u129:58-4100 [022] ..... 4226.927900: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/u129:58-4100 [022] ..... 4226.927902: function: vpe_ring_end_use <-- vpe_ring_test_ring
kworker/7:0-7950 [007] ..... 4227.193678: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/7:0-7950 [007] ..... 4227.197604: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/7:0-7950 [007] ..... 4227.201691: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/7:0-7950 [007] ..... 4228.240479: function: vpe_idle_work_handler <-- process_one_workI have not tested the kernel patch yet, so that will be my next step.
Quoted 20 lines
Here is how to reconstruct the log:
rm -rf crash && mkdir crash
sudo bash -c "cp/sys/fs/pstore/dmesg-efi_pstore-*crash" sudo bash -c "rm -rf/sys/fs/pstore/*"
cat $(find crash/ -name "dmesg-*" | tac) > crash.txtAntheas
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
[AMD Official Use Only - AMD Internal Distribution Only]
I recently encountered a similar issue on Strix.
What I found was that the root cause was GFX failing during hw_init.
Here’s the situation:
Linux AMDGPU boot-up flow:
1.sw_init — This stage initializes the software for each IP block (GFX, VCN, VPE, etc.), and powers them on.
2.hw_init — This stage calls the hardware initialization of each IP block. At this point, VPE begins loading its firmware and configuring hardware.
The issue:
When the problem occurs, the GFX block fails during hw_init. As a result, it requests the SMU to power off all IP blocks. However, at that point, the VPE firmware hasn’t been loaded yet, so it cannot respond to the SMU's power-off request. This causes the system to hang during boot.
Previously, my approach was to remove all the calls to VPE power off (both in the VPE driver and in the SMU deinit function) to help locate the issue. Maybe you could try the same.
________________________________
寄件者: Matthew Schwartz <[email protected]>
已傳送: 星期三, 2025 年 8 月 27 日 08:50
收件者: Antheas Kapenekakis <[email protected]>
副本: Mario Limonciello <[email protected]>; Alex Deucher <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>; David Airlie <[email protected]>; Simona Vetter <[email protected]>; Wentland, Harry <[email protected]>; Rodrigo Siqueira <[email protected]>; Limonciello, Mario <[email protected]>; Lee, Peyton <[email protected]>; Yu, Lang <[email protected]> 主旨: Re: [PATCH v1 1/2] drm/amdgpu/vpe: increase VPE_IDLE_TIMEOUT to fix hang on Strix Halo
Quoted 363 lines · Antheas Kapenekakis
Quoted 307 lines · Matthew Schwartz
Quoted 307 lines · Antheas Kapenekakis
Quoted 300 lines · Mario Limonciello
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
Quoted 6 lines
Quoted 2 lines
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
I can reproduce the hang on a stock 6.17-rc3 kernel on my own Flow Z13, froze within 10 cycles with Antheas’ script. I will setup pstore to get logs from it since nothing appears in my journal after force rebooting.
Matt
Mine does not want to get reproduced right now. I will have to try later.
You will need these kernel arguments:
efi_pstore.pstore_disable=0 pstore.kmsg_bytes=200000Here are some logging commands before the for loop
# clear pstore sudo bash -c "rm -rf /sys/fs/pstore/*"# https://www.ais.com/understanding-pstore-linux-kernel-persistent-storage-file-system/
# Runtime logs
# echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_suspend/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_resume/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/tracing_on# Enable panics on lockups
echo 255 | sudo tee/proc/sys/kernel/sysrqecho 1 | sudo tee /proc/sys/kernel/softlockup_panic echo 1 | sudo tee /proc/sys/kernel/hardlockup_panic echo 1 | sudo tee /proc/sys/kernel/panic_on_oops echo 5 | sudo tee /proc/sys/kernel/panic # echo 64 | sudo tee /proc/sys/kernel/panic_print# Enable these for hangs, shows Thread on hangs
# echo 1 | sudo tee /proc/sys/kernel/softlockup_all_cpu_backtrace # echo 1 | sudo tee /proc/sys/kernel/hardlockup_all_cpu_backtrace# Enable pstore logging on panics
# Needs kernel param:
# efi_pstore.pstore_disable=0 pstore.kmsg_bytes=100000
# First enables, second sets the size to fit all cpus in case of a panic echo Y | sudo tee/sys/module/kernel/parameters/crash_kexec_post_notifiersecho Y | sudo tee/sys/module/printk/parameters/always_kmsg_dump# Enable dynamic debug for various kernel components sudo bash -c "cat >
/sys/kernel/debug/dynamic_debug/control" << EOF filedrivers/acpi/x86/s2idle.c+p
filedrivers/pinctrl/pinctrl-amd.c+p
filedrivers/platform/x86/amd/pmc.c+p
filedrivers/pci/pci-driver.c+p
filedrivers/input/serio/*+p
filedrivers/gpu/drm/amd/pm/*+p
filedrivers/gpu/drm/amd/pm/swsmu/*+p
EOF
Quoted 6 lines
# file drivers/acpi/ec.c +p # file drivers/gpu/drm/amd/* +p # file drivers/gpu/drm/amd/display/dc/core/* -p# Additional debugging for suspend/resume
echo 1 | sudo tee/sys/power/pm_debug_messages
So I ran the commands that you gave above while connected over ssh, and I could actually still interact with the system after the amdgpu failures started. Your suspend script also kept running for a while because of this, and pstore was not necessary.
My dmesg looks very similar to the snippet you posted in the patch contents. Full dmesg is here: https://gist.github.com/matte-schwartz/9ad4b925866d9228923e909618d045d9
I was able to run trace-cmd as Mario suggested, but nothing seemed out of order:
❯ trace-cmd report
kworker/22:6-9326 [022] ..... 4003.204988: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/22:6-9326 [022] ..... 4003.209383: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/22:6-9326 [022] ..... 4003.210152: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/22:6-9326 [022] ..... 4004.263841: function: vpe_idle_work_handler <-- process_one_work
kworker/u129:6-530 [001] ..... 4053.545634: function: amdgpu_pmops_suspend <-- pci_pm_suspend
kworker/u129:18-4060 [002] ..... 4114.908515: function: amdgpu_pmops_resume <-- dpm_run_callback
kworker/u129:18-4060 [023] ..... 4114.931055: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/u129:18-4060 [023] ..... 4114.931057: function: vpe_ring_end_use <-- vpe_ring_test_ring
kworker/7:5-5733 [007] ..... 4115.198936: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/7:5-5733 [007] ..... 4115.203185: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/7:5-5733 [007] ..... 4115.204141: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/7:0-7950 [007] ..... 4116.253971: function: vpe_idle_work_handler <-- process_one_work
kworker/u129:41-4083 [001] ..... 4165.539388: function: amdgpu_pmops_suspend <-- pci_pm_suspend
kworker/u129:58-4100 [001] ..... 4226.906561: function: amdgpu_pmops_resume <-- dpm_run_callback
kworker/u129:58-4100 [022] ..... 4226.927900: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/u129:58-4100 [022] ..... 4226.927902: function: vpe_ring_end_use <-- vpe_ring_test_ring
kworker/7:0-7950 [007] ..... 4227.193678: function: amdgpu_device_delayed_init_work_handler <-- process_one_work
kworker/7:0-7950 [007] ..... 4227.197604: function: vpe_ring_begin_use <-- amdgpu_ring_alloc
kworker/7:0-7950 [007] ..... 4227.201691: function: vpe_ring_end_use <-- amdgpu_ib_schedule
kworker/7:0-7950 [007] ..... 4228.240479: function: vpe_idle_work_handler <-- process_one_workI have not tested the kernel patch yet, so that will be my next step.
Quoted 20 lines
Here is how to reconstruct the log:
rm -rf crash && mkdir crash
sudo bash -c "cp/sys/fs/pstore/dmesg-efi_pstore-*crash" sudo bash -c "rm -rf/sys/fs/pstore/*"
cat $(find crash/ -name "dmesg-*" | tac) > crash.txtAntheas
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 14 lines · Peyton
[AMD Official Use Only - AMD Internal Distribution Only]
I recently encountered a similar issue on Strix.
What I found was that the root cause was GFX failing during
hw_init.Here’s the situation:
Linux AMDGPU boot-up flow:1.
sw_init— This stage initializes the software for each IP block (GFX, VCN, VPE, etc.), and powers them on.
2.hw_init— This stage calls the hardware initialization of each IP block. At this point, VPE begins loading its firmware and configuring hardware.
Quoted 6 lines
The issue:
When the problem occurs, the GFX block fails duringhw_init. As a result, it requests the SMU to power off all IP blocks. However, at that point, the VPE firmware hasn’t been loaded yet, so it cannot respond to the SMU's power-off request. This causes the system to hang during boot.Previously, my approach was to remove all the calls to VPE power off (both in the VPE driver and in the SMU deinit function) to help locate the issue. Maybe you could try the same.
Thanks, I tried both Antheas' patch and your suggestion, and I was still able to trigger the issue with both of them.
On Mario's suggestion, I updated from linux-firmware 20250808 to linux-firmware from git. With this new amdgpu_firmware_info:
VCE feature version: 0, firmware version: 0x00000000
UVD feature version: 0, firmware version: 0x00000000
MC feature version: 0, firmware version: 0x00000000
ME feature version: 35, firmware version: 0x0000001f
PFP feature version: 35, firmware version: 0x0000002c
CE feature version: 0, firmware version: 0x00000000
RLC feature version: 1, firmware version: 0x11530506
RLC SRLC feature version: 0, firmware version: 0x00000000
RLC SRLG feature version: 0, firmware version: 0x00000000
RLC SRLS feature version: 0, firmware version: 0x00000000
RLCP feature version: 1, firmware version: 0x11530506
RLCV feature version: 0, firmware version: 0x00000000
MEC feature version: 35, firmware version: 0x0000001f
IMU feature version: 0, firmware version: 0x0b352300
SOS feature version: 0, firmware version: 0x00000000
ASD feature version: 553648371, firmware version: 0x210000f3
TA XGMI feature version: 0x00000000, firmware version: 0x00000000 TA RAS feature version: 0x00000000, firmware version: 0x00000000 TA HDCP feature version: 0x00000000, firmware version: 0x17000046 TA DTM feature version: 0x00000000, firmware version: 0x12000019 TA RAP feature version: 0x00000000, firmware version: 0x00000000 TA SECUREDISPLAY feature version: 0x00000000, firmware version: 0x00000000 SMC feature version: 0, program: 0, firmware version: 0x00647000 (100.112.0) SDMA0 feature version: 60, firmware version: 0x00000011
VCN feature version: 0, firmware version: 0x09118011
DMCU feature version: 0, firmware version: 0x00000000
DMCUB feature version: 0, firmware version: 0x09002a00
TOC feature version: 0, firmware version: 0x0000000bMES_KIQ feature version: 6, firmware version: 0x0000006c
MES feature version: 1, firmware version: 0x0000007e
VPE feature version: 60, firmware version: 0x00000017
VBIOS version: 113-STRXLGEN-001
I have not been able to trigger any amdgpu failures after 200 cycles on an unpatched 6.17-rc3 kernel.
Thanks,
Matt
Quoted 435 lines
________________________________
寄件者: Matthew Schwartz <[email protected]>
已傳送: 星期三, 2025 年 8 月 27 日 08:50
收件者: Antheas Kapenekakis <[email protected]>
副本: Mario Limonciello <[email protected]>; Alex Deucher <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>; David Airlie <[email protected]>; Simona Vetter <[email protected]>; Wentland, Harry <[email protected]>; Rodrigo Siqueira <[email protected]>; Limonciello, Mario <[email protected]>; Lee, Peyton <[email protected]>; Yu, Lang <[email protected]> 主旨: Re: [PATCH v1 1/2] drm/amdgpu/vpe: increaseVPE_IDLE_TIMEOUTto fix hang on Strix HaloQuoted 363 lines · Antheas Kapenekakis
Quoted 307 lines · Matthew Schwartz
Quoted 307 lines · Antheas Kapenekakis
Quoted 300 lines · Mario Limonciello
Quoted 292 lines · Alex Deucher
Quoted 271 lines · Antheas Kapenekakis
Quoted 157 lines · Mario Limonciello
Quoted 77 lines · Antheas Kapenekakis
Quoted 60 lines · Mario Limonciello
Quoted 56 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 6 lines
Quoted 3 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume.When you say you reproduced with 12s sleep and 8s resume, was that 'amd-s2idle --duration 12 --wait 8'?
I did not use amd-s2idle. I essentially used the script below with a 12 on the wake alarm and 12 on the for loop. I also used pstore for this testing.
for i in {1..200}; do
echo "Suspend attempt $i"
echodate '+%s' -d '+ 60 seconds'| sudo tee/sys/class/rtc/rtc0/wakealarmsudo sh -c 'echo mem >/sys/power/state'Quoted 5 lines
for j in {1..50}; do
# Use repeating sleep in case echo mem returns early sleep 1
done
done👍
Quoted 42 lines
Quoted 34 lines
Quoted 29 lines
The suspected reason here is that 1s that
when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8Quoted 1 line
base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b91s idle timeout has been used by other IPs for a long time. For example JPEG, UVD, VCN all use 1s.
Quoted 5 lines
Can you please confirm both your AGESA and your SMU firmware version? In case you're not aware; you can get AGESA version from SMBIOS string (DMI type 40).
❯ sudo dmidecode | grep AGESA
String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
Quoted 8 lines
Quoted 3 lines
You can get SMU firmware version from this:
❯ grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*grep .
/sys/bus/platform/drivers/amd_pmc/*/smu_*/sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0 /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0Thanks, I'll get some folks to see if we match this AGESA version if we can also reproduce it on reference hardware the same way you did.
Quoted 6 lines
Quoted 2 lines
Are you on the most up to date firmware for your system from the manufacturer?
I updated my bios, pd firmware, and USB device firmware early August, when I was doing this testing.
Quoted 7 lines
Quoted 2 lines
We haven't seen anything like this reported on Strix Halo thus far and we do internal stress testing on s0i3 on reference hardware.
Cant find a reference for it on the bug tracker. I have four bug reports on the bazzite issue tracker, 2 about sleep wake crashes and 2 for runtime crashes, where the culprit would be this. IE runtime gates VPE and causes a crash.
Quoted 4 lines
All on Strix Halo and all tied to VPE? At runtime was VPE in use? By what software?
BTW - Strix and Kraken also have VPE.
All on the Z13. Not tied to VPE necessarily. I just know that I get reports of crashes on the Z13, and with this patch they are fixed for me. It will be part of the next bazzite version so I will get feedback about it.
I don't think software that is using the VPE is relevant. Perhaps for the runtime crashes it is and this patch helps in that case as well. But in my case, the crash is caused after the ungate that runs the tests on resume on the delayed handler.
The Z13 also has some other quirks with spurious wakeups when connected to a charger. So, if systemd is configured to e.g., sleep after 20m, combined with this crash if it stays plugged in overnight in the morning it has crashed.
Quoted 68 lines
Quoted 17 lines
Quoted 3 lines
To me this seems likely to be a platform firmware bug; but I would like to understand the timing of the gate vs ungate on good vs bad.
Perhaps it is. It is either something like that or silicon quality.
Quoted 3 lines
IE is it possible the delayed work handler
amdgpu_device_delayed_init_work_handler()is causing a race withvpe_ring_begin_use()?I don't think so. There is only a single ungate. Also, the crash happens on the gate. So what happens is the device wakes up, the screen turns on, kde clock works, then after a second it freezes, there is a softlock, and the device hangs.
The failed command is always the VPE gate that is triggered after 1s in idle.
Quoted 8 lines
Quoted 3 lines
This should be possible to check without extra instrumentation by using ftrace and looking at the timing of the 2 ring functions and the init work handler and checking good vs bad cycles.
I do not know how to use ftrace. I should also note that after the device freezes around 1/5 cycles will sync the fs, so it is also not a very easy thing to diagnose. The device just stops working. A lot of the logs I got were in pstore by forcing a kernel panic.
Here's how you capture the timing of functions. Each time the function is called there will be an event in the trace buffer.
❯ sudo trace-cmd record -p function -l
amdgpu_device_delayed_init_work_handler-lvpe_idle_work_handler-lvpe_ring_begin_use-lvpe_ring_end_use-lamdgpu_pmops_suspend-lamdgpu_pmops_resumeHere's how you would review the report:
❯ trace-cmd report
cpus=24
kworker/u97:37-18051 [001] ..... 13655.970108: function:amdgpu_pmops_suspend<--pci_pm_suspend
kworker/u97:21-18036 [002] ..... 13666.290715: function:amdgpu_pmops_resume<--dpm_run_callback
kworker/u97:21-18036 [015] ..... 13666.308295: function:vpe_ring_begin_use<--amdgpu_ring_alloc
kworker/u97:21-18036 [015] ..... 13666.308298: function:vpe_ring_end_use<--vpe_ring_test_ringkworker/15:1-12285 [015] ..... 13666.960191: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/15:1-12285 [015] ..... 13666.963970: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/15:1-12285 [015] ..... 13666.965481: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/15:4-16354 [015] ..... 13667.981394: function:
vpe_idle_work_handler<--process_one_workI did this on a Strix system just now to capture that.
You can see that basically the ring gets used before the delayed init work handler, and then again from the ring tests. My concern is if the sequence ever looks different than the above. If it does; we do have a driver race condition.
It would also be helpful to look at the
function_graphtracer.Here's some more documentation about ftrace and trace-cmd.
https://www.kernel.org/doc/html/latest/trace/ftrace.html
https://lwn.net/Articles/410200/Quoted 5 lines
You can probably also get an LLM to help you with building commands if you're not familiar with it.
But if you're hung so bad you can't flush to disk that's going to be a problem without a UART. A few ideas:
Some times it flushes to disk
Quoted 1 line
1) You can use
CONFIG_PSTORE_FTRACEI can look into that
Quoted 12 lines
2) If you add "
tp_printk" to the kernel command line it should make the trace ring buffer flush to kernel log ring buffer. But be warned this is going to change the timing, the issue might go away entirely or have a different failure rate. So hopefully <1> works.Quoted 7 lines
If you say that all IP blocks use 1s, perhaps an alternative solution would be to desync the idle times so they do not happen simultaneously. So 1000, 1200, 1400, etc.
Antheas
Quoted 8 lines
Quoted 4 lines
I don't dobut your your proposal of changing the timing works. I just want to make sure it's the right solution because otherwise we might change the timing or sequence elsewhere in the driver two years from now and re-introduce the problem unintentionally.
If there are other idle timers and only this one changes to 2s, I will agree and say that it would be peculiar. Although 1s seems arbitrary in any case.
All of these timers are arbitrary. Their point is just to provide a future point where we can check if the engine is idle. The idle work handler will either power down the IP if it is idle or re-schedule in the future and try again if there is still work. Making the value longer will use more power as it will wait longer before checking if the engine is idle. Making it shorter will save more power, but adds extra overhead in that the engine will be powered up/down more often. In most cases, the jobs should complete in a few ms. The timer is there to avoid the overhead of powering up/down the block too frequently when applications are using the engine.
Quoted 1 line
Alex
We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or 1001c AGESA on reference system but unfortunately didn't reproduce the issue with a 200 cycle attempt on either kernel or either BIOS (so we had 800 cycles total).
Quoted 6 lines
Quoted 2 lines
I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back to you with 6.17-rc3.
I can reproduce the hang on a stock 6.17-rc3 kernel on my own Flow Z13, froze within 10 cycles with Antheas’ script. I will setup pstore to get logs from it since nothing appears in my journal after force rebooting.
Matt
Mine does not want to get reproduced right now. I will have to try later.
You will need these kernel arguments:
efi_pstore.pstore_disable=0 pstore.kmsg_bytes=200000Here are some logging commands before the for loop
# clear pstore sudo bash -c "rm -rf /sys/fs/pstore/*"# https://www.ais.com/understanding-pstore-linux-kernel-persistent-storage-file-system/
# Runtime logs
# echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_suspend/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/events/power/power_runtime_resume/enable # echo 1 | sudo tee /sys/kernel/debug/tracing/tracing_on# Enable panics on lockups
echo 255 | sudo tee/proc/sys/kernel/sysrqecho 1 | sudo tee /proc/sys/kernel/softlockup_panic echo 1 | sudo tee /proc/sys/kernel/hardlockup_panic echo 1 | sudo tee /proc/sys/kernel/panic_on_oops echo 5 | sudo tee /proc/sys/kernel/panic # echo 64 | sudo tee /proc/sys/kernel/panic_print# Enable these for hangs, shows Thread on hangs
# echo 1 | sudo tee /proc/sys/kernel/softlockup_all_cpu_backtrace # echo 1 | sudo tee /proc/sys/kernel/hardlockup_all_cpu_backtrace# Enable pstore logging on panics
# Needs kernel param:
# efi_pstore.pstore_disable=0 pstore.kmsg_bytes=100000
# First enables, second sets the size to fit all cpus in case of a panic echo Y | sudo tee/sys/module/kernel/parameters/crash_kexec_post_notifiersecho Y | sudo tee/sys/module/printk/parameters/always_kmsg_dump# Enable dynamic debug for various kernel components sudo bash -c "cat >
/sys/kernel/debug/dynamic_debug/control" << EOF filedrivers/acpi/x86/s2idle.c+p
filedrivers/pinctrl/pinctrl-amd.c+p
filedrivers/platform/x86/amd/pmc.c+p
filedrivers/pci/pci-driver.c+p
filedrivers/input/serio/*+p
filedrivers/gpu/drm/amd/pm/*+p
filedrivers/gpu/drm/amd/pm/swsmu/*+p
EOFQuoted 6 lines
# file drivers/acpi/ec.c +p # file drivers/gpu/drm/amd/* +p # file drivers/gpu/drm/amd/display/dc/core/* -p# Additional debugging for suspend/resume
echo 1 | sudo tee/sys/power/pm_debug_messagesSo I ran the commands that you gave above while connected over ssh, and I could actually still interact with the system after the amdgpu failures started. Your suspend script also kept running for a while because of this, and pstore was not necessary.
My dmesg looks very similar to the snippet you posted in the patch contents. Full dmesg is here: https://gist.github.com/matte-schwartz/9ad4b925866d9228923e909618d045d9
I was able to run trace-cmd as Mario suggested, but nothing seemed out of order:
❯ trace-cmd report
kworker/22:6-9326 [022] ..... 4003.204988: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/22:6-9326 [022] ..... 4003.209383: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/22:6-9326 [022] ..... 4003.210152: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/22:6-9326 [022] ..... 4004.263841: function: vpe_idle_work_handler <-- process_one_work kworker/u129:6-530 [001] ..... 4053.545634: function: amdgpu_pmops_suspend <-- pci_pm_suspend kworker/u129:18-4060 [002] ..... 4114.908515: function: amdgpu_pmops_resume <-- dpm_run_callback kworker/u129:18-4060 [023] ..... 4114.931055: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/u129:18-4060 [023] ..... 4114.931057: function: vpe_ring_end_use <-- vpe_ring_test_ring kworker/7:5-5733 [007] ..... 4115.198936: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/7:5-5733 [007] ..... 4115.203185: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/7:5-5733 [007] ..... 4115.204141: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/7:0-7950 [007] ..... 4116.253971: function: vpe_idle_work_handler <-- process_one_work kworker/u129:41-4083 [001] ..... 4165.539388: function: amdgpu_pmops_suspend <-- pci_pm_suspend kworker/u129:58-4100 [001] ..... 4226.906561: function: amdgpu_pmops_resume <-- dpm_run_callback kworker/u129:58-4100 [022] ..... 4226.927900: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/u129:58-4100 [022] ..... 4226.927902: function: vpe_ring_end_use <-- vpe_ring_test_ring kworker/7:0-7950 [007] ..... 4227.193678: function: amdgpu_device_delayed_init_work_handler <-- process_one_work kworker/7:0-7950 [007] ..... 4227.197604: function: vpe_ring_begin_use <-- amdgpu_ring_alloc kworker/7:0-7950 [007] ..... 4227.201691: function: vpe_ring_end_use <-- amdgpu_ib_schedule kworker/7:0-7950 [007] ..... 4228.240479: function: vpe_idle_work_handler <-- process_one_workI have not tested the kernel patch yet, so that will be my next step.
Quoted 20 lines
Here is how to reconstruct the log:
rm -rf crash && mkdir crash
sudo bash -c "cp/sys/fs/pstore/dmesg-efi_pstore-*crash" sudo bash -c "rm -rf/sys/fs/pstore/*"
cat $(find crash/ -name "dmesg-*" | tac) > crash.txtAntheas
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Was your base a bazzite kernel or was it an upstream kernel? I know there are some other patches in bazzite especially relevant to suspend, so I wonder if they could be influencing the timing.
Can you repo on 6.17-rc3?
Quoted 57 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001
Quoted 7 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.
This doesn't make much sense. The VPE idle timeout is arbitrary. The VPE idle work handler checks to see if the block is idle before it powers gates the block. If it's not idle, then the delayed work is rescheduled so changing the timing should not make a difference. We are no powering down VPE while it still has active jobs. It sounds like there is some race condition somewhere else.
Alex
Quoted 28 lines
Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9--
2.50.1
Quoted 67 lines · Alex Deucher
Quoted 57 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 7 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.
Quoted 6 lines
This doesn't make much sense. The VPE idle timeout is arbitrary. The VPE idle work handler checks to see if the block is idle before it powers gates the block. If it's not idle, then the delayed work is rescheduled so changing the timing should not make a difference. We are no powering down VPE while it still has active jobs. It sounds like there is some race condition somewhere else.
On resume, the vpe is ungated and gated instantly, which does not cause any crashes, then the delayed work is scheduled to run two seconds later. Then, the tests run and finish, which start the gate timer. After the timer lapses and the kernel tries to gate VPE, it crashes. I logged all SMU commands and there is no difference between the ones in a crash and not, other than the fact the VPE gate command failed. Which becomes apparent when the next command runs. I will also note that until the idle timer lapses, the system is responsive
Since the VPE is ungated to run the tests, I assume that in my setup it is not used close to resume.
Antheas
Quoted 31 lines
Alex
Quoted 28 lines
Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9--
2.50.1
Quoted 85 lines · Antheas Kapenekakis
Quoted 67 lines · Alex Deucher
Quoted 57 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 7 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Quoted 6 lines
This doesn't make much sense. The VPE idle timeout is arbitrary. The VPE idle work handler checks to see if the block is idle before it powers gates the block. If it's not idle, then the delayed work is rescheduled so changing the timing should not make a difference. We are no powering down VPE while it still has active jobs. It sounds like there is some race condition somewhere else.
On resume, the vpe is ungated and gated instantly, which does not cause any crashes, then the delayed work is scheduled to run two seconds later. Then, the tests run and finish, which start the gate timer. After the timer lapses and the kernel tries to gate VPE, it crashes. I logged all SMU commands and there is no difference between the ones in a crash and not, other than the fact the VPE gate command failed. Which becomes apparent when the next command runs. I will also note that until the idle timer lapses, the system is responsive
Quoted 2 lines
Since the VPE is ungated to run the tests, I assume that in my setup it is not used close to resume.
I should also add that I forced a kernel panic and dumped all CPU backtraces in multiple logs. After the softlock, CPUs were either parked in the scheduler, powered off, or stuck executing an SMU command by e.g., a userspace usage sensor graph. So it is not a deadlock.
Antheas
Quoted 33 lines
Antheas
Quoted 31 lines
Alex
Quoted 28 lines
Fixes 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")Signed-off-by Antheas Kapenekakis <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c +2
-2@@ -34,8 +34,8 @@/* VPE CSA resides in the 4th page of CSA */ #define AMDGPU_CSA_VPE_OFFSET (4096 * 3) -/* 1 second timeout */ -#define VPE_IDLE_TIMEOUT msecs_to_jiffies(1000) +/* 2 second timeout */ +#define VPE_IDLE_TIMEOUT msecs_to_jiffies(2000)#define VPE_MAX_DPM_LEVEL 4#define FIXED1_8_BITS_PER_FRACTIONAL_PART 8base-commit:
c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9--
2.50.1
Quoted 86 lines · Antheas Kapenekakis
Quoted 85 lines · Antheas Kapenekakis
Quoted 67 lines · Alex Deucher
Quoted 57 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 7 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Quoted 6 lines
This doesn't make much sense. The VPE idle timeout is arbitrary. The VPE idle work handler checks to see if the block is idle before it powers gates the block. If it's not idle, then the delayed work is rescheduled so changing the timing should not make a difference. We are no powering down VPE while it still has active jobs. It sounds like there is some race condition somewhere else.
On resume, the vpe is ungated and gated instantly, which does not cause any crashes, then the delayed work is scheduled to run two seconds later. Then, the tests run and finish, which start the gate timer. After the timer lapses and the kernel tries to gate VPE, it crashes. I logged all SMU commands and there is no difference between the ones in a crash and not, other than the fact the VPE gate command failed. Which becomes apparent when the next command runs. I will also note that until the idle timer lapses, the system is responsive
Quoted 9 lines
Quoted 2 lines
Since the VPE is ungated to run the tests, I assume that in my setup it is not used close to resume.
I should also add that I forced a kernel panic and dumped all CPU backtraces in multiple logs. After the softlock, CPUs were either parked in the scheduler, powered off, or stuck executing an SMU command by e.g., a userspace usage sensor graph. So it is not a deadlock.
Can you please confirm if you are on the absolute latest linux-firmware when you reproduced this issue?
Can you please share the debugfs output for amdgpu_firmware_info.
Quoted 98 lines · Mario Limonciello
Quoted 86 lines · Antheas Kapenekakis
Quoted 85 lines · Antheas Kapenekakis
Quoted 67 lines · Alex Deucher
Quoted 57 lines · Antheas Kapenekakis
On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the suspend resumes result in a soft lock around 1 second after the screen turns on (it freezes). This happens due to power gating VPE when it is not used, which happens 1 second after inactivity.
Specifically, the VPE gating after resume is as follows: an initial ungate, followed by a gate in the resume process. Then,
amdgpu_device_delayed_init_work_handlerwith a delay of 2s is scheduled to run tests, one of which is testing VPE invpe_ring_test_ib. This causes an ungate, After that test,vpe_idle_work_handleris scheduled withVPE_IDLE_TIMEOUT(1s).When
vpe_idle_work_handlerruns and tries to gate VPE, it causes the SMU to hang and partially freezes half of the GPU IPs, with the thread that called the command being stuck processing it.Specifically, after that SMU command tries to run, we get the following:
snd_hda_intel0000:c4:00.1: Refused to change power state from D0 to D3hot
...xhci_hcd0000:c4:00.4: Refused to change power state from D0 to D3hot
...
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
[drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
[drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
[drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from0xd3
thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from0x5
thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1]flip_donetimed out
amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000 amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU. Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5, a PCI controller, and 0000:c4.00.2, resume normally.
0x00000032is the PowerDownVpe(50) command which is the common failure point in all failed resumes.On a normal resume, we should get the following power gates:
amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param:0x00010000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param:0x00000000, resp:0x00000001amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param:0x00010000, resp:0x00000001Quoted 7 lines
To fix this, increase
VPE_IDLE_TIMEOUTto 2 seconds. This increases reliability from 4-25 suspends to 200+ (tested) suspends with a cycle time of 12s sleep, 8s resume. The suspected reason here is that 1s that when VPE is used, it needs a bit of time before it can be gated and there was a borderline delay before, which is not enough for Strix Halo. When the VPE is not used, such as on resume, gating it instantly does not seem to cause issues.Quoted 6 lines
This doesn't make much sense. The VPE idle timeout is arbitrary. The VPE idle work handler checks to see if the block is idle before it powers gates the block. If it's not idle, then the delayed work is rescheduled so changing the timing should not make a difference. We are no powering down VPE while it still has active jobs. It sounds like there is some race condition somewhere else.
On resume, the vpe is ungated and gated instantly, which does not cause any crashes, then the delayed work is scheduled to run two seconds later. Then, the tests run and finish, which start the gate timer. After the timer lapses and the kernel tries to gate VPE, it crashes. I logged all SMU commands and there is no difference between the ones in a crash and not, other than the fact the VPE gate command failed. Which becomes apparent when the next command runs. I will also note that until the idle timer lapses, the system is responsive
Quoted 9 lines
Quoted 2 lines
Since the VPE is ungated to run the tests, I assume that in my setup it is not used close to resume.
I should also add that I forced a kernel panic and dumped all CPU backtraces in multiple logs. After the softlock, CPUs were either parked in the scheduler, powered off, or stuck executing an SMU command by e.g., a userspace usage sensor graph. So it is not a deadlock.
Quoted 2 lines
Can you please confirm if you are on the absolute latest linux-firmware when you reproduced this issue?
I was on the latest at the time built from source. I think it was commit 08ee93ff8ffa. There was an update today though it seems.
Quoted 1 line
Can you please share the debugfs output for
amdgpu_firmware_info.
Here is the information from it:
VCE feature version: 0, firmware version: 0x00000000
UVD feature version: 0, firmware version: 0x00000000
MC feature version: 0, firmware version: 0x00000000
ME feature version: 35, firmware version: 0x0000001f
PFP feature version: 35, firmware version: 0x0000002c
CE feature version: 0, firmware version: 0x00000000
RLC feature version: 1, firmware version: 0x11530505
RLC SRLC feature version: 0, firmware version: 0x00000000
RLC SRLG feature version: 0, firmware version: 0x00000000
RLC SRLS feature version: 0, firmware version: 0x00000000
RLCP feature version: 1, firmware version: 0x11530505
RLCV feature version: 0, firmware version: 0x00000000
MEC feature version: 35, firmware version: 0x0000001f
IMU feature version: 0, firmware version: 0x0b352300
SOS feature version: 0, firmware version: 0x00000000
ASD feature version: 553648366, firmware version: 0x210000ee
TA XGMI feature version: 0x00000000, firmware version: 0x00000000 TA RAS feature version: 0x00000000, firmware version: 0x00000000 TA HDCP feature version: 0x00000000, firmware version: 0x17000044 TA DTM feature version: 0x00000000, firmware version: 0x12000018 TA RAP feature version: 0x00000000, firmware version: 0x00000000 TA SECUREDISPLAY feature version: 0x00000000, firmware version: 0x00000000 SMC feature version: 0, program: 0, firmware version: 0x00647000 (100.112.0) SDMA0 feature version: 60, firmware version: 0x0000000e
VCN feature version: 0, firmware version: 0x0911800b
DMCU feature version: 0, firmware version: 0x00000000
DMCUB feature version: 0, firmware version: 0x09002600
TOC feature version: 0, firmware version: 0x0000000bMES_KIQ feature version: 6, firmware version: 0x0000006c
MES feature version: 1, firmware version: 0x0000007c
VPE feature version: 60, firmware version: 0x00000016
VBIOS version: 113-STRXLGEN-001
I see there was an update today though
Antheas