New: [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
https://bugzilla.kernel.org/show_bug.cgi?id=219431
Bug ID: 219431
Summary: [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could
not retrieve highest performance (-19)
Product: Power Management
Version: 2.5
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: cpufreq
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 307075
--> https://bugzilla.kernel.org/attachment.cgi?id=307075&action=edit
dmesg
With https://lore.kernel.org/lkml/[email protected]/
applied (or on 6.12-rc3+), booting on a Ryzen 9 5950X CPU (Zen 3, Vermeer) yields the newly-promoted warning:
``` [ 0.623843] Could not retrieve highest performance (-19) ```
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #1 from Ivan Shapovalov ([email protected]) --- Created attachment 307076
--> https://bugzilla.kernel.org/attachment.cgi?id=307076&action=edit/proc/cpuinfo
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #2 from Ivan Shapovalov ([email protected]) --- Created attachment 307077
--> https://bugzilla.kernel.org/attachment.cgi?id=307077&action=edit/proc/config.gz
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #3 from Mario Limonciello (AMD) ([email protected]) --- Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate dyndbg on the kernel command line for your system on 6.12rc5?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
Ivan Shapovalov ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #307075|0 |1
is obsolete| |
--- Comment #4 from Ivan Shapovalov ([email protected]) --- Created attachment 307079
--> https://bugzilla.kernel.org/attachment.cgi?id=307079&action=edit
dmesg w/ dynamic debug on amd-pstate.c/cppc.c/cppc_acpi.c
(In reply to Mario Limonciello (AMD) from comment #3)
Quoted 2 lines
Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate dyndbg on the kernel command line for your system on 6.12rc5?
Reproduced on 6.12-rc5. There was nothing from amd_pstate.c before the warning message, so I also enabled dynamic debug on cppc.c and cppc_acpi.c.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #5 from Mario Limonciello (AMD) ([email protected]) --- It looks like this message is being emitted specifically from the calls in amd_set_max_freq_ratio().
You system is a shared memory design not a MSR design.
"[ +0.000016] amd_pstate: AMD CPPC shared memory based functionality is supported"
This looks like the calltrace:amd_set_max_freq_ratio
->amd_get_boost_ratio_numerator
->->amd_detect_prefcore
->->->amd_get_highest_perf
->->->->cppc_get_highest_perf
->->->->->cppc_get_perf
The first error you see is "No CPC descriptor for CPU:1" which comes from cppc_get_perf().
Can you please share your acpidump? It sure seems like a BIOS bug to me.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #6 from Oleksandr Natalenko ([email protected]) --- Created attachment 307080
--> https://bugzilla.kernel.org/attachment.cgi?id=307080&action=edit
acpidump, 5950X, ASUS Pro WS X570-ACE
I'm experiencing this issue too. Attaching acpidump output for 5950X and ASUS Pro WS X570-ACE motherboard with BIOS version 4805.
The latest BIOS for my M/B is 4902, but I haven't got a chance to update just yet.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #7 from Mario Limonciello (AMD) ([email protected]) --- Please also share the same dyndbg output. Let's make sure we're talking about the same thing.
Are you guys seeing any functional issues you can elaborate upon?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #8 from Oleksandr Natalenko ([email protected]) --- Created attachment 307081
--> https://bugzilla.kernel.org/attachment.cgi?id=307081&action=edit
dmesg dyndbg, 5950X, ASUS Pro WS X570-ACE
Attached dmesg with dyndbg.
I do not think I can describe any functional limitations I face.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #9 from Ivan Shapovalov ([email protected]) --- Created attachment 307082
--> https://bugzilla.kernel.org/attachment.cgi?id=307082&action=editacpidump -b, 5950X, ASRock X570M-Pro4, BIOS P3.60
Attaching acpidump -b output for my system, which is a 5950X on an ASRock X570M-Pro4 motherboard. The BIOS is also quite old here.
I'm not observing any specific functional limitations *that I can track to this issue*.
However, there is this thing that basically nothing I do to the EPP knobs or the cpufreq governor has any effect on the CPU frequency management behavior. Also, sometimes under ~75% load (cgroup limited) some cores get stuck at <4GHz while others boost to expected ~4.7-ish, but I've no idea how to reproduce this one reliably.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #10 from Ivan Shapovalov ([email protected]) --- (In reply to Mario Limonciello (AMD) from comment #5)
Quoted 6 lines
It looks like this message is being emitted specifically from the calls in
amd_set_max_freq_ratio().You system is a shared memory design not a MSR design.
"[ +0.000016]amd_pstate: AMD CPPC shared memory based functionality is supported"
I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than MSR-based? That's why I mentioned Vermeer in both my original mail and this bug report.
Quoted 8 lines
This looks like the calltrace:
amd_set_max_freq_ratio
->amd_get_boost_ratio_numerator
->->amd_detect_prefcore
->->->amd_get_highest_perf
->->->->cppc_get_highest_perf
->->->->->cppc_get_perf
Quoted 4 lines
The first error you see is "No CPC descriptor for CPU:1" which comes from
cppc_get_perf().Can you please share your acpidump? It sure seems like a BIOS bug to me.
There is also an interesting observation, that at some point *after* this warning we get a pr_debug() from the end of amd_detect_prefcore(), which indicates that the latter is called again and this error does not happen anymore:
``` Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0 Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1 Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19) <...> Oct 28 17:13:08 kernel: amd_pstate: AMD CPPC shared memory based functionality is supported Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: AMD CPPC preferred core is supported (highest perf: 0xe7) ```
It's even more interesting that the debug logs from the amd-pstate flow you described earlier appear to be interleaved with debug logs from acpi_cppc_processor_probe():
``` Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0 Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0 Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1 Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19) Oct 28 17:13:08 kernel: Monitor-Mwait will be used to enter C-1 state Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 1 <-- THIS ```
I'm not familiar with control flow in these subsystems, but isn't it possible that we are simply racing with ACPI (and thus CPPC) subsystem initialization?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #11 from Mario Limonciello (AMD) ([email protected]) ---
Quoted 3 lines
I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than MSR-based? That's why I mentioned Vermeer in both my original mail and this bug report.
Vermeer is a shared memory based design. All the newer designs are MSR based.
Quoted 4 lines
There is also an interesting observation, that at some point *after* this warning we get a
pr_debug()from the end ofamd_detect_prefcore(), which indicates that the latter is called again and this error does not happen anymore:
Yeah that's why I was wondering if there was a functional issue. It seems like this is going to influence some of the frequency invariance code that's set up earlier on.
Quoted 2 lines
I'm not familiar with control flow in these subsystems, but isn't it possible that we are simply racing with ACPI (and thus CPPC) subsystem initialization?
acpi_init() calls
acpi_bus_init()() callsacpi_bus_osc_negotiate_platform_control()
which sets osc_sb_cppc2_support_acked
Here is the more full call path that fails:
acpi_init() callsacpi_scan_init() callsacpi_processor_init()
which loads "processor" kernel module
From "processor" kernel module:
acpi_processor_driver_init() calls acpi_soft_cpu_online() calls __acpi_processor_start() calls acpi_cppc_processor_probe() calls arch_init_invariance_cppc() which is a macro to call init_freq_invariance_cppc().
I'm not sure I understand why CPPC v2 _OSC isn't acked though at the right time.
Can you use ftrace or annotate the functions with some printk's to figure it out perhaps from what I've shared?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #12 from Mario Limonciello (AMD) ([email protected]) --- Something else that would be really useful to understand is whether this is only showing up now because the error level was raised
(IE : https://git.kernel.org/superm1/c/21fb59ab4b976) or if it was there in past kernels too, but because it was debug no one ever saw it.
Try turning up dynamic debug on arch/x86/kernel/acpi/cppc.c on an older kernel that doesn't have that commit and see if you see it there too.
I suspect the answer is yes.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #13 from Mario Limonciello (AMD) ([email protected]) ---
Quoted 1 line
I suspect the answer is yes.
Actually; no it should be new, but it's because the frequency calculation was wrong before.
I found a system with shared mem and I have a theory what's going on is that frequency invariance is attempting to get setup before all CPUs have probed.
Before the preferred cores code frequency invariance would be calculated from the first CPU only as _CPC is probed. But with preferred cores code in place you need ALL _CPC values present to know whether you have preferred cores or not.
I added some dump stack calls to acpi_cppc_processor_probe when the struct is parsed:
ACPI CPPC: Parsed CPC struct for CPU: 0
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd #268 Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024 Call Trace: <TASK> dump_stack_lvl+0x60/0x80 acpi_cppc_processor_probe+0x9fc/0xe50 acpi_soft_cpu_online+0x69/0x160 ? __pfx_acpi_soft_cpu_online+0x10/0x10 cpuhp_invoke_callback+0x308/0x4c0 ? srso_return_thunk+0x5/0x5f ? __pfx_smpboot_thread_fn+0x10/0x10 cpuhp_thread_fun+0x131/0x210 smpboot_thread_fn+0x187/0x220 kthread+0xdd/0x110 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x30/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
Then I added one to the error sequence.
ACPI CPPC: No CPC descriptor for CPU:1
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd #268 Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024 Call Trace: <TASK> dump_stack_lvl+0x60/0x80 cppc_get_perf+0x127/0x1c0 amd_get_highest_perf+0x5e/0x80 amd_detect_prefcore+0x7a/0x130 amd_get_boost_ratio_numerator+0x28/0x90 init_freq_invariance_cppc+0xad/0x140 acpi_cppc_processor_probe+0x7ce/0xe50 acpi_soft_cpu_online+0x69/0x160 ? __pfx_acpi_soft_cpu_online+0x10/0x10 cpuhp_invoke_callback+0x308/0x4c0 ? srso_return_thunk+0x5/0x5f ? __pfx_smpboot_thread_fn+0x10/0x10 cpuhp_thread_fun+0x131/0x210 smpboot_thread_fn+0x187/0x220 kthread+0xdd/0x110 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x30/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
Could not retrieve highest performance (-19)
You can see that in both cases the code runs on CPU0. So it's trying to get values that won't be initialized until CPU1 is ready.
I think this will help "prevent" the error from coming up:
arch/x86/kernel/acpi/cppc.c +1 -1
@@ -190,7 +190,7 @@ int amd_detect_prefcore(bool *detected)break ; } -for_each_present_cpu (cpu) { +for_each_online_cpu (cpu) {u32 tmp;int ret;
But it's going to probably break preferred cores detection, so it's not a correct solution.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #14 from Mario Limonciello (AMD) ([email protected]) --- Created attachment 307084
--> https://bugzilla.kernel.org/attachment.cgi?id=307084&action=edit
possible patch (v1)
Here's a potential patch. This helps on the system I can reproduce this issue, please see if helps you too.
Also if it does help, would you mind if I used your email address in the commit message for the Reported-and-tested-by tag? I can get it from Bugzilla, but I would like your permission first.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #15 from Ivan Shapovalov ([email protected]) --- I'll test this later today, thanks. (I also want to try to dig into _OSC CPCv2 flag not being set, just to definitively confirm if this is a firmware issue or an ordering issue.)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #16 from Oleksandr Natalenko ([email protected]) --- The message disappears with this patch for me. Should I check something else too?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #17 from Mario Limonciello (AMD) ([email protected]) --- Thanks, you can check whether the CPU cores are reporting correct maximum frequencies.
Can I use your email when I post it?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
--- Comment #18 from Oleksandr Natalenko ([email protected]) ---
Quoted 2 lines
you can check whether the CPU cores are reporting correct maximum frequencies.
In amd_pstate_max_freq it's 5084000 on all cores. Running turbostat parallel to xmrig I see ~4.4 GHz on all cores under heavy load. In this regard, for my setup, nothing changed comparing to an unpatched kernel, I guess.
Quoted 1 line
Can I use your email when I post it?
Sure.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219431
Mario Limonciello (AMD) ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
Assignee|[email protected] |[email protected]--- Comment #19 from Mario Limonciello (AMD) ([email protected]) ---
https://lore.kernel.org/linux-acpi/[email protected]/
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.