x86/mce: Lower throttling MCE messages to warnings
Subsystems (3)
PATCH 1/1 x86/mce: Lower throttling MCE messages to warnings
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is not sufficient to cool the CPU at full load and limits can quickly be reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about their severity.
--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/therm_throt.c +1 -1
@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) -pr_crit ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , +pr_warn ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , this_cpu, level ==CORE_LEVEL ?"Core" :"Package" , state->count);
Signature · 1 line
2.23.0
Hi,
Quoted 7 lines · Benjamin Berg
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is not sufficient to cool the CPU at full load and limits can quickly be reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
Quoted 6 lines
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about their severity.
Signed-off-by Benjamin Berg <[email protected]>Tested-by Christian Kellner <[email protected]>
Ah, yes lets please lower the log-prio of these messages:
Regards,
Hans
Quoted 18 lines
--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/mce/therm_throt.c +1
-1@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) -pr_crit ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , +pr_warn ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , this_cpu, level ==CORE_LEVEL ?"Core" :"Package" , state->count);
Borislav Petkov
maintainerQuoted 23 lines · Benjamin Berg
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is not sufficient to cool the CPU at full load and limits can quickly be reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about their severity.
Signed-off-by Benjamin Berg <[email protected]>Tested-by Christian Kellner <[email protected]>--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/mce/therm_throt.c +0
-0
arch/x86/kernel/cpu/mce/therm_throt.c static void therm_throt_process(bool new_event, int event, int level) · line 194
Quoted 8 lines
arch/x86/kernel/cpu/mce/therm_throt.c +1
-1@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) -pr_crit ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , +pr_warn ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , this_cpu, level ==CORE_LEVEL ?"Core" :"Package" , state->count);
This has carried over since its very first addition in
commit 3867eb75b9279c7b0f6840d2ad9f27694ba6c4e4
Author: Dave Jones <[email protected]> Date: Tue Apr 2 20:02:27 2002 -0800
[PATCH] x86 bluesmoke update.
o Make MCE compile time optional (Paul Gortmaker)
o P4 thermal trip monitoring. (Zwane Mwaikambo)
o Non-fatal MCE logging. (Me)
It used to be KERN_EMERG back then, though.
And yes, this issue has come up in the past already so I think I'll take it. I'll just give Intel folks a couple of days to object should there be anything to object to.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 11 lines · Borislav Petkov
Quoted 10 lines · Benjamin Berg
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is not sufficient to cool the CPU at full load and limits can quickly be reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about their severity.
[]
Quoted 1 line
arch/x86/kernel/cpu/mce/therm_throt.c
Quoted 1 line
arch/x86/kernel/cpu/mce/therm_throt.c +0
-0
[]
Quoted 24 lines
arch/x86/kernel/cpu/mce/therm_throt.c static void therm_throt_process(bool new_event, int event, int level) · line 194
Quoted 10 lines
arch/x86/kernel/cpu/mce/therm_throt.c +1
-1@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) -pr_crit ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , +pr_warn ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , this_cpu, level ==CORE_LEVEL ?"Core" :"Package" , state->count);This has carried over since its very first addition in
commit
3867eb75b9279c7b0f6840d2ad9f27694ba6c4e4Author: Dave Jones <[email protected]> Date: Tue Apr 2 20:02:27 2002 -0800[PATCH] x86 bluesmoke update.
o Make MCE compile time optional (Paul Gortmaker)
o P4 thermal trip monitoring. (Zwane Mwaikambo)
o Non-fatal MCE logging. (Me)
Quoted 5 lines
It used to be
KERN_EMERGback then, though.And yes, this issue has come up in the past already so I think I'll take it. I'll just give Intel folks a couple of days to object should there be anything to object to.
Perhaps this should be
pr_warn_ratelimited(...)
as the temperature changes can be relatively quick.
Borislav Petkov
maintainerQuoted 5 lines · Joe Perches
Perhaps this should be
pr_warn_ratelimited(...)as the temperature changes can be relatively quick.
There's already ratelimiting machinery a bit above in the same function.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 8 lines · Borislav Petkov
Quoted 5 lines · Joe Perches
Perhaps this should be
pr_warn_ratelimited(...)as the temperature changes can be relatively quick.
There's already ratelimiting machinery a bit above in the same function.
right, thanks, nevermind...
Hi Benjamin,
Quoted 10 lines · Borislav Petkov
Quoted 9 lines · Benjamin Berg
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is
not sufficient to cool the CPU at full load and limits can quickly be
reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
Quoted 5 lines
Quoted 5 lines
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about
their severity.
I have a patch to address this. Instead of avoiding any critical warnings or wait for 300 seconds for next one, the warning is based on how long the system is working on throttled condition. If for example the fan broke, then the throttling is extended for a long time. Then we better warn.
I am waiting for internal review, and hope to post by tomorrow.
Thanks
Srinivas
Quoted 49 lines
Quoted 22 lines
Signed-off-by Benjamin Berg <[email protected]>Tested-by Christian Kellner <[email protected]>--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/mce/therm_throt.c +1
-1@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level) /* if we just entered the thermal event */ if (new_event) { if (event == THERMAL_THROTTLING_EVENT) - pr_crit("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n", + pr_warn("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n",arch/x86/kernel/cpu/mce/therm_throt.c +0
-0this_cpu, level == CORE_LEVEL ? "Core" :Quoted 3 lines
"Package",
state->count);This has carried over since its very first addition in
commit
3867eb75b9279c7b0f6840d2ad9f27694ba6c4e4Author: Dave Jones <[email protected]> Date: Tue Apr 2 20:02:27 2002 -0800[PATCH] x86 bluesmoke update.
o Make MCE compile time optional (Paul Gortmaker)
o P4 thermal trip monitoring. (Zwane Mwaikambo)
o Non-fatal MCE logging. (Me)It used to be
KERN_EMERGback then, though.And yes, this issue has come up in the past already so I think I'll take
it. I'll just give Intel folks a couple of days to object should there
be anything to object to.Thx.
Hi Srinivas,
Quoted 6 lines · Srinivas Pandruvada
I have a patch to address this. Instead of avoiding any critical warnings or wait for 300 seconds for next one, the warning is based on how long the system is working on throttled condition. If for example the fan broke, then the throttling is extended for a long time. Then we better warn.
I am waiting for internal review, and hope to post by tomorrow.
Nice! I agree that a heuristic seems better than the very simple approach taken in this patch.
Thanks,
Benjamin
Quoted 53 lines
Thanks
SrinivasQuoted 50 lines
Quoted 23 lines
Signed-off-by Benjamin Berg <[email protected]>Tested-by Christian Kellner <[email protected]>--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/mce/therm_throt.c +1
-1@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) - pr_crit("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n", + pr_warn("CPU%d: %s temperature above threshold,cpu clock throttled (total events = %lu)\n",
arch/x86/kernel/cpu/mce/therm_throt.c +0
-0this_cpu, level == CORE_LEVEL ? "Core" :Quoted 3 lines
"Package",
state->count);This has carried over since its very first addition in
commit
3867eb75b9279c7b0f6840d2ad9f27694ba6c4e4Author: Dave Jones <[email protected]> Date: Tue Apr 2 20:02:27 2002 -0800[PATCH] x86 bluesmoke update.
o Make MCE compile time optional (Paul Gortmaker)
o P4 thermal trip monitoring. (Zwane Mwaikambo)
o Non-fatal MCE logging. (Me)It used to be
KERN_EMERGback then, though.And yes, this issue has come up in the past already so I think I'll take
it. I'll just give Intel folks a couple of days to object should there
be anything to object to.Thx.
[tip: ras/core] x86/mce: Lower throttling MCE messages' priority to warning
In reply to Benjamin BergThe following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: 9c3bafaa1fd88e4dd2dba3735a1f1abb0f2c7bb7 Gitweb: https://git.kernel.org/tip/9c3bafaa1fd88e4dd2dba3735a1f1abb0f2c7bb7 Author: Benjamin Berg <[email protected]> AuthorDate: Wed, 09 Oct 2019 17:54:24 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Thu, 17 Oct 2019 09:07:09 +02:00
x86/mce: Lower throttling MCE messages' priority to warning
On modern CPUs it is quite normal that the temperature limits are reached and the CPU is throttled. In fact, often the thermal design is not sufficient to cool the CPU at full load and limits can quickly be reached when a burst in load happens. This will even happen with technologies like RAPL limitting the long term power consumption of the package.
Also, these limits are "softer", as Srinivas explains:
"CPU temperature doesn't have to hit max(TjMax) to get these warnings. OEMs ha[ve] an ability to program a threshold where a thermal interrupt can be generated. In some systems the offset is 20C+ (Read only value).
In recent systems, there is another offset on top of it which can be programmed by OS, once some agent can adjust power limits dynamically. By default this is set to low by the firmware, which I guess the prime motivation of Benjamin to submit the patch."
So these messages do not usually indicate a hardware issue (e.g. insufficient cooling). Log them as warnings to avoid confusion about their severity.
[ bp: Massage commit mesage. ]
--- arch/x86/kernel/cpu/mce/therm_throt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/therm_throt.c +1 -1
@@ -188,7 +188,7 @@ static void therm_throt_process(bool new_event, int event, int level)/* if we just entered the thermal event */ if (new_event) {if (event ==THERMAL_THROTTLING_EVENT ) -pr_crit ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , +pr_warn ("CPU%d: %s temperature above threshold, cpu clock throttled (total events = %lu)\n" , this_cpu, level ==CORE_LEVEL ?"Core" :"Package" , state->count);