Skip to contentSkip to conversation

x86/mce/therm_throt: Fix the access of uninitialized therm_work

PATCH Applied Chuansheng Liu 4 participants 5 messages Jan 7, 2020 – Jan 15, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 x86/mce/therm_throt: Fix the access of uninitialized therm_work

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker. Such race will cause the posssibility that interrupt handler therm_throt_process() accesses uninitialized therm_work, then system hit panic at very early bootup stage.

In my ICL platform, it can be reproduced in several times of reboot stress. With this fix, the system keeps alive for more than 200 times of reboot stress.

V2: Boris shares a good suggestion that we can moving the interrupt unmasking at the end of therm_work initialization.

---
 arch/x86/kernel/cpu/mce/therm_throt.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mce/therm_throt.c +7 -4
index b38010b541d6..528b85664b46 100644
@@ -467,6 +467,7 @@ static int thermal_throttle_online(unsigned int cpu)
 {
 	struct thermal_state *state = &per_cpu(thermal_state, cpu);
 	struct device *dev = get_cpu_device(cpu);
+	u32 l;
 
 	state->package_throttle.level = PACKAGE_LEVEL;
 	state->core_throttle.level = CORE_LEVEL;
@@ -474,6 +475,12 @@ static int thermal_throttle_online(unsigned int cpu)
 	INIT_DELAYED_WORK(&state->package_throttle.therm_work, throttle_active_work);
 	INIT_DELAYED_WORK(&state->core_throttle.therm_work, throttle_active_work);
 
+	/* Unmask the thermal vector after
+	 * therm_works are initialized.
+	 */
+	l = apic_read(APIC_LVTTHMR);
+	apic_write(APIC_LVTTHMR, l & ~APIC_LVT_MASKED);
+
 	return thermal_throttle_add_dev(dev, cpu);
 }
 
@@ -722,10 +729,6 @@ void intel_init_thermal(struct cpuinfo_x86 *c)
 	rdmsr(MSR_IA32_MISC_ENABLE, l, h);
 	wrmsr(MSR_IA32_MISC_ENABLE, l | MSR_IA32_MISC_ENABLE_TM1, h);
 
-	/* Unmask the thermal vector: */
-	l = apic_read(APIC_LVTTHMR);
-	apic_write(APIC_LVTTHMR, l & ~APIC_LVT_MASKED);
-
 	pr_info_once("CPU0: Thermal monitoring enabled (%s)\n",
 		      tm2 ? "TM2" : "TM1");
 
Signature · 1 line
2.17.1
maintainer Acked-by on v2 1/1
#1 lore raw
Quoted 8 lines · Chuansheng Liu

In my ICL platform, it can be reproduced in several times of reboot stress. With this fix, the system keeps alive for more than 200 times of reboot stress.

V2: Boris shares a good suggestion that we can moving the interrupt unmasking at the end of therm_work initialization.

Looks good to me:

maintainer
#2 lore raw
Quoted 5 lines · Tony
Quoted 4 lines · Chuansheng Liu

In my ICL platform, it can be reproduced in several times of reboot stress. With this fix, the system keeps alive for more than 200 times of reboot stress.

Quoted 8 lines
Quoted 4 lines

V2: Boris shares a good suggestion that we can moving the interrupt unmasking at the end of therm_work initialization.

Looks good to me:

Thx.

This "ICL platform" - whatever that is - is this shipping already so that this qualifies for stable@ or can it go the normal path?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw
Quoted 23 lines

-----Original Message-----
From: Borislav Petkov <[email protected]>
Sent: Monday, January 13, 2020 5:05 PM
To: Luck, Tony <[email protected]>
Cc: Liu, Chuansheng <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2] x86/mce/therm_throt: Fix the access of uninitialized therm_work

Quoted 10 lines · Tony
Quoted 8 lines · Chuansheng Liu

In my ICL platform, it can be reproduced in several times of reboot stress. With this fix, the system keeps alive for more than 200 times of reboot stress.

V2: Boris shares a good suggestion that we can moving the interrupt unmasking at the end of therm_work initialization.

Quoted 7 lines
Quoted 3 lines

Looks good to me:

Thx.

This "ICL platform" - whatever that is - is this shipping already so

I just can say ICL(icelake) is shipped platform, I reproduced this issue in one laptop.

merge bot Applied
#4 lore raw

[tip: ras/urgent] x86/mce/therm_throt: Do not access uninitialized therm_work

In reply to Chuansheng Liu
The following commit has been merged into the ras/urgent branch of tip:

The following commit has been merged into the ras/urgent branch of tip:

Commit-ID:     978370956d2046b19313659ce65ed12d5b996626
Gitweb:        https://git.kernel.org/tip/978370956d2046b19313659ce65ed12d5b996626
Author:        Chuansheng Liu <[email protected]>
AuthorDate:    Tue, 07 Jan 2020 00:41:16 
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Wed, 15 Jan 2020 11:31:33 +01:00

x86/mce/therm_throt: Do not access uninitialized therm_work

It is relatively easy to trigger the following boot splat on an Ice Lake client platform. The call stack is like:

kernel BUG at kernel/timer/timer.c:1152!

  Call Trace:
  __queue_delayed_work
  queue_delayed_work_on
  therm_throt_process
  intel_thermal_interrupt
  ...

The reason is that a CPU's thermal interrupt is enabled prior to executing its hotplug onlining callback which will initialize the throttling workqueues.

Such a race can lead to therm_throt_process() accessing an uninitialized therm_work, leading to the above BUG at a very early bootup stage.

Therefore, unmask the thermal interrupt vector only after having setup the workqueues completely.

[ bp: Heavily massage commit message and correct comment formatting. ]

Fixes f6656208f04e ("x86/mce/therm_throt: Optimize notifications of thermal throttle")
---
 arch/x86/kernel/cpu/mce/therm_throt.c |  9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mce/therm_throt.c +5 -4
index b38010b..6c3e1c9 100644
@@ -467,6 +467,7 @@ static int thermal_throttle_online(unsigned int cpu)
 {
 	struct thermal_state *state = &per_cpu(thermal_state, cpu);
 	struct device *dev = get_cpu_device(cpu);
+	u32 l;
 
 	state->package_throttle.level = PACKAGE_LEVEL;
 	state->core_throttle.level = CORE_LEVEL;
@@ -474,6 +475,10 @@ static int thermal_throttle_online(unsigned int cpu)
 	INIT_DELAYED_WORK(&state->package_throttle.therm_work, throttle_active_work);
 	INIT_DELAYED_WORK(&state->core_throttle.therm_work, throttle_active_work);
 
+	/* Unmask the thermal vector after the above workqueues are initialized. */
+	l = apic_read(APIC_LVTTHMR);
+	apic_write(APIC_LVTTHMR, l & ~APIC_LVT_MASKED);
+
 	return thermal_throttle_add_dev(dev, cpu);
 }
 
@@ -722,10 +727,6 @@ void intel_init_thermal(struct cpuinfo_x86 *c)
 	rdmsr(MSR_IA32_MISC_ENABLE, l, h);
 	wrmsr(MSR_IA32_MISC_ENABLE, l | MSR_IA32_MISC_ENABLE_TM1, h);
 
-	/* Unmask the thermal vector: */
-	l = apic_read(APIC_LVTTHMR);
-	apic_write(APIC_LVTTHMR, l & ~APIC_LVT_MASKED);
-
 	pr_info_once("CPU0: Thermal monitoring enabled (%s)\n",
 		      tm2 ? "TM2" : "TM1");