Skip to contentSkip to conversation

x86/mce/therm_throt: Fix the access of uninitialized therm_work

PATCH Superseded Chuansheng Liu 3 participants 9 messages Jan 6, 2020 – Jan 11, 2020

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/mce/therm_throt: Fix the access of uninitialized therm_work

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker. Such race will cause the posssibility that interrupt handler therm_throt_process() accesses uninitialized therm_work, then system hit panic at very early bootup stage.

In my ICL platform, it can be reproduced in several times of reboot stress. With this fix, the system keeps alive for more than 200 times of reboot stress.

---
 arch/x86/kernel/cpu/mce/therm_throt.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mce/therm_throt.c +8 -2
index b38010b541d6..7320eb3ac029 100644
@@ -86,6 +86,7 @@ struct _thermal_state {
 	unsigned long		total_time_ms;
 	bool			rate_control_active;
 	bool			new_event;
+	bool			therm_work_active;
 	u8			level;
 	u8			sample_index;
 	u8			sample_count;
@@ -359,7 +360,9 @@ static void therm_throt_process(bool new_event, int event, int level)
 
 		state->baseline_temp = temp;
 		state->last_interrupt_time = now;
-		schedule_delayed_work_on(this_cpu, &state->therm_work, THERM_THROT_POLL_INTERVAL);
+		if (state->therm_work_active)
+			schedule_delayed_work_on(this_cpu, &state->therm_work,
+					THERM_THROT_POLL_INTERVAL);
 	} else if (old_event && state->last_interrupt_time) {
 		unsigned long throttle_time;
 
@@ -473,7 +476,8 @@ static int thermal_throttle_online(unsigned int cpu)
 
 	INIT_DELAYED_WORK(&state->package_throttle.therm_work, throttle_active_work);
 	INIT_DELAYED_WORK(&state->core_throttle.therm_work, throttle_active_work);
-
+	state->package_throttle.therm_work_active = true;
+	state->core_throttle.therm_work_active = true;
 	return thermal_throttle_add_dev(dev, cpu);
 }
 
@@ -482,6 +486,8 @@ static int thermal_throttle_offline(unsigned int cpu)
 	struct thermal_state *state = &per_cpu(thermal_state, cpu);
 	struct device *dev = get_cpu_device(cpu);
 
+	state->package_throttle.therm_work_active = false;
+	state->core_throttle.therm_work_active = false;
 	cancel_delayed_work(&state->package_throttle.therm_work);
 	cancel_delayed_work(&state->core_throttle.therm_work);
 
Signature · 1 line
2.17.1
maintainer
#1 lore raw
Quoted 10 lines · Chuansheng Liu

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:
Quoted 8 lines

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker.

You mean the unmasking of the thermal vector at the end of intel_init_thermal()?

If so, why don't you move that to the end of the notifier and unmask it only after all the necessary work like setting up the workqueues etc, is done, and save yourself adding yet another silly bool?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#2 lore raw
Quoted 20 lines · Borislav Petkov
Quoted 10 lines · Chuansheng Liu

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:
Quoted 8 lines

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker.

Quoted 6 lines

You mean the unmasking of the thermal vector at the end of intel_init_thermal()?

If so, why don't you move that to the end of the notifier and unmask it only after all the necessary work like setting up the workqueues etc, is done, and save yourself adding yet another silly bool?

A debugging WARN_ON_ONCE() when the workqueue is not initialized yet would also be useful I suspect. This would turn any remaining race-crash boot failure in this area into a warning.

Thanks,

Ingo

#3 lore raw
Quoted 37 lines

-----Original Message-----
From: Ingo Molnar <[email protected]> On Behalf Of Ingo Molnar
Sent: Monday, January 6, 2020 3:11 PM
To: Borislav Petkov <[email protected]>
Cc: Liu, Chuansheng <[email protected]>; [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] x86/mce/therm_throt: Fix the access of uninitialized therm_work

Quoted 23 lines · Borislav Petkov
Quoted 10 lines · Chuansheng Liu

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:
Quoted 8 lines

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker.

You mean the unmasking of the thermal vector at the end of intel_init_thermal()?

Quoted 8 lines
Quoted 3 lines

If so, why don't you move that to the end of the notifier and unmask it only after all the necessary work like setting up the workqueues etc, is done, and save yourself adding yet another silly bool?

A debugging WARN_ON_ONCE() when the workqueue is not initialized yet would also be useful I suspect. This would turn any remaining race-crash boot failure in this area into a warning.

Thanks Ingo.
That's a good suggestion, I can try to make another patch about it later.

Best Regards
Chuansheng

#4 lore raw
In reply to Ingo Molnar

Hi Ingo,

Quoted 35 lines

-----Original Message-----
From: Ingo Molnar <[email protected]> On Behalf Of Ingo Molnar
Sent: Monday, January 6, 2020 3:11 PM
To: Borislav Petkov <[email protected]>
Cc: Liu, Chuansheng <[email protected]>; [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] x86/mce/therm_throt: Fix the access of uninitialized therm_work

Quoted 23 lines · Borislav Petkov
Quoted 10 lines · Chuansheng Liu

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:
Quoted 8 lines

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker.

You mean the unmasking of the thermal vector at the end of intel_init_thermal()?

Quoted 8 lines
Quoted 3 lines

If so, why don't you move that to the end of the notifier and unmask it only after all the necessary work like setting up the workqueues etc, is done, and save yourself adding yet another silly bool?

A debugging WARN_ON_ONCE() when the workqueue is not initialized yet would also be useful I suspect. This would turn any remaining race-crash boot failure in this area into a warning.

Just checked the code, the WARN_ON_ONCE() is already there:
1622 WARN_ON_ONCE(timer->function != delayed_work_timer_fn);

With reproducing it, the corresponding log also shows before panic:

WARNING: CPU: 0 .... at kernel/workqueue.c:1622 __queue_delayed_work+0x73/0x90

Thanks for your reminder.

#5 lore raw
In reply to Borislav Petkov
Quoted 28 lines

-----Original Message-----
From: Borislav Petkov <[email protected]>
Sent: Monday, January 6, 2020 3:08 PM
To: Liu, Chuansheng <[email protected]>
Cc: [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] x86/mce/therm_throt: Fix the access of uninitialized therm_work

Quoted 16 lines · Chuansheng Liu

In ICL platform, it is easy to hit bootup failure with panic in thermal interrupt handler during early bootup stage.

Such issue makes my platform almost can not boot up with latest kernel code.

The call stack is like:
kernel BUG at kernel/timer/timer.c:1152!

Call Trace:

__queue_delayed_work
queue_delayed_work_on
therm_throt_process
intel_thermal_interrupt
...

Quoted 5 lines
Quoted 2 lines

When one CPU is up, the irq is enabled prior to CPU UP notification which will then initialize therm_worker.

You mean the unmasking of the thermal vector at the end of intel_init_thermal()?

Exactly, and there is one local CPU irq enable later too.

Quoted 5 lines

If so, why don't you move that to the end of the notifier and unmask it only after all the necessary work like setting up the workqueues etc, is done, and save yourself adding yet another silly bool?

Thanks for your suggestion, I am just worried about the interrupt delay. I traced there is about 2s gap between unmask interrupt and workqueue Initialization. If you think it is OK to ignore this delay, I will make another simple patch as you suggested😊

Best Regards
Chuansheng

maintainer
#6 lore raw
Quoted 2 lines · Chuansheng

I traced there is about 2s gap between unmask interrupt and workqueue Initialization.

And that is a problem because?

You setup workqueue etc and *then* unmask the irq.

Signature · 1 line
Sent from a small device: formatting sux and brevity is inevitable.
#7 lore raw
Quoted 11 lines

-----Original Message-----
From: Boris Petkov <[email protected]>
Sent: Monday, January 6, 2020 6:01 PM
To: Liu, Chuansheng <[email protected]>
Cc: [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH] x86/mce/therm_throt: Fix the access of uninitialized therm_work

Quoted 9 lines
Quoted 2 lines · Chuansheng

I traced there is about 2s gap between unmask interrupt and workqueue Initialization.

And that is a problem because?

You setup workqueue etc and *then* unmask the irq.

Some previous experience shows:
If there is critical thermal alert, we still can take action in kernel side in this 2s, even though the workqueue is not ready, but interrupt handler can work well.

maintainer
#8 lore raw
Quoted 3 lines · Chuansheng

If there is critical thermal alert, we still can take action in kernel side in this
2s

What critical thermal alert during boot and what action can you take that early that can't wait 2 seconds?

Signature · 1 line
Sent from a small device: formatting sux and brevity is inevitable.