Skip to contentSkip to conversation

Bug: d0e936adbd22 crashes at boot

DISCUSSION Jens Axboe 2 participants 10 messages Sep 3, 2021

Subsystems (2)
lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This was a bit difficult to pin down as it crashes before the display is up, but I managed to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace

Author: Srinivas Pandruvada <[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

cpufreq: intel_pstate: Process HWP Guaranteed change notification

which crashes with a NULL pointer deref in notify_hwp_interrupt() -> queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Signature · 1 line
Jens Axboe
#1 lore raw

Hi Axboe,

Thanks for reporting.

Quoted 12 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This was a bit
difficult to pin down as it crashes before the display is up, but I managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <[email protected]> Date:   Thu Aug 19 19:40:06 2021 -0700

Quoted 7 lines

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

which crashes with a NULL pointer deref in notify_hwp_interrupt() -> queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

(quoted diff) +0 -0

b/drivers/cpufreq/intel_pstate.c
index b4ffe6c8a0d0..6a3c6f60ad12 100644

drivers/cpufreq/intel_pstate.c +0 -1
@@ -1650,7 +1650,10 @@ void notify_hwp_interrupt(void)
                return;
 
        cpudata = all_cpu_data[this_cpu];
-       schedule_delayed_work_on(this_cpu, &cpudata->hwp_notify_work,

msecs_to_jiffies(10));
+ if (cpudata)
+ schedule_delayed_work_on(this_cpu, &cpudata-

Quoted 1 line

hwp_notify_work, msecs_to_jiffies(10));

+ else
+ wrmsrl(MSR_HWP_STATUS, 0);
}

static void intel_pstate_enable_hwp_interrupt(struct cpudata *cpudata)

Thanks,
Srinivas

author
#2 lore raw
Quoted 16 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 12 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This was a bit
difficult to pin down as it crashes before the display is up, but I managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <[email protected]> Date:   Thu Aug 19 19:40:06 2021 -0700

Quoted 8 lines
Quoted 7 lines

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

which crashes with a NULL pointer deref in notify_hwp_interrupt() -> queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look like a real fix and more of a work-around.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 27 lines · Jens Axboe
Quoted 26 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 22 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This was
a bit
difficult to pin down as it crashes before the display is up, but I managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <[email protected]> Date:   Thu Aug 19 19:40:06 2021 -0700

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

which crashes with a NULL pointer deref in notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Quoted 7 lines
Quoted 1 line

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look like
a real fix and more of a work-around.

This platform is sending an HWP interrupt on a CPU which we didn't yet bring it up for pstate control. So somehow firmware decided to send very early during boot, which previously we would have ignored it

Actually try this, with more prevention

(quoted diff) +0 -0

b/drivers/cpufreq/intel_pstate.c
index b4ffe6c8a0d0..6ee88d7640ea 100644

drivers/cpufreq/intel_pstate.c +10 -2
@@ -1645,12 +1645,24 @@ void notify_hwp_interrupt(void)
        if (!hwp_active || !boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
                return;
 
-       rdmsrl(MSR_HWP_STATUS, value);
+       rdmsrl_safe(MSR_HWP_STATUS, &value);
        if (!(value & 0x01))
                return;
 
+       /*
+        * After hwp_active is set and all_cpu_data is allocated, there
+        * is small window.
+        */
+       if (!all_cpu_data) {
+               wrmsrl_safe(MSR_HWP_STATUS, 0);
+               return;
+       }
+
        cpudata = all_cpu_data[this_cpu];
-       schedule_delayed_work_on(this_cpu, &cpudata->hwp_notify_work,

msecs_to_jiffies(10));
+ if (cpudata)
+ schedule_delayed_work_on(this_cpu, &cpudata-

Quoted 1 line

hwp_notify_work, msecs_to_jiffies(10));

+ else
+ wrmsrl_safe(MSR_HWP_STATUS, 0);
}

Quoted 1 line
author
#4 lore raw
Quoted 36 lines · Srinivas Pandruvada
Quoted 27 lines · Jens Axboe
Quoted 26 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 22 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This was
a bit
difficult to pin down as it crashes before the display is up, but I managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace

Author: Srinivas Pandruvada <[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

cpufreq: intel_pstate: Process HWP Guaranteed change
notification

which crashes with a NULL pointer deref in notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Quoted 7 lines
Quoted 1 line

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look like
a real fix and more of a work-around.

Quoted 5 lines

This platform is sending an HWP interrupt on a CPU which we didn't yet bring it up for pstate control. So somehow firmware decided to send very early during boot, which previously we would have ignored it

Actually try this, with more prevention

I can give this a whirl.

Quoted 12 lines
drivers/cpufreq/intel_pstate.c +1 -1
index b4ffe6c8a0d0..6ee88d7640ea 100644
@@ -1645,12 +1645,24 @@ void notify_hwp_interrupt(void)
        if (!hwp_active || !boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
                return;
 
-       rdmsrl(MSR_HWP_STATUS, value);
+       rdmsrl_safe(MSR_HWP_STATUS, &value);
        if (!(value & 0x01))

drivers/cpufreq/intel_pstate.c void notify_hwp_interrupt(void) · line 1659

Quoted 10 lines
drivers/cpufreq/intel_pstate.c +8 -0
                return;
 
+       /*
+        * After hwp_active is set and all_cpu_data is allocated, there
+        * is small window.
+        */
+       if (!all_cpu_data) {
+               wrmsrl_safe(MSR_HWP_STATUS, 0);
+               return;
+       }

What synchronizes the all_cpu_data setup and the interrupt? Can the interrupt come in while it's still being setup?

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 69 lines · Jens Axboe
Quoted 41 lines · Srinivas Pandruvada
Quoted 31 lines · Jens Axboe
Quoted 30 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 26 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This
was
a bit
difficult to pin down as it crashes before the display is up, but I
managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <
[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

which crashes with a NULL pointer deref in
notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Quoted 8 lines
Quoted 1 line

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look
like
a real fix and more of a work-around.

Quoted 6 lines

This platform is sending an HWP interrupt on a CPU which we didn't yet
bring it up for pstate control. So somehow firmware decided to send very early during boot, which previously we would have ignored it

Actually try this, with more prevention

I can give this a whirl.

Quoted 18 lines
drivers/cpufreq/intel_pstate.c +0 -0
index b4ffe6c8a0d0..6ee88d7640ea 100644
@@ -1645,12 +1645,24 @@ void notify_hwp_interrupt(void)
         if (!hwp_active || !boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
                 return;


-       rdmsrl(MSR_HWP_STATUS, value);
+       rdmsrl_safe(MSR_HWP_STATUS, &value);
        if (!(value & 0x01))
                return;

+       /*
+        * After hwp_active is set and all_cpu_data is allocated, there
+        * is small window.

Quoted 8 lines
Quoted 5 lines

+        */
+       if (!all_cpu_data) {
+               wrmsrl_safe(MSR_HWP_STATUS, 0);
+               return;
+       }

What synchronizes the all_cpu_data setup and the interrupt? Can the interrupt come in while it's still being setup?

Yes.
I am working on a change to simulate that case and fix.

Thanks,
Srinivas

Quoted 1 line
#6 lore raw
In reply to Jens Axboe

Hi Axobe,

Quoted 35 lines · Jens Axboe
Quoted 34 lines · Srinivas Pandruvada
Quoted 33 lines · Jens Axboe
Quoted 31 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 19 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This
was
a bit
difficult to pin down as it crashes before the display is up, but I
managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <
[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

Quoted 7 lines

which crashes with a NULL pointer deref in
notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look
like
a real fix and more of a work-around.

Please try the attached.

Thanks,
Srinivas

Quoted 27 lines
Quoted 6 lines

This platform is sending an HWP interrupt on a CPU which we didn't yet
bring it up for pstate control. So somehow firmware decided to send very early during boot, which previously we would have ignored it

Actually try this, with more prevention

I can give this a whirl.

Quoted 18 lines
drivers/cpufreq/intel_pstate.c +0 -0
index b4ffe6c8a0d0..6ee88d7640ea 100644
@@ -1645,12 +1645,24 @@ void notify_hwp_interrupt(void)
         if (!hwp_active || !boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
                 return;


-       rdmsrl(MSR_HWP_STATUS, value);
+       rdmsrl_safe(MSR_HWP_STATUS, &value);
        if (!(value & 0x01))
                return;

+       /*
+        * After hwp_active is set and all_cpu_data is allocated, there
+        * is small window.

Quoted 9 lines
Quoted 5 lines

+        */
+       if (!all_cpu_data) {
+               wrmsrl_safe(MSR_HWP_STATUS, 0);
+               return;
+       }

What synchronizes the all_cpu_data setup and the interrupt? Can the interrupt come in while it's still being setup?

From 380d5a340ebeb172c93a878fd84a12e7bfea9cff Mon Sep 17 00:00:00 2001
From: Srinivas Pandruvada <[email protected]>
Date: Fri, 3 Sep 2021 10:41:35 -0700
Subject: [TEST PATCH] cpufreq: intel_pstate: Fix for HWP interrupt before
driver is ready

In x1 gen9 laptop, one HWP interrupt arrives before driver is ready to handle on that CPU. Here firmware is enabling and sending an interrupt for guarantee change. Since driver didn't have cpudata initialized it will cause NULL pointer when trying to schedule processing of interrupt in a workwqueue.

To avoid this set a cpumask of CPUs for which driver has initialized interrupts. If not initialized simply clear the HWP status.

Since the same thing may happen during S3 resume, clear the cpumask during offline and let it recreate it during online.

---
 drivers/cpufreq/intel_pstate.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
drivers/cpufreq/intel_pstate.c +22 -1
index b4ffe6c8a0d0..5ac86bfa1080 100644
@@ -298,6 +298,8 @@ static bool hwp_boost __read_mostly;
 
 static struct cpufreq_driver *intel_pstate_driver __read_mostly;
 
+static cpumask_t hwp_intr_enable_mask;
+
 #ifdef CONFIG_ACPI
 static bool acpi_ppc;
 #endif
@@ -1067,11 +1069,15 @@ static void intel_pstate_hwp_set(unsigned int cpu)
 	wrmsrl_on_cpu(cpu, MSR_HWP_REQUEST, value);
 }
 
+static void intel_pstate_disable_hwp_interrupt(struct cpudata *cpudata);
+
 static void intel_pstate_hwp_offline(struct cpudata *cpu)
 {
 	u64 value = READ_ONCE(cpu->hwp_req_cached);
 	int min_perf;
 
+	intel_pstate_disable_hwp_interrupt(cpu);
+
 	if (boot_cpu_has(X86_FEATURE_HWP_EPP)) {
 		/*
 		 * In case the EPP has been set to "performance" by the
@@ -1645,20 +1651,35 @@ void notify_hwp_interrupt(void)
 	if (!hwp_active || !boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
 		return;
 
-	rdmsrl(MSR_HWP_STATUS, value);
+	rdmsrl_safe(MSR_HWP_STATUS, &value);
 	if (!(value & 0x01))
 		return;
 
+	if (!cpumask_test_cpu(this_cpu, &hwp_intr_enable_mask)) {
+		wrmsrl_safe(MSR_HWP_STATUS, 0);
+		return;
+	}
+
 	cpudata = all_cpu_data[this_cpu];
 	schedule_delayed_work_on(this_cpu, &cpudata->hwp_notify_work, msecs_to_jiffies(10));
 }
 
+static void intel_pstate_disable_hwp_interrupt(struct cpudata *cpudata)
+{
+
+	if (cpumask_test_and_clear_cpu(cpudata->cpu, &hwp_intr_enable_mask)) {
+		wrmsrl_on_cpu(cpudata->cpu, MSR_HWP_INTERRUPT, 0x00);
+		cancel_delayed_work_sync(&cpudata->hwp_notify_work);
+	}
+}
+
 static void intel_pstate_enable_hwp_interrupt(struct cpudata *cpudata)
 {
 	/* Enable HWP notification interrupt for guaranteed performance change */
 	if (boot_cpu_has(X86_FEATURE_HWP_NOTIFY)) {
 		INIT_DELAYED_WORK(&cpudata->hwp_notify_work, intel_pstate_notify_work);
 		wrmsrl_on_cpu(cpudata->cpu, MSR_HWP_INTERRUPT, 0x01);
+		cpumask_set_cpu(cpudata->cpu, &hwp_intr_enable_mask);
 	}
 }
 
Signature · 1 line
2.17.1
author
#7 lore raw
Quoted 40 lines · Srinivas Pandruvada

Hi Axobe,

Quoted 37 lines · Jens Axboe
Quoted 36 lines · Srinivas Pandruvada
Quoted 35 lines · Jens Axboe
Quoted 23 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 19 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This
was
a bit
difficult to pin down as it crashes before the display is up, but I
managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <
[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

    cpufreq: intel_pstate: Process HWP Guaranteed change notification

Quoted 8 lines
Quoted 7 lines

which crashes with a NULL pointer deref in
notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But why is it NULL? Happy to test patches, but the below doesn't look
like
a real fix and more of a work-around.

Please try the attached.

I'll give it a test spin right now. Please do add a Reported-by tag, though. That's always prudent.

Signature · 1 line
Jens Axboe
author
#8 lore raw
Quoted 47 lines · Jens Axboe
Quoted 46 lines · Srinivas Pandruvada

Hi Axobe,

Quoted 42 lines · Jens Axboe
Quoted 41 lines · Srinivas Pandruvada
Quoted 39 lines · Jens Axboe
Quoted 23 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 19 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9. This
was
a bit
difficult to pin down as it crashes before the display is up, but I
managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <
[email protected]>
Date: Thu Aug 19 19:40:06 2021 -0700

cpufreq: intel_pstate: Process HWP Guaranteed change
notification

Quoted 8 lines
Quoted 7 lines

which crashes with a NULL pointer deref in
notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't look
like
a real fix and more of a work-around.

Quoted 4 lines
Quoted 1 line

Please try the attached.

I'll give it a test spin right now. Please do add a Reported-by tag, though. That's always prudent.

And you can add Tested-by as well, it works for me.

Signature · 1 line
Jens Axboe
#9 lore raw
Quoted 52 lines · Jens Axboe
Quoted 51 lines · Jens Axboe
Quoted 50 lines · Srinivas Pandruvada

Hi Axobe,

Quoted 46 lines · Jens Axboe
Quoted 45 lines · Srinivas Pandruvada
Quoted 43 lines · Jens Axboe
Quoted 26 lines · Srinivas Pandruvada

Hi Axboe,

Thanks for reporting.

Quoted 22 lines · Jens Axboe

Hi,

Booting Linus's tree causes a crash on my laptop, an x1 gen9.
This
was
a bit
difficult to pin down as it crashes before the display is up,
but I
managed
to narrow it down to:

commit d0e936adbd2250cb03f2e840c6651d18edc22ace
Author: Srinivas Pandruvada <
[email protected]>
Date:   Thu Aug 19 19:40:06 2021 -0700

    cpufreq: intel_pstate: Process HWP Guaranteed
change
notification

Quoted 8 lines
Quoted 7 lines

which crashes with a NULL pointer deref in
notify_hwp_interrupt() -

Quoted 1 line

queue_delayed_work_on().

Reverting this change makes the laptop boot fine again.

Does this change fixes your issue?

I would assume so, as it's crashing on cpudata == NULL :-)

But why is it NULL? Happy to test patches, but the below doesn't
look
like
a real fix and more of a work-around.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Please try the attached.

I'll give it a test spin right now. Please do add a Reported-by tag,
though. That's always prudent.

And you can add Tested-by as well, it works for me.

Thanks. I will add both.

-Srinivas

Quoted 1 line