x86/intel: Disable HPET on another Intel Coffee Lake platform

My Lenovo T490s with i7-8665U had been marking TSC as unstable since v5.13, resulting in very sluggish desktop experience...

Kernel logs show:

clocksource: timekeeping watchdog on CPU3: hpet read-back delay of 316000ns, attempt 4, marking unstable
tsc: Marking TSC unstable due to clocksource watchdog
TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
sched_clock: Marking unstable (14539801827657, -530891666)<-(14539319241737, -48307500)
clocksource: Checking clocksource tsc synchronization from CPU 3 to CPUs 0-2,6-7.
clocksource: Switched to clocksource hpet

I have a 8086:3e34 bridge, also known as "Host bridge: Intel Corporation Coffee Lake HOST and DRAM Controller (rev 0c)". Add it to the list.

We should perhaps consider applying this quirk more widely. The Intel documentation does not list my device [1], but linuxhw [2] does, and it seems to list a few more bridges we do not currently cover (3e31, 3ecc, 3e35, 3e0f).

[1] https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/8th-gen-core-family-datasheet-vol-2.pdf
[2] https://github.com/linuxhw/DevicePopulation/blob/master/README.md

Cc [email protected] # v5.13+

Signed-off-by Jakub Kicinski <[email protected]>

---
v2: - add the dmesg output

---
 arch/x86/kernel/early-quirks.c | 2 ++
 1 file changed, 2 insertions(+)

arch/x86/kernel/early-quirks.c +0 -0

index 38837dad46e6..7d2de04f8750 100644

My Lenovo T490s with i7-8665U had been marking TSC as unstable since v5.13, resulting in very sluggish desktop experience...

+++ b/arch/x86/kernel/early-quirks.c

arch/x86/kernel/early-quirks.c +2 -0

@@ -716,6 +716,8 @@ static struct chipset early_qrk[] __initdata = {
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x3e20,
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
+	{ PCI_VENDOR_ID_INTEL, 0x3e34,
+		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x3ec4,
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x8a12,

Quoted 2 lines · On Thu, 16 Sep 2021 19:46:48 -0700 Jakub Kicinski wrote:

My Lenovo T490s with i7-8665U had been marking TSC as unstable since v5.13, resulting in very sluggish desktop experience...

Where do we stand? Waiting for tglx to refactor PC10 detection and use that, or just review delay?

Quoted 10 lines

+++ b/arch/x86/kernel/early-quirks.c

arch/x86/kernel/early-quirks.c +2 -0

@@ -716,6 +716,8 @@ static struct chipset early_qrk[] __initdata = {
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x3e20,
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
+	{ PCI_VENDOR_ID_INTEL, 0x3e34,
+		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x3ec4,
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
 	{ PCI_VENDOR_ID_INTEL, 0x8a12,

Quoted 6 lines · Jakub Kicinski

Quoted 2 lines · On Thu, 16 Sep 2021 19:46:48 -0700 Jakub Kicinski wrote:

My Lenovo T490s with i7-8665U had been marking TSC as unstable since v5.13, resulting in very sluggish desktop experience...

Where do we stand? Waiting for tglx to refactor PC10 detection and use that, or just review delay?

From my point of view, this is an x86 issue, not a PCI one, so I'll defer to the x86 folks.

On recent Intel systems the HPET stops working when the system reaches PC10 idle state.

The approach of adding PCI ids to the early quirks to disable HPET on these systems is a whack a mole game which makes no sense.

Check for PC10 instead and force disable HPET if supported. The check is overbroad as it does not take ACPI, intel_idle enablement and command line parameters into account. That's fine as long as there is at least PMTIMER available to calibrate the TSC frequency. The decision can be overruled by adding "hpet=force" on the kernel command line.

Remove the related early PCI quirks for affected Ice and Coffin Lake systems as they are not longer required.

Fixes Yet another hardware trainwreck

Reported-by Jakub Kicinski <[email protected]>

Not-yet-signed-off-by Thomas Gleixner <[email protected]>

---
Notes: Completely untested. Use at your own peril.

---
 arch/x86/kernel/early-quirks.c |    6 --
 arch/x86/kernel/hpet.c         |   88 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 88 insertions(+), 6 deletions(-)

arch/x86/kernel/early-quirks.c +0 -6

@@ -714,12 +714,6 @@ static struct chipset early_qrk[] __init
         */
        { PCI_VENDOR_ID_INTEL, 0x0f00,
                PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x3e20,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x3ec4,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x8a12,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
        { PCI_VENDOR_ID_BROADCOM, 0x4331,
          PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset},
        {}

arch/x86/kernel/hpet.c +10 -0

@@ -10,6 +10,7 @@
 #include <asm/irq_remapping.h>
 #include <asm/hpet.h>
 #include <asm/time.h>
+#include <asm/mwait.h>
 
 #undef  pr_fmt
 #define pr_fmt(fmt) "hpet: " fmt

@@ -916,6 +917,90 @@ static bool __init hpet_counting(void)
        return false;
 }
 
+static bool __init get_mwait_substates(unsigned int *mwait_substates)
+{
+       unsigned int eax, ebx, ecx;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return false;
+
+       if (!boot_cpu_has(X86_FEATURE_MWAIT))
+               return false;

arch/x86/kernel/hpet.c +68 -0

+
+       return true;
+}
+
+/*
+ * Check whether the system supports PC10. If so force disable HPET as that
+ * stops counting in PC10. This check is overbroad as it does not take any
+ * of the following into account:
+ *
+ *     - ACPI tables
+ *     - Enablement of intel_idle
+ *     - Command line arguments which limit intel_idle C-state support
+ *
+ * That's perfectly fine. HPET is a piece of hardware designed by committee
+ * and the only reasons why it is still in use on modern systems is the
+ * fact that it is impossible to reliably query the TSC frequency via
+ * CPUID or firmware.
+ *
+ * If HPET is functional it is useful for calibrating TSC, but this can be
+ * done via PMTIMER as well which seems to be the last remaining timer on
+ * X86/INTEL platforms that has not been completely wreckaged by feature
+ * creep.
+ *
+ * In theory HPET support should be removed altogether, but there are older
+ * systems out there which depend on it because TSC and APIC timer are
+ * dysfunctional in deeper C-states.
+ *
+ * It's only 20 years now that hardware people have been asked to provide
+ * reliable and discoverable facilities which can be used for timekeeping
+ * and per CPU timer interrupts.
+ *
+ * The probability that this problem is going to be solved in the
+ * forseeable future is close to zero, so the kernel has to be cluttered
+ * with heuristics to keep up with the ever growing amount of hardware and
+ * firmware trainwrecks. Hopefully some day hardware people will understand
+ * that the approach of "This can be fixed in software" is not sustainable.
+ * Hope dies last...
+ */
+static bool __init hpet_is_pc10_damaged(void)
+{
+       unsigned int mwait_substates;
+       unsigned long long pcfg;
+
+       if (!get_mwait_substates(&mwait_substates))
+               return false;
+
+       /* Check whether PC10 substates are supported */
+       if (!(mwait_substates & (0xF << 28)))
+               return false;
+
+       /* Check whether PC10 is enabled in PKG C-state limit */
+       rdmsrl(MSR_PKG_CST_CONFIG_CONTROL, pcfg);
+       if ((pcfg & 0xF) < 8)
+               return false;
+
+       if (hpet_force_user) {
+               pr_warn("HPET force enabled via command line, but dysfunctional in PC10.\n");
+               return false;
+       }
+
+       pr_info("HPET dysfunctional in PC10. Force disabled.\n");
+       boot_hpet_disable = true;
+       return true;
+}
+
 /**
  * hpet_enable - Try to setup the HPET timer. Returns 1 on success.
  */

@@ -929,6 +1014,9 @@ int __init hpet_enable(void)
        if (!is_hpet_capable())
                return 0;
 
+       if (hpet_is_pc10_damaged())
+               return 0;
+
        hpet_set_mapping();
        if (!hpet_virt_address)
                return 0;

Quoted 9 lines

arch/x86/kernel/hpet.c +9 -0

+static bool __init get_mwait_substates(unsigned int *mwait_substates)
+{
+       unsigned int eax, ebx, ecx;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return false;
+
+       if (!boot_cpu_has(X86_FEATURE_MWAIT))
+               return false;

arch/x86/kernel/hpet.c static bool __init hpet_counting(void) · line 938

Quoted 10 lines

arch/x86/kernel/hpet.c +10 -0

+
+       if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
+               return false;
+
+       cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, mwait_substates);
+
+       if (!(ecx & CPUID5_ECX_EXTENSIONS_SUPPORTED) ||
+           !(ecx & CPUID5_ECX_INTERRUPT_BREAK) ||
+           !*mwait_substates)
+               return false;

I would do

return (ecx & CPUID5_ECX_EXTENSIONS_SUPPORTED) && (ecx & CPUID5_ECX_INTERRUPT_BREAK) && *mwait_substates;

And this function could just return the mwait_substates value proper, because returning 0 then would be equivalent to returning 'false' from it as is.

On recent Intel systems the HPET stops working when the system reaches PC10 idle state.

The approach of adding PCI ids to the early quirks to disable HPET on these systems is a whack a mole game which makes no sense.

Check for PC10 instead and force disable HPET if supported. The check is overbroad as it does not take ACPI, intel_idle enablement and command line parameters into account. That's fine as long as there is at least PMTIMER available to calibrate the TSC frequency. The decision can be overruled by adding "hpet=force" on the kernel command line.

Remove the related early PCI quirks for affected Ice Cake and Coffin Lake systems as they are not longer required. That should also cover all other systems, i.e. Tiger Rag and newer generations, which are most likely affected by this as well.

Fixes Yet another hardware trainwreck

Reported-by Jakub Kicinski <[email protected]>

Not-yet-signed-off-by Thomas Gleixner <[email protected]>

On recent Intel systems the HPET stops working when the system reaches PC10 idle state.

The approach of adding PCI ids to the early quirks to disable HPET on these systems is a whack a mole game which makes no sense.

Check for PC10 instead and force disable HPET if supported. The check is overbroad as it does not take ACPI, intel_idle enablement and command line parameters into account. That's fine as long as there is at least PMTIMER available to calibrate the TSC frequency. The decision can be overruled by adding "hpet=force" on the kernel command line.

Remove the related early PCI quirks for affected Ice Cake and Coffin Lake systems as they are not longer required. That should also cover all other systems, i.e. Tiger Rag and newer generations, which are most likely affected by this as well.

Fixes Yet another hardware trainwreck

Reported-by Jakub Kicinski <[email protected]>

Not-yet-signed-off-by Thomas Gleixner <[email protected]>

---
Notes: Completely untested. Use at your own peril.

V2: Move the substate check into the helper function. Adjust function
name accordingly.

---
 arch/x86/kernel/early-quirks.c |    6 ---
 arch/x86/kernel/hpet.c         |   81 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+), 6 deletions(-)

arch/x86/kernel/early-quirks.c +0 -6

@@ -714,12 +714,6 @@ static struct chipset early_qrk[] __init
         */
        { PCI_VENDOR_ID_INTEL, 0x0f00,
                PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x3e20,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x3ec4,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
-       { PCI_VENDOR_ID_INTEL, 0x8a12,
-               PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
        { PCI_VENDOR_ID_BROADCOM, 0x4331,
          PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset},
        {}

arch/x86/kernel/hpet.c +81 -0

@@ -10,6 +10,7 @@
 #include <asm/irq_remapping.h>
 #include <asm/hpet.h>
 #include <asm/time.h>
+#include <asm/mwait.h>
 
 #undef  pr_fmt
 #define pr_fmt(fmt) "hpet: " fmt

@@ -916,6 +917,83 @@ static bool __init hpet_counting(void)
        return false;
 }
 
+static bool __init mwait_pc10_supported(void)
+{
+       unsigned int eax, ebx, ecx, mwait_substates;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return false;
+
+       if (!cpu_feature_enabled(X86_FEATURE_MWAIT))
+               return false;
+
+       if (boot_cpu_data.cpuid_level < CPUID_MWAIT_LEAF)
+               return false;
+
+       cpuid(CPUID_MWAIT_LEAF, &eax, &ebx, &ecx, &mwait_substates);
+
+       return (ecx & CPUID5_ECX_EXTENSIONS_SUPPORTED) &&
+              (ecx & CPUID5_ECX_INTERRUPT_BREAK) &&
+              (mwait_substates & (0xF << 28));
+}
+
+/*
+ * Check whether the system supports PC10. If so force disable HPET as that
+ * stops counting in PC10. This check is overbroad as it does not take any
+ * of the following into account:
+ *
+ *     - ACPI tables
+ *     - Enablement of intel_idle
+ *     - Command line arguments which limit intel_idle C-state support
+ *
+ * That's perfectly fine. HPET is a piece of hardware designed by committee
+ * and the only reasons why it is still in use on modern systems is the
+ * fact that it is impossible to reliably query TSC and CPU frequency via
+ * CPUID or firmware.
+ *
+ * If HPET is functional it is useful for calibrating TSC, but this can be
+ * done via PMTIMER as well which seems to be the last remaining timer on
+ * X86/INTEL platforms that has not been completely wreckaged by feature
+ * creep.
+ *
+ * In theory HPET support should be removed altogether, but there are older
+ * systems out there which depend on it because TSC and APIC timer are
+ * dysfunctional in deeper C-states.
+ *
+ * It's only 20 years now that hardware people have been asked to provide
+ * reliable and discoverable facilities which can be used for timekeeping
+ * and per CPU timer interrupts.
+ *
+ * The probability that this problem is going to be solved in the
+ * forseeable future is close to zero, so the kernel has to be cluttered
+ * with heuristics to keep up with the ever growing amount of hardware and
+ * firmware trainwrecks. Hopefully some day hardware people will understand
+ * that the approach of "This can be fixed in software" is not sustainable.
+ * Hope dies last...
+ */
+static bool __init hpet_is_pc10_damaged(void)
+{
+       unsigned long long pcfg;
+
+       /* Check whether PC10 substates are supported */
+       if (!mwait_pc10_supported())
+               return false;
+
+       /* Check whether PC10 is enabled in PKG C-state limit */
+       rdmsrl(MSR_PKG_CST_CONFIG_CONTROL, pcfg);
+       if ((pcfg & 0xF) < 8)
+               return false;
+
+       if (hpet_force_user) {
+               pr_warn("HPET force enabled via command line, but dysfunctional in PC10.\n");
+               return false;
+       }
+
+       pr_info("HPET dysfunctional in PC10. Force disabled.\n");
+       boot_hpet_disable = true;
+       return true;
+}
+
 /**
  * hpet_enable - Try to setup the HPET timer. Returns 1 on success.
  */

@@ -929,6 +1007,9 @@ int __init hpet_enable(void)
        if (!is_hpet_capable())
                return 0;
 
+       if (hpet_is_pc10_damaged())
+               return 0;
+
        hpet_set_mapping();
        if (!hpet_virt_address)
                return 0;