x86/PCI: Convert force_disable_hpet() to standard quirk
Subsystems (3)
PATCH 1/1 x86/PCI: Convert force_disable_hpet() to standard quirk
From: Bjorn Helgaas <[email protected]>
62187910b0fc ("x86/intel: Add quirk to disable HPET for the Baytrail platform") implemented force_disable_hpet() as a special early quirk. These run before the PCI core is initialized and depend on the x86/pci/early.c accessors that use I/O ports 0xcf8 and 0xcfc.
But force_disable_hpet() doesn't need to be one of these special early quirks. It merely sets "boot_hpet_disable", which is tested by is_hpet_capable(), which is only used by hpet_enable() and hpet_disable(). hpet_enable() is an fs_initcall(), so it runs after the PCI core is initialized.
Convert force_disable_hpet() to the standard PCI quirk mechanism.
--- arch/x86/kernel/early-quirks.c | 24 ------------------------ arch/x86/pci/fixup.c | 25 +++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 24 deletions(-)
arch/x86/kernel/early-quirks.c +0 -24
@@ -604,14 +604,6 @@ static void __init intel_graphics_quirks(int num, int slot, int func) } } -static void __init force_disable_hpet (int num,int slot,int func) -{ -#ifdef CONFIG_HPET_TIMER - boot_hpet_disable = true; -pr_info ("x86/hpet: Will disable the HPET for this platform because it's not reliable\n" ); -#endif -} -#define BCM4331_MMIO_SIZE 16384#define BCM4331_PM_CAP 0x40#define bcma_aread32 (reg) ioread32(mmio + 1 * BCMA_CORE_SIZE + reg)
@@ -701,22 +693,6 @@ static struct chipset early_qrk[] __initdata = { PCI_BASE_CLASS_BRIDGE, 0, intel_remapping_check }, { PCI_VENDOR_ID_INTEL, PCI_ANY_ID, PCI_CLASS_DISPLAY_VGA, PCI_ANY_ID, QFLAG_APPLY_ONCE, intel_graphics_quirks }, -/* -* HPET on the current version of the Baytrail platform has accuracy -* problems: it will halt in deep idle state - so we disable it. -* -* More details can be found in section 18.10.1.3 of the datasheet: -* -* http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf -*/ - { PCI_VENDOR_ID_INTEL, 0x0f00, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x3e20, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x3ec4, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x8a12, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, { PCI_VENDOR_ID_BROADCOM, 0x4331, PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset}, {}
arch/x86/pci/fixup.c +25 -0
@@ -780,3 +780,28 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);DECLARE_PCI_FIXUP_RESUME (PCI_VENDOR_ID_AMD ,0x1601 , pci_amd_enable_64bit_bar);#endif + +/* +* HPET on the current version of the Baytrail platform has accuracy +* problems: it will halt in deep idle state - so we disable it. +* +* More details can be found in section 18.10.1.3 of the datasheet +* (Intel Document Number 332065-003, March 2016): +* +* http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf +*/ +static void force_disable_hpet (struct pci_dev *dev) +{ +#ifdef CONFIG_HPET_TIMER + boot_hpet_disable = true; +pci_info (dev,"x86/hpet: Will disable the HPET for this platform because it's not reliable\n" ); +#endif +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x0f00 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x3e20 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x3ec4 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x8a12 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet);
Signature · 1 line
2.25.1
Quoted 13 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform") implementedforce_disable_hpet()as a special early quirk. These run before the PCI core is initialized and depend on thex86/pci/early.caccessors that use I/O ports0xcf8and0xcfc.But
force_disable_hpet()doesn't need to be one of these special early quirks. It merely sets "boot_hpet_disable", which is tested byis_hpet_capable(), which is only used byhpet_enable()andhpet_disable().hpet_enable()is anfs_initcall(), so it runs after the PCI core is initialized.
Quoted 5 lines
Convert
force_disable_hpet()to the standard PCI quirk mechanism.Signed-off-by Bjorn Helgaas <[email protected]>Cc Feng Tang <[email protected]>Cc Kai-Heng Feng <[email protected]>
I applied this to pci/misc for v5.11. Let me know if you see any issues.
Quoted 83 lines
--- arch/x86/kernel/early-quirks.c | 24 ------------------------ arch/x86/pci/fixup.c | 25 +++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 24 deletions(-)arch/x86/kernel/early-quirks.c +0
-24@@ -604,14 +604,6 @@ static void __init intel_graphics_quirks(int num, int slot, int func) } } -static void __init force_disable_hpet (int num,int slot,int func) -{ -#ifdef CONFIG_HPET_TIMER - boot_hpet_disable = true; -pr_info ("x86/hpet: Will disable the HPET for this platform because it's not reliable\n" ); -#endif -} -#define BCM4331_MMIO_SIZE 16384#define BCM4331_PM_CAP 0x40#define bcma_aread32 (reg) ioread32(mmio + 1 * BCMA_CORE_SIZE + reg)@@ -701,22 +693,6 @@ static struct chipset early_qrk[] __initdata = { PCI_BASE_CLASS_BRIDGE, 0, intel_remapping_check }, { PCI_VENDOR_ID_INTEL, PCI_ANY_ID, PCI_CLASS_DISPLAY_VGA, PCI_ANY_ID, QFLAG_APPLY_ONCE, intel_graphics_quirks }, -/* -* HPET on the current version of the Baytrail platform has accuracy -* problems: it will halt in deep idle state - so we disable it. -* -* More details can be found in section 18.10.1.3 of the datasheet: -* -* http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf -*/ - { PCI_VENDOR_ID_INTEL, 0x0f00, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x3e20, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x3ec4, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, - { PCI_VENDOR_ID_INTEL, 0x8a12, - PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, { PCI_VENDOR_ID_BROADCOM, 0x4331, PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset}, {}arch/x86/pci/fixup.c +25
-0@@ -780,3 +780,28 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);DECLARE_PCI_FIXUP_RESUME (PCI_VENDOR_ID_AMD ,0x1601 , pci_amd_enable_64bit_bar);#endif + +/* +* HPET on the current version of the Baytrail platform has accuracy +* problems: it will halt in deep idle state - so we disable it. +* +* More details can be found in section 18.10.1.3 of the datasheet +* (Intel Document Number 332065-003, March 2016): +* +* http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf +*/ +static void force_disable_hpet (struct pci_dev *dev) +{ +#ifdef CONFIG_HPET_TIMER + boot_hpet_disable = true; +pci_info (dev,"x86/hpet: Will disable the HPET for this platform because it's not reliable\n" ); +#endif +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x0f00 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x3e20 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x3ec4 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet); +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_INTEL ,0x8a12 , +PCI_CLASS_BRIDGE_HOST ,8 , force_disable_hpet);Signature · 1 line
2.25.1
Quoted 10 lines · Bjorn Helgaas
62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform") implementedforce_disable_hpet()as a special early quirk. These run before the PCI core is initialized and depend on thex86/pci/early.caccessors that use I/O ports0xcf8and0xcfc.But
force_disable_hpet()doesn't need to be one of these special early quirks. It merely sets "boot_hpet_disable", which is tested byis_hpet_capable(), which is only used byhpet_enable()andhpet_disable().hpet_enable()is anfs_initcall(), so it runs after the PCI core is initialized.
hpet_enable() is not an fs_initcall(). hpet_late_init() is and that invokes hpet_enable() only for the case that ACPI did not advertise it and the force_hpet quirk provided a base address.
But hpet_enable() is also invoked via:
start_kernel()
late_time_init()
x86_late_time_init()
hpet_time_init()which is way before the PCI core is available and we really don't want to set it up there if it's known to be broken :)
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
e0748539e3d5 ("x86/intel: Disable HPET on Intel Ice Lake platforms") f8edbde885bb ("x86/intel: Disable HPET on Intel Coffee Lake H platforms") fc5db58539b4 ("x86/quirks: Disable HPET on Intel Coffe Lake platforms") 62187910b0fc ("x86/intel: Add quirk to disable HPET for the Baytrail platform")
I might be missing something here, but in general on anything modern HPET is mostly useless.
Thanks,
tglx
Quoted 18 lines · Thomas Gleixner
Quoted 10 lines · Bjorn Helgaas
62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform") implementedforce_disable_hpet()as a special early quirk. These run before the PCI core is initialized and depend on thex86/pci/early.caccessors that use I/O ports0xcf8and0xcfc.But
force_disable_hpet()doesn't need to be one of these special early quirks. It merely sets "boot_hpet_disable", which is tested byis_hpet_capable(), which is only used byhpet_enable()andhpet_disable().hpet_enable()is anfs_initcall(), so it runs after the PCI core is initialized.
hpet_enable()is not anfs_initcall().hpet_late_init()is and that invokeshpet_enable()only for the case that ACPI did not advertise it and theforce_hpetquirk provided a base address.But
hpet_enable()is also invoked via:
Quoted 7 lines
start_kernel()
late_time_init()x86_late_time_init() hpet_time_init()which is way before the PCI core is available and we really don't want to set it up there if it's known to be broken :)
Wow, I really blew that, don't know how I missed that path. Thanks for catching this! I'll drop this patch.
Quoted 3 lines
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
You mean like a CPUID check or something? I'm all in favor of doing something that doesn't depend on PCI.
Quoted 12 lines
e0748539e3d5("x86/intel: Disable HPET on Intel Ice Lake platforms")f8edbde885bb("x86/intel: Disable HPET on Intel Coffee Lake H platforms")fc5db58539b4("x86/quirks: Disable HPET on Intel Coffe Lake platforms")62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform")I might be missing something here, but in general on anything modern HPET is mostly useless.
Thanks,
tglx
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines · Thomas Gleixner
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
You mean like a CPUID check or something? I'm all in favor of doing something that doesn't depend on PCI.
The reason why these are PCI quirks is that the HPET is not part of the CPU core. It's usually part of 00:00:0 (aka. host bridge) and the legacy mess which resides there.
OTOH that thing is integrated into the actual chip nowadays and these quirks are platform quirks, so the CPUID and the PCI vendor/device codes should be the same for a particular model.
But what do I know. This whole platform/cpuid mess is inpenetrable even for people who have access to the Intel internal documentation...
But, the point is that HPET does not provide any value on contemporary CPUs assumed that Intel finally decided that TSC and the TSC deadline timer are crucial system components along with the ability to get the correct frequency of these beasts from firmware/cpuid.
So if parts of a particular chip generation, which we can determine by CPUID (family, model) has issues with HPET, then there is no value in preserving HPET for the N out of M submodels which are not affected even if we can differentiate them by the host bridge device id.
But as I said above, what do I know. The Intel wizards which might have better insight into this should speak up and come forth with objections.
Otherwise I just go and disable HPET support for the CPU models which are covered by these PCI quirks today.
Ideally we can just disable it for anything more modern as well, but of course that requires that future devices have proper frequency enumeration and Intel prevents the OEMs to screw that up with their "value add" BIOS/SMM machinery. Hope dies last...
Thanks,
tglx
Hi Thomas,
Quoted 30 lines · Thomas Gleixner
Quoted 10 lines · Bjorn Helgaas
62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform") implementedforce_disable_hpet()as a special early quirk. These run before the PCI core is initialized and depend on thex86/pci/early.caccessors that use I/O ports0xcf8and0xcfc.But
force_disable_hpet()doesn't need to be one of these special early quirks. It merely sets "boot_hpet_disable", which is tested byis_hpet_capable(), which is only used byhpet_enable()andhpet_disable().hpet_enable()is anfs_initcall(), so it runs after the PCI core is initialized.
hpet_enable()is not anfs_initcall().hpet_late_init()is and that invokeshpet_enable()only for the case that ACPI did not advertise it and theforce_hpetquirk provided a base address.But
hpet_enable()is also invoked via:
start_kernel()
late_time_init()x86_late_time_init() hpet_time_init()which is way before the PCI core is available and we really don't want to set it up there if it's known to be broken :)
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
Quoted 7 lines
e0748539e3d5("x86/intel: Disable HPET on Intel Ice Lake platforms")f8edbde885bb("x86/intel: Disable HPET on Intel Coffee Lake H platforms")fc5db58539b4("x86/quirks: Disable HPET on Intel Coffe Lake platforms")
62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform")
I added this commit, and I can explain some for Baytrail. There was some discussion about the way to disable it:
https://lore.kernel.org/lkml/20140328073718.GA12762@feng-snb/t/
It uses PCI ID early quirk in the hope that later Baytrail stepping
doesn't have the problem. And later on, there was official document
(section 18.10.1.3 http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf)
stating Baytrail's HPET halts in deep idle. So I think your way of
using CPUID to disable Baytrail HPET makes more sense.
Quoted 2 lines
I might be missing something here, but in general on anything modern HPET is mostly useless.
IIUC, nowdays HPET's main use is as a clocksource watchdog monitor. And in one debug case, we found it still useful. The debug platform has early serial console which prints many messages in early boot phase, when the HPET is disabled, the software 'jiffies' clocksource will be used as the monitor. Early printk will disable interrupt will printing message, and this could be quite long for a slow 115200 device, and cause the periodic HW timer interrupt get missed, and make the 'jiffies' clocksource not accurate, which will in turn judge the TSC clocksrouce inaccurate, and disablt it. (Adding Rui, Len for more details)
Thanks,
Feng
Quoted 3 lines
Thanks,
tglx
Feng,
Quoted 14 lines · Feng Tang
Quoted 8 lines · Thomas Gleixner
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
e0748539e3d5("x86/intel: Disable HPET on Intel Ice Lake platforms")f8edbde885bb("x86/intel: Disable HPET on Intel Coffee Lake H platforms")fc5db58539b4("x86/quirks: Disable HPET on Intel Coffe Lake platforms")62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform")I added this commit, and I can explain some for Baytrail. There was some discussion about the way to disable it:
https://lore.kernel.org/lkml/20140328073718.GA12762@feng-snb/t/
Quoted 5 lines
It uses PCI ID early quirk in the hope that later Baytrail stepping
doesn't have the problem. And later on, there was official document
(section 18.10.1.3 http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf)
stating Baytrail's HPET halts in deep idle. So I think your way of
using CPUID to disable Baytrail HPET makes more sense.
Correct.
Quoted 4 lines
Quoted 2 lines
I might be missing something here, but in general on anything modern HPET is mostly useless.
IIUC, nowdays HPET's main use is as a clocksource watchdog monitor.
Plus for the TSC refined calibration, where it is really better than anything else we have _if_ it is functional.
Quoted 9 lines
And in one debug case, we found it still useful. The debug platform has early serial console which prints many messages in early boot phase, when the HPET is disabled, the software 'jiffies' clocksource will be used as the monitor. Early printk will disable interrupt will printing message, and this could be quite long for a slow 115200 device, and cause the periodic HW timer interrupt get missed, and make the 'jiffies' clocksource not accurate, which will in turn judge the TSC clocksrouce inaccurate, and disablt it. (Adding Rui, Len for more details)
Yes, that can happen. But OTOH, we should start to think about the requirements for using the TSC watchdog.
I'm inclined to lift that requirement when the CPU has:
1) X86_FEATURE_CONSTANT_TSC
2) X86_FEATURE_NONSTOP_TSC
3) X86_FEATURE_NONSTOP_TSC_S3
4) X86_FEATURE_TSC_ADJUST
5) At max. 4 sockets
After two decades of horrors we're finally at a point where TSC seems to be halfways reliable and less abused by BIOS tinkerers. TSC_ADJUST was really key as we can now detect even small modifications reliably and the important point is that we can cure them as well (not pretty but better than all other options).
The only nasty one in the list above is #5 because AFAIK there is still no architecural guarantee for TSCs being synchronized on machines with more than 4 sockets. I might be wrong, but then nobody told me.
The only reason I hate to disable HPET upfront at least during boot is that HPET is the best mechanism for the refined TSC calibration. PMTIMER sucks because it's slow and wraps around pretty quick.
So we could do the following even on platforms where HPET stops in some magic PC? state:
- Register it during early boot as clocksource
- Prevent the enablement as clockevent and the chardev hpet timer muck
- Prevent the magic PC? state up to the point where the refined TSC calibration is finished.
- Unregister it once the TSC has taken over as system clocksource and enable the magic PC? state in which HPET gets disfunctional.
Hmm?
Thanks,
tglx
Hi Thomas,
Quoted 46 lines · Thomas Gleixner
Feng,
Quoted 14 lines · Feng Tang
Quoted 8 lines · Thomas Gleixner
Now the more interesting question is why this needs to be a PCI quirk in the first place. Can't we just disable the HPET based on family/model quirks?
e0748539e3d5("x86/intel: Disable HPET on Intel Ice Lake platforms")f8edbde885bb("x86/intel: Disable HPET on Intel Coffee Lake H platforms")fc5db58539b4("x86/quirks: Disable HPET on Intel Coffe Lake platforms")62187910b0fc("x86/intel: Add quirk to disable HPET for the Baytrail platform")I added this commit, and I can explain some for Baytrail. There was some discussion about the way to disable it:
https://lore.kernel.org/lkml/20140328073718.GA12762@feng-snb/t/Quoted 5 lines
It uses PCI ID early quirk in the hope that later Baytrail stepping
doesn't have the problem. And later on, there was official document
(section 18.10.1.3 http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-z8000-datasheet-vol-1.pdf)
stating Baytrail's HPET halts in deep idle. So I think your way of
using CPUID to disable Baytrail HPET makes more sense.Correct.
Quoted 4 lines
Quoted 2 lines
I might be missing something here, but in general on anything modern HPET is mostly useless.
IIUC, nowdays HPET's main use is as a clocksource watchdog monitor.
Plus for the TSC refined calibration, where it is really better than anything else we have _if_ it is functional.
Quoted 9 lines
And in one debug case, we found it still useful. The debug platform has early serial console which prints many messages in early boot phase, when the HPET is disabled, the software 'jiffies' clocksource will be used as the monitor. Early printk will disable interrupt will printing message, and this could be quite long for a slow 115200 device, and cause the periodic HW timer interrupt get missed, and make the 'jiffies' clocksource not accurate, which will in turn judge the TSC clocksrouce inaccurate, and disablt it. (Adding Rui, Len for more details)
Yes, that can happen. But OTOH, we should start to think about the requirements for using the TSC watchdog.
Quoted 6 lines
I'm inclined to lift that requirement when the CPU has:
1)
X86_FEATURE_CONSTANT_TSC
2)X86_FEATURE_NONSTOP_TSC3)
X86_FEATURE_NONSTOP_TSC_S3
IIUC, this feature exists for several generations of Atom platforms, and it is always coupled with 1) and 2), so it could be skipped for the checking.
Quoted 23 lines
4)
X86_FEATURE_TSC_ADJUST5) At max. 4 sockets
After two decades of horrors we're finally at a point where TSC seems to be halfways reliable and less abused by BIOS tinkerers.
TSC_ADJUSTwas really key as we can now detect even small modifications reliably and the important point is that we can cure them as well (not pretty but better than all other options).The only nasty one in the list above is #5 because AFAIK there is still no architecural guarantee for TSCs being synchronized on machines with more than 4 sockets. I might be wrong, but then nobody told me.
The only reason I hate to disable HPET upfront at least during boot is that HPET is the best mechanism for the refined TSC calibration. PMTIMER sucks because it's slow and wraps around pretty quick.
So we could do the following even on platforms where HPET stops in some magic PC? state:
- Register it during early boot as clocksource
Quoted 7 lines
- Prevent the enablement as clockevent and the chardev hpet timer muck
- Prevent the magic PC? state up to the point where the refined TSC calibration is finished.
- Unregister it once the TSC has taken over as system clocksource and enable the magic PC? state in which HPET gets disfunctional.
This looks reasonable to me.
I have thought about lowering the hpet rating to lower than PMTIMER, so it still contributes in early boot phase, and fades out after PMTIMER is initialised.
Thanks,
Feng
Quoted 7 lines
Hmm?
Thanks,
tglx
Feng,
Quoted 7 lines · Feng Tang
Quoted 6 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the requirements for using the TSC watchdog.I'm inclined to lift that requirement when the CPU has:
Quoted 7 lines
Quoted 4 lines
1)
X86_FEATURE_CONSTANT_TSC
2)X86_FEATURE_NONSTOP_TSC3)
X86_FEATURE_NONSTOP_TSC_S3IIUC, this feature exists for several generations of Atom platforms, and it is always coupled with 1) and 2), so it could be skipped for the checking.
Yes, we can ignore that bit as it's not widely available and not required to solve the problem.
Quoted 18 lines
Quoted 18 lines
4)
X86_FEATURE_TSC_ADJUST5) At max. 4 sockets
The only reason I hate to disable HPET upfront at least during boot is that HPET is the best mechanism for the refined TSC calibration. PMTIMER sucks because it's slow and wraps around pretty quick.
So we could do the following even on platforms where HPET stops in some magic PC? state:
- Register it during early boot as clocksource
- Prevent the enablement as clockevent and the chardev hpet timer muck
- Prevent the magic PC? state up to the point where the refined TSC calibration is finished.
Quoted 8 lines
Quoted 2 lines
- Unregister it once the TSC has taken over as system clocksource and enable the magic PC? state in which HPET gets disfunctional.
This looks reasonable to me.
I have thought about lowering the hpet rating to lower than PMTIMER, so it still contributes in early boot phase, and fades out after PMTIMER is initialised.
Not a good idea. pm_timer is initialized before the refined calibration finishes.
Thanks,
tglx
Hi Thomas,
Quoted 42 lines · Thomas Gleixner
Feng,
Quoted 7 lines · Feng Tang
Quoted 6 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the requirements for using the TSC watchdog.I'm inclined to lift that requirement when the CPU has:
Quoted 7 lines
Quoted 4 lines
1)
X86_FEATURE_CONSTANT_TSC
2)X86_FEATURE_NONSTOP_TSC3)
X86_FEATURE_NONSTOP_TSC_S3IIUC, this feature exists for several generations of Atom platforms, and it is always coupled with 1) and 2), so it could be skipped for the checking.
Yes, we can ignore that bit as it's not widely available and not required to solve the problem.
Quoted 21 lines
Quoted 20 lines
4)
X86_FEATURE_TSC_ADJUST5) At max. 4 sockets
The only reason I hate to disable HPET upfront at least during boot is that HPET is the best mechanism for the refined TSC calibration. PMTIMER sucks because it's slow and wraps around pretty quick.
So we could do the following even on platforms where HPET stops in some magic PC? state:
- Register it during early boot as clocksource
- Prevent the enablement as clockevent and the chardev hpet timer muck
- Prevent the magic PC? state up to the point where the refined TSC calibration is finished.
- Unregister it once the TSC has taken over as system clocksource and enable the magic PC? state in which HPET gets disfunctional.
Quoted 8 lines
Quoted 5 lines
This looks reasonable to me.
I have thought about lowering the hpet rating to lower than PMTIMER, so it still contributes in early boot phase, and fades out after PMTIMER is initialised.
Not a good idea.
pm_timeris initialized before the refined calibration finishes.
Yes, you are right. I missed the part.
I dug some old notes, and found another old case (kernel 3.4) that a broken PMTIMER as the watchdog clocksource wrongly judged TSC to be unstable and disabled it, which makes me agree more to the idea of "lift that requirement when the CPU has ..."
If the TSC has those bits to garantee its reliability, then no need to use a less reliable clocksource to monitor it.
Thanks,
Feng
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.
My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
From cf9ce0ecab8851a3745edcad92e072022af3dbd9 Mon Sep 17 00:00:00 2001
From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdog
On IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before acpi_pm timer registered.
This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.
Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued using add_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.
This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
BTW, commit fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.
IMO, the right solution is to only use hardware clocksource as watchdog. Then even if ticks are slow, both the running clocksource and the watchdog returns real time cost, and they still match.
--- kernel/time/clocksource.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/time/clocksource.c +4 -0
@@ -364,6 +364,10 @@ static void clocksource_select_watchdog(bool fallback) watchdog =NULL ;list_for_each_entry (cs, &clocksource_list, list) { +/* Do not use refined-jiffies as clocksource watchdog */ +if (cs->rating <=2 ) +continue ; +/* cs is a clocksource to be watched. */ if (cs->flags &CLOCK_SOURCE_MUST_VERIFY )continue ;
Signature · 1 line
2.17.1
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
I'm inclined to lift that requirement when the CPU has:
1)
X86_FEATURE_CONSTANT_TSC
2)X86_FEATURE_NONSTOP_TSC
3)X86_FEATURE_NONSTOP_TSC_S3
Quoted 12 lines
Quoted 4 lines
IIUC, this feature exists for several generations of Atom platforms,
and it is always coupled with 1) and 2), so it could be skipped for the checking.Yes, we can ignore that bit as it's not widely available and not required to solve the problem.
Quoted 4 lines
Quoted 4 lines
4)
X86_FEATURE_TSC_ADJUST5) At max. 4 sockets
Should we consider some other corner cases like TSC is not used as clocksource? refined_jiffies watchdog can break any other hardware clocksource when it becomes inaccurate.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
The only reason I hate to disable HPET upfront at least during boot is
that HPET is the best mechanism for the refined TSC calibration. PMTIMER
sucks because it's slow and wraps around pretty quick.So we could do the following even on platforms where HPET stops in some
magic PC? state:- Register it during early boot as clocksource
- Prevent the enablement as clockevent and the chardev hpet
timer muck
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
- Prevent the magic PC? state up to the point where the refined TSC calibration is finished.
- Unregister it once the TSC has taken over as system
clocksource and
enable the magic PC? state in which HPET gets disfunctional.
On the other side, for ICL, the HPET problem is observed on early samples, and I didn't reproduce the problem on new ones I have. But I need to confirm internally if it is safe to re-enable it first.
thanks,
rui
Quoted 15 lines
Quoted 7 lines
This looks reasonable to me.
I have thought about lowering the hpet rating to lower than PMTIMER, so it
still contributes in early boot phase, and fades out after PMTIMER is
initialised.Not a good idea.
pm_timeris initialized before the refined calibration
finishes.Thanks,
tglx
Quoted 36 lines · Zhang Rui
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
Quoted 1 line
From
cf9ce0ecab8851a3745edcad92e072022af3dbd9Mon Sep 17 00:00:00 2001From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdogOn IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before
acpi_pmtimer registered.This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued usingadd_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.
Quoted 8 lines
This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
BTW, commit
fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.
Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Thanks,
Liao Yu
Quoted 25 lines
IMO, the right solution is to only use hardware clocksource as watchdog. Then even if ticks are slow, both the running clocksource and the watchdog returns real time cost, and they still match.
Signed-off-by Zhang Rui <[email protected]>--- kernel/time/clocksource.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/time/clocksource.c +4
-0@@ -364,6 +364,10 @@ static void clocksource_select_watchdog(bool fallback) watchdog =NULL ;list_for_each_entry (cs, &clocksource_list, list) { +/* Do not use refined-jiffies as clocksource watchdog */ +if (cs->rating <=2 ) +continue ; +/* cs is a clocksource to be watched. */ if (cs->flags &CLOCK_SOURCE_MUST_VERIFY )continue ;
Quoted 41 lines · Yu Liao
Quoted 40 lines · Zhang Rui
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
Quoted 1 line
From
cf9ce0ecab8851a3745edcad92e072022af3dbd9Mon Sep 17 00:00:00 2001From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdogOn IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before
acpi_pmtimer registered.This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued usingadd_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
Quoted 8 lines
Quoted 4 lines
BTW, commit
fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86 or others), client or sever, if sever, how many sockets (2S/4S/8S)?
The error kernel log will also be helpful.
Thanks,
Feng
Quoted 29 lines
Thanks,
Liao YuQuoted 25 lines
IMO, the right solution is to only use hardware clocksource as watchdog. Then even if ticks are slow, both the running clocksource and the watchdog returns real time cost, and they still match.
Signed-off-by Zhang Rui <[email protected]>--- kernel/time/clocksource.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/time/clocksource.c +4
-0@@ -364,6 +364,10 @@ static void clocksource_select_watchdog(bool fallback) watchdog =NULL ;list_for_each_entry (cs, &clocksource_list, list) { +/* Do not use refined-jiffies as clocksource watchdog */ +if (cs->rating <=2 ) +continue ; +/* cs is a clocksource to be watched. */ if (cs->flags &CLOCK_SOURCE_MUST_VERIFY )continue ;
Quoted 51 lines · Feng Tang
Quoted 41 lines · Yu Liao
Quoted 40 lines · Zhang Rui
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
Quoted 1 line
From
cf9ce0ecab8851a3745edcad92e072022af3dbd9Mon Sep 17 00:00:00 2001From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdogOn IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before
acpi_pmtimer registered.This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued usingadd_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
Quoted 8 lines
Quoted 4 lines
BTW, commit
fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Quoted 6 lines
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86 or others), client or sever, if sever, how many sockets (2S/4S/8S)?
The error kernel log will also be helpful.
Hi, Feng Tang,
It's a X86 Sever. lscpu print the following information:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 224 On-line CPU(s) list: 0-223 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz Stepping: 4 CPU MHz: 3199.379 CPU max MHz: 3800.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 3.5 MiB L1i cache: 3.5 MiB L2 cache: 112 MiB L3 cache: 154 MiB NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223
Part of the kernel log is as follows.
[ 1.144402] smp: Brought up 4 nodes, 224 CPUs [ 1.144402] smpboot: Max logical packages: 4 [ 1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS) [ 1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc-early' as unstable because the skew is too large: [ 1.520010] clocksource: 'refined-jiffies' wd_now: fffb7210 wd_last: fffb7018 mask: ffffffff [ 1.520013] clocksource: 'tsc-early' cs_now: 6606717afddd0 cs_last: 66065eff88ad4 mask: ffffffffffffffff [ 1.520015] tsc: Marking TSC unstable due to clocksource watchdog [ 5.164635] node 0 initialised, 98233092 pages in 4013ms [ 5.209294] node 3 initialised, 98923232 pages in 4057ms [ 5.220001] node 2 initialised, 99054870 pages in 4068ms [ 5.222282] node 1 initialised, 99054870 pages in 4070ms
Thanks,
Xiongfeng
Quoted 35 lines
Thanks,
FengQuoted 20 lines
Thanks,
Liao YuQuoted 17 lines
IMO, the right solution is to only use hardware clocksource as watchdog. Then even if ticks are slow, both the running clocksource and the watchdog returns real time cost, and they still match.
Signed-off-by Zhang Rui <[email protected]>--- kernel/time/clocksource.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/time/clocksource.c +0
-0@@ -364,6 +364,10 @@ static void clocksource_select_watchdog(bool fallback) watchdog =NULL ;Quoted 9 lines
kernel/time/clocksource.c static void clocksource_select_watchdog(bool fallback) · line 373
Quoted 8 lines
kernel/time/clocksource.c +4
-0list_for_each_entry (cs, &clocksource_list, list) { +/* Do not use refined-jiffies as clocksource watchdog */ +if (cs->rating <=2 ) +continue ; +/* cs is a clocksource to be watched. */ if (cs->flags &CLOCK_SOURCE_MUST_VERIFY )continue ;.
Quoted 96 lines · Xiongfeng Wang
Quoted 51 lines · Feng Tang
Quoted 41 lines · Yu Liao
Quoted 40 lines · Zhang Rui
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
Quoted 1 line
From
cf9ce0ecab8851a3745edcad92e072022af3dbd9Mon Sep 17 00:00:00 2001From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdogOn IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before
acpi_pmtimer registered.This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued usingadd_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
Quoted 8 lines
Quoted 4 lines
BTW, commit
fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Quoted 6 lines
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86 or others), client or sever, if sever, how many sockets (2S/4S/8S)?
The error kernel log will also be helpful.
Hi, Feng Tang,
It's a X86 Sever. lscpu print the following information:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 224 On-line CPU(s) list: 0-223 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz Stepping: 4 CPU MHz: 3199.379 CPU max MHz: 3800.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 3.5 MiB L1i cache: 3.5 MiB L2 cache: 112 MiB L3 cache: 154 MiB NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223Part of the kernel log is as follows.
Quoted 14 lines
[ 1.144402] smp: Brought up 4 nodes, 224 CPUs [ 1.144402] smpboot: Max logical packages: 4 [ 1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS) [ 1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc-early' as unstable because the skew is too large: [ 1.520010] clocksource: 'refined-jiffies' wd_now: fffb7210 wd_last: fffb7018 mask: ffffffff [ 1.520013] clocksource: 'tsc-early' cs_now: 6606717afddd0 cs_last: 66065eff88ad4 mask: ffffffffffffffff [ 1.520015] tsc: Marking TSC unstable due to clocksource watchdog [ 5.164635] node 0 initialised, 98233092 pages in 4013ms [ 5.209294] node 3 initialised, 98923232 pages in 4057ms [ 5.220001] node 2 initialised, 99054870 pages in 4068ms [ 5.222282] node 1 initialised, 99054870 pages in 4070ms
Thanks Xiaofeng for the info.
Could you try the below patch? It is kinda extension of
b50db7095fe0 ("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms")
which I have run limited test on some 4 sockets Haswell and Cascadelake AP x86 servers.
Thanks,
Feng
---
arch/x86/kernel/tsc.c +1 -1
@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )tsc_disable_clocksource_watchdog (); }
Quoted 131 lines · Feng Tang
Quoted 96 lines · Xiongfeng Wang
Quoted 51 lines · Feng Tang
Quoted 41 lines · Yu Liao
Quoted 40 lines · Zhang Rui
Quoted 8 lines · Thomas Gleixner
Feng,
Quoted 5 lines · Feng Tang
Quoted 4 lines · Thomas Gleixner
On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
Yes, that can happen. But OTOH, we should start to think about the
requirements for using the TSC watchdog.My original proposal is to disable jiffies and refined-jiffies as the clocksource watchdog, because they are not reliable and it's better to use clocksource that has a hardware counter as watchdog, like the patch below, which I didn't sent out for upstream.
Quoted 1 line
From
cf9ce0ecab8851a3745edcad92e072022af3dbd9Mon Sep 17 00:00:00 2001From: Zhang Rui <[email protected]>
Date: Fri, 19 Jun 2020 22:03:23 +0800
Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdogOn IA platforms, if HPET is disabled, either via x86 early-quirks, or via kernel commandline, refined-jiffies will be used as clocksource watchdog in early boot phase, before
acpi_pmtimer registered.This is not a problem if jiffies are accurate.
But in some cases, for example, when serial console is enabled, it may take several milliseconds to write to the console, with irq disabled, frequently. Thus many ticks may become longer than it should be.Using refined-jiffies as watchdog in this case breaks the system because
a) duration calculated by refined-jiffies watchdog is always consistent with the watchdog timeout issued usingadd_timer(), say, around 500ms.
b) duration calculated by the running clocksource, usually TSC on IA platforms, reflects the real time cost, which may be much larger.
This results in the running clocksource being disabled erroneously.This is reproduced on ICL because HPET is disabled in x86 early-quirks, and also reproduced on a KBL and a WHL platform when HPET is disabled via command line.
Quoted 8 lines
Quoted 4 lines
BTW, commit
fd329f276eca
("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is another example that refined-jiffies causes the same problem when ticks become slow for some other reason.Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Quoted 6 lines
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86 or others), client or sever, if sever, how many sockets (2S/4S/8S)?
The error kernel log will also be helpful.
Hi, Feng Tang,
It's a X86 Sever. lscpu print the following information:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 224 On-line CPU(s) list: 0-223 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz Stepping: 4 CPU MHz: 3199.379 CPU max MHz: 3800.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 3.5 MiB L1i cache: 3.5 MiB L2 cache: 112 MiB L3 cache: 154 MiB NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223Part of the kernel log is as follows.
Quoted 14 lines
[ 1.144402] smp: Brought up 4 nodes, 224 CPUs [ 1.144402] smpboot: Max logical packages: 4 [ 1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS) [ 1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc-early' as unstable because the skew is too large: [ 1.520010] clocksource: 'refined-jiffies' wd_now: fffb7210 wd_last: fffb7018 mask: ffffffff [ 1.520013] clocksource: 'tsc-early' cs_now: 6606717afddd0 cs_last: 66065eff88ad4 mask: ffffffffffffffff [ 1.520015] tsc: Marking TSC unstable due to clocksource watchdog [ 5.164635] node 0 initialised, 98233092 pages in 4013ms [ 5.209294] node 3 initialised, 98923232 pages in 4057ms [ 5.220001] node 2 initialised, 99054870 pages in 4068ms [ 5.222282] node 1 initialised, 99054870 pages in 4070msThanks Xiaofeng for the info.
Could you try the below patch? It is kinda extension of
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms")which I have run limited test on some 4 sockets Haswell and Cascadelake AP x86 servers.
Thanks,
Feng
---arch/x86/kernel/tsc.c +0
-0
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1223
Quoted 8 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )tsc_disable_clocksource_watchdog (); }
Hi Feng,
I tested this patch on a previous server and it fixes the issue.
Thanks,
Yu
On Fri, Sep 30, 2022 at 05:45:29PM +0800, Yu Liao wrote:
[...]
Quoted 91 lines
Quoted 81 lines
Quoted 47 lines
Quoted 11 lines
Quoted 4 lines
Hi, Zhang Rui, we have met the same problem as you mentioned above. I have tested the following modification. It can solve the problem. Do you have plan to push it to upstream ?
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86 or others), client or sever, if sever, how many sockets (2S/4S/8S)?
The error kernel log will also be helpful.
Hi, Feng Tang,
It's a X86 Sever. lscpu print the following information:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 224 On-line CPU(s) list: 0-223 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz Stepping: 4 CPU MHz: 3199.379 CPU max MHz: 3800.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 3.5 MiB L1i cache: 3.5 MiB L2 cache: 112 MiB L3 cache: 154 MiB NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223Part of the kernel log is as follows.
Quoted 14 lines
[ 1.144402] smp: Brought up 4 nodes, 224 CPUs [ 1.144402] smpboot: Max logical packages: 4 [ 1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS) [ 1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc-early' as unstable because the skew is too large: [ 1.520010] clocksource: 'refined-jiffies' wd_now: fffb7210 wd_last: fffb7018 mask: ffffffff [ 1.520013] clocksource: 'tsc-early' cs_now: 6606717afddd0 cs_last: 66065eff88ad4 mask: ffffffffffffffff [ 1.520015] tsc: Marking TSC unstable due to clocksource watchdog [ 5.164635] node 0 initialised, 98233092 pages in 4013ms [ 5.209294] node 3 initialised, 98923232 pages in 4057ms [ 5.220001] node 2 initialised, 99054870 pages in 4068ms [ 5.222282] node 1 initialised, 99054870 pages in 4070msThanks Xiaofeng for the info.
Could you try the below patch? It is kinda extension of
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms")which I have run limited test on some 4 sockets Haswell and Cascadelake AP x86 servers.
Thanks,
Feng
---arch/x86/kernel/tsc.c +0
-0arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1223
Quoted 8 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )tsc_disable_clocksource_watchdog (); }Hi Feng,
Quoted 1 line
I tested this patch on a previous server and it fixes the issue.
Thanks for the testing, please do let us know if there is any TSC problem after long time or stress running.
Plan to send the patch for merging.
Thanks,
Feng
Quoted 4 lines
Thanks,
Yu
Quoted 108 lines · Feng Tang
Quoted 1 line
On Fri, Sep 30, 2022 at 05:45:29PM +0800, Yu Liao wrote:
[...]Quoted 104 lines
Quoted 94 lines
Quoted 63 lines
Quoted 6 lines
Quoted 5 lines
Hi, Zhang Rui, we have met the same problem as you mentioned above. I have
tested the following modification. It can solve the problem. Do you have plan
to push it to upstream ?Quoted 8 lines
Hi Liao Yu,
Could you provoide more details? Like, what ARCH is the platform (x86
or others), client or sever, if sever, how many sockets (2S/4S/8S)?The error kernel log will also be helpful.
Hi, Feng Tang,
It's a X86 Sever. lscpu print the following information:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 224 On-line CPU(s) list: 0-223 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz Stepping: 4 CPU MHz: 3199.379 CPU max MHz: 3800.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 3.5 MiB L1i cache: 3.5 MiB L2 cache: 112 MiB L3 cache: 154 MiB NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223Part of the kernel log is as follows.
[ 1.144402] smp: Brought up 4 nodes, 224 CPUs [ 1.144402] smpboot: Max logical packages: 4 [ 1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS) [ 1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc-early' as unstable because the skew is too large: [ 1.520010] clocksource: 'refined- jiffies' wd_now: fffb7210 wd_last: fffb7018 mask: ffffffff [ 1.520013] clocksource: 'tsc-early'Quoted 8 lines
cs_now:6606717afddd0cs_last:66065eff88ad4mask: ffffffffffffffff[ 1.520015] tsc: Marking TSC unstable due to clocksource watchdog [ 5.164635] node 0 initialised, 98233092 pages in 4013ms [ 5.209294] node 3 initialised, 98923232 pages in 4057ms [ 5.220001] node 2 initialised, 99054870 pages in 4068ms [ 5.222282] node 1 initialised, 99054870 pages in 4070msThanks Xiaofeng for the info.
Could you try the below patch? It is kinda extension of
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms")which I have run limited test on some 4 sockets Haswell and Cascadelake
AP x86 servers.Thanks,
Feng
---arch/x86/kernel/tsc.c +0
-0arch/x86/kernel/tsc.c static void __init · line 1223
Quoted 8 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void) if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) && boot_cpu_has(X86_FEATURE_NONSTOP_TSC) && boot_cpu_has(X86_FEATURE_TSC_ADJUST) && - nr_online_nodes <= 2) + nr_online_nodes <= 8) tsc_disable_clocksource_watchdog(); }Hi Feng,
Quoted 7 lines
Quoted 1 line
I tested this patch on a previous server and it fixes the issue.
Thanks for the testing, please do let us know if there is any TSC problem after long time or stress running.
Plan to send the patch for merging.
Good to know.
This patch fixes the TSC issue for IA.
while the patch at
https://lore.kernel.org/lkml/YzZDLBKbDTbNr45b@feng-clx/T/#m34094630193e8320c6d75e9c8aeabe7633e051d2
impacts more Arches but I'm not sure if similar problem is observed on other Arches as well. So I will hold it for now.
thanks,
rui
On Sat, Oct 01, 2022 at 01:18:05PM +0800, Zhang Rui wrote:
[...]
Quoted 40 lines
Quoted 32 lines
Quoted 32 lines
Quoted 22 lines
Thanks Xiaofeng for the info.
Could you try the below patch? It is kinda extension of
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms")which I have run limited test on some 4 sockets Haswell and Cascadelake
AP x86 servers.Thanks,
Feng
---arch/x86/kernel/tsc.c +0
-0arch/x86/kernel/tsc.c static void __init · line 1223
Quoted 8 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void) if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) && boot_cpu_has(X86_FEATURE_NONSTOP_TSC) && boot_cpu_has(X86_FEATURE_TSC_ADJUST) && - nr_online_nodes <= 2) + nr_online_nodes <= 8) tsc_disable_clocksource_watchdog(); }Hi Feng,
Quoted 7 lines
Quoted 1 line
I tested this patch on a previous server and it fixes the issue.
Thanks for the testing, please do let us know if there is any TSC problem after long time or stress running.
Plan to send the patch for merging.
Quoted 7 lines
Good to know.
This patch fixes the TSC issue for IA.
while the patch at
https://lore.kernel.org/lkml/YzZDLBKbDTbNr45b@feng-clx/T/#m34094630193e8320c6d75e9c8aeabe7633e051d2
impacts more Arches but I'm not sure if similar problem is observed on other Arches as well. So I will hold it for now.
I think your patch is more general and architecture independent. We've seen quite some cases in boot phase on IA platforms, but I guess this is also possible for other ARCHs.
Thanks,
Feng
Quoted 3 lines
thanks,
rui