x86/tsc: Extend the watchdog check exemption to 4S/8S machine
PATCH 1/1 x86/tsc: Extend the watchdog check exemption to 4S/8S machine
There is report again that the tsc clocksource on a 4 sockets x86 Skylake server was wrongly judged as 'unstable' by 'jiffies' watchdog, and disabled [1]. Also we got silimar reports on 8 sockets platform from internal test.
Commit b50db7095fe0 ("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms") was introduce to deal with these false alarms of tsc unstable issues, covering qualified platforms for 2 sockets or smaller ones.
Extend the exemption also to 4/8 sockets to fix the issue.
Rui also proposed another way to disable 'jiffies' as clocksource watchdog [2], which can also solve this specific problem in an architecture independent way, with one limitation that some tsc false alarms are reported by other watchdogs like HPET in post-boot time, while 'jiffies' is mostly used in boot phase before hardware clocksources are initialized.
[1]. https://lore.kernel.org/all/[email protected]/
[2]. https://lore.kernel.org/all/[email protected]/
--- arch/x86/kernel/tsc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/tsc.c +1 -1
@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )tsc_disable_clocksource_watchdog (); }
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerQuoted 34 lines · Feng Tang
There is report again that the tsc clocksource on a 4 sockets x86 Skylake server was wrongly judged as 'unstable' by 'jiffies' watchdog, and disabled [1]. Also we got silimar reports on 8 sockets platform from internal test.
Commit
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms") was introduce to deal with these false alarms of tsc unstable issues, covering qualified platforms for 2 sockets or smaller ones.Extend the exemption also to 4/8 sockets to fix the issue.
Rui also proposed another way to disable 'jiffies' as clocksource watchdog [2], which can also solve this specific problem in an architecture independent way, with one limitation that some tsc false alarms are reported by other watchdogs like HPET in post-boot time, while 'jiffies' is mostly used in boot phase before hardware clocksources are initialized.
[1]. https://lore.kernel.org/all/[email protected]/
[2]. https://lore.kernel.org/all/[email protected]/Reported-by Yu Liao <[email protected]>Tested-by Yu Liao <[email protected]>Signed-off-by Feng Tang <[email protected]>--- arch/x86/kernel/tsc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/tsc.c +0
-0
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 5 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )
So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
AFAIK there is no architectural guarantee for >4 sockets to have a sane TSC. If there is one, the above should be limited to architectures that conform.
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Feng Tang
There is report again that the tsc clocksource on a 4 sockets x86 Skylake server was wrongly judged as 'unstable' by 'jiffies' watchdog, and disabled [1]. Also we got silimar reports on 8 sockets platform from internal test.
Commit
b50db7095fe0("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms") was introduce to deal with these false alarms of tsc unstable issues, covering qualified platforms for 2 sockets or smaller ones.Extend the exemption also to 4/8 sockets to fix the issue.
Rui also proposed another way to disable 'jiffies' as clocksource watchdog [2], which can also solve this specific problem in an architecture independent way, with one limitation that some tsc false alarms are reported by other watchdogs like HPET in post-boot time, while 'jiffies' is mostly used in boot phase before hardware clocksources are initialized.
[1]. https://lore.kernel.org/all/[email protected]/
[2]. https://lore.kernel.org/all/[email protected]/Reported-by Yu Liao <[email protected]>Tested-by Yu Liao <[email protected]>Signed-off-by Feng Tang <[email protected]>--- arch/x86/kernel/tsc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 10 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
Quoted 3 lines
AFAIK there is no architectural guarantee for >4 sockets to have a sane TSC. If there is one, the above should be limited to architectures that conform.
Thanks for the note! Yes, we should be very cautious for 8 sockets machine. Will limit the max sockets to 4, which was also originally suggested by Thomas.
Thanks,
Feng
Quoted 12 lines · Feng Tang
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/tsc.c +0
-0Quoted 8 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 6 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
Quoted 3 lines
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
That's not a great check. Think about numa=fake=4U, for instance. Or a single-socket system with persistent memory and high bandwidth memory.
Basically 'nr_online_nodes' is a software construct. It's going to be really hard to infer anything from it about what the _hardware_ is.
Quoted 17 lines · Dave Hansen
Quoted 12 lines · Feng Tang
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/tsc.c +0
-0Quoted 8 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 6 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
Quoted 3 lines
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
Quoted 5 lines
That's not a great check. Think about numa=fake=4U, for instance. Or a single-socket system with persistent memory and high bandwidth memory.
Basically '
nr_online_nodes' is a software construct. It's going to be really hard to infer anything from it about what the _hardware_ is.
You are right! How to get the socket number was indeed a trouble when I worked on commit b50db7095fe0, the problem is related to the initialization order. This tsc check needs to be done in tsc_init(), while the node_stats[] get initialized in later's call of smp_init().
For the case you mentioned above, I dug out some old logs which showed its init order:
numa=fake=4 on a SKL desktop
================
[ 0.000066] [tsc_early_init()]: nr_online_nodes = 1 [ 0.000068] [tsc_early_init()]: nr_cpu_nodes = 0 [ 0.000070] [tsc_early_init()]: nr_mem_nodes = 0 [ 0.104015] [tsc_init()]: nr_online_nodes = 4 [ 0.104019] [tsc_init()]: nr_cpu_nodes = 0 [ 0.104022] [tsc_init()]: nr_mem_nodes = 4 [ 0.124778] smp: Brought up 4 nodes, 4 CPUs [ 0.760915] [init_tsc_clocksource()]: nr_online_nodes = 4 [ 0.760919] [init_tsc_clocksource()]: nr_cpu_nodes = 4 [ 0.760922] [init_tsc_clocksource()]: nr_mem_nodes = 4
QEMU with 2 CPU-DRAM nodes + 2 Persistent memory nodes
========================================================
[ 0.066651] [tsc_early_init()]: nr_online_nodes = 1 [ 0.067494] [tsc_early_init()]: nr_cpu_nodes = 0 [ 0.068288] [tsc_early_init()]: nr_mem_nodes = 0 [ 0.677694] [tsc_init()]: nr_online_nodes = 4 [ 0.678862] [tsc_init()]: nr_cpu_nodes = 0 [ 0.679962] [tsc_init()]: nr_mem_nodes = 4 [ 1.139240] [init_tsc_clocksource()]: nr_online_nodes = 4 [ 1.140576] [init_tsc_clocksource()]: nr_cpu_nodes = 2 [ 1.141823] [init_tsc_clocksource()]: nr_mem_nodes = 4 [ 1.660100] [kernel_init()]: nr_online_nodes = 4 [ 1.661234] [kernel_init()]: nr_cpu_nodes = 2 [ 1.662300] [kernel_init()]: nr_mem_nodes = 4
The 'nr_online_nodes' was chosed in the hope of that, in worse case the patch is just a nop and won't wrongly lift the check.
One possible solution for this problem is to leverage the SRAT table early init which is called before tsc_init(), and can provide CPU nodes info. Will try this way.
Thanks,
Feng
Quoted 60 lines · Feng Tang
Quoted 17 lines · Dave Hansen
Quoted 12 lines · Feng Tang
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/tsc.c +0
-0Quoted 8 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 6 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
Quoted 3 lines
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
Quoted 5 lines
That's not a great check. Think about numa=fake=4U, for instance. Or a single-socket system with persistent memory and high bandwidth memory.
Basically '
nr_online_nodes' is a software construct. It's going to be really hard to infer anything from it about what the _hardware_ is.You are right! How to get the socket number was indeed a trouble when I worked on commit
b50db7095fe0, the problem is related to the initialization order. This tsc check needs to be done intsc_init(), while the node_stats[] get initialized in later's call ofsmp_init().For the case you mentioned above, I dug out some old logs which showed its init order:
numa=fake=4 on a SKL desktop
================[ 0.000066] [tsc_early_init()]: nr_online_nodes = 1 [ 0.000068] [tsc_early_init()]: nr_cpu_nodes = 0 [ 0.000070] [tsc_early_init()]: nr_mem_nodes = 0 [ 0.104015] [tsc_init()]: nr_online_nodes = 4 [ 0.104019] [tsc_init()]: nr_cpu_nodes = 0 [ 0.104022] [tsc_init()]: nr_mem_nodes = 4 [ 0.124778] smp: Brought up 4 nodes, 4 CPUs [ 0.760915] [init_tsc_clocksource()]: nr_online_nodes = 4 [ 0.760919] [init_tsc_clocksource()]: nr_cpu_nodes = 4 [ 0.760922] [init_tsc_clocksource()]: nr_mem_nodes = 4QEMU with 2 CPU-DRAM nodes + 2 Persistent memory nodes
========================================================[ 0.066651] [tsc_early_init()]: nr_online_nodes = 1 [ 0.067494] [tsc_early_init()]: nr_cpu_nodes = 0 [ 0.068288] [tsc_early_init()]: nr_mem_nodes = 0 [ 0.677694] [tsc_init()]: nr_online_nodes = 4 [ 0.678862] [tsc_init()]: nr_cpu_nodes = 0 [ 0.679962] [tsc_init()]: nr_mem_nodes = 4 [ 1.139240] [init_tsc_clocksource()]: nr_online_nodes = 4 [ 1.140576] [init_tsc_clocksource()]: nr_cpu_nodes = 2 [ 1.141823] [init_tsc_clocksource()]: nr_mem_nodes = 4 [ 1.660100] [kernel_init()]: nr_online_nodes = 4 [ 1.661234] [kernel_init()]: nr_cpu_nodes = 2 [ 1.662300] [kernel_init()]: nr_mem_nodes = 4
Quoted 6 lines
The '
nr_online_nodes' was chosed in the hope of that, in worse case the patch is just a nop and won't wrongly lift the check.One possible solution for this problem is to leverage the SRAT table early init which is called before
tsc_init(), and can provide CPU nodes info. Will try this way.
Th simple patch below is to have a dedicate CPU nodemask and set it in early SRAT CPU parsing, still it has problem when sub-numa is enabled in BIOS where there are more NUMA nodes in SRAT table. (also I'm not sure the change to amdtopology.c is right)
Thanks,
Feng
arch/x86/include/asm/numa.h +1 -0
@@ -31,6 +31,7 @@ extern int numa_off;*/ extern s16 __apicid_to_node[MAX_LOCAL_APIC ];extern nodemask_t numa_nodes_parsed__initdata ; +extern nodemask_t numa_cpu_nodes__initdata ;extern int __init numa_add_memblk (int nodeid,u64 start,u64 end);extern void __init numa_set_distance (int from,int to,int distance);
arch/x86/kernel/tsc.c +2 -1
@@ -29,6 +29,7 @@#include <asm/intel-family.h> #include <asm/i8259.h> #include <asm/uv/uv.h> +#include <asm/numa.h> unsigned int __read_mostly cpu_khz;/* TSC clocks / usec, not used here */ EXPORT_SYMBOL (cpu_khz);
@@ -1218,7 +1219,7 @@ first_dump();if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) +nodes_weight (numa_cpu_nodes) <=2 )tsc_disable_clocksource_watchdog (); }
arch/x86/mm/amdtopology.c +1 -0
@@ -152,6 +152,7 @@ int __init amd_numa_init(void) prevbase = base;numa_add_memblk (nodeid, base, limit);node_set (nodeid, numa_nodes_parsed); +node_set (nodeid, numa_cpu_nodes); }if (nodes_empty (numa_nodes_parsed))
arch/x86/mm/numa.c +1 -0
@@ -21,6 +21,7 @@int numa_off;nodemask_t numa_nodes_parsed__initdata ; +nodemask_t numa_cpu_nodes__initdata ;struct pglist_data *node_data[MAX_NUMNODES ]__read_mostly ;EXPORT_SYMBOL (node_data);
arch/x86/mm/srat.c +2 -0
@@ -59,6 +59,7 @@ acpi_numa_x2apic_affinity_init(struct acpi_srat_x2apic_cpu_affinity *pa) }set_apicid_to_node (apic_id, node);node_set (node, numa_nodes_parsed); +node_set (node, numa_cpu_nodes); printk(KERN_INFO "SRAT: PXM %u -> APIC 0x%04x -> Node %u\n",
@@ -106,6 +107,7 @@ acpi_numa_processor_affinity_init(struct acpi_srat_cpu_affinity *pa)set_apicid_to_node (apic_id, node);node_set (node, numa_nodes_parsed); +node_set (node, numa_cpu_nodes); printk(KERN_INFO "SRAT: PXM %u -> APIC 0x%02x -> Node %u\n",
Peter Zijlstra
maintainerQuoted 4 lines · Feng Tang
Th simple patch below is to have a dedicate CPU nodemask and set it in early SRAT CPU parsing, still it has problem when sub-numa is enabled in BIOS where there are more NUMA nodes in SRAT table. (also I'm not sure the change to
amdtopology.cis right)
No; none of this has anything to do with nodes. This is about sockets.
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Feng Tang
Th simple patch below is to have a dedicate CPU nodemask and set it in early SRAT CPU parsing, still it has problem when sub-numa is enabled in BIOS where there are more NUMA nodes in SRAT table. (also I'm not sure the change to
amdtopology.cis right)No; none of this has anything to do with nodes. This is about sockets.
Exactly. All we try to do is to get a closer number to the socket numbers (also stated in current code comments)
According to our discussion, we haven't found a way to get a very accurate number of sockets, so I plan to (if no objection):
* Send a patch lifting the socket number check from 2 to 4, to fix the issue reported by Yu Liao.
* Send another RFC patch[1], which makes the socket number more accurate, as it solve the 2 problems mentioned by Dave:
- fakenuma (numa=fake=4 etc)
- system with CPU-DRAM nodes + HBM nodes + Persistent Memory nodes but it still can't cover the subnuma enabled case
[1]. https://lore.kernel.org/lkml/Y0UgeUIJSFNR4mQB@feng-clx/
Thanks,
Feng
Peter Zijlstra
maintainerQuoted 17 lines · Dave Hansen
Quoted 12 lines · Feng Tang
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/tsc.c +0
-0Quoted 8 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 6 lines
arch/x86/kernel/tsc.c +1
-1@@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC sync'ed ?
Quoted 3 lines
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
Quoted 5 lines
That's not a great check. Think about numa=fake=4U, for instance. Or a single-socket system with persistent memory and high bandwidth memory.
Basically '
nr_online_nodes' is a software construct. It's going to be really hard to infer anything from it about what the _hardware_ is.
We have both c->phys_proc_id and c->logical_proc_id along with logical_packages.
I'm thinking you want something like max(c->phys_proc_id) <= 4. Because even if you only populate 4 sockets of an 8 socket server you're up a creek without no paddles.
But it all comes down to how much drugs the firmware teams have had :/ It is entirely possible to enumerate with phys_proc_id==42 on a 2 socket system.
Quoted 37 lines · Peter Zijlstra
Quoted 19 lines · Dave Hansen
Quoted 14 lines · Feng Tang
Quoted 6 lines
Quoted 6 lines
arch/x86/kernel/tsc.c +0
-0@@ -1217,7 +1217,7 @@ static void __initcheck_system_tsc_reliable (void)Quoted 8 lines
arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1220
Quoted 5 lines
arch/x86/kernel/tsc.c +1
-1if (boot_cpu_has (X86_FEATURE_CONSTANT_TSC ) &&boot_cpu_has (X86_FEATURE_NONSTOP_TSC ) &&boot_cpu_has (X86_FEATURE_TSC_ADJUST ) && - nr_online_nodes <=2 ) + nr_online_nodes <=8 )So you're saying all 8 socket systems since Broadwell (?) are TSC
sync'ed ?Quoted 3 lines
No, I didn't mean that. I haven't got chance to any 8 sockets machine, and I got a report last month that on one 8S machine, the TSC was judged 'unstable' by HPET as watchdog.
Quoted 8 lines
That's not a great check. Think about numa=fake=4U, for instance. Or a
single-socket system with persistent memory and high bandwidth memory.Basically '
nr_online_nodes' is a software construct. It's going to be
really hard to infer anything from it about what the _hardware_ is.We have both
c->phys_proc_idandc->logical_proc_idalong withlogical_packages.I'm thinking you want something like
max(c->phys_proc_id)<= 4. Because
even if you only populate 4 sockets of an 8 socket server you're up a creek without no paddles.
Quoted 5 lines
But it all comes down to how much drugs the firmware teams have had :/
It is entirely possible to enumerate with phys_proc_id==42 on a 2 socket
system.
topology_max_packages() or variable logical_packages can tell the maximum packages.
But this check_system_tsc_reliable() is done in early boot phase where we have boot cpu only. And the cpu topology is not built up at this stage.
thanks,
rui
Peter Zijlstra
maintainerQuoted 5 lines · Zhang Rui
topology_max_packages()or variablelogical_packagescan tell the maximum packages.
But thischeck_system_tsc_reliable()is done in early boot phase where we have boot cpu only. And the cpu topology is not built up at this stage.
Is there a problem with disabling the TSC watchdog later in boot -- after SMP bringup for example?
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Zhang Rui
topology_max_packages()or variablelogical_packagescan tell the maximum packages.
But thischeck_system_tsc_reliable()is done in early boot phase where we have boot cpu only. And the cpu topology is not built up at this stage.Is there a problem with disabling the TSC watchdog later in boot -- after SMP bringup for example?
Currently the watchdog is disabled inside tsc_init(), right before 'tsc-early' clocksrouce is registered, otherwise it starts to be monitored by 'jiffies' as watchdog. And there has been many cases that 'jiffies' watchdog misjudged tsc as 'unstable' in early boot phase, including recent Yu Liao's report on a 4 socket Skylake server.
Thanks,
Feng