cpu/hotplug: broken sibling thread hotplug
Subsystems (4)
In case guest is booted with one CPU present and then later a sibling CPU is hotplugged [1], it stays offline since SMT is disabled.
Bisects to
73d5e2b47264 ("cpu/hotplug: detect SMT disabled by BIOS")
which used __max_smt_threads to decide disabling SMT and in case [1] only primary CPU thread is present hence SMT is disabled.
Later bc2d8d262cba (cpu/hotplug: Fix SMT supported evaluation), rewrites code path but evaluation criteria still depends on sibling thread being present at boot time, so problem persist.
1) QEMU -smp 1,sockets=2,cores=1,threads=2 -monitor stdio ...
# hotplug sibling thread (qemu) device_add qemu64-x86_64-cpu,socket-id=0,core-id=0,thread-id=1
I've failed to find reasoning behind statement:
"
cpu/hotplug: detect SMT disabled by BIOS
If SMT is disabled in BIOS, the CPU code doesn't properly detect it.
"
Question is
1: why cpu_smt_check_topology_early() at check_bugs()
wasn't sufficient to detect SMT disabled in BIOS and
2: why side-effect of present at boot siblings were used
to keep SMT enabled?
Following quick hack fixes the sibling issue but that's effectively means reverting both above mentioned so we are back to the original issue "If SMT is disabled in BIOS, ..." which roots I weren't able to locate.
---
kernel/cpu.c +1 -2
@@ -415,7 +415,7 @@ void __init cpu_smt_check_topology_early(void)*/ void __init cpu_smt_check_topology (void ) { -if (!cpu_smt_available) +if (!topology_smt_supported ()) cpu_smt_control =CPU_SMT_NOT_SUPPORTED ; } ---
Quoted 22 lines · Igor Mammedov
In case guest is booted with one CPU present and then later a sibling CPU is hotplugged [1], it stays offline since SMT is disabled.
Bisects to
73d5e2b47264("cpu/hotplug: detect SMT disabled by BIOS")
which used__max_smt_threadsto decide disabling SMT and in case [1] only primary CPU thread is present hence SMT is disabled.Later
bc2d8d262cba(cpu/hotplug: Fix SMT supported evaluation), rewrites code path but evaluation criteria still depends on sibling thread being present at boot time, so problem persist.1) QEMU -smp 1,sockets=2,cores=1,threads=2 -monitor stdio ...
# hotplug sibling thread (qemu) device_add qemu64-x86_64-cpu,socket-id=0,core-id=0,thread-id=1I've failed to find reasoning behind statement:
"
cpu/hotplug: detect SMT disabled by BIOS
Quoted 8 lines
If SMT is disabled in BIOS, the CPU code doesn't properly detect it.
"Question is
1: whycpu_smt_check_topology_early()atcheck_bugs()
wasn't sufficient to detect SMT disabled in BIOS and
2: why side-effect of present at boot siblings were used
to keep SMT enabled?
Hi Igor,
Thanks for reporting this.
The original problems with SMT disabled in BIOS were:
1) /sys/devices/system/cpu/smt showed "on"; and
2) vmx_vm_init() was incorrectly showing the L1TF_MSG_SMT warning.
So 73d5e2b47264 ("cpu/hotplug: detect SMT disabled by BIOS") was intended to fix that, by reporting SMT as permanently disabled, when the BIOS has done so.
The problem is, I don't think there's a way to detect a difference between the HW "SMT disabled by BIOS" case and the virt "Sibling not yet plugged in" case.
I'd propose that we consider #1 above to *not* be a problem. Because in the virt case, it's possible that a sibling thread can be brought online later. So it makes sense to default with smt control "on" to allow for that possibility.
The real problem is #2. Which is a simple fix: change vmx_vm_init() to query the current SMT state with sched_smt_active(), instead of looking at the "control" sysfs value.
How about this patch? It's just a revert of 73d5e2b47264 and bc2d8d262cba, plus the 1-line vmx_vm_init() change. If it looks ok to you, I can clean it up and submit an official version.
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -70,7 +70,7 @@ void __init check_bugs(void)* identify_boot_cpu() initialized SMT support information, let the * core code know. */ -cpu_smt_check_topology_early (); +cpu_smt_check_topology ();if (!IS_ENABLED (CONFIG_SMP )) {pr_info ("CPU: " );
arch/x86/kvm/vmx.c +1 -1
@@ -11102,7 +11102,7 @@ static int vmx_vm_init(struct kvm *kvm)* Warn upon starting the first VM in a potentially * insecure environment. */ -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (sched_smt_active ())pr_warn_once (L1TF_MSG_SMT );if (l1tf_vmx_mitigation ==VMENTER_L1D_FLUSH_NEVER )pr_warn_once (L1TF_MSG_L1D );
include/linux/cpu.h +0 -2
@@ -180,12 +180,10 @@ enum cpuhp_smt_control {#if defined(CONFIG_SMP ) && defined(CONFIG_HOTPLUG_SMT )extern enum cpuhp_smt_control cpu_smt_control;extern void cpu_smt_disable (bool force); -extern void cpu_smt_check_topology_early (void );extern void cpu_smt_check_topology (void );#else # define cpu_smt_control (CPU_SMT_ENABLED)static inline void cpu_smt_disable (bool force) { } -static inline void cpu_smt_check_topology_early (void ) { }static inline void cpu_smt_check_topology (void ) { }#endif
kernel/cpu.c +4 -28
@@ -360,8 +360,6 @@ void __weak arch_smt_update(void) { }enum cpuhp_smt_control cpu_smt_control__read_mostly =CPU_SMT_ENABLED ;EXPORT_SYMBOL_GPL (cpu_smt_control); -static bool cpu_smt_available__read_mostly ; -void __init cpu_smt_disable (bool force) { if (cpu_smt_control == CPU_SMT_FORCE_DISABLED ||
@@ -378,25 +376,11 @@ void __init cpu_smt_disable(bool force)/* * The decision whether SMT is supported can only be done after the full -* CPU identification. Called from architecture code before non boot CPUs -* are brought up. -*/ -void __init cpu_smt_check_topology_early (void ) -{ -if (!topology_smt_supported ()) - cpu_smt_control =CPU_SMT_NOT_SUPPORTED ; -} - -/* -* If SMT was disabled by BIOS, detect it here, after the CPUs have been -* brought online. This ensures the smt/l1tf sysfs entries are consistent -* with reality. cpu_smt_available is set to true during the bringup of non -* boot CPUs when a SMT sibling is detected. Note, this may overwrite -* cpu_smt_control's previous setting. +* CPU identification. Called from architecture code. */ void __init cpu_smt_check_topology (void ) { -if (!cpu_smt_available) +if (!topology_smt_supported ()) cpu_smt_control =CPU_SMT_NOT_SUPPORTED ; }
@@ -409,18 +393,10 @@ early_param("nosmt", smt_cmdline_disable);static inline bool cpu_smt_allowed (unsigned int cpu) { -if (topology_is_primary_thread (cpu)) +if (cpu_smt_control ==CPU_SMT_ENABLED )return true; -/* -* If the CPU is not a 'primary' thread and the booted_once bit is -* set then the processor has SMT support. Store this information -* for the late check of SMT support in cpu_smt_check_topology(). -*/ -if (per_cpu (cpuhp_state, cpu).booted_once) - cpu_smt_available = true; - -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (topology_is_primary_thread (cpu))return true;/*
kernel/smp.c +0 -2
@@ -584,8 +584,6 @@ void __init smp_init(void) num_nodes, (num_nodes >1 ?"s" :"" ), num_cpus, (num_cpus > 1 ? "s" : "")); -/* Final decision about SMT support */ -cpu_smt_check_topology ();/* Any cleanup work */ smp_cpus_done (setup_max_cpus); }
Quoted 3 lines · Josh Poimboeuf
How about this patch? It's just a revert of
73d5e2b47264andbc2d8d262cba, plus the 1-linevmx_vm_init()change. If it looks ok to you, I can clean it up and submit an official version.
This one actually compiles...
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -71,7 +71,7 @@ void __init check_bugs(void)* identify_boot_cpu() initialized SMT support information, let the * core code know. */ -cpu_smt_check_topology_early (); +cpu_smt_check_topology ();if (!IS_ENABLED (CONFIG_SMP )) {pr_info ("CPU: " );
arch/x86/kvm/vmx/vmx.c +2 -1
@@ -26,6 +26,7 @@#include <linux/mod_devicetable.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/sched/smt.h> #include <linux/slab.h> #include <linux/tboot.h> #include <linux/trace_events.h>
@@ -6816,7 +6817,7 @@ static int vmx_vm_init(struct kvm *kvm)* Warn upon starting the first VM in a potentially * insecure environment. */ -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (sched_smt_active ())pr_warn_once (L1TF_MSG_SMT );if (l1tf_vmx_mitigation ==VMENTER_L1D_FLUSH_NEVER )pr_warn_once (L1TF_MSG_L1D );
include/linux/cpu.h +0 -2
@@ -180,12 +180,10 @@ enum cpuhp_smt_control {#if defined(CONFIG_SMP ) && defined(CONFIG_HOTPLUG_SMT )extern enum cpuhp_smt_control cpu_smt_control;extern void cpu_smt_disable (bool force); -extern void cpu_smt_check_topology_early (void );extern void cpu_smt_check_topology (void );#else # define cpu_smt_control (CPU_SMT_ENABLED)static inline void cpu_smt_disable (bool force) { } -static inline void cpu_smt_check_topology_early (void ) { }static inline void cpu_smt_check_topology (void ) { }#endif
kernel/cpu.c +4 -28
@@ -387,8 +387,6 @@ void __weak arch_smt_update(void) { }enum cpuhp_smt_control cpu_smt_control__read_mostly =CPU_SMT_ENABLED ;EXPORT_SYMBOL_GPL (cpu_smt_control); -static bool cpu_smt_available__read_mostly ; -void __init cpu_smt_disable (bool force) { if (cpu_smt_control == CPU_SMT_FORCE_DISABLED ||
@@ -406,25 +404,11 @@ void __init cpu_smt_disable(bool force)/* * The decision whether SMT is supported can only be done after the full -* CPU identification. Called from architecture code before non boot CPUs -* are brought up. -*/ -void __init cpu_smt_check_topology_early (void ) -{ -if (!topology_smt_supported ()) - cpu_smt_control =CPU_SMT_NOT_SUPPORTED ; -} - -/* -* If SMT was disabled by BIOS, detect it here, after the CPUs have been -* brought online. This ensures the smt/l1tf sysfs entries are consistent -* with reality. cpu_smt_available is set to true during the bringup of non -* boot CPUs when a SMT sibling is detected. Note, this may overwrite -* cpu_smt_control's previous setting. +* CPU identification. Called from architecture code. */ void __init cpu_smt_check_topology (void ) { -if (!cpu_smt_available) +if (!topology_smt_supported ()) cpu_smt_control =CPU_SMT_NOT_SUPPORTED ; }
@@ -437,18 +421,10 @@ early_param("nosmt", smt_cmdline_disable);static inline bool cpu_smt_allowed (unsigned int cpu) { -if (topology_is_primary_thread (cpu)) +if (cpu_smt_control ==CPU_SMT_ENABLED )return true; -/* -* If the CPU is not a 'primary' thread and the booted_once bit is -* set then the processor has SMT support. Store this information -* for the late check of SMT support in cpu_smt_check_topology(). -*/ -if (per_cpu (cpuhp_state, cpu).booted_once) - cpu_smt_available = true; - -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (topology_is_primary_thread (cpu))return true;/*
kernel/smp.c +0 -2
@@ -584,8 +584,6 @@ void __init smp_init(void) num_nodes, (num_nodes >1 ?"s" :"" ), num_cpus, (num_cpus > 1 ? "s" : "")); -/* Final decision about SMT support */ -cpu_smt_check_topology ();/* Any cleanup work */ smp_cpus_done (setup_max_cpus); }
Quoted 6 lines · On Fri, 25 Jan 2019 11:02:03 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 3 lines · Josh Poimboeuf
How about this patch? It's just a revert of
73d5e2b47264andbc2d8d262cba, plus the 1-linevmx_vm_init()change. If it looks ok to you, I can clean it up and submit an official version.This one actually compiles...
Looks good to me,
(one small question below)
[...]
Quoted 7 lines
static inline bool
cpu_smt_allowed(unsigned int cpu){kernel/cpu.c +1
-2-if (topology_is_primary_thread (cpu)) +if (cpu_smt_control ==CPU_SMT_ENABLED )return true; -/*
kernel/cpu.c early_param("nosmt", smt_cmdline_disable); · line 428
Quoted 10 lines
kernel/cpu.c +1
-8-* If the CPU is not a 'primary' thread and the booted_once bit is -* set then the processor has SMT support. Store this information -* for the late check of SMT support in cpu_smt_check_topology(). -*/ -if (per_cpu (cpuhp_state, cpu).booted_once) - cpu_smt_available = true; - -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (topology_is_primary_thread (cpu))return true;
why did you swap cpu_smt_control and topology_is_primary_thread checks?
Quoted 2 lines
/*
Quoted 21 lines · Igor Mammedov
Quoted 6 lines · On Fri, 25 Jan 2019 11:02:03 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 3 lines · Josh Poimboeuf
How about this patch? It's just a revert of
73d5e2b47264andbc2d8d262cba, plus the 1-linevmx_vm_init()change. If it looks ok to you, I can clean it up and submit an official version.This one actually compiles...
Looks good to me,
(one small question below)[...]
Quoted 6 lines
static inline bool
cpu_smt_allowed(unsigned int cpu){kernel/cpu.c +1
-1-if (topology_is_primary_thread (cpu)) +if (cpu_smt_control ==CPU_SMT_ENABLED )return true;
Quoted 12 lines
kernel/cpu.c early_param("nosmt", smt_cmdline_disable); · line 428
Quoted 11 lines
kernel/cpu.c +1
-9-/* -* If the CPU is not a 'primary' thread and the booted_once bit is -* set then the processor has SMT support. Store this information -* for the late check of SMT support in cpu_smt_check_topology(). -*/ -if (per_cpu (cpuhp_state, cpu).booted_once) - cpu_smt_available = true; - -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (topology_is_primary_thread (cpu))return true;why did you swap
cpu_smt_controlandtopology_is_primary_threadchecks?
That's just the revert of bc2d8d262cba5.
Signature · 1 line
Josh
Quoted 35 lines · On Mon, 28 Jan 2019 06:52:52 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 21 lines · Igor Mammedov
Quoted 6 lines · On Fri, 25 Jan 2019 11:02:03 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 3 lines · Josh Poimboeuf
How about this patch? It's just a revert of
73d5e2b47264andbc2d8d262cba, plus the 1-linevmx_vm_init()change. If it looks ok to you, I can clean it up and submit an official version.This one actually compiles...
Looks good to me,
(one small question below)[...]
Quoted 6 lines
static inline bool
cpu_smt_allowed(unsigned int cpu){kernel/cpu.c +1
-1-if (topology_is_primary_thread (cpu)) +if (cpu_smt_control ==CPU_SMT_ENABLED )return true;Quoted 12 lines
kernel/cpu.c early_param("nosmt", smt_cmdline_disable); · line 428
Quoted 11 lines
kernel/cpu.c +1
-9-/* -* If the CPU is not a 'primary' thread and the booted_once bit is -* set then the processor has SMT support. Store this information -* for the late check of SMT support in cpu_smt_check_topology(). -*/ -if (per_cpu (cpuhp_state, cpu).booted_once) - cpu_smt_available = true; - -if (cpu_smt_control ==CPU_SMT_ENABLED ) +if (topology_is_primary_thread (cpu))return true;why did you swap
cpu_smt_controlandtopology_is_primary_threadchecks?
Quoted 1 line
That's just the revert of
bc2d8d262cba5.
ok. waiting for formal patch
pls, CC [email protected] so Paolo could have a chance to review (pick it up if Thomas is ok with approach)