x86/paravirt: Disable virt spinlock on bare metal
Subsystems (3)
PATCH v3 1/1 x86/paravirt: Disable virt spinlock on bare metal
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a 'virt_spin_lock_key' static key to switch between the two modes.
The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commit ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning"). When CONFIG_PARAVIRT_SPINLOCKS is disabled the virt_spin_lock_key is incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.
Set the default value of virt_spin_lock_key to false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), the virt_spin_lock_key is disabled accordingly. The relation is described as below:
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_key N N Y N
---
v2._v3:
Change the default value of virt_spin_lock_key from true to false. Enable this key when it is in the VM, and disable it when needed. This makes the code more readable. (Nikolay Borisov)
Dropped Reviewed-by because the code has been changed.v1->v2:
Refine the commit log per Dave's suggestion.
Simplify the fix by directly disabling the virt_spin_lock_key on bare metal. Collect Reviewed-by from Juergen.
--- arch/x86/include/asm/qspinlock.h | 4 ++-- arch/x86/kernel/paravirt.c | 7 +++---- 2 files changed, 5 insertions(+), 6 deletions(-)
arch/x86/include/asm/qspinlock.h +2 -2
@@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)#ifdef CONFIG_PARAVIRT /* -* virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. +* virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack. * * Native (and PV wanting native due to vCPU pinning) should disable this key. * It is done in this backwards fashion to only have a single direction change, * which removes ordering between native_pv_spin_init() and HV setup. */ -DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);/* * Shortcut for the queued_spin_lock_slowpath() function that allows
arch/x86/kernel/paravirt.c +3 -4
@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key); +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
Signature · 1 line
2.25.1
Quoted 76 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commitce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled accordingly. The relation is described as below:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y NFixes ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")Suggested-by Dave Hansen <[email protected]>Suggested-by Qiuxu Zhuo <[email protected]>Suggested-by Nikolay Borisov <[email protected]>Reported-by Prem Nath Dey <[email protected]>Reported-by Xiaoping Zhou <[email protected]>Signed-off-by Chen Yu <[email protected]>---
v2._v3:
Change the default value ofvirt_spin_lock_keyfrom true to false. Enable this key when it is in the VM, and disable it when needed. This makes the code more readable. (Nikolay Borisov)
Dropped Reviewed-by because the code has been changed.v1->v2:
Refine the commit log per Dave's suggestion.
Simplify the fix by directly disabling thevirt_spin_lock_keyon bare metal. Collect Reviewed-by from Juergen.--- arch/x86/include/asm/qspinlock.h | 4 ++-- arch/x86/kernel/paravirt.c | 7 +++---- 2 files changed, 5 insertions(+), 6 deletions(-)arch/x86/include/asm/qspinlock.h +2
-2@@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)#ifdef CONFIG_PARAVIRT /* -* virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. +* virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack. * * Native (and PV wanting native due to vCPU pinning) should disable this key. * It is done in this backwards fashion to only have a single direction change, * which removes ordering between native_pv_spin_init() and HV setup. */ -DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);/* * Shortcut for the queued_spin_lock_slowpath() function that allows arch/x86/kernel/paravirt.c +0
-0
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 58
Quoted 9 lines
arch/x86/kernel/paravirt.c +1
-2@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { - if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) &&
Actually now shouldn't the CONFIG_PARAVIRT_SPINLOCKS check be retained? Otherwise we'll have the virtspinlock enabled even if we are a guest but CONFIG_PARAVIRT_SPINLOCKS is disabled, no ?
Quoted 7 lines
arch/x86/kernel/paravirt.c +2
-2- !boot_cpu_has(X86_FEATURE_HYPERVISOR)) -static_branch_disable (&virt_spin_lock_key); +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commitce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled accordingly. The relation is described as below:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y N
Quoted 10 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 58
Quoted 5 lines
arch/x86/kernel/paravirt.c +1
-2-DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { - if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) &&Actually now shouldn't the
CONFIG_PARAVIRT_SPINLOCKScheck be retained? Otherwise we'll have the virtspinlock enabled even if we are a guest butCONFIG_PARAVIRT_SPINLOCKSis disabled, no ?
It seems to be the expected behavior? If CONFIG_PARAVIRT_SPINLOCKS is disabled, should the virt_spin_lock_key be enabled in the guest?
The previous behavior before commit ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning"): kvm_spinlock_init() is NULL if CONFIG_PARAVIRT_SPINLOCKS is disabled, and static_branch_disable(&virt_spin_lock_key) can not be invoked, so the virt_spin_lock_key keeps enabled.
thanks,
Chenyu
Quoted 44 lines · Chen Yu
Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commitce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled accordingly. The relation is described as below:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y NQuoted 10 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 58
Quoted 5 lines
arch/x86/kernel/paravirt.c +1
-2-DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { - if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) &&Actually now shouldn't the
CONFIG_PARAVIRT_SPINLOCKScheck be retained? Otherwise we'll have the virtspinlock enabled even if we are a guest butCONFIG_PARAVIRT_SPINLOCKSis disabled, no ?
Quoted 2 lines
It seems to be the expected behavior? If
CONFIG_PARAVIRT_SPINLOCKSis disabled, should thevirt_spin_lock_keybe enabled in the guest?
No, but if it's disabled and we are under a hypervisor shouldn't the virt spinlock be kept disabled? As it stands now everytime we are under a hypervisor the virt spinlock is enabled irrespective of the PARAVIRT_SPINLOCK config state.
Quoted 8 lines
The previous behavior before commit
ce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"):kvm_spinlock_init()is NULL ifCONFIG_PARAVIRT_SPINLOCKSis disabled, andstatic_branch_disable(&virt_spin_lock_key)can not be invoked, so thevirt_spin_lock_keykeeps enabled.thanks,
Chenyu
Quoted 8 lines
From: Nikolay Borisov <[email protected]>
[...]Quoted 1 line
Quoted 1 line
Actually now shouldn't the
CONFIG_PARAVIRT_SPINLOCKScheck beretained?
Quoted 4 lines
Quoted 3 lines
Otherwise we'll have the virtspinlock enabled even if we are a guest but
CONFIG_PARAVIRT_SPINLOCKSis disabled, no ?
Quoted 5 lines
Quoted 2 lines
It seems to be the expected behavior? If
CONFIG_PARAVIRT_SPINLOCKSis disabled, should thevirt_spin_lock_keybe enabled in the guest?No, but if it's disabled and we are under a hypervisor shouldn't the virt spinlock be kept disabled?
No, the virt_spin_lock_key shouldn't be kept disabled.
According to the comments [1], in the hypervisor if CONFIG_PARAVIRT_SPINLOCKS is disabled, the virt_spin_lock_key should be enabled to fall back to the TAS spinlock.
[1] https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/qspinlock.h#L94
According to the comments [2]:
So my understanding is that in hypervisor keeping virt_spin_lock_key enabled allows the spinlock fallback to TAS if PV spinlock is not supported (either CONFIG_PARAVIRT_SPINLOCKS=n or the host doesn't support the PV feature)
[2] https://github.com/torvalds/linux/blob/master/arch/x86/kernel/kvm.c#L1073
Quoted 3 lines
As it stands now everytime we are under a
hypervisor the virt spinlock is enabled irrespective of thePARAVIRT_SPINLOCKconfig state.
According to [1] [2], yes, I think so,
-Qiuxu
Quoted 29 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commitce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled accordingly. The relation is described as below:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y N
Quoted 7 lines
Fixes ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")Suggested-by Dave Hansen <[email protected]>Suggested-by Qiuxu Zhuo <[email protected]>Suggested-by Nikolay Borisov <[email protected]>Reported-by Prem Nath Dey <[email protected]>Reported-by Xiaoping Zhou <[email protected]>Signed-off-by Chen Yu <[email protected]>
Quoted 29 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.The static key is always enabled by default (run in guest mode) and should be disabled for bare metal (and in some guests that want native behavior).
Performance drop is reported when running encode/decode workload and BenchSEE cache sub-workload.
Bisect points to commitce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is preferred(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled accordingly. The relation is described as below:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y N
Quoted 8 lines
Fixes ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")Suggested-by Dave Hansen <[email protected]>Suggested-by Qiuxu Zhuo <[email protected]>Suggested-by Nikolay Borisov <[email protected]>Reported-by Prem Nath Dey <[email protected]>Reported-by Xiaoping Zhou <[email protected]>Signed-off-by Chen Yu <[email protected]>
Hi,
A gentle ping of this patch. Or should I need to send a new version with updated Reviewed-by tag from Nikolay?
thanks,
Chenyu