x86/paravirt: Disable virt spinlock on bare metal
Subsystems (3)
PATCH v4 1/1 x86/paravirt: Disable virt spinlock on bare metal
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a 'virt_spin_lock_key' static key to switch between the two modes.
In current code, the static key is always enabled by default when running in guest mode. The key is disabled for bare metal (and in some guests that want native behavior).
Large performance regression is reported when running encode/decode workload and BenchSEE cache sub-workload on the bare metal. Bisect points to commit ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning"). When CONFIG_PARAVIRT_SPINLOCKS is disabled, the virt_spin_lock_key is incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.
Set the default value of virt_spin_lock_key to false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is detected(paravirt-spinlock eg), the virt_spin_lock_key is disabled. According to the description above, the final effect will be as followed:
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_key N N Y N
To summarize, the virt_spin_lock_key is disabled on the bare metal no matter what other condidtion is. And the virt_spin_lock_key is also disabled when other spinlock mechanism is detected in the VM guest.
---v3->v4:
Refine the commit log.
Added Reviewed-by tag from Nikolay.v2->v3:
Change the default value of virt_spin_lock_key from true to false. Enable this key when it is in the VM, and disable it when needed. This makes the code more readable. (Nikolay Borisov)
Dropped Reviewed-by because the code has been changed.v1->v2:
Refine the commit log per Dave's suggestion.
Simplify the fix by directly disabling the virt_spin_lock_key on bare metal. Collect Reviewed-by from Juergen.
--- arch/x86/include/asm/qspinlock.h | 4 ++-- arch/x86/kernel/paravirt.c | 7 +++---- 2 files changed, 5 insertions(+), 6 deletions(-)
arch/x86/include/asm/qspinlock.h +2 -2
@@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)#ifdef CONFIG_PARAVIRT /* -* virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. +* virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack. * * Native (and PV wanting native due to vCPU pinning) should disable this key. * It is done in this backwards fashion to only have a single direction change, * which removes ordering between native_pv_spin_init() and HV setup. */ -DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);/* * Shortcut for the queued_spin_lock_slowpath() function that allows
arch/x86/kernel/paravirt.c +3 -4
@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key); +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
Signature · 1 line
2.25.1
Quoted 8 lines · Chen Yu
#ifdef
CONFIG_PARAVIRT
/*
- *virt_spin_lock_key- enables (by default) thevirt_spin_lock()hijack.
+ *virt_spin_lock_key- disables (by default) thevirt_spin_lock()hijack. *
* Native (and PV wanting native due to vCPU pinning) should disable this key.
* It is done in this backwards fashion to only have a single direction change,
* which removes ordering betweennative_pv_spin_init()and HV setup.
This comment is bogus now.
Hi Thomas,
Quoted 10 lines · Thomas Gleixner
Quoted 8 lines · Chen Yu
#ifdef
CONFIG_PARAVIRT
/*
- *virt_spin_lock_key- enables (by default) thevirt_spin_lock()hijack.
+ *virt_spin_lock_key- disables (by default) thevirt_spin_lock()hijack. *
* Native (and PV wanting native due to vCPU pinning) should disable this key.
* It is done in this backwards fashion to only have a single direction change,
* which removes ordering betweennative_pv_spin_init()and HV setup.
Quoted 2 lines
This comment is bogus now.
Yes, now it could be two direction: enabled first and disabled later. Let me refine the comments.
thanks,
Chenyu
Chenyu,
Sorry to bother you, I am porting pv spinlock to LoongArch platform, I do not know the history about function virt_spin_lock().
When CONFIG_PARAVIRT_SPINLOCKS is enabled, there is pv_enabled() before virt_spin_lock(), it seems that virt_spin_lock is never called in this condition.
if (pv_enabled())
goto pv_queue;
if (virt_spin_lock(lock))
return;When CONFIG_PARAVIRT_SPINLOCKS is disabled, there is no pv qspinlock compiled, so what is the usage about function virt_spin_lock()? Is it to switch spinlock method from test-set spinlock to qspinlock? why is there such requirement for spinlock switching method?
Regards
Bibo Mao
Quoted 102 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.In current code, the static key is always enabled by default when running in guest mode. The key is disabled for bare metal (and in some guests that want native behavior).
Large performance regression is reported when running encode/decode workload and BenchSEE cache sub-workload on the bare metal. Bisect points to commit
ce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled, thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is detected(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled. According to the description above, the final effect will be as followed:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y NTo summarize, the
virt_spin_lock_keyis disabled on the bare metal no matter what other condidtion is. And thevirt_spin_lock_keyis also disabled when other spinlock mechanism is detected in the VM guest.Fixes ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")Suggested-by Dave Hansen <[email protected]>Suggested-by Qiuxu Zhuo <[email protected]>Suggested-by Nikolay Borisov <[email protected]>Reported-by Prem Nath Dey <[email protected]>Reported-by Xiaoping Zhou <[email protected]>Reviewed-by Nikolay Borisov <[email protected]>Signed-off-by Chen Yu <[email protected]>---
v3->v4:
Refine the commit log.
Added Reviewed-by tag from Nikolay.v2->v3:
Change the default value ofvirt_spin_lock_keyfrom true to false. Enable this key when it is in the VM, and disable it when needed. This makes the code more readable. (Nikolay Borisov)
Dropped Reviewed-by because the code has been changed.v1->v2:
Refine the commit log per Dave's suggestion.
Simplify the fix by directly disabling thevirt_spin_lock_keyon bare metal. Collect Reviewed-by from Juergen.--- arch/x86/include/asm/qspinlock.h | 4 ++-- arch/x86/kernel/paravirt.c | 7 +++---- 2 files changed, 5 insertions(+), 6 deletions(-)arch/x86/include/asm/qspinlock.h +2
-2@@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)#ifdef CONFIG_PARAVIRT /* -* virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. +* virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack. * * Native (and PV wanting native due to vCPU pinning) should disable this key. * It is done in this backwards fashion to only have a single direction change, * which removes ordering between native_pv_spin_init() and HV setup. */ -DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);/* * Shortcut for the queued_spin_lock_slowpath() function that allows arch/x86/kernel/paravirt.c +3
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key); +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
Hi Bibo,
Quoted 8 lines · maobibo
Chenyu,
Sorry to bother you, I am porting pv spinlock to LoongArch platform, I do not know the history about function
virt_spin_lock().When
CONFIG_PARAVIRT_SPINLOCKSis enabled, there ispv_enabled()beforevirt_spin_lock(), it seems thatvirt_spin_lockis never called in this condition.
Right, if CONFIG_PARAVIRT_SPINLOCKS is enabled, virt_spin_lock() will not be checked at all and go directly to pv_queue lock section.
Quoted 10 lines
if (pv_enabled()) goto pv_queue; if (virt_spin_lock(lock)) return;When
CONFIG_PARAVIRT_SPINLOCKSis disabled, there is no pv qspinlock compiled, so what is the usage about functionvirt_spin_lock()? Is it to switch spinlock method from test-set spinlock to qspinlock? why is there such requirement for spinlock switching method?
Firstly, according to the commit 2aa79af64263 ("locking/qspinlock: Revert to test-and-set on hypervisors"), virt_spin_lock() was originally introduced to avoid the over-preemption of native qspinlock(when paravirt-spinlock is disabled).
Secondly, there is user requirement to use native qspinlock over test-set lock in the VM. For example, nopvspin parameter is parsed in xen_init_spinlocks(). commit 9043442b43b1 ("locking/paravirt: Use new static key for controlling call of virt_spin_lock()") was introduced to adjust the key from user boot up commandline. so the key switches between test-set and the native qspinlock. I've Cced Juergen and Peter in case I misunderstood.
thanks,
Chenyu
Quoted 26 lines · Chen Yu
Hi Bibo,
Quoted 8 lines · maobibo
Chenyu,
Sorry to bother you, I am porting pv spinlock to LoongArch platform, I do not know the history about function
virt_spin_lock().When
CONFIG_PARAVIRT_SPINLOCKSis enabled, there ispv_enabled()beforevirt_spin_lock(), it seems thatvirt_spin_lockis never called in this condition.Right, if
CONFIG_PARAVIRT_SPINLOCKSis enabled,virt_spin_lock()will not be checked at all and go directly topv_queuelock section.Quoted 10 lines
if (pv_enabled()) goto pv_queue; if (virt_spin_lock(lock)) return;When
CONFIG_PARAVIRT_SPINLOCKSis disabled, there is no pv qspinlock compiled, so what is the usage about functionvirt_spin_lock()? Is it to switch spinlock method from test-set spinlock to qspinlock? why is there such requirement for spinlock switching method?
Quoted 4 lines
Firstly, according to the commit
2aa79af64263("locking/qspinlock: Revert to test-and-set on hypervisors"),virt_spin_lock()was originally introduced to avoid the over-preemption of native qspinlock(when paravirt-spinlock is disabled).
Chenyu,
Thanks for the information. It seems that test-and-set spinlock is suggested rather than qspinlock on VM when paravirt-spinlock is disabled.
Regards
Bibo Mao
Quoted 11 lines
Secondly, there is user requirement to use native qspinlock over test-set lock in the VM. For example, nopvspin parameter is parsed in
xen_init_spinlocks(). commit9043442b43b1("locking/paravirt: Use new static key for controlling call ofvirt_spin_lock()") was introduced to adjust the key from user boot up commandline. so the key switches between test-set and the native qspinlock. I've Cced Juergen and Peter in case I misunderstood.thanks,
Chenyu
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 67 lines · Chen Yu
The kernel can change spinlock behavior when running as a guest. But this guest-friendly behavior causes performance problems on bare metal. So there's a '
virt_spin_lock_key' static key to switch between the two modes.In current code, the static key is always enabled by default when running in guest mode. The key is disabled for bare metal (and in some guests that want native behavior).
Large performance regression is reported when running encode/decode workload and BenchSEE cache sub-workload on the bare metal. Bisect points to commit
ce0a1b608bfc("x86/paravirt: Silence unusednative_pv_lock_init()function warning"). WhenCONFIG_PARAVIRT_SPINLOCKSis disabled, thevirt_spin_lock_keyis incorrectly set to true on bare metal. The qspinlock degenerates to test-and-set spinlock, which decrease the performance on bare metal.Set the default value of
virt_spin_lock_keyto false. If booting in a VM, enable this key. Later during the VM initialization, if other high-efficient spinlock is detected(paravirt-spinlock eg), thevirt_spin_lock_keyis disabled. According to the description above, the final effect will be as followed:X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y NTo summarize, the
virt_spin_lock_keyis disabled on the bare metal no matter what other condidtion is. And thevirt_spin_lock_keyis also disabled when other spinlock mechanism is detected in the VM guest.Fixes ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")Suggested-by Dave Hansen <[email protected]>Suggested-by Qiuxu Zhuo <[email protected]>Suggested-by Nikolay Borisov <[email protected]>Reported-by Prem Nath Dey <[email protected]>Reported-by Xiaoping Zhou <[email protected]>Reviewed-by Nikolay Borisov <[email protected]>Signed-off-by Chen Yu <[email protected]>---
v3->v4:
Refine the commit log.
Added Reviewed-by tag from Nikolay.v2->v3:
Change the default value ofvirt_spin_lock_keyfrom true to false. Enable this key when it is in the VM, and disable it when needed. This makes the code more readable. (Nikolay Borisov)
Dropped Reviewed-by because the code has been changed.v1->v2:
Refine the commit log per Dave's suggestion.
Simplify the fix by directly disabling thevirt_spin_lock_keyon bare metal. Collect Reviewed-by from Juergen.--- arch/x86/include/asm/qspinlock.h | 4 ++-- arch/x86/kernel/paravirt.c | 7 +++---- 2 files changed, 5 insertions(+), 6 deletions(-)arch/x86/include/asm/qspinlock.h +0
-0@@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)#ifdef CONFIG_PARAVIRT
arch/x86/include/asm/qspinlock.h static inline bool vcpu_is_preempted(long cpu) · line 75
Quoted 10 lines
arch/x86/include/asm/qspinlock.h +2
-2/* -* virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack. +* virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack. * * Native (and PV wanting native due to vCPU pinning) should disable this key. * It is done in this backwards fashion to only have a single direction change, * which removes ordering between native_pv_spin_init() and HV setup. */ -DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);
@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock)
{
int val;- if (!static_branch_likely(&virt_spin_lock_key))
+ if (!static_branch_unlikely(&virt_spin_lock_key))
return false;
Do we need change it with static_branch_unlikely() if value of key is false by fault?
Quoted 12 lines
arch/x86/include/asm/qspinlock.h +0
-0/* * Shortcut for the queued_spin_lock_slowpath() function that allows arch/x86/kernel/paravirt.c +0
-1@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key);
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 10 lines
arch/x86/kernel/paravirt.c +3
-3+DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key); +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }
From my point, the sentence static_branch_disable(&virt_spin_lock_key) can be removed in file arch/x86/xen/spinlock.c and arch/x86/kernel/kvm.c, since function native_smp_prepare_boot_cpu() is already called by xen_smp_prepare_boot_cpu() and kvm_smp_prepare_boot_cpu().
Regards
Bibo Mao
Quoted 3 lines
arch/x86/kernel/paravirt.c +0
-0static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
Hi Bibo,
Quoted 23 lines · maobibo
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 8 lines · Chen Yu
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y Narch/x86/include/asm/qspinlock.h +1
-1-DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);(quoted diff) +1
-1@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) { int val; - if (!static_branch_likely(&virt_spin_lock_key)) + if (!static_branch_unlikely(&virt_spin_lock_key)) return false;
Quoted 2 lines
Do we need change it with
static_branch_unlikely()if value of key is false by fault?
My understanding is that, firstly, whether it is likely() or unlikely() depends on the 'expected' value of the key, rather than its default initialized value. The compiler can arrange the if 'jump' condition to avoid the overhead of branch jump(to keep the instruction pipeline) as much as possible. Secondly, before this patch, the 'expected' value of virt_spin_lock_key is 'true', so I'm not sure if we should change its behavior. Although it seems that in most VM guest, with para-virt spinlock enabled, this key should be false at most time, but just in case of any regression...
Quoted 16 lines
Quoted 16 lines
/*
* Shortcut for thequeued_spin_lock_slowpath()function that allowsarch/x86/kernel/paravirt.c +1
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key);
Quoted 9 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 3 lines
arch/x86/kernel/paravirt.c +2
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }From my point, the sentence
static_branch_disable(&virt_spin_lock_key)can be removed in filearch/x86/xen/spinlock.candarch/x86/kernel/kvm.c, since functionnative_smp_prepare_boot_cpu()is already called byxen_smp_prepare_boot_cpu()andkvm_smp_prepare_boot_cpu().
The key is enabled by native_smp_prepare_boot_cpu() for VM guest as the initial value(default to true). And later either arch/x86/xen/spinlock.c or arch/x86/kernel/kvm.c disable this key in a on-demand manner.
thanks,
Chenyu
Hi Chenyu,
Quoted 26 lines · Chen Yu
Hi Bibo,
Quoted 23 lines · maobibo
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 8 lines · Chen Yu
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y Narch/x86/include/asm/qspinlock.h +1
-1-DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);(quoted diff) +1
-1@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) { int val; - if (!static_branch_likely(&virt_spin_lock_key)) + if (!static_branch_unlikely(&virt_spin_lock_key)) return false;
Quoted 12 lines
Quoted 2 lines
Do we need change it with
static_branch_unlikely()if value of key is false by fault?My understanding is that, firstly, whether it is
likely()orunlikely()depends on the 'expected' value of the key, rather than its default initialized value. The compiler can arrange the if 'jump' condition to avoid the overhead of branch jump(to keep the instruction pipeline) as much as possible. Secondly, before this patch, the 'expected' value ofvirt_spin_lock_keyis 'true', so I'm not sure if we should change its behavior. Although it seems that in most VM guest, with para-virt spinlock enabled, this key should be false at most time, but just in case of any regression...
yes, it does not inflect the result, it is a trivial thing and depends on personal like or dislike.
Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
/*
* Shortcut for thequeued_spin_lock_slowpath()function that allowsarch/x86/kernel/paravirt.c +1
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key);Quoted 9 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 3 lines
arch/x86/kernel/paravirt.c +2
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }From my point, the sentence
static_branch_disable(&virt_spin_lock_key)can
be removed in filearch/x86/xen/spinlock.candarch/x86/kernel/kvm.c, since functionnative_smp_prepare_boot_cpu()is already called byxen_smp_prepare_boot_cpu()andkvm_smp_prepare_boot_cpu().
Quoted 3 lines
The key is enabled by
native_smp_prepare_boot_cpu()for VM guest as the initial value(default to true). And later eitherarch/x86/xen/spinlock.corarch/x86/kernel/kvm.cdisable this key in a on-demand manner.
I understand that you only care about host machine and do not want to change behavior of VM. Only that from the view of VM, there are two conditions such as:
1. If option CONFIG_PARAVIRT_SPINLOCKS is disabled, virt_spin_lock_key is disabled with your patch. VM will use test-set spinlock rather than qspinlock to avoid the over-preemption of native qspinlock, just the same with commit 2aa79af64263. And it is the same for all the hypervisor types.
2. If option CONFIG_PARAVIRT_SPINLOCKS is enable, pv spinlock cannot be used because some reasons, such as host hypervisor has no KVM_FEATURE_PV_UNHALT feature or nopvspin kernel parameter is added. The behavior to use test-set spinlock or native qspinlock is different on different hypervisor types.
Even on KVM hypervisor, if KVM_FEATURE_PV_UNHALT is not supported, test-set spinlock will be used on VM; For nopvspin kernel parameter, native spinlock is used on VM. What is the rule for this? :)
Regards
Bibo Mao
Quoted 4 lines
thanks,
Chenyu
Quoted 83 lines · maobibo
Hi Chenyu,
Quoted 26 lines · Chen Yu
Hi Bibo,
Quoted 23 lines · maobibo
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 8 lines · Chen Yu
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y Narch/x86/include/asm/qspinlock.h +1
-1-DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);(quoted diff) +1
-1@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) { int val; - if (!static_branch_likely(&virt_spin_lock_key)) + if (!static_branch_unlikely(&virt_spin_lock_key)) return false;Quoted 12 lines
Quoted 2 lines
Do we need change it with
static_branch_unlikely()if value of key is false by fault?My understanding is that, firstly, whether it is
likely()orunlikely()depends on the 'expected' value of the key, rather than its default initialized value. The compiler can arrange the if 'jump' condition to avoid the overhead of branch jump(to keep the instruction pipeline) as much as possible. Secondly, before this patch, the 'expected' value ofvirt_spin_lock_keyis 'true', so I'm not sure if we should change its behavior. Although it seems that in most VM guest, with para-virt spinlock enabled, this key should be false at most time, but just in case of any regression...yes, it does not inflect the result, it is a trivial thing and depends on personal like or dislike.
Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
/*
* Shortcut for thequeued_spin_lock_slowpath()function that allowsarch/x86/kernel/paravirt.c +1
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key);Quoted 9 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 3 lines
arch/x86/kernel/paravirt.c +2
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }From my point, the sentence
static_branch_disable(&virt_spin_lock_key)can
be removed in filearch/x86/xen/spinlock.candarch/x86/kernel/kvm.c, since functionnative_smp_prepare_boot_cpu()is already called byxen_smp_prepare_boot_cpu()andkvm_smp_prepare_boot_cpu().Quoted 3 lines
The key is enabled by
native_smp_prepare_boot_cpu()for VM guest as the initial value(default to true). And later eitherarch/x86/xen/spinlock.corarch/x86/kernel/kvm.cdisable this key in a on-demand manner.I understand that you only care about host machine and do not want to change behavior of VM. Only that from the view of VM, there are two conditions such as:
1. If option
CONFIG_PARAVIRT_SPINLOCKSis disabled,virt_spin_lock_keyis disabled with your patch. VM will use test-set spinlock rather than qspinlock to avoid the over-preemption of native qspinlock, just the same with commit2aa79af64263. And it is the same for all the hypervisor types.
Quoted 9 lines
2. If option
CONFIG_PARAVIRT_SPINLOCKSis enable, pv spinlock cannot be used because some reasons, such as host hypervisor has noKVM_FEATURE_PV_UNHALTfeature or nopvspin kernel parameter is added. The behavior to use test-set spinlock or native qspinlock is different on different hypervisor types.Even on KVM hypervisor, if
KVM_FEATURE_PV_UNHALTis not supported, test-set spinlock will be used on VM; For nopvspin kernel parameter, native spinlock is used on VM. What is the rule for this? :)
If CONFIG_PARAVIRT_SPINLOCKS is enabled, the test-set spinlock has nothing to do with the lock path, because if pv_enabled() is true, it will skip the test-set spinlock and go to pv_queue section. If for some reason the pv spinlock can not be used because KVM_FEATURE_PV_UNHALT is not supported, it will fall into the default qpinlock without pv-qspinlock(no pv_wait hook because it is NULL).
thanks,
Chenyu
Hi Chenyu,
Quoted 89 lines · Chen Yu
Quoted 88 lines · maobibo
Hi Chenyu,
Quoted 26 lines · Chen Yu
Hi Bibo,
Quoted 23 lines · maobibo
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 8 lines · Chen Yu
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y Narch/x86/include/asm/qspinlock.h +1
-1-DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);(quoted diff) +1
-1@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) { int val; - if (!static_branch_likely(&virt_spin_lock_key)) + if (!static_branch_unlikely(&virt_spin_lock_key)) return false;Quoted 12 lines
Quoted 2 lines
Do we need change it with
static_branch_unlikely()if value of key is false by fault?My understanding is that, firstly, whether it is
likely()orunlikely()depends on the 'expected' value of the key, rather than its default initialized value. The compiler can arrange the if 'jump' condition to avoid the overhead of branch jump(to keep the instruction pipeline) as much as possible. Secondly, before this patch, the 'expected' value ofvirt_spin_lock_keyis 'true', so I'm not sure if we should change its behavior. Although it seems that in most VM guest, with para-virt spinlock enabled, this key should be false at most time, but just in case of any regression...yes, it does not inflect the result, it is a trivial thing and depends on personal like or dislike.
Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
/*
* Shortcut for thequeued_spin_lock_slowpath()function that allowsarch/x86/kernel/paravirt.c +1
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key);Quoted 9 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 3 lines
arch/x86/kernel/paravirt.c +2
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }From my point, the sentence
static_branch_disable(&virt_spin_lock_key)can
be removed in filearch/x86/xen/spinlock.candarch/x86/kernel/kvm.c, since functionnative_smp_prepare_boot_cpu()is already called byxen_smp_prepare_boot_cpu()andkvm_smp_prepare_boot_cpu().Quoted 3 lines
The key is enabled by
native_smp_prepare_boot_cpu()for VM guest as the initial value(default to true). And later eitherarch/x86/xen/spinlock.corarch/x86/kernel/kvm.cdisable this key in a on-demand manner.I understand that you only care about host machine and do not want to change behavior of VM. Only that from the view of VM, there are two conditions such as:
1. If option
CONFIG_PARAVIRT_SPINLOCKSis disabled,virt_spin_lock_keyis disabled with your patch. VM will use test-set spinlock rather than qspinlock to avoid the over-preemption of native qspinlock, just the same with commit2aa79af64263. And it is the same for all the hypervisor types.2. If option
CONFIG_PARAVIRT_SPINLOCKSis enable, pv spinlock cannot be used because some reasons, such as host hypervisor has noKVM_FEATURE_PV_UNHALTfeature or nopvspin kernel parameter is added. The behavior to use test-set spinlock or native qspinlock is different on different hypervisor types.
Quoted 7 lines
Quoted 4 lines
Even on KVM hypervisor, if
KVM_FEATURE_PV_UNHALTis not supported, test-set spinlock will be used on VM; For nopvspin kernel parameter, native spinlock is used on VM. What is the rule for this? :)If
CONFIG_PARAVIRT_SPINLOCKSis enabled, the test-set spinlock has nothing to do with the lock path, because ifpv_enabled()is true, it will skip the
IIRC, if CONFIG_PARAVIRT_SPINLOCKS is enabled, there is two qspinlock path: native_queued_spin_lock_slowpath() and __pv_queued_spin_lock_slowpath(). pv_enabled is false for native qspinlock path native_queued_spin_lock_slowpath(), test-set spinlock can be used in function native_queued_spin_lock_slowpath(). pv_enabled() is true only for function __pv_queued_spin_lock_slowpath().
Quoted 3 lines
test-set spinlock and go to
pv_queuesection. If for some reason the pv spinlock can not be used becauseKVM_FEATURE_PV_UNHALTis not supported, it will fall into the default qpinlock without pv-qspinlock(nopv_waithook because it is NULL).
yes, if pv spinlock cannot be used, native_queued_spin_lock_slowpath() will be called for spin_lock_slowpath, then there will be native qspinlock and test-test spinlock.
Regards
Bibo Mao
Quoted 4 lines
thanks,
Chenyu
Quoted 97 lines · maobibo
Hi Chenyu,
Quoted 94 lines · Chen Yu
Quoted 92 lines · maobibo
Hi Chenyu,
Quoted 26 lines · Chen Yu
Hi Bibo,
Quoted 23 lines · maobibo
Chenyu,
I do not know much about x86, just give some comments(probably incorrected) from the code.
Quoted 8 lines · Chen Yu
X86_FEATURE_HYPERVISOR Y Y Y N CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N PV spinlock Y N N Y/N
virt_spin_lock_keyN N Y Narch/x86/include/asm/qspinlock.h +1
-1-DECLARE_STATIC_KEY_TRUE (virt_spin_lock_key); +DECLARE_STATIC_KEY_FALSE (virt_spin_lock_key);(quoted diff) +1
-1@@ -87,7 +87,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) { int val; - if (!static_branch_likely(&virt_spin_lock_key)) + if (!static_branch_unlikely(&virt_spin_lock_key)) return false;Quoted 12 lines
Quoted 2 lines
Do we need change it with
static_branch_unlikely()if value of key is false by fault?My understanding is that, firstly, whether it is
likely()orunlikely()depends on the 'expected' value of the key, rather than its default initialized value. The compiler can arrange the if 'jump' condition to avoid the overhead of branch jump(to keep the instruction pipeline) as much as possible. Secondly, before this patch, the 'expected' value ofvirt_spin_lock_keyis 'true', so I'm not sure if we should change its behavior. Although it seems that in most VM guest, with para-virt spinlock enabled, this key should be false at most time, but just in case of any regression...yes, it does not inflect the result, it is a trivial thing and depends on personal like or dislike.
Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
/*
* Shortcut for thequeued_spin_lock_slowpath()function that allowsarch/x86/kernel/paravirt.c +1
-4@@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);#endif -DEFINE_STATIC_KEY_TRUE (virt_spin_lock_key); +DEFINE_STATIC_KEY_FALSE (virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) && - !boot_cpu_has (X86_FEATURE_HYPERVISOR )) -static_branch_disable (&virt_spin_lock_key);Quoted 9 lines
arch/x86/kernel/paravirt.c DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text); · line 60
Quoted 3 lines
arch/x86/kernel/paravirt.c +2
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +static_branch_enable (&virt_spin_lock_key); }From my point, the sentence
static_branch_disable(&virt_spin_lock_key)can
be removed in filearch/x86/xen/spinlock.candarch/x86/kernel/kvm.c, since functionnative_smp_prepare_boot_cpu()is already called byxen_smp_prepare_boot_cpu()andkvm_smp_prepare_boot_cpu().Quoted 3 lines
The key is enabled by
native_smp_prepare_boot_cpu()for VM guest as the initial value(default to true). And later eitherarch/x86/xen/spinlock.corarch/x86/kernel/kvm.cdisable this key in a on-demand manner.I understand that you only care about host machine and do not want to change behavior of VM. Only that from the view of VM, there are two conditions such as:
1. If option
CONFIG_PARAVIRT_SPINLOCKSis disabled,virt_spin_lock_keyis disabled with your patch. VM will use test-set spinlock rather than qspinlock to avoid the over-preemption of native qspinlock, just the same with commit2aa79af64263. And it is the same for all the hypervisor types.2. If option
CONFIG_PARAVIRT_SPINLOCKSis enable, pv spinlock cannot be used because some reasons, such as host hypervisor has noKVM_FEATURE_PV_UNHALTfeature or nopvspin kernel parameter is added. The behavior to use test-set spinlock or native qspinlock is different on different hypervisor types.Even on KVM hypervisor, if
KVM_FEATURE_PV_UNHALTis not supported, test-set spinlock will be used on VM; For nopvspin kernel parameter, native spinlock is used on VM. What is the rule for this? :)
Quoted 9 lines
Quoted 2 lines
If
CONFIG_PARAVIRT_SPINLOCKSis enabled, the test-set spinlock has nothing to do with the lock path, because ifpv_enabled()is true, it will skip theIIRC, if
CONFIG_PARAVIRT_SPINLOCKSis enabled, there is two qspinlock path:native_queued_spin_lock_slowpath()and__pv_queued_spin_lock_slowpath().pv_enabledis false for native qspinlock pathnative_queued_spin_lock_slowpath(), test-set spinlock can be used in functionnative_queued_spin_lock_slowpath().pv_enabled()is true only for function__pv_queued_spin_lock_slowpath().
Thanks for this explaination in detail!
Quoted 7 lines
Quoted 3 lines
test-set spinlock and go to
pv_queuesection. If for some reason the pv spinlock can not be used becauseKVM_FEATURE_PV_UNHALTis not supported, it will fall into the default qpinlock without pv-qspinlock(nopv_waithook because it is NULL).yes, if pv spinlock cannot be used,
native_queued_spin_lock_slowpath()will be called forspin_lock_slowpath, then there will be native qspinlock and test-test spinlock.
If I understand correctly, your concern about current logic is that, when CONFIG_PARAVIRT_SPINLOCKS is set but unfortunately disabled at runtime, there is inconsistence between using native qspinlock and test-set. My guess is that, nopvspin is for user who wants non-paravirt and native qspin, no matter it is xen or the kvm, all the other cases prefer test-set lock, no?
thanks,
Chenyu