locking/qspinlock/x86: Avoid test-and-set when PV_DEDICATED is set
Subsystems (5)
PATCH v2 1/1 locking/qspinlock/x86: Avoid test-and-set when PV_DEDICATED is set
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the PV_UNHALT flag.
This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the PV_DEDICATED KVM feature flag. When the PV_DEDICATED flag is not set, the code will still fall back to test-and-set, but when the PV_DEDICATED flag is set, the code will use the regular queue spinlock implementation.
---
V2:
- rebase on top of tip/master
Documentation/virtual/kvm/cpuid.txt | 6 ++++++ arch/x86/include/asm/qspinlock.h | 4 ++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + 3 files changed, 11 insertions(+)
Documentation/virtual/kvm/cpuid.txt +6 -0
@@ -54,6 +54,12 @@ KVM_FEATURE_PV_UNHALT || 7 || guest checks this feature bit || || before enabling paravirtualized || || spinlock support. ------------------------------------------------------------------------------ +KVM_FEATURE_PV_DEDICATED || 8 || guest checks this feature bit + || || to determine if they run on + || || dedicated vCPUs, allowing opti- + || || mizations such as usage of + || || qspinlocks. +------------------------------------------------------------------------------ KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side || || per-cpu warps are expected in || || kvmclock.
arch/x86/include/asm/qspinlock.h +4 -0
@@ -2,6 +2,8 @@#define _ASM_X86_QSPINLOCK_H#include <linux/jump_label.h> +#include <linux/kvm_para.h> +#include <asm/cpufeature.h> #include <asm-generic/qspinlock_types.h> #include <asm/paravirt.h>
@@ -57,6 +59,8 @@ static inline bool virt_spin_lock(struct qspinlock *lock)if (!static_branch_likely (&virt_spin_lock_key))return false; +if (kvm_para_has_feature (KVM_FEATURE_PV_DEDICATED )) +return false;/* * On hypervisors without PARAVIRT_SPINLOCKS support we fall * back to a Test-and-Set spinlock, because fair locks have
arch/x86/include/uapi/asm/kvm_para.h +1 -0
@@ -24,6 +24,7 @@#define KVM_FEATURE_STEAL_TIME 5#define KVM_FEATURE_PV_EOI 6#define KVM_FEATURE_PV_UNHALT 7 +#define KVM_FEATURE_PV_DEDICATED 8/* The last 8 bits are used to indicate how to interpret the flags field * in pvclock structure. If no bits are set, all flags are ignored.
Signature · 1 line
2.7.4
Paolo Bonzini
maintainerQuoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
Thanks,
Paolo
Quoted 77 lines
Cc Paolo Bonzini <[email protected]>Cc "Radim Krčmář" <[email protected]>Cc Jonathan Corbet <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Waiman Long <[email protected]>Cc Jan H. Schoenherr <[email protected]>Cc Anthony Liguori <[email protected]>Suggested-by Matt Wilson <[email protected]>Signed-off-by Eduardo Valentin <[email protected]>---
V2:
- rebase on top of tip/masterDocumentation/virtual/kvm/cpuid.txt | 6 ++++++ arch/x86/include/asm/qspinlock.h | 4 ++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + 3 files changed, 11 insertions(+)Documentation/virtual/kvm/cpuid.txt +6
-0@@ -54,6 +54,12 @@ KVM_FEATURE_PV_UNHALT || 7 || guest checks this feature bit || || before enabling paravirtualized || || spinlock support. ------------------------------------------------------------------------------ +KVM_FEATURE_PV_DEDICATED || 8 || guest checks this feature bit + || || to determine if they run on + || || dedicated vCPUs, allowing opti- + || || mizations such as usage of + || || qspinlocks. +------------------------------------------------------------------------------ KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side || || per-cpu warps are expected in || || kvmclock.arch/x86/include/asm/qspinlock.h +4
-0@@ -2,6 +2,8 @@#define _ASM_X86_QSPINLOCK_H#include <linux/jump_label.h> +#include <linux/kvm_para.h> +#include <asm/cpufeature.h> #include <asm-generic/qspinlock_types.h> #include <asm/paravirt.h> @@ -57,6 +59,8 @@ static inline bool virt_spin_lock(struct qspinlock *lock)if (!static_branch_likely (&virt_spin_lock_key))return false; +if (kvm_para_has_feature (KVM_FEATURE_PV_DEDICATED )) +return false;/* * On hypervisors without PARAVIRT_SPINLOCKS support we fall * back to a Test-and-Set spinlock, because fair locks have arch/x86/include/uapi/asm/kvm_para.h +1
-0@@ -24,6 +24,7 @@#define KVM_FEATURE_STEAL_TIME 5#define KVM_FEATURE_PV_EOI 6#define KVM_FEATURE_PV_UNHALT 7 +#define KVM_FEATURE_PV_DEDICATED 8/* The last 8 bits are used to indicate how to interpret the flags field * in pvclock structure. If no bits are set, all flags are ignored.
Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.
Quoted 3 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
Quoted 4 lines
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
The way I think of this is this is a flag set by host side so the guest adapts accordingly.
If the admin in guest side wants to ignore what the host is flagging, that is a different story.
Quoted 83 lines
Thanks,
Paolo
Quoted 77 lines
Cc Paolo Bonzini <[email protected]>Cc "Radim Krčmář" <[email protected]>Cc Jonathan Corbet <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Waiman Long <[email protected]>Cc Jan H. Schoenherr <[email protected]>Cc Anthony Liguori <[email protected]>Suggested-by Matt Wilson <[email protected]>Signed-off-by Eduardo Valentin <[email protected]>---
V2:
- rebase on top of tip/masterDocumentation/virtual/kvm/cpuid.txt | 6 ++++++ arch/x86/include/asm/qspinlock.h | 4 ++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + 3 files changed, 11 insertions(+)Documentation/virtual/kvm/cpuid.txt +6
-0@@ -54,6 +54,12 @@ KVM_FEATURE_PV_UNHALT || 7 || guest checks this feature bit || || before enabling paravirtualized || || spinlock support. ------------------------------------------------------------------------------ +KVM_FEATURE_PV_DEDICATED || 8 || guest checks this feature bit + || || to determine if they run on + || || dedicated vCPUs, allowing opti- + || || mizations such as usage of + || || qspinlocks. +------------------------------------------------------------------------------ KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side || || per-cpu warps are expected in || || kvmclock.arch/x86/include/asm/qspinlock.h +4
-0@@ -2,6 +2,8 @@#define _ASM_X86_QSPINLOCK_H#include <linux/jump_label.h> +#include <linux/kvm_para.h> +#include <asm/cpufeature.h> #include <asm-generic/qspinlock_types.h> #include <asm/paravirt.h> @@ -57,6 +59,8 @@ static inline bool virt_spin_lock(struct qspinlock *lock)if (!static_branch_likely (&virt_spin_lock_key))return false; +if (kvm_para_has_feature (KVM_FEATURE_PV_DEDICATED )) +return false;/* * On hypervisors without PARAVIRT_SPINLOCKS support we fall * back to a Test-and-Set spinlock, because fair locks have arch/x86/include/uapi/asm/kvm_para.h +1
-0@@ -24,6 +24,7 @@#define KVM_FEATURE_STEAL_TIME 5#define KVM_FEATURE_PV_EOI 6#define KVM_FEATURE_PV_UNHALT 7 +#define KVM_FEATURE_PV_DEDICATED 8/* The last 8 bits are used to indicate how to interpret the flags field * in pvclock structure. If no bits are set, all flags are ignored.
Signature · 2 lines
All the best, Eduardo Valentin
Quoted 11 lines · Eduardo Valentin
Quoted 10 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.
Quoted 4 lines
Quoted 3 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
See https://lkml.org/lkml/2017/11/1/655
Cheers,
Longman
Paolo Bonzini
maintainerQuoted 12 lines · Eduardo Valentin
Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.
Quoted 4 lines
Quoted 2 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
[PATCH-tip v2 0/2] x86/paravirt: Enable users to choose PV lock type
https://lkml.org/lkml/2017/11/1/655
Quoted 9 lines
Quoted 3 lines
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
The way I think of this is this is a flag set by host side so the guest adapts accordingly.
If the admin in guest side wants to ignore what the host is flagging, that is a different story.
Okay, this makes sense. But perhaps it should be a separate CPUID leaf, such as "configuration hints", rather than properly a feature.
Paolo
Longman,
Quoted 12 lines · Waiman Long
Quoted 11 lines · Eduardo Valentin
Quoted 10 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
Oh I see, thanks. I think that patch would help, but I believe the series and this patch are complementary.
Paolo, back to your question, I think this patch still makes sense in combination with Waiman's series for the following case:
+ * If this argument is not specified, the kernel will automatically choose
+ * an appropriate one depending on X86_FEATURE_HYPERVISOR and hypervisor
+ * specific settings.
+ */
In this case, the hypervisor can still flag PV_DEDICATED and the guest would not pick test&set, when
that scenario is desirable.
Quoted 4 lines
Cheers,
Longman
Signature · 2 lines
All the best, Eduardo Valentin
Quoted 25 lines · Paolo Bonzini
Quoted 12 lines · Eduardo Valentin
Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.Quoted 4 lines
Quoted 2 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
[PATCH-tip v2 0/2] x86/paravirt: Enable users to choose PV lock type
https://lkml.org/lkml/2017/11/1/655Quoted 4 lines
Quoted 3 lines
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
Quoted 8 lines
Quoted 5 lines
The way I think of this is this is a flag set by host side so the guest adapts accordingly.
If the admin in guest side wants to ignore what the host is flagging, that is a different story.
Okay, this makes sense. But perhaps it should be a separate CPUID leaf, such as "configuration hints", rather than properly a feature.
Oh OK, you don't think this starts to deviate from the feature concept. But would the PV_UNHALT also go to "configuration hints" bucket?
Another way to see this is we have three locking feature options to select from, so we need at least two bits here.
Quoted 2 lines
Paolo
Signature · 2 lines
All the best, Eduardo Valentin
Paolo Bonzini
maintainerQuoted 29 lines · Eduardo Valentin
Quoted 28 lines · Paolo Bonzini
Quoted 12 lines · Eduardo Valentin
Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.Quoted 4 lines
Quoted 2 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
[PATCH-tip v2 0/2] x86/paravirt: Enable users to choose PV lock type
https://lkml.org/lkml/2017/11/1/655Quoted 7 lines
Quoted 3 lines
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
The way I think of this is this is a flag set by host side so the guest adapts accordingly.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If the admin in guest side wants to ignore what the host is flagging, that is a different story.
Okay, this makes sense. But perhaps it should be a separate CPUID leaf, such as "configuration hints", rather than properly a feature.
Oh OK, you don't think this starts to deviate from the feature concept. But would the
PV_UNHALTalso go to "configuration hints" bucket?
PV_UNHALT says whether the pvqspinlock API is available, PV_DEDICATED says whether it should be used.
Quoted 2 lines
Another way to see this is we have three locking feature options to select from, so we need at least two bits here.
PV_DEDICATED = 1, PV_UNHALT = anything: default is qspinlock PV_DEDICATED = 0, PV_UNHALT = 1: default is pvqspinlock PV_DEDICATED = 0, PV_UNHALT = 0: default is tas
What do you think?
Paolo
Quoted 42 lines · Paolo Bonzini
Quoted 29 lines · Eduardo Valentin
Quoted 28 lines · Paolo Bonzini
Quoted 12 lines · Eduardo Valentin
Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Eduardo Valentin
Currently, the existing qspinlock implementation will fallback to test-and-set if the hypervisor has not set the
PV_UNHALTflag.This patch gives the opportunity to guest kernels to select between test-and-set and the regular queueu fair lock implementation based on the
PV_DEDICATEDKVM feature flag. When thePV_DEDICATEDflag is not set, the code will still fall back to test-and-set, but when thePV_DEDICATEDflag is set, the code will use the regular queue spinlock implementation.Quoted 4 lines
Quoted 2 lines
Have you seen Waiman's series that lets you specify this on the guest command line instead? Would this be acceptable for your use case?
No, can you please share a link to it? is it already merged to tip/master?
[PATCH-tip v2 0/2] x86/paravirt: Enable users to choose PV lock type
https://lkml.org/lkml/2017/11/1/655Quoted 7 lines
Quoted 3 lines
(In other words, is there a difference for you between making the host vs. guest administrator toggle the feature? "@amazon.com" means you are the host admin, how would you use it?)
The way I think of this is this is a flag set by host side so the guest adapts accordingly.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If the admin in guest side wants to ignore what the host is flagging, that is a different story.
Okay, this makes sense. But perhaps it should be a separate CPUID leaf, such as "configuration hints", rather than properly a feature.
Oh OK, you don't think this starts to deviate from the feature concept. But would the
PV_UNHALTalso go to "configuration hints" bucket?
PV_UNHALTsays whether the pvqspinlock API is available,PV_DEDICATEDsays whether it should be used.
Quoted 8 lines
Quoted 2 lines
Another way to see this is we have three locking feature options to select from, so we need at least two bits here.
PV_DEDICATED= 1,PV_UNHALT= anything: default is qspinlockPV_DEDICATED= 0,PV_UNHALT= 1: default is pvqspinlockPV_DEDICATED= 0,PV_UNHALT= 0: default is tasWhat do you think?
Sounds reasonable, and it is almost what the patch does. But to achieve the above table, we need in include the following chunk:
arch/x86/kernel/kvm.c +2 -0
@@ -642,6 +642,8 @@ void __init kvm_spinlock_init(void) {if (!kvm_para_available ())return ; +if (kvm_para_has_feature (KVM_FEATURE_PV_DEDICATED )) +return ;/* Does host kernel support KVM_FEATURE_PV_UNHALT? */ if (!kvm_para_has_feature (KVM_FEATURE_PV_UNHALT ))return ;
Now we get:PV_DEDICATED PV_UNHALT IMPLEMENTATION
1 X qspinlock 0 1 pvspinlock 0 0 tas
Do you still think PV_DEDICATED goes as a "configuration hint", given that it would take precedence on a feature (PV_UNHALT)? Or do we keep everything as features (two bits to represent selection of three features)?
BR,
Quoted 2 lines
Paolo
Signature · 2 lines
All the best, Eduardo Valentin