KVM: x86: synthesize TSA CPUID bits via SCATTERED_F()
PATCH 1/1 KVM: x86: synthesize TSA CPUID bits via SCATTERED_F()
KVM incorrectly synthesizes TSA_SQ_NO and TSA_L1_NO when running on AMD Family 19h CPUs by using SYNTHESIZED_F(), which unconditionally enables features for KVM-only CPUID leaves (as is the case with CPUID_8000_0021_ECX), regardless of the kernel's synthesis logic in tsa_init(). This is due to the following logic in kvm_cpu_cap_init():
if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];
This can cause an unexpected failure on Family 19h CPUs during SEV-SNP guest setup, when userspace issues SNP_LAUNCH_UPDATE, as setting these bits in the CPUID page on vulnerable CPUs is explicitly rejected by SNP firmware.
Switch to SCATTERED_F(), so that the bits are only set if the features have been force-set by the kernel in tsa_init(), or if they are reported in the raw CPUID.
--- arch/x86/kvm/cpuid.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/cpuid.c +2 -2
@@ -1230,8 +1230,8 @@ void kvm_set_cpu_caps(void) );kvm_cpu_cap_init (CPUID_8000_0021_ECX , -SYNTHESIZED_F (TSA_SQ_NO ), - SYNTHESIZED_F(TSA_L1_NO), +SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO), ); kvm_cpu_cap_init(CPUID_8000_0022_EAX,
base-commit: 0de4a0eec25b9171f2a2abb1a820e125e6797770
Signature · 1 line
2.51.0
Borislav Petkov
maintainerQuoted 29 lines · Carlos López
KVM incorrectly synthesizes
TSA_SQ_NOandTSA_L1_NOwhen running on AMD Family 19h CPUs by usingSYNTHESIZED_F(), which unconditionally enables features for KVM-only CPUID leaves (as is the case withCPUID_8000_0021_ECX), regardless of the kernel's synthesis logic intsa_init(). This is due to the following logic inkvm_cpu_cap_init():if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];This can cause an unexpected failure on Family 19h CPUs during SEV-SNP guest setup, when userspace issues
SNP_LAUNCH_UPDATE, as setting these bits in the CPUID page on vulnerable CPUs is explicitly rejected by SNP firmware.Switch to
SCATTERED_F(), so that the bits are only set if the features have been force-set by the kernel intsa_init(), or if they are reported in the raw CPUID.Fixes 31272abd5974 ("KVM: SVM: Advertise TSA CPUID bits to guests")Signed-off-by Carlos López <[email protected]>--- arch/x86/kvm/cpuid.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/kvm/cpuid.c +0
-0
arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 1234
Quoted 5 lines
arch/x86/kvm/cpuid.c +0
-2@@ -1230,8 +1230,8 @@ void kvm_set_cpu_caps(void) ); kvm_cpu_cap_init(CPUID_8000_0021_ECX, - SYNTHESIZED_F(TSA_SQ_NO), - SYNTHESIZED_F(TSA_L1_NO),
Well:
/*
* Synthesized Feature - For features that are synthesized into boot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*/
#define SYNTHESIZED_F(name)
arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 1234
Quoted 2 lines
arch/x86/kvm/cpuid.c +2
-0+SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO),
And scattered are of the same type.
Sean, what's the subtle difference here?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJim Mattson
Reviewed-byQuoted 47 lines · Borislav Petkov
Quoted 29 lines · Carlos López
KVM incorrectly synthesizes
TSA_SQ_NOandTSA_L1_NOwhen running on AMD Family 19h CPUs by usingSYNTHESIZED_F(), which unconditionally enables features for KVM-only CPUID leaves (as is the case withCPUID_8000_0021_ECX), regardless of the kernel's synthesis logic intsa_init(). This is due to the following logic inkvm_cpu_cap_init():if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];This can cause an unexpected failure on Family 19h CPUs during SEV-SNP guest setup, when userspace issues
SNP_LAUNCH_UPDATE, as setting these bits in the CPUID page on vulnerable CPUs is explicitly rejected by SNP firmware.Switch to
SCATTERED_F(), so that the bits are only set if the features have been force-set by the kernel intsa_init(), or if they are reported in the raw CPUID.Fixes 31272abd5974 ("KVM: SVM: Advertise TSA CPUID bits to guests")Signed-off-by Carlos López <[email protected]>--- arch/x86/kvm/cpuid.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/kvm/cpuid.c +0
-0arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 1234
Quoted 5 lines
arch/x86/kvm/cpuid.c +0
-2@@ -1230,8 +1230,8 @@ void kvm_set_cpu_caps(void) ); kvm_cpu_cap_init(CPUID_8000_0021_ECX, - SYNTHESIZED_F(TSA_SQ_NO), - SYNTHESIZED_F(TSA_L1_NO),Well:
/*
* Synthesized Feature - For features that are synthesized intoboot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#defineSYNTHESIZED_F(name)
Quoted 6 lines
arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 1234
Quoted 2 lines
arch/x86/kvm/cpuid.c +2
-0+SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO),And scattered are of the same type.
Sean, what's the subtle difference here?
SYNTHESIZED_F() sets the bit unconditionally. SCATTERED_F() propagates the bit (if set) from the host's cpufeature flags.
Using SYNTHESIZED_F() is fine, but it needs to be applied conditionally. Per AMD's guidance in
https://www.amd.com/content/dam/amd/en/documents/resources/bulletin/technical-guidance-for-mitigating-transient-scheduler-attacks.pdf:
As of the date of this paper, AMD’s analysis has determined that only Family 19h CPUs are vulnerable to TSA. AMD CPUs that are older than Family 19h are not vulnerable to TSA but do not set TSA_L1_NO or TSA_SQ_NO. Bare-metal software that detects such a CPU should assume the CPU is not vulnerable to TSA. Hypervisor software should synthesize the value of the TSA_L1_NO and TSA_SQ_NO CPUID bits on such platforms so guest software can rely on CPUID to detect if TSA mitigations are required.
So, the original code should have checked for family and model if running on bare metal, and should not have synthesized the feature for Zen3 and Zen4.
However, as Carlos notes, this logic is already present in tsa_init(). Even though unaffected hardware doesn't report the TSA_NO CPUID bits, tsa_init() sets the corresponding cpufeature flags. Rather than replicate the tsa_init() logic in kvm, SCATTERED_F() can be used to propagate the cpufeature flags from the host.
Borislav Petkov
maintainerQuoted 11 lines · Jim Mattson
Quoted 11 lines
/*
* Synthesized Feature - For features that are synthesized intoboot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#defineSYNTHESIZED_F(name)Quoted 2 lines
arch/x86/kvm/cpuid.c +2
-0+SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO),
Quoted 6 lines
Quoted 3 lines
And scattered are of the same type.
Sean, what's the subtle difference here?
SYNTHESIZED_F()sets the bit unconditionally.SCATTERED_F()propagates the bit (if set) from the host's cpufeature flags.
Yah, and I was hinting at the scarce documentation.
SYNTHESIZED_F() is "Primarily used for mitigation related feature flags." SCATTERED_F() is "For features that are scattered by cpufeatures.h."
And frankly, I don't understand why there needs to be a difference whether the feature is scattered or synthesized. If the flag is set on baremetal, then it is and it being set, denotes what it means. And if it is not set, then it means the absence of that feature.
It is that simple.
Then it becomes a decision of the hypervisor whether to expose it to the guest or not.
Not whether it is synthesized or scattered.
But maybe I'm missing an aspect which is important for virt...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 17 lines · Borislav Petkov
Quoted 15 lines · Jim Mattson
Quoted 14 lines
/*
* Synthesized Feature - For features that are synthesized intoboot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#defineSYNTHESIZED_F(name)arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 1234
Quoted 2 lines
arch/x86/kvm/cpuid.c +2
-0+SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO),And scattered are of the same type.
Sean, what's the subtle difference here?
Quoted 7 lines
Quoted 2 lines
SYNTHESIZED_F()sets the bit unconditionally.SCATTERED_F()propagates the bit (if set) from the host's cpufeature flags.Yah, and I was hinting at the scarce documentation.
SYNTHESIZED_F()is "Primarily used for mitigation related feature flags."SCATTERED_F()is "For features that are scattered bycpufeatures.h."
Ugh. I have to rescind my Reviewed-by. IIUC, SCATTERED_F() implies a logical and with hardware CPUID, which means that the current proposal will never set the ITS_NO bits.
I clearly don't have any idea how the new infrastructure works, and there's too much CPP nonsense for me to care. I'll just defer to Sean.
Quoted 19 lines
And frankly, I don't understand why there needs to be a difference whether the feature is scattered or synthesized. If the flag is set on baremetal, then it is and it being set, denotes what it means. And if it is not set, then it means the absence of that feature.
It is that simple.
Then it becomes a decision of the hypervisor whether to expose it to the guest or not.
Not whether it is synthesized or scattered.
But maybe I'm missing an aspect which is important for virt...
--
Regards/Gruss,
Boris.
Hi,
Quoted 21 lines · Jim Mattson
Quoted 20 lines · Borislav Petkov
Quoted 17 lines · Jim Mattson
Quoted 8 lines
/*
* Synthesized Feature - For features that are synthesized intoboot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#defineSYNTHESIZED_F(name)Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ SCATTERED_F(TSA_SQ_NO), + SCATTERED_F(TSA_L1_NO),And scattered are of the same type.
Sean, what's the subtle difference here?
SYNTHESIZED_F()sets the bit unconditionally.SCATTERED_F()propagates the bit (if set) from the host's cpufeature flags.
Quoted 8 lines
Quoted 4 lines
Yah, and I was hinting at the scarce documentation.
SYNTHESIZED_F()is "Primarily used for mitigation related feature flags."SCATTERED_F()is "For features that are scattered bycpufeatures.h."Ugh. I have to rescind my Reviewed-by. IIUC,
SCATTERED_F()implies a logical and with hardware CPUID, which means that the current proposal will never set theITS_NObits.
Right, I see what you mean now. SCATTERED_F() will set kvm_cpu_caps correctly, but then this will clear the bits, because kvm_cpu_cap_synthesized is now 0:
kvm_cpu_caps[leaf] &= (raw_cpuid_get(cpuid) |
kvm_cpu_cap_synthesized);So to me it seems like SYNTHESIZED_F() is just wrong, since it always enables bits for KVM-only leafs. So how about the following (I think Binbin Wu suggests this in his other email):
arch/x86/kvm/cpuid.c +2 -1
@@ -769,7 +769,8 @@ do { \*/ #define SYNTHESIZED_F (name) \ ({ \ - kvm_cpu_cap_synthesized |= feature_bit(name); \ + if (boot_cpu_has(X86_FEATURE_##name)) \ + kvm_cpu_cap_synthesized |= feature_bit(name); \ F(name); \ })
Sean Christopherson
maintainerQuoted 23 lines · Carlos López
Hi,
Quoted 20 lines · Jim Mattson
Quoted 19 lines · Borislav Petkov
Quoted 17 lines · Jim Mattson
Quoted 8 lines
/*
* Synthesized Feature - For features that are synthesized intoboot_cpu_data,
* i.e. may not be present in the raw CPUID, but can still be advertised to
* userspace. Primarily used for mitigation related feature flags.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#defineSYNTHESIZED_F(name)Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ SCATTERED_F(TSA_SQ_NO), + SCATTERED_F(TSA_L1_NO),And scattered are of the same type.
Sean, what's the subtle difference here?
SYNTHESIZED_F()sets the bit unconditionally.SCATTERED_F()propagates the bit (if set) from the host's cpufeature flags.
As noted below, SYNTHESIZED_F() isn't entirely unconditional. kvm_cpu_cap_init() factors in boot_cpu_data.x86_capability, the problem here is that SYNTHESIZED_F() is now being used for KVM-only leafs, and so the common code doesn't work as intended.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Yah, and I was hinting at the scarce documentation.
Maybe I could add a table showing how the XXX_F() macros map to various controls?
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
SYNTHESIZED_F()is "Primarily used for mitigation related feature flags."SCATTERED_F()is "For features that are scattered bycpufeatures.h."Ugh. I have to rescind my Reviewed-by. IIUC,
SCATTERED_F()implies a logical and with hardware CPUID, which means that the current proposal will never set theITS_NObits.
Quoted 8 lines
Right, I see what you mean now.
SCATTERED_F()will setkvm_cpu_capscorrectly, but then this will clear the bits, becausekvm_cpu_cap_synthesizedis now 0:kvm_cpu_caps[leaf] &= (raw_cpuid_get(cpuid) | kvm_cpu_cap_synthesized);So to me it seems like
SYNTHESIZED_F()is just wrong,
It was right when I wrote it :-)
Quoted 1 line
since it always enables bits for KVM-only leafs.
Yes, I didn't anticipate synthesizing flags into KVM-only leafs.
Quoted 8 lines
So how about the following
(I think Binbin Wu suggests this in his other email):arch/x86/kvm/cpuid.c +0
-0
arch/x86/kvm/cpuid.c do { \ · line 773
Quoted 6 lines
arch/x86/kvm/cpuid.c +2
-1@@ -769,7 +769,8 @@ do { \*/ #define SYNTHESIZED_F (name) \ ({ \ - kvm_cpu_cap_synthesized |= feature_bit(name); \ + if (boot_cpu_has(X86_FEATURE_##name)) \ + kvm_cpu_cap_synthesized |= feature_bit(name); \
I would rather do this:
arch/x86/kvm/cpuid.c +4 -1
@@ -770,7 +770,10 @@ do { \#define SYNTHESIZED_F (name) \ ({ \ kvm_cpu_cap_synthesized |= feature_bit(name); \ - F(name); \ + \ + BUILD_BUG_ON(X86_FEATURE_##name >= MAX_CPU_FEATURES); \ + if (boot_cpu_has(X86_FEATURE_##name)) \ + F(name); \ })/*
because I'd like to keep kvm_cpu_cap_synthesized unconditional and have kvm_cpu_cap_features reflect what is supported. And with
because everything was fine before that commit (though it was set up to fail).
Quoted 3 lines
arch/x86/kvm/cpuid.c +0
-0F (name); \ })
Borislav Petkov
maintainerQuoted 1 line · Sean Christopherson
Maybe I could add a table showing how the
XXX_F()macros map to various controls?
Perhaps start first, please, with explaining the difference between SYNTHESIZED_F and SCATTERED_F and why you even need it?
I mean, KVM and guests only care whether X86_FEATURE flags are set or not, no? Not how they get defined underneath...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 5 lines · Borislav Petkov
Quoted 1 line · Sean Christopherson
Maybe I could add a table showing how the
XXX_F()macros map to various controls?Perhaps start first, please, with explaining the difference between
SYNTHESIZED_FandSCATTERED_Fand why you even need it?
SCATTERED_F requires the feature flag to be present in raw CPUID, SYNTHESIZED_F does not.
Quoted 1 line
I mean, KVM and guests only care whether
X86_FEATUREflags are set or not, no?
Nope. KVM cares about what KVM can virtualize/emulate, and about helping userspace accurately represent the virtual CPU that will be enumerated to the guest.
F : Features that must be present in boot_cpu_data and raw CPUID SCATTERED_F : Same as F(), but are scattered by the kernel X86_64_F : Same as F(), but are restricted to 64-bit kernels EMULATED_F : Always supported; the feature is unconditionally emulated in software SYNTHESIZED_F : Features that must be present in boot_cpu_data, but may or
may not be in raw CPUID. May also be scattered.
PASSTHROUGH_F : Features that must be present in raw CPUID, but may or may
not be present in boot_cpu_data
ALIASED_1_EDX_F : Features in 0x8000_0001.EDX that are duplicates of identical 0x1.EDX features
VENDOR_F : Features that are controlled by vendor code, often because
they are guarded by a vendor specific module param. Rules vary, but typically they are handled like basic F() features
RUNTIME_F : Features that KVM dynamically sets/clears at runtime, but that
are never adveristed to userspace. E.g. OSXSAVE and OSPKE.
Quoted 1 line
Not how they get defined underneath...
Borislav Petkov
maintainerQuoted 2 lines · Sean Christopherson
Nope. KVM cares about what KVM can virtualize/emulate, and about helping userspace accurately represent the virtual CPU that will be enumerated to the guest.
So why don't you key on that in those macros instead of how they're defined?
EXPOSE_TO_GUEST_F()
and then underneath we can figure out how to expose them.
We could have a helper table which determines what each feature is and how it should interact with raw host CPUID or something slicker.
Quoted 5 lines
F : Features that must be present in boot_cpu_data and raw CPUID SCATTERED_F : Same as F(), but are scattered by the kernel X86_64_F : Same as F(), but are restricted to 64-bit kernels EMULATED_F : Always supported; the feature is unconditionally emulated in software SYNTHESIZED_F : Features that must be present in boot_cpu_data, but may or
Quoted 9 lines
may not be in raw CPUID. May also be scattered.
PASSTHROUGH_F: Features that must be present in raw CPUID, but may or may
not be present inboot_cpu_data
ALIASED_1_EDX_F: Features in 0x8000_0001.EDX that are duplicates of identical 0x1.EDX features
VENDOR_F: Features that are controlled by vendor code, often because
they are guarded by a vendor specific module param. Rules vary, but typically they are handled like basicF()features
RUNTIME_F: Features that KVM dynamically sets/clears at runtime, but that
are never adveristed to userspace. E.g. OSXSAVE and OSPKE.
And for the time being, I'd love if this were somewhere in arch/x86/kvm/cpuid.c so that it is clear how one should use those macros.
The end goal of having the user not care about which macro to use would be the ultimate, super-duper thing tho.
I'd say.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 9 lines · Borislav Petkov
Quoted 2 lines · Sean Christopherson
Nope. KVM cares about what KVM can virtualize/emulate, and about helping userspace accurately represent the virtual CPU that will be enumerated to the guest.
So why don't you key on that in those macros instead of how they're defined?
EXPOSE_TO_GUEST_F()and then underneath we can figure out how to expose them.
Huh? That's what the macros do, they describe KVM's handling of the associated feature. SYNTHESIZED is a bit weird because it bleeds some kernel details into KVM, but ultimately it's still KVM decision as to whether or not "forced" features can be synthesized for the guest.
Quoted 18 lines
We could have a helper table which determines what each feature is and how it should interact with raw host CPUID or something slicker.
Quoted 5 lines
F : Features that must be present in boot_cpu_data and raw CPUID SCATTERED_F : Same as F(), but are scattered by the kernel X86_64_F : Same as F(), but are restricted to 64-bit kernels EMULATED_F : Always supported; the feature is unconditionally emulated in software SYNTHESIZED_F : Features that must be present in boot_cpu_data, but may orQuoted 9 lines
may not be in raw CPUID. May also be scattered.
PASSTHROUGH_F: Features that must be present in raw CPUID, but may or may
not be present inboot_cpu_data
ALIASED_1_EDX_F: Features in 0x8000_0001.EDX that are duplicates of identical 0x1.EDX features
VENDOR_F: Features that are controlled by vendor code, often because
they are guarded by a vendor specific module param. Rules vary, but typically they are handled like basicF()features
RUNTIME_F: Features that KVM dynamically sets/clears at runtime, but that
are never adveristed to userspace. E.g. OSXSAVE and OSPKE.
Quoted 2 lines
And for the time being, I'd love if this were somewhere in
arch/x86/kvm/cpuid.cso that it is clear how one should use those macros.
I'll a patch with the above and more guidance.
Quoted 2 lines
The end goal of having the user not care about which macro to use would be the ultimate, super-duper thing tho.
And impossible, for all intents and purposes. The user/contributor/developer needs to define KVM's handling semantics *somehwere*. Sure, we could to that in a big array or something, but that's just a different way of dressing up the same pig. All of this very much is an ugly pig, but it's the concepts and mechanics that are ugly and convoluted.
E.g. if we define a giant array or table, the contributor will need to map the feature to one of the above macros.
In other words, kvm_initialize_cpu_caps() _is_ the helper table. If someone wants to try and do better, by all means, have at it. But I won't hold my breath.
Borislav Petkov
maintainerQuoted 7 lines · Sean Christopherson
Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Sean Christopherson
Nope. KVM cares about what KVM can virtualize/emulate, and about helping userspace accurately represent the virtual CPU that will be enumerated to the guest.
So why don't you key on that in those macros instead of how they're defined?
Quoted 8 lines
Quoted 3 lines
EXPOSE_TO_GUEST_F()and then underneath we can figure out how to expose them.
Huh? That's what the macros do, they describe KVM's handling of the associated feature. SYNTHESIZED is a bit weird because it bleeds some kernel details into KVM, but ultimately it's still KVM decision as to whether or not "forced" features can be synthesized for the guest.
My point is that you have to know *which* macro of all the available ones you need to use, in order to expose the feature. This thread is a case-in-point example about how it can get confusing. And it dudn't have to...
Quoted 6 lines
Quoted 6 lines
We could have a helper table which determines what each feature is and how it should interact with raw host CPUID or something slicker.
Quoted 2 lines
F : Features that must be present in
boot_cpu_dataand raw CPUID
SCATTERED_F: Same asF(), but are scattered by the kernel
Right, so what happens if we unscatter a leaf?
And why does it matter to KVM if the baremetal feature is scattered or not? KVM should only care whether the kernel has set it or not.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
X86_64_F: Same asF(), but are restricted to 64-bit kernels
EMULATED_F: Always supported; the feature is unconditionally emulated in software
And an emulated feature *can* be scattered or synthesized or whatever...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
SYNTHESIZED_F: Features that must be present inboot_cpu_data, but may or
may not be in raw CPUID. May also be scattered.
So which one do I use here?
This is the confusion I'm taking about.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
PASSTHROUGH_F: Features that must be present in raw CPUID, but may or may
not be present inboot_cpu_data
Maybe there's a reason for it but why would the guest care if the feature is present in raw CPUID or not? The hypervisor controls what the guest sees in CPUID...
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
ALIASED_1_EDX_F: Features in 0x8000_0001.EDX that are duplicates of identical 0x1.EDX features
VENDOR_F: Features that are controlled by vendor code, often because
they are guarded by a vendor specific module param. Rules vary, but typically they are handled like basicF()features
RUNTIME_F: Features that KVM dynamically sets/clears at runtime, but that
are never adveristed to userspace. E.g. OSXSAVE and OSPKE.
Also, we're rewriting the whole CPUID handling on baremetal and someday the CPUID table in the kernel will be the only thing you query - not the CPUID insn. Then those names above become wrong/obsolete.
Quoted 10 lines
Quoted 2 lines
And for the time being, I'd love if this were somewhere in
arch/x86/kvm/cpuid.cso that it is clear how one should use those macros.I'll a patch with the above and more guidance.
Quoted 2 lines
The end goal of having the user not care about which macro to use would be the ultimate, super-duper thing tho.
And impossible, for all intents and purposes. The user/contributor/developer needs to define KVM's handling semantics *somehwere*.
I still don't get this: why does KVM need to know whether a X86_FEATURE is scattered or synthesized or whatnot?
Quoted 3 lines
Sure, we could to that in a big array or something, but that's just a different way of dressing up the same pig. All of this very much is an ugly pig, but it's the concepts and mechanics that are ugly and convoluted.
Well, if we're redoing how feature flags and CPUID leafs etc are being handled on baremetal, why not extend that handling so that KVM can put info there too, about each feature and how it is going to be exposed to the guest instead of doing a whole bucket of _F() macros?
Quoted 2 lines
E.g. if we define a giant array or table, the contributor will need to map the feature to one of the above macros.
We are on the way to a giant array/table anyway:
https://lore.kernel.org/r/[email protected]
Quoted 1 line
In other words,
kvm_initialize_cpu_caps()_is_ the helper table.
$ git grep kvm_initialize_cpu_caps $
I'm on current Linus/master.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 12 lines · Borislav Petkov
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Sean Christopherson
Nope. KVM cares about what KVM can virtualize/emulate, and about helping userspace accurately represent the virtual CPU that will be enumerated to the guest.
So why don't you key on that in those macros instead of how they're defined?
EXPOSE_TO_GUEST_F()and then underneath we can figure out how to expose them.
Quoted 8 lines
Quoted 4 lines
Huh? That's what the macros do, they describe KVM's handling of the associated feature. SYNTHESIZED is a bit weird because it bleeds some kernel details into KVM, but ultimately it's still KVM decision as to whether or not "forced" features can be synthesized for the guest.
My point is that you have to know *which* macro of all the available ones you need to use, in order to expose the feature. This thread is a case-in-point example about how it can get confusing. And it dudn't have to...
Yes, it did have to. These are the choices. Whether they're in raw C code, macro hell, or a spreadsheet, the choices remain the same.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
We could have a helper table which determines what each feature is and how it should interact with raw host CPUID or something slicker.
Quoted 2 lines
F : Features that must be present in
boot_cpu_dataand raw CPUID
SCATTERED_F: Same asF(), but are scattered by the kernelRight, so what happens if we unscatter a leaf?
The build will fail. Explicitly because of this:
BUILD_BUG_ON(X86_FEATURE_##name >= MAX_CPU_FEATURES); \
and also because attempting to defined the CPUID_fn_idx_REG enum will collide with the existing enum in kvm_only_cpuid_leafs, and we'll go unscatter the KVM code.
Even without those safeguards, everything would be totally fine, the "overhead" is negligible. In other words, scattered leafs require a bit of extra code to handle correctly in KVM, whereas normal F() leaves Just Work.
Quoted 2 lines
And why does it matter to KVM if the baremetal feature is scattered or not? KVM should only care whether the kernel has set it or not.
Because KVM needs to query features in _guest_ CPUID, and so the bit position must match the architectural values. KVM also cares if the feature is present in raw CPUID.
E.g. X86_FEATURE_SGX1 is bit 8 in Linux-defined word 11. But in CPUID, SGX1 is bit 0 in CPUID.0x12.0.EAX. If KVM tried to query bit 8 in CPUID.0x12.0.EAX, it would read garbage.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
X86_64_F: Same asF(), but are restricted to 64-bit kernels
EMULATED_F: Always supported; the feature is unconditionally emulated in softwareAnd an emulated feature *can* be scattered or synthesized or whatever...
Yep.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
SYNTHESIZED_F: Features that must be present inboot_cpu_data, but may or
may not be in raw CPUID. May also be scattered.So which one do I use here?
This is the confusion I'm taking about.
For the TSA stuff? SYNTHESIZED_F(), because KVM's ABI is to advertise support for the "features" even if they're not present in raw CPUID, so long as they're supported by the host kernel.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
PASSTHROUGH_F: Features that must be present in raw CPUID, but may or may
not be present inboot_cpu_dataMaybe there's a reason for it but why would the guest care if the feature is present in raw CPUID or not? The hypervisor controls what the guest sees in CPUID...
The VMM controls what the guest sees, _KVM_ does not.
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
ALIASED_1_EDX_F: Features in 0x8000_0001.EDX that are duplicates of identical 0x1.EDX features
VENDOR_F: Features that are controlled by vendor code, often because
they are guarded by a vendor specific module param. Rules vary, but typically they are handled like basicF()features
RUNTIME_F: Features that KVM dynamically sets/clears at runtime, but that
are never adveristed to userspace. E.g. OSXSAVE and OSPKE.Also, we're rewriting the whole CPUID handling on baremetal and someday the CPUID table in the kernel will be the only thing you query - not the CPUID insn.
Perhaps. But the only if the table provides both the kernel's configuration *and* raw CPUID, and is 100% comprehensive. And even if that happens, it won't change anything about KVM's macros, except I guess remove the need for SCATTERED_F() if the tables are truly comprehensive.
Quoted 1 line
Then those names above become wrong/obsolete.
No, because the concepts won't change. The code may look different, and KVM may need to #define a pile of things to do what it needs to do, but the semantics of how KVM supports various features isn't changing.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
And for the time being, I'd love if this were somewhere in
arch/x86/kvm/cpuid.cso that it is clear how one should use those macros.I'll a patch with the above and more guidance.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The end goal of having the user not care about which macro to use would be the ultimate, super-duper thing tho.
And impossible, for all intents and purposes. The user/contributor/developer needs to define KVM's handling semantics *somehwere*.
I still don't get this: why does KVM need to know whether a
X86_FEATUREis scattered or synthesized or whatnot?
See above regarding scattered. As for synthesized, KVM is paranoid and so by default, requires features to be supported by the host kernel *and* present in raw CPUID in order to advertise support to the guest. Whether or not the paranoia is justified is arguable, but in practice it costs KVM almost nothing, and at the very least, IMO it's very helpful to document KVM's exact expectations/rules.
Quoted 8 lines
Quoted 3 lines
Sure, we could to that in a big array or something, but that's just a different way of dressing up the same pig. All of this very much is an ugly pig, but it's the concepts and mechanics that are ugly and convoluted.
Well, if we're redoing how feature flags and CPUID leafs etc are being handled on baremetal, why not extend that handling so that KVM can put info there too, about each feature and how it is going to be exposed to the guest instead of doing a whole bucket of
_F()macros?
Because IMO, that would be a huge net negative. I have zero desire to go lookup a table to figure out KVM's rules for supporting a given feature, and even less desire to have to route KVM-internal changes through a giant shared table. I'm also skeptical that a table would provide as many safeguards as the macro magic, at least not without a lot more development.
Quoted 6 lines
Quoted 2 lines
E.g. if we define a giant array or table, the contributor will need to map the feature to one of the above macros.
We are on the way to a giant array/table anyway:
Using something like that for the core kernel makes a lot of sense. But I don't see what would be gained by shoehorning KVM's ABI into that table.
Quoted 6 lines
Quoted 1 line
In other words,
kvm_initialize_cpu_caps()_is_ the helper table.$ git grep kvm_initialize_cpu_caps $I'm on current Linus/master.
Ah, sorry, it's kvm_set_cpu_caps() until this pull request:
Borislav Petkov
maintainerQuoted 3 lines · Sean Christopherson
See above regarding scattered. As for synthesized, KVM is paranoid and so by default, requires features to be supported by the host kernel *and* present in raw CPUID in order to advertise support to the guest.
Yes, it will check for X86_FEATURE to be and then look at CPUID.
Quoted 5 lines
Because IMO, that would be a huge net negative. I have zero desire to go lookup a table to figure out KVM's rules for supporting a given feature, and even less desire to have to route KVM-internal changes through a giant shared table. I'm also skeptical that a table would provide as many safeguards as the macro magic, at least not without a lot more development.
Lemme cut to the chase because it seems to me like my point is not coming across:
We're changing how CPUID is handled on baremetal. Consider how much trouble there was and is between how the baremetal kernel handles CPUID features and how KVM wants to handle them and how those should be independent but they aren't and if we change baremetal handling - i.e., unscatter a leaf - we break KVM, yadda yadda, and all the friction over the years.
Now we have the chance to define that cleanly and also accomodate KVM's needs.
As in, you add a CPUID flag in baremetal and then in the representation of that flag in our internal structures, there are KVM-specific attributes which are used by it to do that feature flag's representation to guests.
The new scheme will get rid of the scattered crap as it is not needed anymore
- we'll have the *whole* CPUID leaf hierarchy. Now wouldn't it be lovely to have a
.kvm_flags = EMULATED_F | X86_64_F ... RUNTIME_F
which is per CPUID feature bit and which KVM code queries?
SCATTERED_F, SYNTHESIZED_F, PASSTHROUGH_F become obsolete.
No need for those macros, adding new CPUID feature flags would mean simply adding those .kvm_flags things which denote what KVM does with the feature. Not how it is defined internally.
And then everything JustWorks(tm) naturally without having to deal with those macros.
And you'd get rid of the KVM-only CPUID leafs too because everything will be in one central place.
Now why wouldn't you want that wonderful and charming thing?!
:-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 7 lines · Borislav Petkov
Quoted 3 lines · Sean Christopherson
See above regarding scattered. As for synthesized, KVM is paranoid and so by default, requires features to be supported by the host kernel *and* present in raw CPUID in order to advertise support to the guest.
Yes, it will check for
X86_FEATUREto be and then look at CPUID.
Quoted 8 lines
Quoted 5 lines
Because IMO, that would be a huge net negative. I have zero desire to go lookup a table to figure out KVM's rules for supporting a given feature, and even less desire to have to route KVM-internal changes through a giant shared table. I'm also skeptical that a table would provide as many safeguards as the macro magic, at least not without a lot more development.
Lemme cut to the chase because it seems to me like my point is not coming across:
I understand what the goal is, I just don't want to buy what you're selling.
Quoted 5 lines
We're changing how CPUID is handled on baremetal. Consider how much trouble there was and is between how the baremetal kernel handles CPUID features and how KVM wants to handle them and how those should be independent but they aren't and if we change baremetal handling - i.e., unscatter a leaf - we break KVM, yadda yadda, and all the friction over the years.
Those problems are _entirely_ limited to the fact that the kernel's feature tracking isn't 100% comprehensive. It's completely orthogonal to KVM's enumeration of its supported feature.
Quoted 6 lines
Now we have the chance to define that cleanly and also accomodate KVM's needs.
As in, you add a CPUID flag in baremetal and then in the representation of that flag in our internal structures, there are KVM-specific attributes which are used by it to do that feature flag's representation to guests.
Quoted 7 lines
The new scheme will get rid of the scattered crap as it is not needed anymore
- we'll have the *whole* CPUID leaf hierarchy. Now wouldn't it be lovely to have a.kvm_flags = EMULATED_F | X86_64_F ... RUNTIME_Fwhich is per CPUID feature bit and which KVM code queries?
No, it honestly sounds rather awful.
Quoted 1 line
SCATTERED_F,SYNTHESIZED_F,PASSTHROUGH_Fbecome obsolete.
SCATTERED_F will become obsolete, SYNTHESIZED_F and PASSTHROUGH_F will not. They cannot. It's impossible to express three states with one bit.
SYNTHESIZED_F PASSTHROUGH_F F raw CPUID DONT_CARE Y Y kernel caps Y DONT_CARE Y
If the kernel tracks both raw CPUID *and* kernel caps, then KVM can use the table without having to (re)do CPUID when configuring KVM's feature set. But KVM would still need to have processing for SYNTHESIZED_F, PASSTHROUGH_F, and F, to derive the correct state from the raw+kernel tables.
Quoted 9 lines
No need for those macros, adding new CPUID feature flags would mean simply adding those .kvm_flags things which denote what KVM does with the feature. Not how it is defined internally.
And then everything JustWorks(tm) naturally without having to deal with those macros.
And you'd get rid of the KVM-only CPUID leafs too because everything will be in one central place.
Again, that's achieved purely by tracking the full CPUID hierarchy. It has nothing to do with EMULATED_F, X86_64_F, RUNTIME_F, SYNTHESIZED_F, PASSTHROUGH_F, etc.
Quoted 1 line
Now why wouldn't you want that wonderful and charming thing?!
Because from my perspective, centralizing *everything* is all pain, no gain. It would bleed KVM details into the broader kernel, unnecessarily limit KVM's ability to change how KVM emulates/virtualizes features, and require querying a lookaside table to understand KVM's rules/handling. No thanks.
Borislav Petkov
maintainerQuoted 2 lines · Sean Christopherson
Those problems are _entirely_ limited to the fact that the kernel's feature tracking isn't 100% comprehensive.
Thus the rewrite. :)
Quoted 4 lines
If the kernel tracks both raw CPUID *and* kernel caps, then KVM can use the table without having to (re)do CPUID when configuring KVM's feature set. But KVM would still need to have processing for
SYNTHESIZED_F,PASSTHROUGH_F, and F, to derive the correct state from the raw+kernel tables.
That's what I meant - the macros and the confusion which one to use would go away.
Quoted 4 lines
Because from my perspective, centralizing *everything* is all pain, no gain. It would bleed KVM details into the broader kernel, unnecessarily limit KVM's ability to change how KVM emulates/virtualizes features, and require querying a lookaside table to understand KVM's rules/handling. No thanks.
The point is not to limit KVM's ability but *augment* the internal representation so that it *accomodates* KVM fully. But ok, your call.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 8 lines · Borislav Petkov
Quoted 4 lines · Sean Christopherson
If the kernel tracks both raw CPUID *and* kernel caps, then KVM can use the table without having to (re)do CPUID when configuring KVM's feature set. But KVM would still need to have processing for
SYNTHESIZED_F,PASSTHROUGH_F, and F, to derive the correct state from the raw+kernel tables.That's what I meant - the macros and the confusion which one to use would go away.
Again, the macros would go away, but they would simply be replaced by labels in a table. I.e. the "confusion" won't go away, because it can't simply disappear. That knowledge must live somewhere.
Quoted 8 lines · Carlos López
KVM incorrectly synthesizes
TSA_SQ_NOandTSA_L1_NOwhen running on AMD Family 19h CPUs by usingSYNTHESIZED_F(), which unconditionally enables features for KVM-only CPUID leaves (as is the case withCPUID_8000_0021_ECX), regardless of the kernel's synthesis logic intsa_init(). This is due to the following logic inkvm_cpu_cap_init():if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];
Since TSA_SQ_NO and TSA_L1_NO are defined in CPUID_8000_0021_ECX, and CPUID_8000_0021_ECX > NCAPINTS, the code above doesn't take effect.
The code makes the two bits set unconditionally is:
SYNTHESIZED_F() sets kvm_cpu_cap_synthesized
and later
kvm_cpu_caps[leaf] &= (raw_cpuid_get(cpuid) |
kvm_cpu_cap_synthesized);Quoted 9 lines
This can cause an unexpected failure on Family 19h CPUs during SEV-SNP guest setup, when userspace issues
SNP_LAUNCH_UPDATE, as setting these bits in the CPUID page on vulnerable CPUs is explicitly rejected by SNP firmware.Switch to
SCATTERED_F(), so that the bits are only set if the features have been force-set by the kernel intsa_init(), or if they are reported in the raw CPUID.
When you switch to SCATTERED_F(), if the two bits are not in raw cpuid, the two bits will not be set to kvm_cpu_caps[CPUID_8000_0021_ECX], regardless the setting initialized in tsa_init().
TSA_SQ_NO and TSA_L1_NO are conditional kernel synthesis bits, should SYNTHESIZED_F() check that the related bit is actually synthesized into boot_cpu_data before setting to kvm_cpu_cap_synthesized?
Quoted 24 lines
Fixes 31272abd5974 ("KVM: SVM: Advertise TSA CPUID bits to guests")Signed-off-by Carlos López <[email protected]>--- arch/x86/kvm/cpuid.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/kvm/cpuid.c +2
-2@@ -1230,8 +1230,8 @@ void kvm_set_cpu_caps(void) );kvm_cpu_cap_init (CPUID_8000_0021_ECX , -SYNTHESIZED_F (TSA_SQ_NO ), - SYNTHESIZED_F(TSA_L1_NO), +SCATTERED_F (TSA_SQ_NO ), + SCATTERED_F(TSA_L1_NO), ); kvm_cpu_cap_init(CPUID_8000_0022_EAX,base-commit:
0de4a0eec25b9171f2a2abb1a820e125e6797770
Hi,
Quoted 9 lines · Binbin Wu
Quoted 6 lines · Carlos López
KVM incorrectly synthesizes
TSA_SQ_NOandTSA_L1_NOwhen running on AMD Family 19h CPUs by usingSYNTHESIZED_F(), which unconditionally enables features for KVM-only CPUID leaves (as is the case withCPUID_8000_0021_ECX), regardless of the kernel's synthesis logic intsa_init(). This is due to the following logic inkvm_cpu_cap_init():
Quoted 5 lines
Quoted 2 lines
if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];Since
TSA_SQ_NOandTSA_L1_NOare defined inCPUID_8000_0021_ECX, andCPUID_8000_0021_ECX> NCAPINTS, the code above doesn't take effect.
I should have explained it better, but that is precisely what I meant. Since the &= never takes place, kvm_cpu_cap_features always has the bits set. In other words: the branch always evaluates to false for KVM-only CPUID leaves, so the bits are always set.
Quoted 12 lines
The code makes the two bits set unconditionally is:
SYNTHESIZED_F()setskvm_cpu_cap_synthesized
and laterkvm_cpu_caps[leaf] &= (raw_cpuid_get(cpuid) | kvm_cpu_cap_synthesized);Quoted 6 lines
This can cause an unexpected failure on Family 19h CPUs during SEV-SNP guest setup, when userspace issues
SNP_LAUNCH_UPDATE, as setting these bits in the CPUID page on vulnerable CPUs is explicitly rejected by SNP firmware.
Quoted 7 lines
Quoted 3 lines
Switch to
SCATTERED_F(), so that the bits are only set if the features have been force-set by the kernel intsa_init(), or if they are reported in the raw CPUID.When you switch to
SCATTERED_F(), if the two bits are not in raw cpuid, the two bits will not be set to kvm_cpu_caps[CPUID_8000_0021_ECX], regardless the setting initialized intsa_init().
Yes, I think you're right.
Quoted 3 lines
TSA_SQ_NOandTSA_L1_NOare conditional kernel synthesis bits, shouldSYNTHESIZED_F()check that the related bit is actually synthesized intoboot_cpu_databefore setting tokvm_cpu_cap_synthesized?
Right, I thought this is what SCATTERED_F() already did, but I missed that there is the logical AND with the raw CPUID later. See the diff in my reply to Jim Mattson, I think that is what you propose as well.