Skip to contentSkip to conversation

x86: Activate DOITM

PATCH In review Andrew Cooper 4 participants 14 messages Oct 4, 2022 – Oct 5, 2022

lore
Share & tools
lore
#0 lore raw

cover x86: Activate DOITM

Slightly RFC. I'm a little uncertain about putting doitm into cpuid= but it's the least bad option I can think of.

Andrew Cooper (2):
x86/cpuid: Infrastructure to support pseudo feature identifiers
x86: Activate Data Operand Invariant Timing Mode by default

 xen/arch/x86/cpu/common.c            | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/cpuid.c                 | 22 ++++++++++++++++++----
 xen/arch/x86/include/asm/processor.h |  2 ++
 xen/tools/gen-cpuid.py               | 15 +++++++++++++++
 4 files changed, 64 insertions(+), 4 deletions(-)
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/2 x86/cpuid: Infrastructure to support pseudo feature identifiers

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

* Pass the feature name into the parse callback.
* Exclude a feature value of ~0u from falling into the general set/clear bit paths.
* In gen-cpuid.py, insert a placeholder to collect all the pseudo feature names.

No practical change.

---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
CC: Henry Wang <[email protected]>
CC: Marek Marczykowski-Górecki <[email protected]>
CC: Demi Marie Obenour <[email protected]>

---
 xen/arch/x86/cpuid.c   | 17 +++++++++++++----
 xen/tools/gen-cpuid.py | 13 +++++++++++++
 2 files changed, 26 insertions(+), 4 deletions(-)
xen/arch/x86/cpuid.c +13 -4
index 822f9ace1087..112ee63a9449 100644
@@ -39,7 +39,8 @@ static const struct feature_name {
  * function pointer call in the middle of the loop.
  */
 static int __init always_inline parse_cpuid(
-    const char *s, void (*callback)(unsigned int feat, bool val))
+    const char *s, void (*callback)(const char *name,
+                                    unsigned int feat, bool val))
 {
     const char *ss;
     int val, rc = 0;
@@ -81,7 +82,7 @@ static int __init always_inline parse_cpuid(
 
             if ( (val = parse_boolean(mid->name, s, ss)) >= 0 )
             {
-                callback(mid->bit, val);
+                callback(mid->name, mid->bit, val);
                 mid = NULL;
             }
 
@@ -101,8 +102,12 @@ static int __init always_inline parse_cpuid(
     return rc;
 }
 
-static void __init cf_check _parse_xen_cpuid(unsigned int feat, bool val)
+static void __init cf_check _parse_xen_cpuid(
+    const char *name, unsigned int feat, bool val)
 {
+    if ( unlikely(feat == ~0u) )
+        return;
+
     if ( !val )
         setup_clear_cpu_cap(feat);
     else if ( feat == X86_FEATURE_RDRAND &&
@@ -120,8 +125,12 @@ static bool __initdata dom0_cpuid_cmdline;
 static uint32_t __initdata dom0_enable_feat[FSCAPINTS];
 static uint32_t __initdata dom0_disable_feat[FSCAPINTS];
 
-static void __init cf_check _parse_dom0_cpuid(unsigned int feat, bool val)
+static void __init cf_check _parse_dom0_cpuid(
+    const char *name, unsigned int feat, bool val)
 {
+    if ( unlikely(feat == ~0u) )
+        return;
+
     __set_bit  (feat, val ? dom0_enable_feat  : dom0_disable_feat);
     __clear_bit(feat, val ? dom0_disable_feat : dom0_enable_feat );
 }
xen/tools/gen-cpuid.py +13 -0
index 4f7c8d78cce7..f3045b3bfd36 100755
@@ -297,6 +297,19 @@ def crunch_numbers(state):
         RTM: [TSXLDTRK],
     }
 
+    #
+    # Pseudo feature names.  These don't map to a feature bit, but are
+    # inserted into the values dictionary so they can be parsed and handled
+    # specially
+    #
+    pseduo_names = (
+    )
+
+    for n in pseduo_names:
+        if n in state.values:
+            raise Fail("Pseduo feature name %s aliases real feature" % (n, ))
+        state.values[n] = 0xffffffff
+
     deep_features = tuple(sorted(deps.keys()))
     state.deep_deps = {}
 
Signature · 1 line
2.11.0
#2 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 8 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

* Pass the feature name into the parse callback.
* Exclude a feature value of ~0u from falling into the general set/clear bit paths.
* In gen-cpuid.py, insert a placeholder to collect all the pseudo feature names.

Hmm, I was envisioning this to be fitted into the existing model in a less adhoc way: (parts of) MSRs holding feature bits aren't very different from individual (pairs of) registers of CPUID output (in the case of ARCH_CAPS there would be a perhaps just abstract mask limiting things to the subset of bits which actually act as feature indicators). Hence I'd have expected them to gain proper entries in the public interface (cpufeatureset.h) and then be represented / processed the same way in featuresets and policies. All that would be left out at this point would be the exposing of the bit to guests (in patch 2, assuming the split into two patches is then actually still warranted), integration into guest_rdmsr(), and at least some of the tool stack side (xen-cpuid, for example, could easily learn of such right away).

However, since I'm pretty sure you've considered such an approach, I guess I might be overlooking some caveat?

Quoted 8 lines
xen/tools/gen-cpuid.py +2 -0
@@ -297,6 +297,19 @@ def crunch_numbers(state):
         RTM: [TSXLDTRK],
     }
 
+    #
+    # Pseudo feature names.  These don't map to a feature bit, but are

xen/tools/gen-cpuid.py def crunch_numbers(state): · line 311

Quoted 10 lines
xen/tools/gen-cpuid.py +10 -0
+    # inserted into the values dictionary so they can be parsed and handled
+    # specially
+    #
+    pseduo_names = (
+    )
+
+    for n in pseduo_names:
+        if n in state.values:
+            raise Fail("Pseduo feature name %s aliases real feature" % (n, ))
+        state.values[n] = 0xffffffff

Throughout this hunk: s/pseduo/pseudo/g.

Jan

#3 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 22 lines · Jan Beulich
Quoted 8 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

* Pass the feature name into the parse callback.
* Exclude a feature value of ~0u from falling into the general set/clear bit paths.
* In gen-cpuid.py, insert a placeholder to collect all the pseudo feature names.

Hmm, I was envisioning this to be fitted into the existing model in a less adhoc way: (parts of) MSRs holding feature bits aren't very different from individual (pairs of) registers of CPUID output (in the case of ARCH_CAPS there would be a perhaps just abstract mask limiting things to the subset of bits which actually act as feature indicators). Hence I'd have expected them to gain proper entries in the public interface (cpufeatureset.h) and then be represented / processed the same way in featuresets and policies. All that would be left out at this point would be the exposing of the bit to guests (in patch 2, assuming the split into two patches is then actually still warranted), integration into guest_rdmsr(), and at least some of the tool stack side (xen-cpuid, for example, could easily learn of such right away).

Quoted 2 lines

However, since I'm pretty sure you've considered such an approach, I guess I might be overlooking some caveat?

I have on multiple occasions considered putting MSR_ARCH_CAPS into the existing X86_FEATURE_* infrastructure.  In the future, it's likely the right course of action to take.

However, doing so now will break speculation safety for guests in some cases.  The featureset API intended to be safe to treat as a regular bitmap, and this is how it is used in practice.

Honestly, I didn't imagine that we'd get bits like RSBA and RRSBA that need to be treated with opposite polarity to be levelled safely.

Even if we do end up putting MSR_ARCH_CAPS in X86_FEATURE_*, we still need to remove the featureset api (replaced by the cpu policy infrastructure and libx86) to retain migration safety.

Quoted 7 lines
Quoted 7 lines
xen/tools/gen-cpuid.py +0 -0
@@ -297,6 +297,19 @@ def crunch_numbers(state):
         RTM: [TSXLDTRK],
     }
 
Quoted 13 lines

xen/tools/gen-cpuid.py def crunch_numbers(state): · line 311

Quoted 12 lines
xen/tools/gen-cpuid.py +12 -0
+    #
+    # Pseudo feature names.  These don't map to a feature bit, but are
+    # inserted into the values dictionary so they can be parsed and handled
+    # specially
+    #
+    pseduo_names = (
+    )
+
+    for n in pseduo_names:
+        if n in state.values:
+            raise Fail("Pseduo feature name %s aliases real feature" % (n, ))
+        state.values[n] = 0xffffffff

Throughout this hunk: s/pseduo/pseudo/g.

Oops, yes.  Will fix.

~Andrew

#4 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 23 lines · Andrew Cooper
Quoted 22 lines · Jan Beulich
Quoted 8 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

* Pass the feature name into the parse callback.
* Exclude a feature value of ~0u from falling into the general set/clear bit paths.
* In gen-cpuid.py, insert a placeholder to collect all the pseudo feature names.

Hmm, I was envisioning this to be fitted into the existing model in a less adhoc way: (parts of) MSRs holding feature bits aren't very different from individual (pairs of) registers of CPUID output (in the case of ARCH_CAPS there would be a perhaps just abstract mask limiting things to the subset of bits which actually act as feature indicators). Hence I'd have expected them to gain proper entries in the public interface (cpufeatureset.h) and then be represented / processed the same way in featuresets and policies. All that would be left out at this point would be the exposing of the bit to guests (in patch 2, assuming the split into two patches is then actually still warranted), integration into guest_rdmsr(), and at least some of the tool stack side (xen-cpuid, for example, could easily learn of such right away).

Quoted 8 lines
Quoted 2 lines

However, since I'm pretty sure you've considered such an approach, I guess I might be overlooking some caveat?

I have on multiple occasions considered putting MSR_ARCH_CAPS into the existing X86_FEATURE_* infrastructure.  In the future, it's likely the right course of action to take.

However, doing so now will break speculation safety for guests in some

Considering further text - s/speculation/migration/, I guess?

Quoted 9 lines

cases.  The featureset API intended to be safe to treat as a regular bitmap, and this is how it is used in practice.

Honestly, I didn't imagine that we'd get bits like RSBA and RRSBA that need to be treated with opposite polarity to be levelled safely.

Even if we do end up putting MSR_ARCH_CAPS in X86_FEATURE_*, we still need to remove the featureset api (replaced by the cpu policy infrastructure and libx86) to retain migration safety.

Well, I didn't mean to suggest to put all of the feature-like bits there right away. Just the one bit we're after would do for now. Afaict that wouldn't affect migration safety, while it would allow doing things in Xen in a more "natural" way.

Jan

#5 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 24 lines · Jan Beulich
Quoted 23 lines · Andrew Cooper
Quoted 22 lines · Jan Beulich
Quoted 8 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

* Pass the feature name into the parse callback.
* Exclude a feature value of ~0u from falling into the general set/clear bit paths.
* In gen-cpuid.py, insert a placeholder to collect all the pseudo feature names.

Hmm, I was envisioning this to be fitted into the existing model in a less adhoc way: (parts of) MSRs holding feature bits aren't very different from individual (pairs of) registers of CPUID output (in the case of ARCH_CAPS there would be a perhaps just abstract mask limiting things to the subset of bits which actually act as feature indicators). Hence I'd have expected them to gain proper entries in the public interface (cpufeatureset.h) and then be represented / processed the same way in featuresets and policies. All that would be left out at this point would be the exposing of the bit to guests (in patch 2, assuming the split into two patches is then actually still warranted), integration into guest_rdmsr(), and at least some of the tool stack side (xen-cpuid, for example, could easily learn of such right away).

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

However, since I'm pretty sure you've considered such an approach, I guess I might be overlooking some caveat?

I have on multiple occasions considered putting MSR_ARCH_CAPS into the existing X86_FEATURE_* infrastructure.  In the future, it's likely the right course of action to take.

However, doing so now will break speculation safety for guests in some

Considering further text - s/speculation/migration/, I guess?

More "speculation safety on migrate".

Except its not just on migrate.  It can go wrong for suspend/resume on the same host across a reboot which changes the microcode version.

Quoted 6 lines
Quoted 6 lines

cases.  The featureset API intended to be safe to treat as a regular bitmap, and this is how it is used in practice.

Honestly, I didn't imagine that we'd get bits like RSBA and RRSBA that need to be treated with opposite polarity to be levelled safely.

Quoted 7 lines
Quoted 3 lines

Even if we do end up putting MSR_ARCH_CAPS in X86_FEATURE_*, we still need to remove the featureset api (replaced by the cpu policy infrastructure and libx86) to retain migration safety.

Well, I didn't mean to suggest to put all of the feature-like bits there right away. Just the one bit we're after would do for now. Afaict that wouldn't affect migration safety, while it would allow doing things in Xen in a more "natural" way.

That requires reworking how the MSR policies get set up, patches for which have been stagnent on xen-devel, as well as a reasonable amount of plumbing because the featureset<->policy conversions which only have CPUID data right now.

If I go down this route, realise it is going to have to go out in an XSA so will be backported, and that you're also committing to taking the VMX MSRs in due course.  (which is all on my plan, but I don't want to start the work now and have an argument down the line.)

~Andrew

#6 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

In reply to Andrew Cooper
Quoted 2 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

Why we prefer this logic over just giving such pseudo features a synthetic bit or akin?

Could we have a bit more of a description about what would be considered a pseudo feature?

Thanks, Roger.

#7 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 5 lines · Roger Pau Monné
Quoted 2 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

Why we prefer this logic over just giving such pseudo features a synthetic bit or akin?

Synthetic bits are (intentionally) not available to cmdline parsing.  We need something that is available for parsing.

Quoted 2 lines

Could we have a bit more of a description about what would be considered a pseudo feature?

I don't really have anything further to add beyond the comment in gen-cpuid.py

It's a misc collection of things requiring special handling.

~Andrew

#8 lore raw

Re: [PATCH 1/2] x86/cpuid: Infrastructure to support pseudo feature identifiers

Quoted 9 lines · Andrew Cooper
Quoted 5 lines · Roger Pau Monné
Quoted 2 lines · Andrew Cooper

A future change will want a cpuid-like identifier which doesn't have a mapping to a feature bit.

Why we prefer this logic over just giving such pseudo features a synthetic bit or akin?

Synthetic bits are (intentionally) not available to cmdline parsing.  We need something that is available for parsing.

I think Jan expressed my view better, in that it would be nicer to just have the MSR_ARCH_CAPS bits in the featureset, and listed in cpufeatureset.h like we handle CPUID features.

Maybe we want to go your proposed route now if that's easier however.

Thanks, Roger.

#9 lore raw

PATCH 2/2 x86: Activate Data Operand Invariant Timing Mode by default

In reply to Andrew Cooper

Intel IceLake and later CPUs have microarchitectural behaviours which cause data-dependent timing behaviour. This is not an issue for 99% of software, but it is a problem for cryptography routines. On these CPUs, a new architectural feature, DOITM, was retrofitted in microcode.

For now, Xen can't enumerate DOITM to guest kernels; getting this working is still in progress. The consequence is that guest kernels will incorrectly conclude that they are safe.

To maintain the safety of current software, activate DOITM unilaterally. This will be relaxed in the future when we can enumerate the feature properly to guests.

As an emergency stopgap, this behaviour can be disabled by specifying cpuid=no-doitm on Xen's command line, but is not guaranteed ABI moving forward.

---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
CC: Henry Wang <[email protected]>
CC: Marek Marczykowski-Górecki <[email protected]>
CC: Demi Marie Obenour <[email protected]>

---
 xen/arch/x86/cpu/common.c            | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/cpuid.c                 |  5 +++++
 xen/arch/x86/include/asm/processor.h |  2 ++
 xen/tools/gen-cpuid.py               |  2 ++
 4 files changed, 38 insertions(+)
xen/arch/x86/cpu/common.c +29 -0
index 0412dbc915e5..8c46a4db430a 100644
@@ -209,6 +209,34 @@ void ctxt_switch_levelling(const struct vcpu *next)
 		alternative_vcall(ctxt_switch_masking, next);
 }
 
+bool __ro_after_init opt_doitm = true;
+
+static void doitm_init(void)
+{
+    uint64_t val;
+
+    if ( !opt_doitm || !cpu_has_arch_caps )
+        return;
+
+    rdmsrl(MSR_ARCH_CAPABILITIES, val);
+    if ( !(val & ARCH_CAPS_DOITM) )
+        return;
+
+    /*
+     * We are currently unable to enumerate MSR_ARCH_CAPS to guest.  As a
+     * consequence, guest kernels will believe they're safe even when they are
+     * not.
+     *
+     * Until we can enumerate DOITM properly for guests, set it unilaterally.
+     * This prevents otherwise-correct crypto from becoming vulnerable to
+     * timing sidechannels.
+     */
+
+    rdmsrl(MSR_UARCH_MISC_CTRL, val);
+    val |= UARCH_CTRL_DOITM;
+    wrmsrl(MSR_UARCH_MISC_CTRL, val);
+}
+
 bool_t opt_cpu_info;
 boolean_param("cpuinfo", opt_cpu_info);
 
@@ -532,6 +560,7 @@ void identify_cpu(struct cpuinfo_x86 *c)
 	/* Now the feature flags better reflect actual CPU features! */
 
 	xstate_init(c);
+	doitm_init();
 
 #ifdef NOISY_CAPS
 	printk(KERN_DEBUG "CPU: After all inits, caps:");
xen/arch/x86/cpuid.c +5 -0
index 112ee63a9449..09c1ee18fd95 100644
@@ -106,7 +106,12 @@ static void __init cf_check _parse_xen_cpuid(
     const char *name, unsigned int feat, bool val)
 {
     if ( unlikely(feat == ~0u) )
+    {
+        if ( strcmp(name, "doitm") == 0 )
+            opt_doitm = val;
+
         return;
+    }
 
     if ( !val )
         setup_clear_cpu_cap(feat);
xen/arch/x86/include/asm/processor.h +2 -0
index 8e2816fae9b9..2978416e6c5b 100644
@@ -637,6 +637,8 @@ enum ap_boot_method {
 };
 extern enum ap_boot_method ap_boot_method;
 
+extern bool opt_doitm;
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __ASM_X86_PROCESSOR_H */
xen/tools/gen-cpuid.py +2 -0
index f3045b3bfd36..78a3a5c1941f 100755
@@ -303,6 +303,8 @@ def crunch_numbers(state):
     # specially
     #
     pseduo_names = (
+        # Data Operand Invariant Timing Mode.  Lives in MSR_ARCH_CAPS
+        "doitm",
     )
 
     for n in pseduo_names:
Signature · 1 line
2.11.0
#10 lore raw

Re: [PATCH 2/2] x86: Activate Data Operand Invariant Timing Mode by default

Quoted 124 lines · Andrew Cooper

Intel IceLake and later CPUs have microarchitectural behaviours which cause data-dependent timing behaviour. This is not an issue for 99% of software, but it is a problem for cryptography routines. On these CPUs, a new architectural feature, DOITM, was retrofitted in microcode.

For now, Xen can't enumerate DOITM to guest kernels; getting this working is still in progress. The consequence is that guest kernels will incorrectly conclude that they are safe.

To maintain the safety of current software, activate DOITM unilaterally. This will be relaxed in the future when we can enumerate the feature properly to guests.

As an emergency stopgap, this behaviour can be disabled by specifying cpuid=no-doitm on Xen's command line, but is not guaranteed ABI moving forward.

---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
CC: Henry Wang <[email protected]>
CC: Marek Marczykowski-Górecki <[email protected]>
CC: Demi Marie Obenour <[email protected]>

---
 xen/arch/x86/cpu/common.c            | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/cpuid.c                 |  5 +++++
 xen/arch/x86/include/asm/processor.h |  2 ++
 xen/tools/gen-cpuid.py               |  2 ++
 4 files changed, 38 insertions(+)
xen/arch/x86/cpu/common.c +29 -0
index 0412dbc915e5..8c46a4db430a 100644
@@ -209,6 +209,34 @@ void ctxt_switch_levelling(const struct vcpu *next)
 		alternative_vcall(ctxt_switch_masking, next);
 }
 
+bool __ro_after_init opt_doitm = true;
+
+static void doitm_init(void)
+{
+    uint64_t val;
+
+    if ( !opt_doitm || !cpu_has_arch_caps )
+        return;
+
+    rdmsrl(MSR_ARCH_CAPABILITIES, val);
+    if ( !(val & ARCH_CAPS_DOITM) )
+        return;
+
+    /*
+     * We are currently unable to enumerate MSR_ARCH_CAPS to guest.  As a
+     * consequence, guest kernels will believe they're safe even when they are
+     * not.
+     *
+     * Until we can enumerate DOITM properly for guests, set it unilaterally.
+     * This prevents otherwise-correct crypto from becoming vulnerable to
+     * timing sidechannels.
+     */
+
+    rdmsrl(MSR_UARCH_MISC_CTRL, val);
+    val |= UARCH_CTRL_DOITM;
+    wrmsrl(MSR_UARCH_MISC_CTRL, val);
+}
+
 bool_t opt_cpu_info;
 boolean_param("cpuinfo", opt_cpu_info);
 
@@ -532,6 +560,7 @@ void identify_cpu(struct cpuinfo_x86 *c)
 	/* Now the feature flags better reflect actual CPU features! */
 
 	xstate_init(c);
+	doitm_init();
 
 #ifdef NOISY_CAPS
 	printk(KERN_DEBUG "CPU: After all inits, caps:");
xen/arch/x86/cpuid.c +5 -0
index 112ee63a9449..09c1ee18fd95 100644
@@ -106,7 +106,12 @@ static void __init cf_check _parse_xen_cpuid(
     const char *name, unsigned int feat, bool val)
 {
     if ( unlikely(feat == ~0u) )
+    {
+        if ( strcmp(name, "doitm") == 0 )
+            opt_doitm = val;
+
         return;
+    }
 
     if ( !val )
         setup_clear_cpu_cap(feat);
xen/arch/x86/include/asm/processor.h +2 -0
index 8e2816fae9b9..2978416e6c5b 100644
@@ -637,6 +637,8 @@ enum ap_boot_method {
 };
 extern enum ap_boot_method ap_boot_method;
 
+extern bool opt_doitm;
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __ASM_X86_PROCESSOR_H */
xen/tools/gen-cpuid.py +2 -0
index f3045b3bfd36..78a3a5c1941f 100755
@@ -303,6 +303,8 @@ def crunch_numbers(state):
     # specially
     #
     pseduo_names = (
+        # Data Operand Invariant Timing Mode.  Lives in MSR_ARCH_CAPS
+        "doitm",
     )
 
     for n in pseduo_names:
Signature · 1 line
2.11.0

I can’t review the actual implementation, but the idea looks good to me.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
#11 lore raw

Re: [PATCH 2/2] x86: Activate Data Operand Invariant Timing Mode by default

In reply to Andrew Cooper
Quoted 56 lines · Andrew Cooper

Intel IceLake and later CPUs have microarchitectural behaviours which cause data-dependent timing behaviour. This is not an issue for 99% of software, but it is a problem for cryptography routines. On these CPUs, a new architectural feature, DOITM, was retrofitted in microcode.

For now, Xen can't enumerate DOITM to guest kernels; getting this working is still in progress. The consequence is that guest kernels will incorrectly conclude that they are safe.

To maintain the safety of current software, activate DOITM unilaterally. This will be relaxed in the future when we can enumerate the feature properly to guests.

As an emergency stopgap, this behaviour can be disabled by specifying cpuid=no-doitm on Xen's command line, but is not guaranteed ABI moving forward.

---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
CC: Henry Wang <[email protected]>
CC: Marek Marczykowski-Górecki <[email protected]>
CC: Demi Marie Obenour <[email protected]>

---
 xen/arch/x86/cpu/common.c            | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/cpuid.c                 |  5 +++++
 xen/arch/x86/include/asm/processor.h |  2 ++
 xen/tools/gen-cpuid.py               |  2 ++
 4 files changed, 38 insertions(+)
xen/arch/x86/cpu/common.c +16 -0
index 0412dbc915e5..8c46a4db430a 100644
@@ -209,6 +209,34 @@ void ctxt_switch_levelling(const struct vcpu *next)
 		alternative_vcall(ctxt_switch_masking, next);
 }
 
+bool __ro_after_init opt_doitm = true;
+
+static void doitm_init(void)
+{
+    uint64_t val;
+
+    if ( !opt_doitm || !cpu_has_arch_caps )
+        return;
+
+    rdmsrl(MSR_ARCH_CAPABILITIES, val);
+    if ( !(val & ARCH_CAPS_DOITM) )
+        return;
+
+    /*
+     * We are currently unable to enumerate MSR_ARCH_CAPS to guest.  As a
+     * consequence, guest kernels will believe they're safe even when they are

xen/arch/x86/cpu/common.c void ctxt_switch_levelling(const struct vcpu *next) · line 237

Quoted 10 lines
xen/arch/x86/cpu/common.c +10 -0
+     * not.
+     *
+     * Until we can enumerate DOITM properly for guests, set it unilaterally.
+     * This prevents otherwise-correct crypto from becoming vulnerable to
+     * timing sidechannels.
+     */
+
+    rdmsrl(MSR_UARCH_MISC_CTRL, val);
+    val |= UARCH_CTRL_DOITM;
+    wrmsrl(MSR_UARCH_MISC_CTRL, val);

Is it possible for the firmware to have enabled DOITM and Xen needing to clear it if !opt_doitm?

Thanks, Roger.

#12 lore raw

Re: [PATCH 2/2] x86: Activate Data Operand Invariant Timing Mode by default

Quoted 28 lines · Roger Pau Monné
Quoted 27 lines · Andrew Cooper
xen/arch/x86/cpu/common.c +21 -0
@@ -209,6 +209,34 @@ void ctxt_switch_levelling(const struct vcpu *next)
 		alternative_vcall(ctxt_switch_masking, next);
 }
 
+bool __ro_after_init opt_doitm = true;
+
+static void doitm_init(void)
+{
+    uint64_t val;
+
+    if ( !opt_doitm || !cpu_has_arch_caps )
+        return;
+
+    rdmsrl(MSR_ARCH_CAPABILITIES, val);
+    if ( !(val & ARCH_CAPS_DOITM) )
+        return;
+
+    /*
+     * We are currently unable to enumerate MSR_ARCH_CAPS to guest.  As a
+     * consequence, guest kernels will believe they're safe even when they are
+     * not.
+     *
+     * Until we can enumerate DOITM properly for guests, set it unilaterally.
+     * This prevents otherwise-correct crypto from becoming vulnerable to
+     * timing sidechannels.
Quoted 8 lines

xen/arch/x86/cpu/common.c void ctxt_switch_levelling(const struct vcpu *next) · line 237

Quoted 5 lines
xen/arch/x86/cpu/common.c +5 -0
+     */
+
+    rdmsrl(MSR_UARCH_MISC_CTRL, val);
+    val |= UARCH_CTRL_DOITM;
+    wrmsrl(MSR_UARCH_MISC_CTRL, val);

Is it possible for the firmware to have enabled DOITM and Xen needing to clear it if !opt_doitm?

I think a firmware setup option is quite plausible to expect, such that safety can also be achieved underneath an unaware OS. Note how in my earlier patch I did specifically set the bit both ways, for this very reason.

Jan

#13 lore raw

Re: [PATCH 2/2] x86: Activate Data Operand Invariant Timing Mode by default

Quoted 36 lines · Jan Beulich
Quoted 27 lines · Roger Pau Monné
Quoted 26 lines · Andrew Cooper
xen/arch/x86/cpu/common.c +20 -0
@@ -209,6 +209,34 @@ void ctxt_switch_levelling(const struct vcpu *next)
 		alternative_vcall(ctxt_switch_masking, next);
 }
 
+bool __ro_after_init opt_doitm = true;
+
+static void doitm_init(void)
+{
+    uint64_t val;
+
+    if ( !opt_doitm || !cpu_has_arch_caps )
+        return;
+
+    rdmsrl(MSR_ARCH_CAPABILITIES, val);
+    if ( !(val & ARCH_CAPS_DOITM) )
+        return;
+
+    /*
+     * We are currently unable to enumerate MSR_ARCH_CAPS to guest.  As a
+     * consequence, guest kernels will believe they're safe even when they are
+     * not.
+     *
+     * Until we can enumerate DOITM properly for guests, set it unilaterally.
+     * This prevents otherwise-correct crypto from becoming vulnerable to
Quoted 8 lines

xen/arch/x86/cpu/common.c void ctxt_switch_levelling(const struct vcpu *next) · line 237

Quoted 6 lines
xen/arch/x86/cpu/common.c +6 -0
+     * timing sidechannels.
+     */
+
+    rdmsrl(MSR_UARCH_MISC_CTRL, val);
+    val |= UARCH_CTRL_DOITM;
+    wrmsrl(MSR_UARCH_MISC_CTRL, val);

Is it possible for the firmware to have enabled DOITM and Xen needing to clear it if !opt_doitm?

Quoted 4 lines

I think a firmware setup option is quite plausible to expect, such that safety can also be achieved underneath an unaware OS. Note how in my earlier patch I did specifically set the bit both ways, for this very reason.

Firmware is not likely to set it, but we should cope with the case when we're somewhere along a kexec chain.

I'll adjust.

~Andrew