Skip to contentSkip to conversation

target/i386: add support for FRED

PATCH Superseded Xin Li 3 participants 8 messages Sep 1, 2023 – Sep 26, 2023 KVM

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Xin Li

author
#0 lore raw

cover target/i386: add support for FRED

This patch set adds support for the Intel flexible return and event delivery (FRED) architecture to allow Qemu to run KVM guests with FRED enabled.

The FRED architecture defines simple new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context.

Search for the latest FRED spec in most search engines with this search pattern:

site:intel.com FRED (flexible return and event delivery) specification

Xin Li (4):
target/i386: add support for FRED in CPUID enumeration target/i386: mark CR4.FRED not reserved target/i386: enumerate VMX nested-exception support target/i386: add live migration support for FRED

 target/i386/cpu.c     |  7 +++++-
 target/i386/cpu.h     | 40 +++++++++++++++++++++++++++++++-
 target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
 target/i386/machine.c | 10 ++++++++
 4 files changed, 109 insertions(+), 2 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/4 target/i386: add support for FRED in CPUID enumeration

FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).

In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.

The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS.

Add CPUID definitions for FRED/LKGS, and expose them to KVM guests only.

Because FRED relies on LKGS, add it to feature dependency map.

---
 target/i386/cpu.c | 6 +++++-
 target/i386/cpu.h | 4 ++++
 2 files changed, 9 insertions(+), 1 deletion(-)
target/i386/cpu.c +5 -1
index 00f913b638..3dba6b46d9 100644
@@ -963,7 +963,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "avx-vnni", "avx512-bf16", NULL, "cmpccxadd",
             NULL, NULL, "fzrm", "fsrs",
             "fsrc", NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
+            NULL, "fred", "lkgs", NULL,
             NULL, "amx-fp16", NULL, "avx-ifma",
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
@@ -1549,6 +1549,10 @@ static FeatureDep feature_dependencies[] = {
         .from = { FEAT_VMX_SECONDARY_CTLS,  VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE },
         .to = { FEAT_7_0_ECX,               CPUID_7_0_ECX_WAITPKG },
     },
+    {
+        .from = { FEAT_7_1_EAX,             CPUID_7_1_EAX_LKGS },
+        .to = { FEAT_7_1_EAX,               CPUID_7_1_EAX_FRED },
+    },
 };
 
 typedef struct X86RegisterInfo32 {
target/i386/cpu.h +4 -0
index a6000e93bd..064decbc85 100644
@@ -932,6 +932,10 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,
 #define CPUID_7_1_EDX_AVX_NE_CONVERT    (1U << 5)
 /* PREFETCHIT0/1 Instructions */
 #define CPUID_7_1_EDX_PREFETCHITI       (1U << 14)
+/* Flexible return and event delivery (FRED) */
+#define CPUID_7_1_EAX_FRED              (1U << 17)
+/* Load into IA32_KERNEL_GS_BASE (LKGS) */
+#define CPUID_7_1_EAX_LKGS              (1U << 18)
 
 /* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */
 #define CPUID_7_2_EDX_MCDT_NO           (1U << 5)
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/4 target/i386: mark CR4.FRED not reserved

In reply to Xin Li

The CR4.FRED bit, i.e., CR4[32], is no longer a reserved bit when FRED is exposed to guests, otherwise it is still a reserved bit.

---
 target/i386/cpu.h | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
target/i386/cpu.h +11 -1
index 064decbc85..924819a64c 100644
@@ -262,6 +262,12 @@ typedef enum X86Seg {
 #define CR4_PKE_MASK   (1U << 22)
 #define CR4_PKS_MASK   (1U << 24)
 
+#ifdef TARGET_X86_64
+#define CR4_FRED_MASK   (1ULL << 32)
+#else
+#define CR4_FRED_MASK   0
+#endif
+
 #define CR4_RESERVED_MASK \
 (~(target_ulong)(CR4_VME_MASK | CR4_PVI_MASK | CR4_TSD_MASK \
                 | CR4_DE_MASK | CR4_PSE_MASK | CR4_PAE_MASK \
@@ -269,7 +275,8 @@ typedef enum X86Seg {
                 | CR4_OSFXSR_MASK | CR4_OSXMMEXCPT_MASK | CR4_UMIP_MASK \
                 | CR4_LA57_MASK \
                 | CR4_FSGSBASE_MASK | CR4_PCIDE_MASK | CR4_OSXSAVE_MASK \
-                | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK))
+                | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK \
+                | CR4_FRED_MASK))
 
 #define DR6_BD          (1 << 13)
 #define DR6_BS          (1 << 14)
@@ -2481,6 +2488,9 @@ static inline uint64_t cr4_reserved_bits(CPUX86State *env)
     if (!(env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_PKS)) {
         reserved_bits |= CR4_PKS_MASK;
     }
+    if (!(env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED)) {
+        reserved_bits |= CR4_FRED_MASK;
+    }
     return reserved_bits;
 }
 
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/4 target/i386: enumerate VMX nested-exception support

In reply to Xin Li

Allow VMX nested-exception support to be exposed in KVM guests, thus nested KVM guests can enumerate it.

---
 target/i386/cpu.c | 1 +
 1 file changed, 1 insertion(+)
target/i386/cpu.c +1 -0
index 3dba6b46d9..ba579e1fb7 100644
@@ -1340,6 +1340,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .feat_names = {
             [54] = "vmx-ins-outs",
             [55] = "vmx-true-ctls",
+            [58] = "vmx-nested-exception",
         },
         .msr = {
             .index = MSR_IA32_VMX_BASIC,
Signature · 1 line
2.34.1
maintainer
#4 lore raw

Re: [PATCH 3/4] target/i386: enumerate VMX nested-exception support

Quoted 14 lines · Xin Li

Allow VMX nested-exception support to be exposed in KVM guests, thus nested KVM guests can enumerate it.

---
  target/i386/cpu.c | 1 +
  1 file changed, 1 insertion(+)
target/i386/cpu.c +0 -0
index 3dba6b46d9..ba579e1fb7 100644

target/i386/cpu.c FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 1346

Quoted 7 lines
target/i386/cpu.c +1 -0
@@ -1340,6 +1340,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
          .feat_names = {
              [54] = "vmx-ins-outs",
              [55] = "vmx-true-ctls",
+            [58] = "vmx-nested-exception",
          },
          .msr = {
              .index = MSR_IA32_VMX_BASIC,

Please also add it to scripts/kvm/vmxcap, and rebase on top of the recent introduction of MSR_VMX_BASIC_ANY_ERRCODE.

Paolo

author
#5 lore raw

PATCH 4/4 target/i386: add live migration support for FRED

In reply to Xin Li

FRED CPU states are managed in 10 FRED MSRs, in addtion to a few existing CPU registers and MSRs, e.g., the CR4.FRED bit.

Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.

---
 target/i386/cpu.h     | 24 +++++++++++++++++++
 target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
 target/i386/machine.c | 10 ++++++++
 3 files changed, 88 insertions(+)
target/i386/cpu.h +24 -0
index 924819a64c..a36a1a58c4 100644
@@ -529,6 +529,20 @@ typedef enum X86Seg {
 #define MSR_IA32_XFD                    0x000001c4
 #define MSR_IA32_XFD_ERR                0x000001c5
 
+#define MSR_IA32_PL0_SSP                0x000006a4       /* Stack level 0 shadow stack pointer in ring 0 */
+
+/* FRED MSRs */
+#define MSR_IA32_FRED_RSP0              0x000001cc       /* Stack level 0 regular stack pointer */
+#define MSR_IA32_FRED_RSP1              0x000001cd       /* Stack level 1 regular stack pointer */
+#define MSR_IA32_FRED_RSP2              0x000001ce       /* Stack level 2 regular stack pointer */
+#define MSR_IA32_FRED_RSP3              0x000001cf       /* Stack level 3 regular stack pointer */
+#define MSR_IA32_FRED_STKLVLS           0x000001d0       /* FRED exception stack levels */
+#define MSR_IA32_FRED_SSP0              MSR_IA32_PL0_SSP /* Stack level 0 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP1              0x000001d1       /* Stack level 1 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP2              0x000001d2       /* Stack level 2 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP3              0x000001d3       /* Stack level 3 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_CONFIG            0x000001d4       /* FRED Entrypoint and interrupt stack level */
+
 #define MSR_IA32_BNDCFGS                0x00000d90
 #define MSR_IA32_XSS                    0x00000da0
 #define MSR_IA32_UMWAIT_CONTROL         0xe1
@@ -1680,6 +1694,16 @@ typedef struct CPUArchState {
     target_ulong cstar;
     target_ulong fmask;
     target_ulong kernelgsbase;
+    target_ulong fred_rsp0;
+    target_ulong fred_rsp1;
+    target_ulong fred_rsp2;
+    target_ulong fred_rsp3;
+    target_ulong fred_stklvls;
+    target_ulong fred_ssp0;
+    target_ulong fred_ssp1;
+    target_ulong fred_ssp2;
+    target_ulong fred_ssp3;
+    target_ulong fred_config;
 #endif
 
     uint64_t tsc_adjust;
target/i386/kvm/kvm.c +54 -0

72 lines not on this page · raw message

target/i386/machine.c +10 -0
index c7ac8084b2..5c722a49c5 100644
@@ -1652,6 +1652,16 @@ const VMStateDescription vmstate_x86_cpu = {
         VMSTATE_UINT64(env.cstar, X86CPU),
         VMSTATE_UINT64(env.fmask, X86CPU),
         VMSTATE_UINT64(env.kernelgsbase, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp0, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp1, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp2, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp3, X86CPU),
+        VMSTATE_UINT64(env.fred_stklvls, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp0, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp1, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp2, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp3, X86CPU),
+        VMSTATE_UINT64(env.fred_config, X86CPU),
 #endif
         VMSTATE_UINT32(env.smbase, X86CPU),
 
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH 4/4] target/i386: add live migration support for FRED

Quoted 145 lines · On 9/1/2023 1:30 PM, Li, Xin3 wrote:

FRED CPU states are managed in 10 FRED MSRs, in addtion to a few existing CPU registers and MSRs, e.g., the CR4.FRED bit.

Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.

---
  target/i386/cpu.h     | 24 +++++++++++++++++++
  target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
  target/i386/machine.c | 10 ++++++++
  3 files changed, 88 insertions(+)
target/i386/cpu.h +24 -0
index 924819a64c..a36a1a58c4 100644
@@ -529,6 +529,20 @@ typedef enum X86Seg {
  #define MSR_IA32_XFD                    0x000001c4
  #define MSR_IA32_XFD_ERR                0x000001c5
 
+#define MSR_IA32_PL0_SSP                0x000006a4       /* Stack level 0 shadow stack pointer in ring 0 */
+
+/* FRED MSRs */
+#define MSR_IA32_FRED_RSP0              0x000001cc       /* Stack level 0 regular stack pointer */
+#define MSR_IA32_FRED_RSP1              0x000001cd       /* Stack level 1 regular stack pointer */
+#define MSR_IA32_FRED_RSP2              0x000001ce       /* Stack level 2 regular stack pointer */
+#define MSR_IA32_FRED_RSP3              0x000001cf       /* Stack level 3 regular stack pointer */
+#define MSR_IA32_FRED_STKLVLS           0x000001d0       /* FRED exception stack levels */
+#define MSR_IA32_FRED_SSP0              MSR_IA32_PL0_SSP /* Stack level 0 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP1              0x000001d1       /* Stack level 1 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP2              0x000001d2       /* Stack level 2 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP3              0x000001d3       /* Stack level 3 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_CONFIG            0x000001d4       /* FRED Entrypoint and interrupt stack level */
+
  #define MSR_IA32_BNDCFGS                0x00000d90
  #define MSR_IA32_XSS                    0x00000da0
  #define MSR_IA32_UMWAIT_CONTROL         0xe1
@@ -1680,6 +1694,16 @@ typedef struct CPUArchState {
      target_ulong cstar;
      target_ulong fmask;
      target_ulong kernelgsbase;
+    target_ulong fred_rsp0;
+    target_ulong fred_rsp1;
+    target_ulong fred_rsp2;
+    target_ulong fred_rsp3;
+    target_ulong fred_stklvls;
+    target_ulong fred_ssp0;
+    target_ulong fred_ssp1;
+    target_ulong fred_ssp2;
+    target_ulong fred_ssp3;
+    target_ulong fred_config;
  #endif
 
      uint64_t tsc_adjust;
target/i386/kvm/kvm.c +54 -0
index 639a242ad8..4b241c82d8 100644
@@ -3401,6 +3401,18 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
          kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, env->kernelgsbase);
          kvm_msr_entry_add(cpu, MSR_FMASK, env->fmask);
          kvm_msr_entry_add(cpu, MSR_LSTAR, env->lstar);
+        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, env->fred_rsp0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, env->fred_rsp1);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, env->fred_rsp2);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, env->fred_rsp3);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, env->fred_stklvls);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, env->fred_ssp0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, env->fred_ssp1);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, env->fred_ssp2);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, env->fred_ssp3);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, env->fred_config);
+        }
      }
  #endif
 
@@ -3901,6 +3913,18 @@ static int kvm_get_msrs(X86CPU *cpu)
          kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, 0);
          kvm_msr_entry_add(cpu, MSR_FMASK, 0);
          kvm_msr_entry_add(cpu, MSR_LSTAR, 0);
+        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, 0);
+        }
      }
  #endif
      kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, 0);
@@ -4123,6 +4147,36 @@ static int kvm_get_msrs(X86CPU *cpu)
          case MSR_LSTAR:
              env->lstar = msrs[i].data;
              break;
+        case MSR_IA32_FRED_RSP0:
+            env->fred_rsp0 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP1:
+            env->fred_rsp1 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP2:
+            env->fred_rsp2 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP3:
+            env->fred_rsp3 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_STKLVLS:
+            env->fred_stklvls = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP0:
+            env->fred_ssp0 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP1:
+            env->fred_ssp1 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP2:
+            env->fred_ssp2 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP3:
+            env->fred_ssp3 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_CONFIG:
+            env->fred_config = msrs[i].data;
+            break;
  #endif
          case MSR_IA32_TSC:
              env->tsc = msrs[i].data;
target/i386/machine.c +3 -0
index c7ac8084b2..5c722a49c5 100644
@@ -1652,6 +1652,16 @@ const VMStateDescription vmstate_x86_cpu = {
          VMSTATE_UINT64(env.cstar, X86CPU),
          VMSTATE_UINT64(env.fmask, X86CPU),
          VMSTATE_UINT64(env.kernelgsbase, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp0, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp1, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp2, X86CPU),

target/i386/machine.c const VMStateDescription vmstate_x86_cpu = { · line 1667

Quoted 10 lines
target/i386/machine.c +7 -0
+        VMSTATE_UINT64(env.fred_rsp3, X86CPU),
+        VMSTATE_UINT64(env.fred_stklvls, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp0, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp1, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp2, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp3, X86CPU),
+        VMSTATE_UINT64(env.fred_config, X86CPU),
  #endif
          VMSTATE_UINT32(env.smbase, X86CPU),
 

IMO, it's better to split this patch into two, one is for FRED MSR access, the other is for LM support.

And move the FRED MSRs from basic x86_cpu part to .subsections part because FRED doesn't belong to basic CPU registers.  In the .subsections part, you may define a struct to hold FRED MSRs then add  .needed callback helper for QEMU to query whether FRED MSRs are necessary for live migration.

See other features, e.g., vmstate_msr_intel_sgx

author
#7 lore raw

RE: [PATCH 4/4] target/i386: add live migration support for FRED

Quoted 6 lines
Quoted 5 lines

FRED CPU states are managed in 10 FRED MSRs, in addtion to a few existing CPU registers and MSRs, e.g., the CR4.FRED bit.

Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.

Quoted 7 lines

IMO, it's better to split this patch into two, one is for FRED MSR access, the other is for LM support.

And move the FRED MSRs from basic x86_cpu part to .subsections part because FRED doesn't belong to basic CPU registers.  In the .subsections part, you may define a struct to hold FRED MSRs then add  .needed callback helper for QEMU to query whether FRED MSRs are necessary for live migration.

Good point!

Quoted 2 lines

See other features, e.g., vmstate_msr_intel_sgx