Skip to contentSkip to conversation

hvm/apic: repurpose the reporting of the APIC assist options

PATCH In review Roger Pau Monne 5 participants 16 messages Nov 4, 2022 – Nov 14, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 hvm/apic: repurpose the reporting of the APIC assist options

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware virtualized APIC')

---
I find the logic in vmx_vlapic_msr_changed() hard to follow, but I don't want to rewrite the function logic at this point.
---
Changes since v1:
- Fix Viridian MSR tip conditions.

---
 xen/arch/x86/hvm/viridian/viridian.c |  2 +-
 xen/arch/x86/hvm/vmx/vmcs.c          |  8 ++++----
 xen/arch/x86/hvm/vmx/vmx.c           | 25 ++++++++++++++++++-------
 xen/arch/x86/traps.c                 |  4 +---
 4 files changed, 24 insertions(+), 15 deletions(-)
xen/arch/x86/hvm/viridian/viridian.c +1 -1
index 25dca93e8b..44eb3d0519 100644
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;
         if ( viridian_feature_mask(d) & HVMPV_hcall_ipi )
             res->a |= CPUID4A_SYNTHETIC_CLUSTER_IPI;
xen/arch/x86/hvm/vmx/vmcs.c +4 -4
index a1aca1ec04..7bb96e1a8e 100644
@@ -1136,7 +1136,7 @@ static int construct_vmcs(struct vcpu *v)
 
     if ( !has_assisted_xapic(d) )
         v->arch.hvm.vmx.secondary_exec_control &=
-            ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+            ~SECONDARY_EXEC_APIC_REGISTER_VIRT;
 
     if ( cpu_has_vmx_secondary_exec_control )
         __vmwrite(SECONDARY_VM_EXEC_CONTROL,
@@ -2156,10 +2156,10 @@ int __init vmx_vmcs_init(void)
     if ( !ret )
     {
         /* Check whether hardware supports accelerated xapic and x2apic. */
-        assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+        assisted_xapic_available = cpu_has_vmx_tpr_shadow &&
+                                   cpu_has_vmx_apic_reg_virt;
         assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
-                                    (cpu_has_vmx_apic_reg_virt ||
-                                     cpu_has_vmx_virtual_intr_delivery);
+                                    cpu_has_vmx_apic_reg_virt;
         register_keyhandler('v', vmcs_dump, "dump VT-x VMCSs", 1);
     }
 
xen/arch/x86/hvm/vmx/vmx.c +18 -7
index e624b415c9..bf0fe3355c 100644
@@ -3405,25 +3405,29 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)
 
 void vmx_vlapic_msr_changed(struct vcpu *v)
 {
+    bool virtualize_x2apic_mode = has_assisted_x2apic(v->domain) ||
+                                  (cpu_has_vmx_virtualize_x2apic_mode &&
+                                   cpu_has_vmx_virtual_intr_delivery);
     struct vlapic *vlapic = vcpu_vlapic(v);
     unsigned int msr;
 
-    if ( !has_assisted_xapic(v->domain) &&
-         !has_assisted_x2apic(v->domain) )
+    if ( !cpu_has_vmx_virtualize_apic_accesses &&
+         !virtualize_x2apic_mode )
         return;
 
     vmx_vmcs_enter(v);
     v->arch.hvm.vmx.secondary_exec_control &=
         ~(SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
-          SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE);
+          SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE |
+          SECONDARY_EXEC_APIC_REGISTER_VIRT);
     if ( !vlapic_hw_disabled(vlapic) &&
          (vlapic_base_address(vlapic) == APIC_DEFAULT_PHYS_BASE) )
     {
-        if ( has_assisted_x2apic(v->domain) && vlapic_x2apic_mode(vlapic) )
+        if ( virtualize_x2apic_mode && vlapic_x2apic_mode(vlapic) )
         {
             v->arch.hvm.vmx.secondary_exec_control |=
                 SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE;
-            if ( cpu_has_vmx_apic_reg_virt )
+            if ( has_assisted_x2apic(v->domain) )
             {
                 for ( msr = MSR_X2APIC_FIRST;
                       msr <= MSR_X2APIC_LAST; msr++ )
@@ -3432,6 +3436,10 @@ void vmx_vlapic_msr_changed(struct vcpu *v)
                 vmx_set_msr_intercept(v, MSR_X2APIC_PPR, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMICT, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMCCT, VMX_MSR_R);
+
+                v->arch.hvm.vmx.secondary_exec_control |=
+                    SECONDARY_EXEC_APIC_REGISTER_VIRT;
+
             }
             if ( cpu_has_vmx_virtual_intr_delivery )
             {
@@ -3440,9 +3448,12 @@ void vmx_vlapic_msr_changed(struct vcpu *v)
                 vmx_clear_msr_intercept(v, MSR_X2APIC_SELF, VMX_MSR_W);
             }
         }
-        else if ( has_assisted_xapic(v->domain) )
+        else if ( vlapic_xapic_mode(vlapic) )
             v->arch.hvm.vmx.secondary_exec_control |=
-                SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+                (cpu_has_vmx_virtualize_apic_accesses ?
+                    SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES : 0) |
+                (has_assisted_xapic(v->domain) ?
+                    SECONDARY_EXEC_APIC_REGISTER_VIRT : 0);
     }
     if ( !(v->arch.hvm.vmx.secondary_exec_control &
            SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE) )
xen/arch/x86/traps.c +1 -3
index 7207390118..5c0aabe8a3 100644
@@ -1124,8 +1124,7 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
         if ( !is_hvm_domain(d) || subleaf != 0 )
             break;
 
-        if ( cpu_has_vmx_apic_reg_virt &&
-             has_assisted_xapic(d) )
+        if ( has_assisted_xapic(d) )
             res->a |= XEN_HVM_CPUID_APIC_ACCESS_VIRT;
 
         /*
@@ -1135,7 +1134,6 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
          * vmx_vlapic_msr_changed()).
          */
         if ( has_assisted_x2apic(d) &&
-             cpu_has_vmx_apic_reg_virt &&
              cpu_has_vmx_virtual_intr_delivery )
             res->a |= XEN_HVM_CPUID_X2APIC_VIRT;
 
Signature · 1 line
2.37.3
Reviewed-by on v2 1/1
#1 lore raw
Quoted 17 lines · Roger Pau Monne

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Quoted 8 lines
Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware virtualized APIC')

---
I find the logic in vmx_vlapic_msr_changed() hard to follow, but I don't want to rewrite the function logic at this point.
---
Changes since v1:
- Fix Viridian MSR tip conditions.

#2 lore raw

Hi Roger,

Quoted 21 lines

Subject: Re: [PATCH for-4.17 v2] hvm/apic: repurpose the reporting of the APIC assist options

Quoted 17 lines · Roger Pau Monne

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Quoted 8 lines
Quoted 1 line
Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware

virtualized APIC')

Quoted 4 lines

---
Changes since v1:
- Fix Viridian MSR tip conditions.

Kind regards,
Henry

#3 lore raw
In reply to Roger Pau Monne

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

Isn't this too restrictive when considering x2APIC? IOW is there anything wrong with leaving this as is?

xen/arch/x86/hvm/viridian/viridian.c void vmx_vlapic_msr_changed(struct vcpu *v) · line 3443

Quoted 9 lines
xen/arch/x86/hvm/viridian/viridian.c +4 -0
@@ -3432,6 +3436,10 @@ void vmx_vlapic_msr_changed(struct vcpu *v)
                 vmx_set_msr_intercept(v, MSR_X2APIC_PPR, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMICT, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMCCT, VMX_MSR_R);
+
+                v->arch.hvm.vmx.secondary_exec_control |=
+                    SECONDARY_EXEC_APIC_REGISTER_VIRT;
+
             }

Nit: stray trailing blank line inside the block.

Everything else looks plausible to me, but from prior discussion I wonder whether the result isn't still going to be too coarse grained for Andrew's taste.

Jan

#4 lore raw
Quoted 13 lines · Jan Beulich

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

Isn't this too restrictive when considering x2APIC? IOW is there anything wrong with leaving this as is?

Using cpu_has_vmx_apic_reg_virt won't be correct, as a domain can have it disabled now after this change.

When using x2APIC accesses will already be done using MSRs, so the hint is not useful in that mode.

Quoted 11 lines

xen/arch/x86/hvm/viridian/viridian.c void vmx_vlapic_msr_changed(struct vcpu *v) · line 3443

Quoted 9 lines
xen/arch/x86/hvm/viridian/viridian.c +4 -0
@@ -3432,6 +3436,10 @@ void vmx_vlapic_msr_changed(struct vcpu *v)
                 vmx_set_msr_intercept(v, MSR_X2APIC_PPR, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMICT, VMX_MSR_R);
                 vmx_set_msr_intercept(v, MSR_X2APIC_TMCCT, VMX_MSR_R);
+
+                v->arch.hvm.vmx.secondary_exec_control |=
+                    SECONDARY_EXEC_APIC_REGISTER_VIRT;
+
             }

Nit: stray trailing blank line inside the block.

Oh, thanks. I will wait for Andrews feedback then, I think the extra blank can likely be removed at commit if we agree this is OK.

Quoted 3 lines

Everything else looks plausible to me, but from prior discussion I wonder whether the result isn't still going to be too coarse grained for Andrew's taste.

Ack, thanks, I think this is the best that we can do given the status of the release, but would likely need to be quick or else it's gonna be too late.

Roger.

#5 lore raw
In reply to Jan Beulich

Hi Andrew,

Quoted 7 lines

Subject: Re: [PATCH for-4.17 v2] hvm/apic: repurpose the reporting of the APIC assist options

Everything else looks plausible to me, but from prior discussion I wonder whether the result isn't still going to be too coarse grained for Andrew's taste.

If you have time, would you mind providing any feedback on this patch so we can proceed based on your feedback? Thanks very much!

Kind regards,
Henry

Quoted 2 lines

Jan

#6 lore raw
In reply to Roger Pau Monne
Quoted 37 lines · Roger Pau Monne

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware virtualized APIC')

---
I find the logic in vmx_vlapic_msr_changed() hard to follow, but I don't want to rewrite the function logic at this point.
---
Changes since v1:
- Fix Viridian MSR tip conditions.

---
 xen/arch/x86/hvm/viridian/viridian.c |  2 +-
 xen/arch/x86/hvm/vmx/vmcs.c          |  8 ++++----
 xen/arch/x86/hvm/vmx/vmx.c           | 25 ++++++++++++++++++-------
 xen/arch/x86/traps.c                 |  4 +---
 4 files changed, 24 insertions(+), 15 deletions(-)
xen/arch/x86/hvm/viridian/viridian.c +0 -0
index 25dca93e8b..44eb3d0519 100644

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 6 lines
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

But doing this correctly needs a per-domain vintr setting, which we don't have yet.

It is marginally less broken with this change, than without, but that's not saying much.

Quoted 16 lines
xen/arch/x86/hvm/viridian/viridian.c +0 -0
         if ( viridian_feature_mask(d) & HVMPV_hcall_ipi )
             res->a |= CPUID4A_SYNTHETIC_CLUSTER_IPI;
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
index a1aca1ec04..7bb96e1a8e 100644
@@ -1136,7 +1136,7 @@ static int construct_vmcs(struct vcpu *v)
 
     if ( !has_assisted_xapic(d) )
         v->arch.hvm.vmx.secondary_exec_control &=
-            ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+            ~SECONDARY_EXEC_APIC_REGISTER_VIRT;
 
     if ( cpu_has_vmx_secondary_exec_control )
         __vmwrite(SECONDARY_VM_EXEC_CONTROL,

xen/arch/x86/hvm/vmx/vmcs.c int __init vmx_vmcs_init(void) · line 2162

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +3 -3
@@ -2156,10 +2156,10 @@ int __init vmx_vmcs_init(void)
     if ( !ret )
     {
         /* Check whether hardware supports accelerated xapic and x2apic. */
-        assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+        assisted_xapic_available = cpu_has_vmx_tpr_shadow &&
+                                   cpu_has_vmx_apic_reg_virt;
         assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
-                                    (cpu_has_vmx_apic_reg_virt ||
-                                     cpu_has_vmx_virtual_intr_delivery);
+                                    cpu_has_vmx_apic_reg_virt;

apic reg virt already depends on tpr shadow, so that part of the condition is redundant.

virtualise x2apic mode and apic reg virt aren't dependent, but they do only ever appear together in hardware.

Keeping the conditionals might be ok to combat a bad outer hypervisor, but ...

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmcs.c +0 -0
         register_keyhandler('v', vmcs_dump, "dump VT-x VMCSs", 1);
     }
 
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index e624b415c9..bf0fe3355c 100644

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3410

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -3405,25 +3405,29 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)
 
 void vmx_vlapic_msr_changed(struct vcpu *v)
 {
+    bool virtualize_x2apic_mode = has_assisted_x2apic(v->domain) ||
+                                  (cpu_has_vmx_virtualize_x2apic_mode &&
+                                   cpu_has_vmx_virtual_intr_delivery);

... this is still wrong, and ...

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3416

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -2
     struct vlapic *vlapic = vcpu_vlapic(v);
     unsigned int msr;
 
-    if ( !has_assisted_xapic(v->domain) &&
-         !has_assisted_x2apic(v->domain) )
+    if ( !cpu_has_vmx_virtualize_apic_accesses &&
+         !virtualize_x2apic_mode )
         return;

... this surely cannot be right.

While attempting to figure ^ out, I've found yet another regression vs 4.16.  Because virt intr delivery is set in the execution controls system-wide and not controlled per domain, we'll take a vmentry failure on SKX/CLX/ICX when trying to build an HVM domain without xAPIC acceleration.

This, combined with the ABI errors (/misfeatures) that we really don't want to escape into the world but I haven't finished fixing yet, means that the only appropriate course of action is to revert.

I'd really hoped to avoid a full revert, but we've run out of time.

~Andrew

#7 lore raw
Quoted 13 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

Jan

#8 lore raw
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

It is not by accident that "virtual interrupt delivery", introduced in IvyBridge, is exactly the missing registers (on top of "use TPR Shadow" which is even older) to make windows performance less bad.

~Andrew

#9 lore raw
Quoted 28 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

Quoted 3 lines

It is not by accident that "virtual interrupt delivery", introduced in IvyBridge, is exactly the missing registers (on top of "use TPR Shadow" which is even older) to make windows performance less bad.

Sorry, sent too early.

This also firmly highlights why the API/ABI is broken.  It has successfully fooled all the other maintainers into doing the wrong thing, even after recently discussing the complexity and subtly in full as part of the XSA-412 security fix.

~Andrew

#10 lore raw
Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

Quoted 7 lines
Quoted 3 lines

It is not by accident that "virtual interrupt delivery", introduced in IvyBridge, is exactly the missing registers (on top of "use TPR Shadow" which is even older) to make windows performance less bad.

Sorry, sent too early.

This also firmly highlights why the API/ABI is broken.

I'm not seeing how you are making this connection: the context here is strictly about a Viridian hint which Xen has been wrongly reporting, but has nothing to do with the APIC assist API/ABI stuff. It was wrong before the introduction of APIC assist, and it's also wrong after.

Also see my other reply - I'm dubious whether this hint is useful for any Windows version that supports x2APIC (which seems to be >= Windows Server 2008), because we expose x2APIC to guests regardless of whether the underlying platform APIC supports such mode.

Thanks, Roger.

#11 lore raw
Quoted 33 lines · Roger Pau Monne
Quoted 32 lines · Andrew Cooper
Quoted 31 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

It is not by accident that "virtual interrupt delivery", introduced in IvyBridge, is exactly the missing registers (on top of "use TPR Shadow" which is even older) to make windows performance less bad.

Quoted 8 lines
Quoted 3 lines

Sorry, sent too early.

This also firmly highlights why the API/ABI is broken.

I'm not seeing how you are making this connection: the context here is strictly about a Viridian hint which Xen has been wrongly reporting, but has nothing to do with the APIC assist API/ABI stuff. It was wrong before the introduction of APIC assist, and it's also wrong after.

And now it has a layer of obfuscation which makes the bug less obvious.

Quoted 4 lines

Also see my other reply - I'm dubious whether this hint is useful for any Windows version that supports x2APIC (which seems to be >= Windows Server 2008), because we expose x2APIC to guests regardless of whether the underlying platform APIC supports such mode.

It's not about whether a version of Windows supports x2APIC.  Its whether windows turns x2APIC on.

Short of having an emulated IOMMU, or an absence of an IO-APIC (neither of which we do), guests wont turn x2APIC on.

I know we have the magic hack for IO-APIC with >8 bit destinations, but that only got enumerated in the Xen leaves (IIRC?), so only Linux can pick it up.

The hint is still very relevant for any version of windows running in xAPIC mode which, at a guess, is all of them under Xen.

~Andrew

#12 lore raw
Quoted 36 lines · Andrew Cooper
Quoted 35 lines · Roger Pau Monne
Quoted 34 lines · Andrew Cooper
Quoted 28 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

Quoted 3 lines

It is not by accident that "virtual interrupt delivery", introduced in IvyBridge, is exactly the missing registers (on top of "use TPR Shadow" which is even older) to make windows performance less bad.

Sorry, sent too early.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

This also firmly highlights why the API/ABI is broken.

I'm not seeing how you are making this connection: the context here is strictly about a Viridian hint which Xen has been wrongly reporting, but has nothing to do with the APIC assist API/ABI stuff. It was wrong before the introduction of APIC assist, and it's also wrong after.

And now it has a layer of obfuscation which makes the bug less obvious.

Still, that's not an argument as to why the API/ABI is broken, just a remark that the current usage here needs fixing (which it does).

Quoted 7 lines
Quoted 4 lines

Also see my other reply - I'm dubious whether this hint is useful for any Windows version that supports x2APIC (which seems to be >= Windows Server 2008), because we expose x2APIC to guests regardless of whether the underlying platform APIC supports such mode.

It's not about whether a version of Windows supports x2APIC.  Its whether windows turns x2APIC on.

From my previous conversation with Paul I got the impression that Windows would choose x2APIC based solely on the CPUID bit:

https://lore.kernel.org/xen-devel/[email protected]/

Quoted 6 lines

Short of having an emulated IOMMU, or an absence of an IO-APIC (neither of which we do), guests wont turn x2APIC on.

I know we have the magic hack for IO-APIC with >8 bit destinations, but that only got enumerated in the Xen leaves (IIRC?), so only Linux can pick it up.

We don't have the hack yet, just the CPUID bit reserved.

Thanks, Roger.

#13 lore raw
In reply to Andrew Cooper
Quoted 23 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 9 lines · Roger Pau Monne
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 4 lines

To me this connection you suggest looks entirely unobvious, so would you mind expanding as to why you're thinking so? The hint to the guest here is related to how it would best access certain registers (aiui), which to me looks orthogonal to how interrupt delivery works.

I refer you again to the diagram.  (For everyone else on xen-devel, I put this together when fixing XSA-412 because Intel's APIC acceleration controls are entirely unintuitive.)

Quoted 5 lines

It is "virtual interrupt delivery" which controls EOI/ICR acceleration, and not "apic register virtualisation".  There's a decade worth of hardware where this logic is an anti-optimsiation, by telling windows to use a VMExit-ing mechanism even when microcode would have avoided the VMExit.

Okay, I agree that I was mislead by the name of the control. Together with Paul (re)clarifying what (virtual) MSRs the CPUID hint is about I agree that it wants to be "virtual interrupt delivery" alone which is checked here. Which in turn makes this a change orthogonal to the other APIC assist work.

Jan

#14 lore raw
In reply to Andrew Cooper
Quoted 125 lines · Andrew Cooper
Quoted 37 lines · Roger Pau Monne

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware virtualized APIC')

---
I find the logic in vmx_vlapic_msr_changed() hard to follow, but I don't want to rewrite the function logic at this point.
---
Changes since v1:
- Fix Viridian MSR tip conditions.

---
 xen/arch/x86/hvm/viridian/viridian.c |  2 +-
 xen/arch/x86/hvm/vmx/vmcs.c          |  8 ++++----
 xen/arch/x86/hvm/vmx/vmx.c           | 25 ++++++++++++++++++-------
 xen/arch/x86/traps.c                 |  4 +---
 4 files changed, 24 insertions(+), 15 deletions(-)
xen/arch/x86/hvm/viridian/viridian.c +0 -0
index 25dca93e8b..44eb3d0519 100644

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 6 lines
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

But doing this correctly needs a per-domain vintr setting, which we don't have yet.

It is marginally less broken with this change, than without, but that's not saying much.

Quoted 16 lines
xen/arch/x86/hvm/viridian/viridian.c +0 -0
         if ( viridian_feature_mask(d) & HVMPV_hcall_ipi )
             res->a |= CPUID4A_SYNTHETIC_CLUSTER_IPI;
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
index a1aca1ec04..7bb96e1a8e 100644
@@ -1136,7 +1136,7 @@ static int construct_vmcs(struct vcpu *v)
 
     if ( !has_assisted_xapic(d) )
         v->arch.hvm.vmx.secondary_exec_control &=
-            ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+            ~SECONDARY_EXEC_APIC_REGISTER_VIRT;
 
     if ( cpu_has_vmx_secondary_exec_control )
         __vmwrite(SECONDARY_VM_EXEC_CONTROL,

xen/arch/x86/hvm/vmx/vmcs.c int __init vmx_vmcs_init(void) · line 2162

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +3 -3
@@ -2156,10 +2156,10 @@ int __init vmx_vmcs_init(void)
     if ( !ret )
     {
         /* Check whether hardware supports accelerated xapic and x2apic. */
-        assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+        assisted_xapic_available = cpu_has_vmx_tpr_shadow &&
+                                   cpu_has_vmx_apic_reg_virt;
         assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
-                                    (cpu_has_vmx_apic_reg_virt ||
-                                     cpu_has_vmx_virtual_intr_delivery);
+                                    cpu_has_vmx_apic_reg_virt;

apic reg virt already depends on tpr shadow, so that part of the condition is redundant.

virtualise x2apic mode and apic reg virt aren't dependent, but they do only ever appear together in hardware.

Keeping the conditionals might be ok to combat a bad outer hypervisor, but ...

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmcs.c +0 -0
         register_keyhandler('v', vmcs_dump, "dump VT-x VMCSs", 1);
     }
 
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index e624b415c9..bf0fe3355c 100644

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3410

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -3405,25 +3405,29 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)
 
 void vmx_vlapic_msr_changed(struct vcpu *v)
 {
+    bool virtualize_x2apic_mode = has_assisted_x2apic(v->domain) ||
+                                  (cpu_has_vmx_virtualize_x2apic_mode &&
+                                   cpu_has_vmx_virtual_intr_delivery);

... this is still wrong, and ...

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3416

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -2
     struct vlapic *vlapic = vcpu_vlapic(v);
     unsigned int msr;
 
-    if ( !has_assisted_xapic(v->domain) &&
-         !has_assisted_x2apic(v->domain) )
+    if ( !cpu_has_vmx_virtualize_apic_accesses &&
+         !virtualize_x2apic_mode )
         return;

... this surely cannot be right.

While attempting to figure ^ out, I've found yet another regression vs 4.16.  Because virt intr delivery is set in the execution controls system-wide and not controlled per domain, we'll take a vmentry failure on SKX/CLX/ICX when trying to build an HVM domain without xAPIC acceleration.

Quoted 5 lines

This, combined with the ABI errors (/misfeatures) that we really don't want to escape into the world but I haven't finished fixing yet, means that the only appropriate course of action is to revert.

I'd really hoped to avoid a full revert, but we've run out of time.

Can we wait for the revert until Monday, it's a public holiday here today and won't be able to reply to the comments.

Thanks, Roger.

#15 lore raw
In reply to Andrew Cooper
Quoted 48 lines · Andrew Cooper
Quoted 37 lines · Roger Pau Monne

The current reporting of the hardware assisted APIC options is done by checking "virtualize APIC accesses" which is not very helpful, as that feature doesn't avoid a vmexit, instead it does provide some help in order to detect APIC MMIO accesses in vmexit processing.

Repurpose the current reporting of xAPIC assistance to instead report such feature as present when there's support for "TPR shadow" and "APIC register virtualization" because in that case some xAPIC MMIO register accesses are handled directly by the hardware, without requiring a vmexit.

For symetry also change assisted x2APIC reporting to require "virtualize x2APIC mode" and "APIC register virtualization", dropping the option to also be reported when "virtual interrupt delivery" is available. Presence of the "virtual interrupt delivery" feature will be reported using a different option.

Fixes 2ce11ce249 ('x86/HVM: allow per-domain usage of hardware virtualized APIC')

---
I find the logic in vmx_vlapic_msr_changed() hard to follow, but I don't want to rewrite the function logic at this point.
---
Changes since v1:
- Fix Viridian MSR tip conditions.

---
 xen/arch/x86/hvm/viridian/viridian.c |  2 +-
 xen/arch/x86/hvm/vmx/vmcs.c          |  8 ++++----
 xen/arch/x86/hvm/vmx/vmx.c           | 25 ++++++++++++++++++-------
 xen/arch/x86/traps.c                 |  4 +---
 4 files changed, 24 insertions(+), 15 deletions(-)
xen/arch/x86/hvm/viridian/viridian.c +0 -0
index 25dca93e8b..44eb3d0519 100644

xen/arch/x86/hvm/viridian/viridian.c void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf, · line 201

Quoted 6 lines
xen/arch/x86/hvm/viridian/viridian.c +1 -1
@@ -197,7 +197,7 @@ void cpuid_viridian_leaves(const struct vcpu *v, uint32_t leaf,
         res->a = CPUID4A_RELAX_TIMER_INT;
         if ( viridian_feature_mask(d) & HVMPV_hcall_remote_tlb_flush )
             res->a |= CPUID4A_HCALL_REMOTE_TLB_FLUSH;
-        if ( !cpu_has_vmx_apic_reg_virt )
+        if ( !has_assisted_xapic(d) )
             res->a |= CPUID4A_MSR_BASED_APIC;

This check is broken before and after.  It needs to be keyed on virtualised interrupt delivery, not register acceleration.

Quoted 5 lines

But doing this correctly needs a per-domain vintr setting, which we don't have yet.

It is marginally less broken with this change, than without, but that's not saying much.

I took a look at the HyperV TLFS spec but the table that lists the CPUID bits don't explicitly name which registers are accessed using MSRs rather than MMIO when the 'MSR_BASED_APIC' suggestion is set on CPUID.

It's my understanding that the hint is not useful anymore, as Xen exposes x2APIC by default, and that's what any new-ish version of Windows should use, in which case all APIC registers are already accessed using MSRs and the hint is moot.

Quoted 30 lines
Quoted 16 lines
xen/arch/x86/hvm/viridian/viridian.c +0 -0
         if ( viridian_feature_mask(d) & HVMPV_hcall_ipi )
             res->a |= CPUID4A_SYNTHETIC_CLUSTER_IPI;
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
index a1aca1ec04..7bb96e1a8e 100644
@@ -1136,7 +1136,7 @@ static int construct_vmcs(struct vcpu *v)
 
     if ( !has_assisted_xapic(d) )
         v->arch.hvm.vmx.secondary_exec_control &=
-            ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+            ~SECONDARY_EXEC_APIC_REGISTER_VIRT;
 
     if ( cpu_has_vmx_secondary_exec_control )
         __vmwrite(SECONDARY_VM_EXEC_CONTROL,

xen/arch/x86/hvm/vmx/vmcs.c int __init vmx_vmcs_init(void) · line 2162

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +3 -3
@@ -2156,10 +2156,10 @@ int __init vmx_vmcs_init(void)
     if ( !ret )
     {
         /* Check whether hardware supports accelerated xapic and x2apic. */
-        assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+        assisted_xapic_available = cpu_has_vmx_tpr_shadow &&
+                                   cpu_has_vmx_apic_reg_virt;
         assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
-                                    (cpu_has_vmx_apic_reg_virt ||
-                                     cpu_has_vmx_virtual_intr_delivery);
+                                    cpu_has_vmx_apic_reg_virt;

apic reg virt already depends on tpr shadow, so that part of the condition is redundant.

Quoted 2 lines

virtualise x2apic mode and apic reg virt aren't dependent, but they do only ever appear together in hardware.

I would keep those as-is for sanity, it's easier to spot the dependencies between them. And then it's also possible that we want to introduce a control for tpr_shadow, and which point having the conditional here is a good reminder that virtualize_apic_accesses depends on that feature being available and enabled.

Quoted 23 lines

Keeping the conditionals might be ok to combat a bad outer hypervisor, but ...

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmcs.c +0 -0
         register_keyhandler('v', vmcs_dump, "dump VT-x VMCSs", 1);
     }
 
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index e624b415c9..bf0fe3355c 100644

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3410

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -3405,25 +3405,29 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)
 
 void vmx_vlapic_msr_changed(struct vcpu *v)
 {
+    bool virtualize_x2apic_mode = has_assisted_x2apic(v->domain) ||
+                                  (cpu_has_vmx_virtualize_x2apic_mode &&
+                                   cpu_has_vmx_virtual_intr_delivery);

... this is still wrong, and ...

Quoted 3 lines
xen/arch/x86/hvm/vmx/vmx.c +0 -0
     struct vlapic *vlapic = vcpu_vlapic(v);
     unsigned int msr;
 
Quoted 7 lines

xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3416

Quoted 5 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -2
-    if ( !has_assisted_xapic(v->domain) &&
-         !has_assisted_x2apic(v->domain) )
+    if ( !cpu_has_vmx_virtualize_apic_accesses &&
+         !virtualize_x2apic_mode )
         return;

... this surely cannot be right.

It's indeed missing a has_assisted_xapic(v->domain), so it should be:

if ( !cpu_has_vmx_virtualize_apic_accesses &&

     !has_assisted_xapic(v->domain) &&
     !virtualize_x2apic_mode )
    return;

Logic in this function is not the best one IMO, as I've already mentioned in a post-commit remark.

Quoted 6 lines

While attempting to figure ^ out, I've found yet another regression vs 4.16.  Because virt intr delivery is set in the execution controls system-wide and not controlled per domain, we'll take a vmentry failure on SKX/CLX/ICX when trying to build an HVM domain without xAPIC acceleration.

I've tried creating the following guest on one of our Ice Lake boxes (with and without this patch applied):

type="pvh"
name = "test"
memory = 1024
vcpus = 1

kernel = "/root/vmlinuz-6.1.0-rc4+"
extra = "console=hvc0"

assisted_xapic=0

And it seem s to boot just fine, no vmentry failure.

Quoted 3 lines

This, combined with the ABI errors (/misfeatures) that we really don't want to escape into the world but I haven't finished fixing yet, means that the only appropriate course of action is to revert.

Hm, I'm confused by this, as it is still not clear to me what's wrong with the ABI. Is it the way in which we expose the flags? Or is it the naming?

AFAIK we want to have a flag to toggle apic_reg_virt, which is the proposal here. Should it be named differently?

Quoted 1 line

I'd really hoped to avoid a full revert, but we've run out of time.

I've also hoped so.

Thanks, Roger.