Skip to contentSkip to conversation

xen+tools: Report Interrupt Controller Virtualization capabilities on x86

PATCH In review Jane Malalane 3 participants 4 messages Jul 8, 2022 – Jul 11, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v11 1/2 xen+tools: Report Interrupt Controller Virtualization capabilities on x86

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the minimum, supports virtualize_apic_accesses as this feature alone means that an access to the APIC page will cause an APIC-access VM exit. An APIC-access VM exit provides a VMM with information about the access causing the VM exit, unlike a regular EPT fault, thus simplifying some internal handling.

HW assisted x2APIC virtualization will be reported if HW supports virtualize_x2apic_mode and, at least, either apic_reg_virt or virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that AVIC support can be introduced in a future patch. Unlike Intel that has multiple controls for APIC Virtualization, AMD has one global 'AVIC Enable' control bit, so fine-graining of APIC virtualization control cannot be done on a common interface.

---
CC: George Dunlap <[email protected]>
CC: Nick Rosbrook <[email protected]>
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Christian Lindig <[email protected]>
CC: David Scott <[email protected]>
CC: "Roger Pau Monné" <[email protected]>
CC: Jun Nakajima <[email protected]>
CC: Kevin Tian <[email protected]>

v11:
* explicity define arch_cap_list in stub_xc_physinfo for non-x86 arch

v10:
* Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()

v9:
* Move assisted_x{2}apic_available to vmx_vmcs_init() so they get declared at boot time, after vmx_secondary_exec_control is set

v8:
* Improve commit message

v7:
* Make sure assisted_x{2}apic_available evaluates to false, to ensure Xen builds, when !CONFIG_HVM
* Fix coding style issues

v6:
* Limit abi check to x86
* Fix coding style issue

v5:
* Have assisted_xapic_available solely depend on cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available depend on cpu_has_vmx_virtualize_x2apic_mode and cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
* Fallback to the original v2/v1 conditions for setting assisted_xapic_available and assisted_x2apic_available so that in the future APIC virtualization can be exposed on AMD hardware since fine-graining of "AVIC" is not supported, i.e., AMD solely uses "AVIC Enable". This also means that sysctl mimics what's exposed in CPUID

v3:
* Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
* Have assisted_x2apic_available only depend on cpu_has_vmx_virtualize_x2apic_mode

v2:
* Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
* Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
* Set assisted_x{2}apic_available to be conditional upon "bsp" and annotate it with __ro_after_init
* Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to _X86_ASSISTED_X{2}APIC
* Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within sysctl.h
* Fix padding introduced in struct xen_sysctl_physinfo and bump XEN_SYSCTL_INTERFACE_VERSION

---
 tools/golang/xenlight/helpers.gen.go |  4 ++++
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h                |  7 +++++++
 tools/libs/gnttab/st8ojiU6           |  0
 tools/libs/light/libxl.c             |  3 +++
 tools/libs/light/libxl_arch.h        |  4 ++++
 tools/libs/light/libxl_arm.c         |  5 +++++
 tools/libs/light/libxl_types.idl     |  2 ++
 tools/libs/light/libxl_x86.c         | 11 +++++++++++
 tools/ocaml/libs/xc/xenctrl.ml       |  9 +++++++++
 tools/ocaml/libs/xc/xenctrl.mli      |  8 ++++++++
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 18 ++++++++++++++++--
 tools/xl/xl_info.c                   |  6 ++++--
 xen/arch/x86/hvm/hvm.c               |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c          |  7 +++++++
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +++++
 xen/arch/x86/sysctl.c                |  4 ++++
 xen/include/public/sysctl.h          | 11 ++++++++++-
 18 files changed, 104 insertions(+), 5 deletions(-)
 create mode 100644 tools/libs/gnttab/st8ojiU6
tools/golang/xenlight/helpers.gen.go +4 -0
index b746ff1081..dd4e6c9f14 100644
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
tools/golang/xenlight/types.gen.go +2 -0
index b1e84d5258..87be46c745 100644
@@ -1014,6 +1014,8 @@ CapVmtrace bool
 CapVpmu bool
 CapGnttabV1 bool
 CapGnttabV2 bool
+CapAssistedXapic bool
+CapAssistedX2Apic bool
 }
 
 type Connectorinfo struct {
tools/include/libxl.h +7 -0
index 835dfabc50..3a68e1b55a 100644
@@ -536,6 +536,13 @@
 #define LIBXL_HAVE_NIC_TRUSTED 1
 
 /*
+ * LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has
+ * cap_assisted_xapic and cap_assisted_x2apic fields, which indicates
+ * the availability of x{2}APIC hardware assisted virtualization.
+ */
+#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
tools/libs/gnttab/st8ojiU6 new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tools/libs/light/libxl.c +3 -0
index a0bf7d186f..6d699951e2 100644
@@ -15,6 +15,7 @@
 #include "libxl_osdeps.h"
 
 #include "libxl_internal.h"
+#include "libxl_arch.h"
 
 int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags, xentoollog_logger * lg)
@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo)
     physinfo->cap_gnttab_v2 =
         !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2);
 
+    libxl__arch_get_physinfo(physinfo, &xcphysinfo);
+
     GC_FREE;
     return 0;
 }
tools/libs/light/libxl_arch.h +4 -0
index 1522ecb97f..207ceac6a1 100644
@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc,
                              uint64_t *out);
 
 _hidden
+void libxl__arch_get_physinfo(libxl_physinfo *physinfo,
+                              const xc_physinfo_t *xcphysinfo);
+
+_hidden
 void libxl__arch_update_domain_config(libxl__gc *gc,
                                       libxl_domain_config *dst,
                                       const libxl_domain_config *src);
tools/libs/light/libxl_arm.c +5 -0
index eef1de0939..39fdca1b49 100644
@@ -1431,6 +1431,11 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
     return rc;
 }
 
+void libxl__arch_get_physinfo(libxl_physinfo *physinfo,
+                              const xc_physinfo_t *xcphysinfo)
+{
+}
+
 void libxl__arch_update_domain_config(libxl__gc *gc,
                                       libxl_domain_config *dst,
                                       const libxl_domain_config *src)
tools/libs/light/libxl_types.idl +2 -0
index 89962218b4..47f6a30cde 100644
@@ -1070,6 +1070,8 @@ libxl_physinfo = Struct("physinfo", [
     ("cap_vpmu", bool),
     ("cap_gnttab_v1", bool),
     ("cap_gnttab_v2", bool),
+    ("cap_assisted_xapic", bool),
+    ("cap_assisted_x2apic", bool),
     ], dir=DIR_OUT)
 
 libxl_connectorinfo = Struct("connectorinfo", [
tools/libs/light/libxl_x86.c +11 -0
index 1feadebb18..e0a06ecfe3 100644
@@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
     return rc;
 }
 
+void libxl__arch_get_physinfo(libxl_physinfo *physinfo,
+                              const xc_physinfo_t *xcphysinfo)
+{
+    physinfo->cap_assisted_xapic =
+        !!(xcphysinfo->arch_capabilities &
+           XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC);
+    physinfo->cap_assisted_x2apic =
+        !!(xcphysinfo->arch_capabilities &
+           XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC);
+}
+
 void libxl__arch_update_domain_config(libxl__gc *gc,
                                       libxl_domain_config *dst,
                                       const libxl_domain_config *src)
tools/ocaml/libs/xc/xenctrl.ml +9 -0
index 8eab6f60eb..6fa30ddb56 100644
@@ -128,6 +128,14 @@ type physinfo_cap_flag =
 	| CAP_Gnttab_v1
 	| CAP_Gnttab_v2
 
+
+type x86_physinfo_arch_cap_flag =
+	| CAP_X86_ASSISTED_XAPIC
+	| CAP_X86_ASSISTED_X2APIC
+
+type physinfo_arch_cap_flag =
+	| X86 of x86_physinfo_arch_cap_flag
+
 type physinfo =
 {
 	threads_per_core : int;
@@ -141,6 +149,7 @@ type physinfo =
 	(* XXX hw_cap *)
 	capabilities     : physinfo_cap_flag list;
 	max_nr_cpus      : int;
+	arch_capabilities : physinfo_arch_cap_flag list;
 }
 
 type version =
tools/ocaml/libs/xc/xenctrl.mli +8 -0
index d3014a2708..01774da768 100644
@@ -113,6 +113,13 @@ type physinfo_cap_flag =
   | CAP_Gnttab_v1
   | CAP_Gnttab_v2
 
+type x86_physinfo_arch_cap_flag =
+  | CAP_X86_ASSISTED_XAPIC
+  | CAP_X86_ASSISTED_X2APIC
+
+type physinfo_arch_cap_flag =
+  | X86 of x86_physinfo_arch_cap_flag
+
 type physinfo = {
   threads_per_core : int;
   cores_per_socket : int;
@@ -124,6 +131,7 @@ type physinfo = {
   scrub_pages      : nativeint;
   capabilities     : physinfo_cap_flag list;
   max_nr_cpus      : int; (** compile-time max possible number of nr_cpus *)
+  arch_capabilities : physinfo_arch_cap_flag list;
 }
 type version = { major : int; minor : int; extra : string; }
 type compile_info = {
tools/ocaml/libs/xc/xenctrl_stubs.c +16 -2
index 513ee142d2..42c2bcd333 100644
@@ -718,7 +718,7 @@ CAMLprim value stub_xc_send_debug_keys(value xch, value keys)
 CAMLprim value stub_xc_physinfo(value xch)
 {
 	CAMLparam1(xch);
-	CAMLlocal2(physinfo, cap_list);
+	CAMLlocal4(physinfo, cap_list, x86_arch_cap_list, arch_cap_list);
 	xc_physinfo_t c_physinfo;
 	int r;
 
@@ -737,7 +737,7 @@ CAMLprim value stub_xc_physinfo(value xch)
 		/* ! XEN_SYSCTL_PHYSCAP_ XEN_SYSCTL_PHYSCAP_MAX max */
 		(c_physinfo.capabilities);
 
-	physinfo = caml_alloc_tuple(10);
+	physinfo = caml_alloc_tuple(11);
 	Store_field(physinfo, 0, Val_int(c_physinfo.threads_per_core));
 	Store_field(physinfo, 1, Val_int(c_physinfo.cores_per_socket));
 	Store_field(physinfo, 2, Val_int(c_physinfo.nr_cpus));
@@ -749,6 +749,20 @@ CAMLprim value stub_xc_physinfo(value xch)
 	Store_field(physinfo, 8, cap_list);
 	Store_field(physinfo, 9, Val_int(c_physinfo.max_cpu_id + 1));
 
+#if defined(__i386__) || defined(__x86_64__)
+	x86_arch_cap_list = c_bitmap_to_ocaml_list
+		/* ! x86_physinfo_arch_cap_flag CAP_X86_ none */
+		/* ! XEN_SYSCTL_PHYSCAP_X86_ XEN_SYSCTL_PHYSCAP_X86_MAX max */
+		(c_physinfo.arch_capabilities);
+	/*
+	 * arch_capabilities: physinfo_arch_cap_flag list;
+	 */
+	arch_cap_list = x86_arch_cap_list;
+#else
+	arch_cap_list = Val_emptylist;
+#endif
+	Store_field(physinfo, 10, arch_cap_list);
+
 	CAMLreturn(physinfo);
 }
 
tools/xl/xl_info.c +4 -2
index 712b7638b0..3205270754 100644
@@ -210,7 +210,7 @@ static void output_physinfo(void)
          info.hw_cap[4], info.hw_cap[5], info.hw_cap[6], info.hw_cap[7]
         );
 
-    maybe_printf("virt_caps              :%s%s%s%s%s%s%s%s%s%s%s\n",
+    maybe_printf("virt_caps              :%s%s%s%s%s%s%s%s%s%s%s%s%s\n",
          info.cap_pv ? " pv" : "",
          info.cap_hvm ? " hvm" : "",
          info.cap_hvm && info.cap_hvm_directio ? " hvm_directio" : "",
@@ -221,7 +221,9 @@ static void output_physinfo(void)
          info.cap_vmtrace ? " vmtrace" : "",
          info.cap_vpmu ? " vpmu" : "",
          info.cap_gnttab_v1 ? " gnttab-v1" : "",
-         info.cap_gnttab_v2 ? " gnttab-v2" : ""
+         info.cap_gnttab_v2 ? " gnttab-v2" : "",
+         info.cap_assisted_xapic ? " assisted_xapic" : "",
+         info.cap_assisted_x2apic ? " assisted_x2apic" : ""
         );
 
     vinfo = libxl_get_version_info(ctx);
xen/arch/x86/hvm/hvm.c +3 -0
index 5b16fb4cd8..0a32a948db 100644
@@ -117,6 +117,9 @@ static const char __initconst warning_hvm_fep[] =
 static bool_t __initdata opt_altp2m_enabled = 0;
 boolean_param("altp2m", opt_altp2m_enabled);
 
+bool __ro_after_init assisted_xapic_available;
+bool __ro_after_init assisted_x2apic_available;
+
 static int cf_check cpu_callback(
     struct notifier_block *nfb, unsigned long action, void *hcpu)
 {
xen/arch/x86/hvm/vmx/vmcs.c +7 -0
index 56fed2db03..7329622dd4 100644
@@ -2146,7 +2146,14 @@ int __init vmx_vmcs_init(void)
     ret = _vmx_cpu_up(true);
 
     if ( !ret )
+    {
+        /* Check whether hardware supports accelerated xapic and x2apic. */
+        assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+        assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
+                                    (cpu_has_vmx_apic_reg_virt ||
+                                     cpu_has_vmx_virtual_intr_delivery);
         register_keyhandler('v', vmcs_dump, "dump VT-x VMCSs", 1);
+    }
 
     return ret;
 }
xen/arch/x86/include/asm/hvm/hvm.h +5 -0
index caaeacabc7..8d162b2c99 100644
@@ -388,6 +388,9 @@ int hvm_get_param(struct domain *d, uint32_t index, uint64_t *value);
 #define hvm_tsc_scaling_ratio(d) \
     ((d)->arch.hvm.tsc_scaling_ratio)
 
+extern bool assisted_xapic_available;
+extern bool assisted_x2apic_available;
+
 #define hvm_get_guest_time(v) hvm_get_guest_time_fixed(v, 0)
 
 #define hvm_paging_enabled(v) \
@@ -901,6 +904,8 @@ static inline void hvm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)
 #define hvm_tsc_scaling_supported false
 #define hap_has_1gb false
 #define hap_has_2mb false
+#define assisted_xapic_available false
+#define assisted_x2apic_available false
 
 #define hvm_paging_enabled(v) ((void)(v), false)
 #define hvm_wp_enabled(v) ((void)(v), false)
xen/arch/x86/sysctl.c +4 -0
index f82abc2488..716525f72f 100644
@@ -135,6 +135,10 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
         pi->capabilities |= XEN_SYSCTL_PHYSCAP_hap;
     if ( IS_ENABLED(CONFIG_SHADOW_PAGING) )
         pi->capabilities |= XEN_SYSCTL_PHYSCAP_shadow;
+    if ( assisted_xapic_available )
+        pi->arch_capabilities |= XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC;
+    if ( assisted_x2apic_available )
+        pi->arch_capabilities |= XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC;
 }
 
 long arch_do_sysctl(
xen/include/public/sysctl.h +10 -1
index 60c8711483..5672906729 100644
@@ -35,7 +35,7 @@
 #include "domctl.h"
 #include "physdev.h"
 
-#define XEN_SYSCTL_INTERFACE_VERSION 0x00000014
+#define XEN_SYSCTL_INTERFACE_VERSION 0x00000015
 
 /*
  * Read console content from Xen buffer ring.
@@ -111,6 +111,13 @@ struct xen_sysctl_tbuf_op {
 /* Max XEN_SYSCTL_PHYSCAP_* constant.  Used for ABI checking. */
 #define XEN_SYSCTL_PHYSCAP_MAX XEN_SYSCTL_PHYSCAP_gnttab_v2
 
+/* The platform supports x{2}apic hardware assisted emulation. */
+#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC  (1u << 0)
+#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC (1u << 1)
+
+/* Max XEN_SYSCTL_PHYSCAP_X86_* constant. Used for ABI checking. */
+#define XEN_SYSCTL_PHYSCAP_X86_MAX XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC
+
 struct xen_sysctl_physinfo {
     uint32_t threads_per_core;
     uint32_t cores_per_socket;
@@ -120,6 +127,8 @@ struct xen_sysctl_physinfo {
     uint32_t max_node_id; /* Largest possible node ID on this host */
     uint32_t cpu_khz;
     uint32_t capabilities;/* XEN_SYSCTL_PHYSCAP_??? */
+    uint32_t arch_capabilities;/* XEN_SYSCTL_PHYSCAP_{X86,ARM,...}_??? */
+    uint32_t pad;
     uint64_aligned_t total_pages;
     uint64_aligned_t free_pages;
     uint64_aligned_t scrub_pages;
Signature · 1 line
2.11.0
#2 lore raw
Quoted 11 lines · Christian Lindig

On 8 Jul 2022, at 14:55, Jane Malalane <[email protected]<mailto:[email protected]>> wrote:

tools/ocaml/libs/xc/xenctrl.ml       |  9 +++++++++
tools/ocaml/libs/xc/xenctrl.mli      |  8 ++++++++
tools/ocaml/libs/xc/xenctrl_stubs.c  | 18 ++++++++++++++++—

Hello Jan,

I believe I now have all the acks needed for this patch series. Regarding the commit message of patch 2, I thought that explanation made it clearer but I can always remove it.

Thank you,

Jane.

#3 lore raw
Quoted 7 lines · Jane Malalane
Quoted 6 lines · Christian Lindig

On 8 Jul 2022, at 14:55, Jane Malalane <[email protected]<mailto:[email protected]>> wrote:

tools/ocaml/libs/xc/xenctrl.ml       |  9 +++++++++
tools/ocaml/libs/xc/xenctrl.mli      |  8 ++++++++
tools/ocaml/libs/xc/xenctrl_stubs.c  | 18 ++++++++++++++++—
Quoted 5 lines
Quoted 1 line

I believe I now have all the acks needed for this patch series. Regarding the commit message of patch 2, I thought that explanation made it clearer but I can always remove it.

I haven't seen v11 2/2 anywhere, so I'm afraid I can't check what you're referring to.

Jan