xen+tools: Report Interrupt Controller Virtualization capabilities on x86
PATCH v11 1/2 xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.
HW assisted xAPIC virtualization will be reported if HW, at the minimum, supports virtualize_apic_accesses as this feature alone means that an access to the APIC page will cause an APIC-access VM exit. An APIC-access VM exit provides a VMM with information about the access causing the VM exit, unlike a regular EPT fault, thus simplifying some internal handling.
HW assisted x2APIC virtualization will be reported if HW supports virtualize_x2apic_mode and, at least, either apic_reg_virt or virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().
For that purpose, also add an arch-specific "capabilities" parameter to struct xen_sysctl_physinfo.
Note that this interface is intended to be compatible with AMD so that AVIC support can be introduced in a future patch. Unlike Intel that has multiple controls for APIC Virtualization, AMD has one global 'AVIC Enable' control bit, so fine-graining of APIC virtualization control cannot be done on a common interface.
---
CC: George Dunlap <[email protected]>
CC: Nick Rosbrook <[email protected]>
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Christian Lindig <[email protected]>
CC: David Scott <[email protected]>
CC: "Roger Pau Monné" <[email protected]>
CC: Jun Nakajima <[email protected]>
CC: Kevin Tian <[email protected]>
v11:
* explicity define arch_cap_list in stub_xc_physinfo for non-x86 arch
v10:
* Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()
v9:
* Move assisted_x{2}apic_available to vmx_vmcs_init() so they get declared at boot time, after vmx_secondary_exec_control is set
v8:
* Improve commit message
v7:
* Make sure assisted_x{2}apic_available evaluates to false, to ensure Xen builds, when !CONFIG_HVM
* Fix coding style issues
v6:
* Limit abi check to x86
* Fix coding style issue
v5:
* Have assisted_xapic_available solely depend on cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available depend on cpu_has_vmx_virtualize_x2apic_mode and cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery
v4:
* Fallback to the original v2/v1 conditions for setting assisted_xapic_available and assisted_x2apic_available so that in the future APIC virtualization can be exposed on AMD hardware since fine-graining of "AVIC" is not supported, i.e., AMD solely uses "AVIC Enable". This also means that sysctl mimics what's exposed in CPUID
v3:
* Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
* Have assisted_x2apic_available only depend on cpu_has_vmx_virtualize_x2apic_mode
v2:
* Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
* Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
* Set assisted_x{2}apic_available to be conditional upon "bsp" and annotate it with __ro_after_init
* Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to _X86_ASSISTED_X{2}APIC
* Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within sysctl.h
* Fix padding introduced in struct xen_sysctl_physinfo and bump XEN_SYSCTL_INTERFACE_VERSION
--- tools/golang/xenlight/helpers.gen.go | 4 ++++ tools/golang/xenlight/types.gen.go | 2 ++ tools/include/libxl.h | 7 +++++++ tools/libs/gnttab/st8ojiU6 | 0 tools/libs/light/libxl.c | 3 +++ tools/libs/light/libxl_arch.h | 4 ++++ tools/libs/light/libxl_arm.c | 5 +++++ tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 11 +++++++++++ tools/ocaml/libs/xc/xenctrl.ml | 9 +++++++++ tools/ocaml/libs/xc/xenctrl.mli | 8 ++++++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 18 ++++++++++++++++-- tools/xl/xl_info.c | 6 ++++-- xen/arch/x86/hvm/hvm.c | 3 +++ xen/arch/x86/hvm/vmx/vmcs.c | 7 +++++++ xen/arch/x86/include/asm/hvm/hvm.h | 5 +++++ xen/arch/x86/sysctl.c | 4 ++++ xen/include/public/sysctl.h | 11 ++++++++++- 18 files changed, 104 insertions(+), 5 deletions(-) create mode 100644 tools/libs/gnttab/st8ojiU6
tools/golang/xenlight/helpers.gen.go +4 -0
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace) x.CapVpmu = bool(xc.cap_vpmu) x.CapGnttabV1 = bool(xc.cap_gnttab_v1) x.CapGnttabV2 = bool(xc.cap_gnttab_v2) +x.CapAssistedXapic = bool(xc.cap_assisted_xapic) +x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic) return nil}
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace) xc.cap_vpmu = C.bool(x.CapVpmu) xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1) xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2) +xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic) +xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic) return nil }
tools/golang/xenlight/types.gen.go +2 -0
@@ -1014,6 +1014,8 @@ CapVmtrace bool CapVpmu bool CapGnttabV1 bool CapGnttabV2 bool +CapAssistedXapic bool +CapAssistedX2Apic bool } type Connectorinfo struct {
tools/include/libxl.h +7 -0
@@ -536,6 +536,13 @@#define LIBXL_HAVE_NIC_TRUSTED 1/* +* LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has +* cap_assisted_xapic and cap_assisted_x2apic fields, which indicates +* the availability of x{2}APIC hardware assisted virtualization. +*/ +#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility
tools/libs/gnttab/st8ojiU6 new file +0 -0
tools/libs/light/libxl.c +3 -0
@@ -15,6 +15,7 @@#include "libxl_osdeps.h" #include "libxl_internal.h" +#include "libxl_arch.h" int libxl_ctx_alloc (libxl_ctx **pctx,int version,unsigned flags,xentoollog_logger * lg)
@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) physinfo->cap_gnttab_v2 = !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2); +libxl__arch_get_physinfo (physinfo, &xcphysinfo); +GC_FREE ;return 0 ; }
tools/libs/light/libxl_arch.h +4 -0
@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc, uint64_t *out);_hidden +void libxl__arch_get_physinfo(libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo); + +_hidden void libxl__arch_update_domain_config(libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src);
tools/libs/light/libxl_arm.c +5 -0
@@ -1431,6 +1431,11 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)
tools/libs/light/libxl_types.idl +2 -0
@@ -1070,6 +1070,8 @@ libxl_physinfo = Struct("physinfo", [ ("cap_vpmu", bool), ("cap_gnttab_v1", bool), ("cap_gnttab_v2", bool), + ("cap_assisted_xapic", bool), + ("cap_assisted_x2apic", bool), ], dir=DIR_OUT) libxl_connectorinfo = Struct("connectorinfo", [
tools/libs/light/libxl_x86.c +11 -0
@@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ + physinfo->cap_assisted_xapic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ); + physinfo->cap_assisted_x2apic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ); +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)
tools/ocaml/libs/xc/xenctrl.ml +9 -0
@@ -128,6 +128,14 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 + +type x86_physinfo_arch_cap_flag = + | CAP_X86_ASSISTED_XAPIC + | CAP_X86_ASSISTED_X2APIC + +type physinfo_arch_cap_flag = + | X86 of x86_physinfo_arch_cap_flag + type physinfo = { threads_per_core : int;
@@ -141,6 +149,7 @@ type physinfo = (* XXX hw_cap *) capabilities : physinfo_cap_flag list; max_nr_cpus : int; + arch_capabilities : physinfo_arch_cap_flag list; } type version =
tools/ocaml/libs/xc/xenctrl.mli +8 -0
@@ -113,6 +113,13 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type x86_physinfo_arch_cap_flag = + | CAP_X86_ASSISTED_XAPIC + | CAP_X86_ASSISTED_X2APIC + +type physinfo_arch_cap_flag = + | X86 of x86_physinfo_arch_cap_flag + type physinfo = { threads_per_core : int; cores_per_socket : int;
@@ -124,6 +131,7 @@ type physinfo = { scrub_pages : nativeint; capabilities : physinfo_cap_flag list; max_nr_cpus : int; (** compile-time max possible number of nr_cpus *) + arch_capabilities : physinfo_arch_cap_flag list; } type version = { major : int; minor : int; extra : string; } type compile_info = {
tools/ocaml/libs/xc/xenctrl_stubs.c +16 -2
@@ -718,7 +718,7 @@ CAMLprim value stub_xc_send_debug_keys(value xch, value keys) CAMLprim value stub_xc_physinfo(value xch) {CAMLparam1 (xch); -CAMLlocal2 (physinfo, cap_list); +CAMLlocal4 (physinfo, cap_list, x86_arch_cap_list, arch_cap_list);xc_physinfo_t c_physinfo;int r;
@@ -737,7 +737,7 @@ CAMLprim value stub_xc_physinfo(value xch)/* ! XEN_SYSCTL_PHYSCAP_ XEN_SYSCTL_PHYSCAP_MAX max */ (c_physinfo.capabilities); - physinfo =caml_alloc_tuple (10 ); + physinfo =caml_alloc_tuple (11 );Store_field (physinfo,0 ,Val_int (c_physinfo.threads_per_core));Store_field (physinfo,1 ,Val_int (c_physinfo.cores_per_socket));Store_field (physinfo,2 ,Val_int (c_physinfo.nr_cpus));
@@ -749,6 +749,20 @@ CAMLprim value stub_xc_physinfo(value xch)Store_field (physinfo,8 , cap_list);Store_field (physinfo,9 ,Val_int (c_physinfo.max_cpu_id +1 )); +#if defined(__i386__) || defined(__x86_64__) + x86_arch_cap_list =c_bitmap_to_ocaml_list +/* ! x86_physinfo_arch_cap_flag CAP_X86_ none */ +/* ! XEN_SYSCTL_PHYSCAP_X86_ XEN_SYSCTL_PHYSCAP_X86_MAX max */ + (c_physinfo.arch_capabilities); +/* +* arch_capabilities: physinfo_arch_cap_flag list; +*/ + arch_cap_list = x86_arch_cap_list; +#else + arch_cap_list = Val_emptylist; +#endif +Store_field (physinfo,10 , arch_cap_list); +CAMLreturn (physinfo); }
tools/xl/xl_info.c +4 -2
@@ -210,7 +210,7 @@ static void output_physinfo(void) info.hw_cap[4 ], info.hw_cap[5 ], info.hw_cap[6 ], info.hw_cap[7 ] ); - maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s\n", + maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s%s%s\n", info.cap_pv ? " pv" : "", info.cap_hvm ? " hvm" : "", info.cap_hvm && info.cap_hvm_directio ? " hvm_directio" : "",
@@ -221,7 +221,9 @@ static void output_physinfo(void) info.cap_vmtrace ?" vmtrace" :"" , info.cap_vpmu ?" vpmu" :"" , info.cap_gnttab_v1 ?" gnttab-v1" :"" , - info.cap_gnttab_v2 ?" gnttab-v2" :"" + info.cap_gnttab_v2 ?" gnttab-v2" :"" , + info.cap_assisted_xapic ?" assisted_xapic" :"" , + info.cap_assisted_x2apic ?" assisted_x2apic" :"" ); vinfo =libxl_get_version_info (ctx);
xen/arch/x86/hvm/hvm.c +3 -0
@@ -117,6 +117,9 @@ static const char __initconst warning_hvm_fep[] =static bool_t __initdata opt_altp2m_enabled =0 ;boolean_param ("altp2m" , opt_altp2m_enabled); +bool __ro_after_init assisted_xapic_available; +bool __ro_after_init assisted_x2apic_available; + static int cf_check cpu_callback(struct notifier_block *nfb,unsigned long action,void *hcpu) {
xen/arch/x86/hvm/vmx/vmcs.c +7 -0
@@ -2146,7 +2146,14 @@ int __init vmx_vmcs_init(void) ret =_vmx_cpu_up (true);if ( !ret ) + { +/* Check whether hardware supports accelerated xapic and x2apic. */ + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode && + (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery);register_keyhandler ('v' , vmcs_dump,"dump VT-x VMCSs" ,1 ); + }return ret; }
xen/arch/x86/include/asm/hvm/hvm.h +5 -0
@@ -388,6 +388,9 @@ int hvm_get_param(struct domain *d, uint32_t index, uint64_t *value);#define hvm_tsc_scaling_ratio (d) \ ((d)->arch.hvm.tsc_scaling_ratio) +extern bool assisted_xapic_available; +extern bool assisted_x2apic_available; +#define hvm_get_guest_time (v) hvm_get_guest_time_fixed(v, 0)#define hvm_paging_enabled (v) \
@@ -901,6 +904,8 @@ static inline void hvm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)#define hvm_tsc_scaling_supported false#define hap_has_1gb false#define hap_has_2mb false +#define assisted_xapic_available false +#define assisted_x2apic_available false#define hvm_paging_enabled (v) ((void)(v), false)#define hvm_wp_enabled (v) ((void)(v), false)
xen/arch/x86/sysctl.c +4 -0
@@ -135,6 +135,10 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi) pi->capabilities |= XEN_SYSCTL_PHYSCAP_hap;if (IS_ENABLED (CONFIG_SHADOW_PAGING ) ) pi->capabilities |= XEN_SYSCTL_PHYSCAP_shadow; +if ( assisted_xapic_available ) + pi->arch_capabilities |=XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ; +if ( assisted_x2apic_available ) + pi->arch_capabilities |=XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ; } long arch_do_sysctl(
xen/include/public/sysctl.h +10 -1
@@ -35,7 +35,7 @@#include "domctl.h" #include "physdev.h" -#define XEN_SYSCTL_INTERFACE_VERSION 0x00000014 +#define XEN_SYSCTL_INTERFACE_VERSION 0x00000015/* * Read console content from Xen buffer ring.
@@ -111,6 +111,13 @@ struct xen_sysctl_tbuf_op {/* Max XEN_SYSCTL_PHYSCAP_* constant. Used for ABI checking. */ #define XEN_SYSCTL_PHYSCAP_MAX XEN_SYSCTL_PHYSCAP_gnttab_v2 +/* The platform supports x{2}apic hardware assisted emulation. */ +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC (1u << 0) +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC (1u << 1) + +/* Max XEN_SYSCTL_PHYSCAP_X86_* constant. Used for ABI checking. */ +#define XEN_SYSCTL_PHYSCAP_X86_MAX XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC +struct xen_sysctl_physinfo {uint32_t threads_per_core;uint32_t cores_per_socket;
@@ -120,6 +127,8 @@ struct xen_sysctl_physinfo {uint32_t max_node_id;/* Largest possible node ID on this host */ uint32_t cpu_khz;uint32_t capabilities;/* XEN_SYSCTL_PHYSCAP_??? */ +uint32_t arch_capabilities;/* XEN_SYSCTL_PHYSCAP_{X86,ARM,...}_??? */ +uint32_t pad;uint64_aligned_t total_pages;uint64_aligned_t free_pages;uint64_aligned_t scrub_pages;
Signature · 1 line
2.11.0
On 8 Jul 2022, at 14:55, Jane Malalane <[email protected]<mailto:[email protected]>> wrote:
tools/ocaml/libs/xc/xenctrl.ml | 9 +++++++++ tools/ocaml/libs/xc/xenctrl.mli | 8 ++++++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 18 ++++++++++++++++—
Quoted 11 lines · Christian Lindig
On 8 Jul 2022, at 14:55, Jane Malalane <[email protected]<mailto:[email protected]>> wrote:
tools/ocaml/libs/xc/xenctrl.ml | 9 +++++++++ tools/ocaml/libs/xc/xenctrl.mli | 8 ++++++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 18 ++++++++++++++++—
Hello Jan,
I believe I now have all the acks needed for this patch series. Regarding the commit message of patch 2, I thought that explanation made it clearer but I can always remove it.
Thank you,
Jane.
Quoted 7 lines · Jane Malalane
Quoted 6 lines · Christian Lindig
On 8 Jul 2022, at 14:55, Jane Malalane <[email protected]<mailto:[email protected]>> wrote:
tools/ocaml/libs/xc/xenctrl.ml | 9 +++++++++ tools/ocaml/libs/xc/xenctrl.mli | 8 ++++++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 18 ++++++++++++++++—
Quoted 5 lines
Quoted 1 line
I believe I now have all the acks needed for this patch series. Regarding the commit message of patch 2, I thought that explanation made it clearer but I can always remove it.
I haven't seen v11 2/2 anywhere, so I'm afraid I can't check what you're referring to.
Jan