xen: Report and use hardware APIC virtualization capabilities
v2 cover xen: Report and use hardware APIC virtualization capabilities
Jane Malalane (2):
xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
x86/xen: Allow per-domain usage of hardware virtualized APIC
docs/man/xl.cfg.5.pod.in | 10 +++++++++ docs/man/xl.conf.5.pod.in | 12 ++++++++++ tools/golang/xenlight/helpers.gen.go | 16 +++++++++++++ tools/golang/xenlight/types.gen.go | 6 +++++ tools/include/libxl.h | 14 ++++++++++++ tools/libs/light/libxl.c | 3 +++ tools/libs/light/libxl_arch.h | 9 ++++++-- tools/libs/light/libxl_arm.c | 12 ++++++++-- tools/libs/light/libxl_create.c | 23 +++++++++++-------- tools/libs/light/libxl_types.idl | 4 ++++ tools/libs/light/libxl_x86.c | 42 +++++++++++++++++++++++++++++++++-- tools/ocaml/libs/xc/xenctrl.ml | 7 ++++++ tools/ocaml/libs/xc/xenctrl.mli | 7 ++++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 2 +- tools/xl/xl.c | 8 +++++++ tools/xl/xl.h | 2 ++ tools/xl/xl_info.c | 6 +++-- tools/xl/xl_parse.c | 16 +++++++++++++ xen/arch/x86/domain.c | 28 ++++++++++++++++++++++- xen/arch/x86/hvm/vmx/vmcs.c | 13 +++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 14 +++++------- xen/arch/x86/include/asm/domain.h | 3 +++ xen/arch/x86/include/asm/hvm/domain.h | 6 +++++ xen/arch/x86/sysctl.c | 7 ++++++ xen/arch/x86/traps.c | 8 +++---- xen/include/public/arch-x86/xen.h | 2 ++ xen/include/public/sysctl.h | 8 ++++++- 27 files changed, 255 insertions(+), 33 deletions(-)
Signature · 1 line
2.11.0
PATCH v2 1/2 xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.
For that purpose, also add an arch-specific "capabilities" parameter to struct xen_sysctl_physinfo.
---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Bertrand Marquis <[email protected]>
CC: Jun Nakajima <[email protected]>
CC: Kevin Tian <[email protected]>
CC: "Roger Pau Monné" <[email protected]>
v2:
* Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
* Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
* Set assisted_x{2}apic_available to be conditional upon "bsp" and annotate it with __ro_after_init
* Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to .._X86_ASSISTED_X{2}APIC
* Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within sysctl.h
* Fix padding introduced in struct xen_sysctl_physinfo and bump XEN_SYSCTL_INTERFACE_VERSION
--- tools/golang/xenlight/helpers.gen.go | 4 ++++ tools/golang/xenlight/types.gen.go | 6 ++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl.c | 3 +++ tools/libs/light/libxl_arch.h | 4 ++++ tools/libs/light/libxl_arm.c | 5 +++++ tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 11 +++++++++++ tools/ocaml/libs/xc/xenctrl.ml | 5 +++++ tools/ocaml/libs/xc/xenctrl.mli | 5 +++++ tools/xl/xl_info.c | 6 ++++-- xen/arch/x86/hvm/vmx/vmcs.c | 9 +++++++++ xen/arch/x86/include/asm/domain.h | 3 +++ xen/arch/x86/sysctl.c | 7 +++++++ xen/include/public/sysctl.h | 8 +++++++- 15 files changed, 82 insertions(+), 3 deletions(-)
tools/golang/xenlight/helpers.gen.go +4 -0
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace) x.CapVpmu = bool(xc.cap_vpmu) x.CapGnttabV1 = bool(xc.cap_gnttab_v1) x.CapGnttabV2 = bool(xc.cap_gnttab_v2) +x.CapAssistedXapic = bool(xc.cap_assisted_xapic) +x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic) return nil}
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace) xc.cap_vpmu = C.bool(x.CapVpmu) xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1) xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2) +xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic) +xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic) return nil }
tools/golang/xenlight/types.gen.go +6 -0
@@ -389,6 +389,10 @@ RunHotplugScripts Defbool DriverDomain Defbool Passthrough Passthrough XendSuspendEvtchnCompat Defbool +ArchX86 struct { +AssistedXapic Defbool +AssistedX2Apic Defbool +} } type DomainRestoreParams struct {
@@ -1014,6 +1018,8 @@ CapVmtrace bool CapVpmu bool CapGnttabV1 bool CapGnttabV2 bool +CapAssistedXApic bool +CapAssistedX2apic bool } type Connectorinfo struct {
tools/include/libxl.h +7 -0
@@ -528,6 +528,13 @@#define LIBXL_HAVE_MAX_GRANT_VERSION 1/* +* LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has +* cap_assisted_x{2}apic fields, which indicates the availability of x{2}APIC +* hardware assisted virtualization. +*/ +#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility
tools/libs/light/libxl.c +3 -0
@@ -15,6 +15,7 @@#include "libxl_osdeps.h" #include "libxl_internal.h" +#include "libxl_arch.h" int libxl_ctx_alloc (libxl_ctx **pctx,int version,unsigned flags,xentoollog_logger * lg)
@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) physinfo->cap_gnttab_v2 = !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2); +libxl__arch_get_physinfo (physinfo, &xcphysinfo); +GC_FREE ;return 0 ; }
tools/libs/light/libxl_arch.h +4 -0
@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc, uint64_t *out);_hidden +void libxl__arch_get_physinfo(libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo); + +_hidden void libxl__arch_update_domain_config(libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src);
tools/libs/light/libxl_arm.c +5 -0
@@ -1431,6 +1431,11 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)
tools/libs/light/libxl_types.idl +2 -0
@@ -1068,6 +1068,8 @@ libxl_physinfo = Struct("physinfo", [ ("cap_vpmu", bool), ("cap_gnttab_v1", bool), ("cap_gnttab_v2", bool), + ("cap_assisted_xapic", bool), + ("cap_assisted_x2apic", bool), ], dir=DIR_OUT) libxl_connectorinfo = Struct("connectorinfo", [
tools/libs/light/libxl_x86.c +11 -0
@@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ + physinfo->cap_assisted_xapic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ); + physinfo->cap_assisted_x2apic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ); +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)
tools/ocaml/libs/xc/xenctrl.ml +5 -0
@@ -127,6 +127,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int;
@@ -139,6 +143,7 @@ type physinfo = scrub_pages : nativeint; (* XXX hw_cap *) capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list; max_nr_cpus : int; }
tools/ocaml/libs/xc/xenctrl.mli +5 -0
@@ -112,6 +112,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int; cores_per_socket : int;
@@ -122,6 +126,7 @@ type physinfo = { free_pages : nativeint; scrub_pages : nativeint; capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list; max_nr_cpus : int; (** compile-time max possible number of nr_cpus *) } type version = { major : int; minor : int; extra : string; }
tools/xl/xl_info.c +4 -2
@@ -210,7 +210,7 @@ static void output_physinfo(void) info.hw_cap[4 ], info.hw_cap[5 ], info.hw_cap[6 ], info.hw_cap[7 ] ); - maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s\n", + maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s%s%s\n", info.cap_pv ? " pv" : "", info.cap_hvm ? " hvm" : "", info.cap_hvm && info.cap_hvm_directio ? " hvm_directio" : "",
@@ -221,7 +221,9 @@ static void output_physinfo(void) info.cap_vmtrace ?" vmtrace" :"" , info.cap_vpmu ?" vpmu" :"" , info.cap_gnttab_v1 ?" gnttab-v1" :"" , - info.cap_gnttab_v2 ?" gnttab-v2" :"" + info.cap_gnttab_v2 ?" gnttab-v2" :"" , + info.cap_assisted_xapic ?" assisted_xapic" :"" , + info.cap_assisted_x2apic ?" assisted_x2apic" :"" ); vinfo =libxl_get_version_info (ctx);
xen/arch/x86/hvm/vmx/vmcs.c +9 -0
@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode; + } +/* The IA32_VMX_EPT_VPID_CAP MSR exists only when EPT or VPID available */ if ( _vmx_secondary_exec_control & (SECONDARY_EXEC_ENABLE_EPT |SECONDARY_EXEC_ENABLE_VPID ) )
xen/arch/x86/include/asm/domain.h +3 -0
@@ -756,6 +756,9 @@ static inline void pv_inject_sw_interrupt(unsigned int vector) : is_pv_32bit_domain(d) ? PV32_VM_ASSIST_MASK \ : PV64_VM_ASSIST_MASK) +extern bool assisted_xapic_available; +extern bool assisted_x2apic_available; +#endif /* __ASM_DOMAIN_H__ */ /*
xen/arch/x86/sysctl.c +7 -0
@@ -69,6 +69,9 @@ struct l3_cache_info {unsigned long size; }; +bool __ro_after_init assisted_xapic_available; +bool __ro_after_init assisted_x2apic_available; +static void l3_cache_get (void *arg) {struct cpuid4_info info;
@@ -135,6 +138,10 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi) pi->capabilities |= XEN_SYSCTL_PHYSCAP_hap;if (IS_ENABLED (CONFIG_SHADOW_PAGING ) ) pi->capabilities |= XEN_SYSCTL_PHYSCAP_shadow; +if ( assisted_xapic_available ) + pi->arch_capabilities |=XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ; +if ( assisted_x2apic_available ) + pi->arch_capabilities |=XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ; } long arch_do_sysctl(
xen/include/public/sysctl.h +7 -1
@@ -35,7 +35,7 @@#include "domctl.h" #include "physdev.h" -#define XEN_SYSCTL_INTERFACE_VERSION 0x00000014 +#define XEN_SYSCTL_INTERFACE_VERSION 0x00000015/* * Read console content from Xen buffer ring.
@@ -111,6 +111,10 @@ struct xen_sysctl_tbuf_op {/* Max XEN_SYSCTL_PHYSCAP_* constant. Used for ABI checking. */ #define XEN_SYSCTL_PHYSCAP_MAX XEN_SYSCTL_PHYSCAP_gnttab_v2 +/* The platform supports x{2}apic hardware assisted emulation. */ +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC (1u << 0) +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC (1u << 1) +struct xen_sysctl_physinfo {uint32_t threads_per_core;uint32_t cores_per_socket;
@@ -120,6 +124,8 @@ struct xen_sysctl_physinfo {uint32_t max_node_id;/* Largest possible node ID on this host */ uint32_t cpu_khz;uint32_t capabilities;/* XEN_SYSCTL_PHYSCAP_??? */ +uint32_t arch_capabilities;/* XEN_SYSCTL_PHYSCAP_X86{ARM}_??? */ +uint32_t pad;/* Must be zero. */ uint64_aligned_t total_pages;uint64_aligned_t free_pages;uint64_aligned_t scrub_pages;
Signature · 1 line
2.11.0
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 10 lines · Jane Malalane
Add
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapicandXEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apicto report accelerated xapic and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.For that purpose, also add an arch-specific "capabilities" parameter to struct
xen_sysctl_physinfo.Signed-off-by Jane Malalane <[email protected]>Suggested-by Andrew Cooper <[email protected]>
Tag order should be inverted, first Suggested-by, then SoB.
Quoted 71 lines
---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Bertrand Marquis <[email protected]>
CC: Jun Nakajima <[email protected]>
CC: Kevin Tian <[email protected]>
CC: "Roger Pau Monné" <[email protected]>v2:
* Use one macroLIBXL_HAVE_PHYSINFO_ASSISTED_APICinstead of two
* Pass xcpyshinfo as a pointer inlibxl__arch_get_physinfo
* Set assisted_x{2}apic_available to be conditional upon "bsp" and annotate it with__ro_after_init
* Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to .._X86_ASSISTED_X{2}APIC
* Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained withinsysctl.h
* Fix padding introduced in structxen_sysctl_physinfoand bumpXEN_SYSCTL_INTERFACE_VERSION--- tools/golang/xenlight/helpers.gen.go | 4 ++++ tools/golang/xenlight/types.gen.go | 6 ++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl.c | 3 +++ tools/libs/light/libxl_arch.h | 4 ++++ tools/libs/light/libxl_arm.c | 5 +++++ tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 11 +++++++++++ tools/ocaml/libs/xc/xenctrl.ml | 5 +++++ tools/ocaml/libs/xc/xenctrl.mli | 5 +++++ tools/xl/xl_info.c | 6 ++++-- xen/arch/x86/hvm/vmx/vmcs.c | 9 +++++++++ xen/arch/x86/include/asm/domain.h | 3 +++ xen/arch/x86/sysctl.c | 7 +++++++ xen/include/public/sysctl.h | 8 +++++++- 15 files changed, 82 insertions(+), 3 deletions(-)tools/golang/xenlight/helpers.gen.go +4
-0@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace) x.CapVpmu = bool(xc.cap_vpmu) x.CapGnttabV1 = bool(xc.cap_gnttab_v1) x.CapGnttabV2 = bool(xc.cap_gnttab_v2) +x.CapAssistedXapic = bool(xc.cap_assisted_xapic) +x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic) return nil}@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace) xc.cap_vpmu = C.bool(x.CapVpmu) xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1) xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2) +xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic) +xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic) return nil }tools/golang/xenlight/types.gen.go +0
-0
tools/golang/xenlight/types.gen.go RunHotplugScripts Defbool · line 394
Quoted 6 lines
tools/golang/xenlight/types.gen.go +3
-0@@ -389,6 +389,10 @@ RunHotplugScripts Defbool DriverDomain Defbool Passthrough Passthrough XendSuspendEvtchnCompat Defbool +ArchX86 struct { +AssistedXapic Defbool +AssistedX2Apic Defbool
Don't you need some indentation here?
Also name would better be Assistedx{2}APIC IMO if possible. Having a capital 'X' and lowercase 'apic' looks really strange.
Quoted 134 lines
tools/golang/xenlight/types.gen.go +3
-0+} } type DomainRestoreParams struct {@@ -1014,6 +1018,8 @@ CapVmtrace bool CapVpmu bool CapGnttabV1 bool CapGnttabV2 bool +CapAssistedXApic bool +CapAssistedX2apic bool } type Connectorinfo struct {tools/include/libxl.h +7
-0@@ -528,6 +528,13 @@#define LIBXL_HAVE_MAX_GRANT_VERSION 1/* +* LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has +* cap_assisted_x{2}apic fields, which indicates the availability of x{2}APIC +* hardware assisted virtualization. +*/ +#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility tools/libs/light/libxl.c +3
-0@@ -15,6 +15,7 @@#include "libxl_osdeps.h" #include "libxl_internal.h" +#include "libxl_arch.h" int libxl_ctx_alloc (libxl_ctx **pctx,int version,unsigned flags,xentoollog_logger * lg)@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) physinfo->cap_gnttab_v2 = !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2); +libxl__arch_get_physinfo (physinfo, &xcphysinfo); +GC_FREE ;return 0 ; }tools/libs/light/libxl_arch.h +4
-0@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc, uint64_t *out);_hidden +void libxl__arch_get_physinfo(libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo); + +_hidden void libxl__arch_update_domain_config(libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src);tools/libs/light/libxl_arm.c +5
-0@@ -1431,6 +1431,11 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)tools/libs/light/libxl_types.idl +2
-0@@ -1068,6 +1068,8 @@ libxl_physinfo = Struct("physinfo", [ ("cap_vpmu", bool), ("cap_gnttab_v1", bool), ("cap_gnttab_v2", bool), + ("cap_assisted_xapic", bool), + ("cap_assisted_x2apic", bool), ], dir=DIR_OUT) libxl_connectorinfo = Struct("connectorinfo", [tools/libs/light/libxl_x86.c +11
-0@@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ + physinfo->cap_assisted_xapic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ); + physinfo->cap_assisted_x2apic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ); +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)tools/ocaml/libs/xc/xenctrl.ml +4
-0@@ -127,6 +127,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int;
tools/ocaml/libs/xc/xenctrl.ml type physinfo = · line 146
Quoted 4 lines
tools/ocaml/libs/xc/xenctrl.ml +1
-0@@ -139,6 +143,7 @@ type physinfo = scrub_pages : nativeint; (* XXX hw_cap *) capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list;
I know very little about Ocaml, but I think you are not setting this field anywhere? I would expect a call to ocaml_list_to_c_bitmap and then you will likely need to define XEN_SYSCTL_PHYSCAP_X86_MAX so you can check the options. See XEN_SYSCTL_PHYSCAP_MAX for example.
Quoted 89 lines
tools/ocaml/libs/xc/xenctrl.ml +0
-0max_nr_cpus : int; }tools/ocaml/libs/xc/xenctrl.mli +5
-0@@ -112,6 +112,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int; cores_per_socket : int;@@ -122,6 +126,7 @@ type physinfo = { free_pages : nativeint; scrub_pages : nativeint; capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list; max_nr_cpus : int; (** compile-time max possible number of nr_cpus *) } type version = { major : int; minor : int; extra : string; }tools/xl/xl_info.c +4
-2@@ -210,7 +210,7 @@ static void output_physinfo(void) info.hw_cap[4 ], info.hw_cap[5 ], info.hw_cap[6 ], info.hw_cap[7 ] ); - maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s\n", + maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s%s%s\n", info.cap_pv ? " pv" : "", info.cap_hvm ? " hvm" : "", info.cap_hvm && info.cap_hvm_directio ? " hvm_directio" : "",@@ -221,7 +221,9 @@ static void output_physinfo(void) info.cap_vmtrace ?" vmtrace" :"" , info.cap_vpmu ?" vpmu" :"" , info.cap_gnttab_v1 ?" gnttab-v1" :"" , - info.cap_gnttab_v2 ?" gnttab-v2" :"" + info.cap_gnttab_v2 ?" gnttab-v2" :"" , + info.cap_assisted_xapic ?" assisted_xapic" :"" , + info.cap_assisted_x2apic ?" assisted_x2apic" :"" ); vinfo =libxl_get_version_info (ctx);xen/arch/x86/hvm/vmx/vmcs.c +9
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode; + } +/* The IA32_VMX_EPT_VPID_CAP MSR exists only when EPT or VPID available */ if ( _vmx_secondary_exec_control & (SECONDARY_EXEC_ENABLE_EPT |SECONDARY_EXEC_ENABLE_VPID ) )xen/arch/x86/include/asm/domain.h +3
-0@@ -756,6 +756,9 @@ static inline void pv_inject_sw_interrupt(unsigned int vector) : is_pv_32bit_domain(d) ? PV32_VM_ASSIST_MASK \ : PV64_VM_ASSIST_MASK) +extern bool assisted_xapic_available; +extern bool assisted_x2apic_available; +#endif /* __ASM_DOMAIN_H__ */ /* xen/arch/x86/sysctl.c +0
-0
xen/arch/x86/sysctl.c struct l3_cache_info { · line 73
Quoted 5 lines
xen/arch/x86/sysctl.c +2
-0@@ -69,6 +69,9 @@ struct l3_cache_info {unsigned long size; }; +bool __ro_after_init assisted_xapic_available; +bool __ro_after_init assisted_x2apic_available;
You could likely shorten this by dropping the _available suffix.
Thanks, Roger.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 78 lines · Roger Pau Monné
Quoted 10 lines · Jane Malalane
Add
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapicandXEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apicto report accelerated xapic and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.For that purpose, also add an arch-specific "capabilities" parameter to struct
xen_sysctl_physinfo.Signed-off-by Jane Malalane <[email protected]>Suggested-by Andrew Cooper <[email protected]>Tag order should be inverted, first Suggested-by, then SoB.
Quoted 64 lines
---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Bertrand Marquis <[email protected]>
CC: Jun Nakajima <[email protected]>
CC: Kevin Tian <[email protected]>
CC: "Roger Pau Monné" <[email protected]>v2:
* Use one macroLIBXL_HAVE_PHYSINFO_ASSISTED_APICinstead of two
* Pass xcpyshinfo as a pointer inlibxl__arch_get_physinfo
* Set assisted_x{2}apic_available to be conditional upon "bsp" and annotate it with__ro_after_init
* Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to.._X86_ASSISTED_X{2}APIC * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within sysctl.h * Fix padding introduced in struct xen_sysctl_physinfo and bump XEN_SYSCTL_INTERFACE_VERSION --- tools/golang/xenlight/helpers.gen.go | 4 ++++ tools/golang/xenlight/types.gen.go | 6 ++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl.c | 3 +++ tools/libs/light/libxl_arch.h | 4 ++++ tools/libs/light/libxl_arm.c | 5 +++++ tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 11 +++++++++++ tools/ocaml/libs/xc/xenctrl.ml | 5 +++++ tools/ocaml/libs/xc/xenctrl.mli | 5 +++++ tools/xl/xl_info.c | 6 ++++-- xen/arch/x86/hvm/vmx/vmcs.c | 9 +++++++++ xen/arch/x86/include/asm/domain.h | 3 +++ xen/arch/x86/sysctl.c | 7 +++++++ xen/include/public/sysctl.h | 8 +++++++- 15 files changed, 82 insertions(+), 3 deletions(-)tools/golang/xenlight/helpers.gen.go +4
-0@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace) x.CapVpmu = bool(xc.cap_vpmu) x.CapGnttabV1 = bool(xc.cap_gnttab_v1) x.CapGnttabV2 = bool(xc.cap_gnttab_v2) +x.CapAssistedXapic = bool(xc.cap_assisted_xapic) +x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic) return nil}@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace) xc.cap_vpmu = C.bool(x.CapVpmu) xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1) xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2) +xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic) +xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
Quoted 15 lines
tools/golang/xenlight/types.gen.go RunHotplugScripts Defbool · line 394
Quoted 13 lines
tools/golang/xenlight/helpers.gen.go +0
-0return nil }tools/golang/xenlight/types.gen.go +3
-0@@ -389,6 +389,10 @@ RunHotplugScripts Defbool DriverDomain Defbool Passthrough Passthrough XendSuspendEvtchnCompat Defbool +ArchX86 struct { +AssistedXapic Defbool +AssistedX2Apic DefboolDon't you need some indentation here?
I hadn't realized it appeared like this here (and the same happens for other parts of my code as I'm seeing now) because the git output is correct. I will fix it.
Quoted 3 lines
Also name would better be Assistedx{2}APIC IMO if possible. Having a capital 'X' and lowercase 'apic' looks really strange.
Okay.
Quoted 136 lines
Quoted 136 lines
tools/golang/xenlight/types.gen.go +3
-0+} } type DomainRestoreParams struct {@@ -1014,6 +1018,8 @@ CapVmtrace bool CapVpmu bool CapGnttabV1 bool CapGnttabV2 bool +CapAssistedXApic bool +CapAssistedX2apic bool } type Connectorinfo struct {tools/include/libxl.h +7
-0@@ -528,6 +528,13 @@#define LIBXL_HAVE_MAX_GRANT_VERSION 1/* +* LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has +* cap_assisted_x{2}apic fields, which indicates the availability of x{2}APIC +* hardware assisted virtualization. +*/ +#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility tools/libs/light/libxl.c +3
-0@@ -15,6 +15,7 @@#include "libxl_osdeps.h" #include "libxl_internal.h" +#include "libxl_arch.h" int libxl_ctx_alloc (libxl_ctx **pctx,int version,unsigned flags,xentoollog_logger * lg)@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) physinfo->cap_gnttab_v2 = !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2); +libxl__arch_get_physinfo (physinfo, &xcphysinfo); +GC_FREE ;return 0 ; }tools/libs/light/libxl_arch.h +4
-0@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc, uint64_t *out);_hidden +void libxl__arch_get_physinfo(libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo); + +_hidden void libxl__arch_update_domain_config(libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src);tools/libs/light/libxl_arm.c +5
-0@@ -1431,6 +1431,11 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)tools/libs/light/libxl_types.idl +2
-0@@ -1068,6 +1068,8 @@ libxl_physinfo = Struct("physinfo", [ ("cap_vpmu", bool), ("cap_gnttab_v1", bool), ("cap_gnttab_v2", bool), + ("cap_assisted_xapic", bool), + ("cap_assisted_x2apic", bool), ], dir=DIR_OUT) libxl_connectorinfo = Struct("connectorinfo", [tools/libs/light/libxl_x86.c +11
-0@@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,return rc; } +void libxl__arch_get_physinfo (libxl_physinfo *physinfo, +const xc_physinfo_t *xcphysinfo) +{ + physinfo->cap_assisted_xapic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC ); + physinfo->cap_assisted_x2apic = + !!(xcphysinfo->arch_capabilities & +XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC ); +} +void libxl__arch_update_domain_config (libxl__gc *gc,libxl_domain_config *dst,const libxl_domain_config *src)tools/ocaml/libs/xc/xenctrl.ml +4
-0@@ -127,6 +127,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int;@@ -139,6 +143,7 @@ type physinfo = scrub_pages : nativeint;
Quoted 8 lines
tools/ocaml/libs/xc/xenctrl.ml type physinfo = · line 146
Quoted 3 lines
tools/ocaml/libs/xc/xenctrl.ml +1
-0(* XXX hw_cap *) capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list;I know very little about Ocaml, but I think you are not setting this field anywhere? I would expect a call to
ocaml_list_to_c_bitmapand then you will likely need to defineXEN_SYSCTL_PHYSCAP_X86_MAXso you can check the options. SeeXEN_SYSCTL_PHYSCAP_MAXfor example.
Yes, you're right, I will add that in the v3.
Quoted 93 lines
Quoted 93 lines
tools/ocaml/libs/xc/xenctrl.ml +0
-0max_nr_cpus : int; }tools/ocaml/libs/xc/xenctrl.mli +5
-0@@ -112,6 +112,10 @@ type physinfo_cap_flag = | CAP_Gnttab_v1 | CAP_Gnttab_v2 +type physinfo_cap_arch_flag = + | CAP_ARCH_ASSISTED_XAPIC + | CAP_ARCH_ASSISTED_X2APIC + type physinfo = { threads_per_core : int; cores_per_socket : int;@@ -122,6 +126,7 @@ type physinfo = { free_pages : nativeint; scrub_pages : nativeint; capabilities : physinfo_cap_flag list; + arch_capabilities : physinfo_cap_arch_flag list; max_nr_cpus : int; (** compile-time max possible number of nr_cpus *) } type version = { major : int; minor : int; extra : string; }tools/xl/xl_info.c +4
-2@@ -210,7 +210,7 @@ static void output_physinfo(void) info.hw_cap[4 ], info.hw_cap[5 ], info.hw_cap[6 ], info.hw_cap[7 ] ); - maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s\n", + maybe_printf("virt_caps :%s%s%s%s%s%s%s%s%s%s%s%s%s\n", info.cap_pv ? " pv" : "", info.cap_hvm ? " hvm" : "", info.cap_hvm && info.cap_hvm_directio ? " hvm_directio" : "",@@ -221,7 +221,9 @@ static void output_physinfo(void) info.cap_vmtrace ?" vmtrace" :"" , info.cap_vpmu ?" vpmu" :"" , info.cap_gnttab_v1 ?" gnttab-v1" :"" , - info.cap_gnttab_v2 ?" gnttab-v2" :"" + info.cap_gnttab_v2 ?" gnttab-v2" :"" , + info.cap_assisted_xapic ?" assisted_xapic" :"" , + info.cap_assisted_x2apic ?" assisted_x2apic" :"" ); vinfo =libxl_get_version_info (ctx);xen/arch/x86/hvm/vmx/vmcs.c +9
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode; + } +/* The IA32_VMX_EPT_VPID_CAP MSR exists only when EPT or VPID available */ if ( _vmx_secondary_exec_control & (SECONDARY_EXEC_ENABLE_EPT |SECONDARY_EXEC_ENABLE_VPID ) )xen/arch/x86/include/asm/domain.h +3
-0@@ -756,6 +756,9 @@ static inline void pv_inject_sw_interrupt(unsigned int vector) : is_pv_32bit_domain(d) ? PV32_VM_ASSIST_MASK \ : PV64_VM_ASSIST_MASK) +extern bool assisted_xapic_available; +extern bool assisted_x2apic_available; +#endif /* __ASM_DOMAIN_H__ */ /* xen/arch/x86/sysctl.c +0
-0@@ -69,6 +69,9 @@ struct l3_cache_info {unsigned long size; };
Quoted 4 lines
xen/arch/x86/sysctl.c struct l3_cache_info { · line 73
Quoted 2 lines
xen/arch/x86/sysctl.c +2
-0+bool __ro_after_init assisted_xapic_available; +bool __ro_after_init assisted_x2apic_available;You could likely shorten this by dropping the
_availablesuffix.
Okay.
Thanks,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 15 lines · Jane Malalane
Quoted 14 lines · Roger Pau Monné
tools/golang/xenlight/types.gen.go RunHotplugScripts Defbool · line 394
Quoted 11 lines · Jane Malalane
tools/golang/xenlight/types.gen.go +3
-0@@ -389,6 +389,10 @@ RunHotplugScripts Defbool DriverDomain Defbool Passthrough Passthrough XendSuspendEvtchnCompat Defbool +ArchX86 struct { +AssistedXapic Defbool +AssistedX2Apic DefboolDon't you need some indentation here?
Quoted 7 lines
I hadn't realized it appeared like this here (and the same happens for other parts of my code as I'm seeing now) because the git output is correct. I will fix it.
Quoted 3 lines
Also name would better be Assistedx{2}APIC IMO if possible. Having a capital 'X' and lowercase 'apic' looks really strange.
Okay.
This is a generated file, you can't change indentation or fields names. It would be rebuilt automatically if you had golang installed and where rebuilding all the tools.
There's two ways to generate it, you could install golang and build all the tools. Or without golang: justcd tools/golang/xenlight; make types.gen.go. Both should regenerate both "helpers.gen.go" "types.gen.go" files.
There's an even easier way, tell the committer to regen the files when committing :-).
Cheers,
Signature · 1 line
Anthony PERARD
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 29 lines · Anthony PERARD
Quoted 15 lines · Jane Malalane
Quoted 14 lines · Roger Pau Monné
tools/golang/xenlight/types.gen.go RunHotplugScripts Defbool · line 394
Quoted 11 lines · Jane Malalane
tools/golang/xenlight/types.gen.go +3
-0@@ -389,6 +389,10 @@ RunHotplugScripts Defbool DriverDomain Defbool Passthrough Passthrough XendSuspendEvtchnCompat Defbool +ArchX86 struct { +AssistedXapic Defbool +AssistedX2Apic DefboolDon't you need some indentation here?
Quoted 7 lines
I hadn't realized it appeared like this here (and the same happens for other parts of my code as I'm seeing now) because the git output is correct. I will fix it.
Quoted 3 lines
Also name would better be Assistedx{2}APIC IMO if possible. Having a capital 'X' and lowercase 'apic' looks really strange.
Okay.
This is a generated file, you can't change indentation or fields names. It would be rebuilt automatically if you had golang installed and where rebuilding all the tools.
Quoted 7 lines
There's two ways to generate it, you could install golang and build all the tools. Or without golang: just
cd tools/golang/xenlight; make types.gen.go. Both should regenerate both "helpers.gen.go" "types.gen.go" files.There's an even easier way, tell the committer to regen the files when committing :-).
Oh, yeah I forgot they were automatically generated. Right, thank you.
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
In reply to Jane MalalaneQuoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0
xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;
I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with virtualize_apic_accesses available, while for x2APIC we require virtualize_x2apic_mode plus either apic_reg_virt or virtual_intr_delivery.
I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as virtualize_x2apic_mode is available.
This will likely have some effect on patch 2 also, as you will have to adjust vmx_vlapic_msr_changed.
Thanks, Roger.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 23 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.
Quoted 8 lines
I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only virtualize_x2apic_mode set as, in this case, a VM exit will be avoided only when trying to access the TPR register.
Thanks,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.
Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.
I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES or SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE doesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.
So maybe we should only report hardware assisted support when the mentioned features are present together with SECONDARY_EXEC_APIC_REGISTER_VIRT?
Thanks, Roger.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?
Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Jan
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.
Quoted 9 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Jan
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just VIRTUALIZE_APIC_ACCESSES or VIRTUALIZE_X2APIC_MODE but still claim that x{2}apic is virtualized if no MSR accesses are intercepted with XEN_HVM_CPUID_X2APIC_VIRT (in traps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.
Thanks,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
Jan
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.
Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.
and going for assisted_x2apic_available =cpu_has_vmx_virtualize_x2apic_mode.
This would prevent the customer from expecting full acceleration when apic_register_virt and/or virtual_intr_delivery aren't available whilst still offering some if they are not available as Xen currently does. In a future patch, we could also expose and add config options for these controls if we wanted to.
Thank you for your help,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 80 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
Review
Quoted 6 lines
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.
But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Jan
Quoted 12 lines
and going for
assisted_x2apic_available=cpu_has_vmx_virtualize_x2apic_mode.This would prevent the customer from expecting full acceleration when
apic_register_virtand/orvirtual_intr_deliveryaren't available whilst still offering some if they are not available as Xen currently does. In a future patch, we could also expose and add config options for these controls if we wanted to.Thank you for your help,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
Thanks,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 94 lines · Jane Malalane
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Quoted 5 lines
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
I agree here, albeit I'd like to mention that EPT faults are also VM exits. All my earlier comment was about is that this piece of doc wants to express reality, whichever way it is that things end up being implemented.
Jan
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 103 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 94 lines · Jane Malalane
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Quoted 5 lines
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
Quoted 4 lines
I agree here, albeit I'd like to mention that EPT faults are also VM exits. All my earlier comment was about is that this piece of doc wants to express reality, whichever way it is that things end up being implemented.
Oh yes. Right, I see how this info could be misleading.
How about this?...
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
+
+B<(x86 only)> Enables or disables hardware assisted virtualization for +xAPIC. With this option enabled, a memory-mapped APIC access will be +decoded by hardware and either issue a VM exit with an exit reason +instead of an EPT fault or altogether avoid a VM exit. Notice +full virtualization for xAPIC can only be achieved if hardware +supports “APIC-register virtualization” and “virtual-interrupt +delivery”. The default is settable via L<xl.conf(5)>.
+=item B<assisted_x2apic=BOOLEAN>
+
+B<(x86 only)> Enables or disables hardware assisted virtualization for +x2APIC. With this option enabled, an MSR-Based APIC access will either +issue a VM exit or altogether avoid one. Notice full virtualization +for x2APIC can only be achieved if hardware supports “APIC-register +virtualization” and “virtual-interrupt delivery”. The default is +settable via L<xl.conf(5)>.
...because with only VIRTUALIZE_APIC_ACCESSES enabled, hardware decodes accesses to the xAPIC page and the VM exit gives an exit reason. And if VIRTUALIZE_X2APIC_MODE is set, although no assistance is provided w.r.t. to decoding x2APIC accesses as the MSR that the VM tried to access is already part of the vmexit information, VM exits for accesses to the TPR MSR are avoided, regardless of whether shadow TPR is set or not for e.g.
I hope this makes sense but I welcome any other suggestions/corrections.
Thank you,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 104 lines · Jane Malalane
Quoted 103 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 99 lines · Jane Malalane
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 5 lines · Jane Malalane
xen/arch/x86/hvm/vmx/vmcs.c +0
-0xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(bool bsp) · line 352
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmcs.c +7
-0@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp) MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch); } +/* Check whether hardware supports accelerated xapic and x2apic. */ +if ( bsp ) + { + assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses; + assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt || + cpu_has_vmx_virtual_intr_delivery) && + cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
Quoted 8 lines
Quoted 4 lines
I agree here, albeit I'd like to mention that EPT faults are also VM exits. All my earlier comment was about is that this piece of doc wants to express reality, whichever way it is that things end up being implemented.
Oh yes. Right, I see how this info could be misleading.
How about this?...
Getting close. The thing I can't judge is whether this level of technical detail is suitable for this doc. Just one further remark:
Review
Quoted 6 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +5
-0+ +B<(x86 only)> Enables or disables hardware assisted virtualization for +xAPIC. With this option enabled, a memory-mapped APIC access will be +decoded by hardware and either issue a VM exit with an exit reason +instead of an EPT fault or altogether avoid a VM exit. Notice
As said before, EPT faults also are VM exits and also provide an exit reason. Therefore maybe "... and either issue a VM exit with a more specific exit reason than an EPT fault would provide, or altogether avoid a VM exit" or "... and either issue a more specific VM exit than just an EPT fault, or altogether avoid a VM exit"?
Jan
Quoted 27 lines
(quoted diff) +11
-0+full virtualization for xAPIC can only be achieved if hardware +supports “APIC-register virtualization” and “virtual-interrupt +delivery”. The default is settable via L<xl.conf(5)>. +=item B<assisted_x2apic=BOOLEAN> + +B<(x86 only)> Enables or disables hardware assisted virtualization for +x2APIC. With this option enabled, an MSR-Based APIC access will either +issue a VM exit or altogether avoid one. Notice full virtualization +for x2APIC can only be achieved if hardware supports “APIC-register +virtualization” and “virtual-interrupt delivery”. The default is +settable via L<xl.conf(5)>....because with only
VIRTUALIZE_APIC_ACCESSESenabled, hardware decodes accesses to the xAPIC page and the VM exit gives an exit reason. And ifVIRTUALIZE_X2APIC_MODEis set, although no assistance is provided w.r.t. to decoding x2APIC accesses as the MSR that the VM tried to access is already part of the vmexit information, VM exits for accesses to the TPR MSR are avoided, regardless of whether shadow TPR is set or not for e.g.I hope this makes sense but I welcome any other suggestions/corrections.
Thank you,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 112 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 109 lines · Jane Malalane
Quoted 107 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 94 lines · Jane Malalane
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
Quoted 15 lines · Jane Malalane
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 7ab15e07a0..4060aef1bd 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch);
}+ /* Check whether hardware supports accelerated xapic and x2apic. */
+ if ( bsp )
+ {
+ assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+ assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
+ cpu_has_vmx_virtual_intr_delivery) &&
+ cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for xAPIC mode we report hw assisted support only with
virtualize_apic_accessesavailable, while for x2APIC we requirevirtualize_x2apic_modeplus eitherapic_reg_virtorvirtual_intr_delivery.I think we likely need to be more consistent here, and report hw assisted x2APIC support as long as
virtualize_x2apic_modeis available.This will likely have some effect on patch 2 also, as you will have to adjust
vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Quoted 5 lines
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
I agree here, albeit I'd like to mention that EPT faults are also VM exits. All my earlier comment was about is that this piece of doc wants to express reality, whichever way it is that things end up being implemented.
Quoted 6 lines
Quoted 3 lines
Oh yes. Right, I see how this info could be misleading.
How about this?...
Getting close. The thing I can't judge is whether this level of technical detail is suitable for this doc. Just one further remark:
Unsure too.
Quoted 4 lines
Quoted 4 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +3
-0+ +B<(x86 only)> Enables or disables hardware assisted virtualization for +xAPIC. With this option enabled, a memory-mapped APIC access will be
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+decoded by hardware and either issue a VM exit with an exit reason +instead of an EPT fault or altogether avoid a VM exit. NoticeAs said before, EPT faults also are VM exits and also provide an exit reason. Therefore maybe "... and either issue a VM exit with a more specific exit reason than an EPT fault would provide, or altogether avoid a VM exit" or "... and either issue a more specific VM exit than just an EPT fault, or altogether avoid a VM exit"?
Yes, that's better.
Thank you,
Jane.
Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86
Quoted 135 lines · Jane Malalane
Quoted 112 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 109 lines · Jane Malalane
Quoted 107 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 94 lines · Jane Malalane
Quoted 82 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 79 lines · Jane Malalane
Quoted 70 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 58 lines · Jane Malalane
Quoted 50 lines · Jan Beulich
[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
Quoted 47 lines · Roger Pau Monné
Quoted 31 lines · Jane Malalane
Quoted 30 lines · Roger Pau Monné
On Mon, Feb 07, 2022 at 06:21:00PM +0000, Jane Malalane wrote:
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 7ab15e07a0..4060aef1bd 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
MSR_IA32_VMX_PROCBASED_CTLS2, &mismatch);
}+ /* Check whether hardware supports accelerated xapic and x2apic. */
+ if ( bsp )
+ {
+ assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
+ assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
+ cpu_has_vmx_virtual_intr_delivery) &&
+ cpu_has_vmx_virtualize_x2apic_mode;I've been think about this, and it seems kind of asymmetric that for
xAPIC mode we report hw assisted support only with
virtualize_apic_accesses available, while for x2APIC we require
virtualize_x2apic_mode plus either apic_reg_virt or
virtual_intr_delivery.I think we likely need to be more consistent here, and report hw
assisted x2APIC support as long as virtualize_x2apic_mode is
available.This will likely have some effect on patch 2 also, as you will have to
adjust vmx_vlapic_msr_changed.Quoted 6 lines
Quoted 1 line
Thanks, Roger.
Any other thoughts on this? As on one hand it is asymmetric but also there isn't much assistance with only
virtualize_x2apic_modeset as, in this case, a VM exit will be avoided only when trying to access the TPR register.I've been thinking about this, and reporting hardware assisted x{2}APIC virtualization with just
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESorSECONDARY_EXEC_VIRTUALIZE_X2APIC_MODEdoesn't seem very helpful. While those provide some assistance to the VMM in order to handle APIC accesses, it will still require a trap into the hypervisor to handle most of the accesses.Quoted 7 lines
Quoted 3 lines
So maybe we should only report hardware assisted support when the mentioned features are present together with
SECONDARY_EXEC_APIC_REGISTER_VIRT?Not sure - "some assistance" seems still a little better than none at all. Which route to go depends on what exactly we intend the bit to be used for.
Quoted 8 lines
True. I intended this bit to be specifically for enabling assisted_x{2}apic. So, would it be inconsistent to report hardware assistance with just
VIRTUALIZE_APIC_ACCESSESorVIRTUALIZE_X2APIC_MODEbut still claim that x{2}apic is virtualized if no MSR accesses are intercepted withXEN_HVM_CPUID_X2APIC_VIRT(intraps.c) so that, as you say, the guest gets at least "some assistance" instead of none but we still claim x{2}apic virtualization when it is actually complete? Maybe I could also add a comment alluding to this in the xl documentation.Quoted 6 lines
To rephrase my earlier point: Which kind of decisions are the consumer(s) of us reporting hardware assistance going to take? In how far is there a risk that "some assistance" is overall going to lead to a loss of performance? I guess I'd need to see comment and actual code all in one place ...
So, I was thinking of adding something along the lines of:
Quoted 10 lines
Review
Quoted 7 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +6
-0+Enables or disables hardware assisted virtualization for xAPIC. This +allows accessing APIC registers without a VM-exit. Notice enabling +this does not guarantee full virtualization for xAPIC, as this can +only be achieved if hardware supports “APIC-register virtualization” +and “virtual-interrupt delivery”. The default is settable via +L<xl.conf(5)>.But isn't this contradictory? Doesn't lack of APIC-register virtualization mean VM exits upon (most) accesses?
Quoted 5 lines
Yes, it does mean. I guess the alternative wouuld be then to require APIC-register virtualization for enabling xAPIC. But also, although this doesn't provide much acceleration, even getting a VM exit is some assistance if compared to instead getting an EPT fault and having to decode the access.
I agree here, albeit I'd like to mention that EPT faults are also VM exits. All my earlier comment was about is that this piece of doc wants to express reality, whichever way it is that things end up being implemented.
Quoted 6 lines
Quoted 3 lines
Oh yes. Right, I see how this info could be misleading.
How about this?...
Getting close. The thing I can't judge is whether this level of technical detail is suitable for this doc. Just one further remark:
Unsure too.
Quoted 4 lines
Quoted 4 lines
+=item B<assisted_xapic=BOOLEAN> B<(x86 only)>
(quoted diff) +3
-0+ +B<(x86 only)> Enables or disables hardware assisted virtualization for +xAPIC. With this option enabled, a memory-mapped APIC access will beQuoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+decoded by hardware and either issue a VM exit with an exit reason +instead of an EPT fault or altogether avoid a VM exit. NoticeAs said before, EPT faults also are VM exits and also provide an exit reason. Therefore maybe "... and either issue a VM exit with a more specific exit reason than an EPT fault would provide, or altogether avoid a VM exit" or "... and either issue a more specific VM exit than just an EPT fault, or altogether avoid a VM exit"?
Quoted 1 line
Yes, that's better.
I would avoid mentioning EPT, as that's an Intel specific technology. Could we instead use 'p2m fault'?
Thanks, Roger.
PATCH v2 2/2 x86/xen: Allow per-domain usage of hardware virtualized APIC
In reply to Jane MalalaneIntroduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
A per-domain option is added to xl in order to select the usage of x{2}APIC hardware assisted vitualization, as well as a global configuration option.
Having all APIC interaction exit to Xen for emulation is slow and can induce much overhead. Hardware can speed up x{2}APIC by running APIC read/write accesses without taking a VM exit.
Being able to disable x{2}APIC hardware assisted vitualization can be useful for testing and debugging purposes.
---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Christian Lindig <[email protected]>
CC: David Scott <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: "Roger Pau Monné" <[email protected]>
v2:
* Add a LIBXL_HAVE_ASSISTED_APIC macro
* Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
* Add a return statement in now "int" libxl__arch_domain_build_info_setdefault
* Preserve libxl__arch_domain_build_info_setdefault 's location in libxl_create.c
* Correct x{2}apic default setting logic in libxl__arch_domain_prepare_config
* Correct logic for parsing assisted_x{2}apic host/guest options in xl_parse.c and initialize them to -1 in xl.c
* Use guest options directly in vmx_vlapic_msr_changed
* Fix indentation of bool assisted_x{2}apic in struct hvm_domain
* Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
--- docs/man/xl.cfg.5.pod.in | 10 ++++++++++ docs/man/xl.conf.5.pod.in | 12 ++++++++++++ tools/golang/xenlight/helpers.gen.go | 12 ++++++++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl_arch.h | 5 +++-- tools/libs/light/libxl_arm.c | 7 +++++-- tools/libs/light/libxl_create.c | 23 ++++++++++++++--------- tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 31 +++++++++++++++++++++++++++++-- tools/ocaml/libs/xc/xenctrl.ml | 2 ++ tools/ocaml/libs/xc/xenctrl.mli | 2 ++ tools/ocaml/libs/xc/xenctrl_stubs.c | 2 +- tools/xl/xl.c | 8 ++++++++ tools/xl/xl.h | 2 ++ tools/xl/xl_parse.c | 16 ++++++++++++++++ xen/arch/x86/domain.c | 28 +++++++++++++++++++++++++++- xen/arch/x86/hvm/vmx/vmcs.c | 4 ++++ xen/arch/x86/hvm/vmx/vmx.c | 14 +++++--------- xen/arch/x86/include/asm/hvm/domain.h | 6 ++++++ xen/arch/x86/traps.c | 8 ++++---- xen/include/public/arch-x86/xen.h | 2 ++ 21 files changed, 173 insertions(+), 30 deletions(-)
docs/man/xl.cfg.5.pod.in +10 -0
@@ -1862,6 +1862,16 @@ firmware tables when using certain older guest Operating Systems. These tables have been superseded by newer constructs within the ACPI tables. +=item B<assisted_xAPIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for xapic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + +=item B<assisted_x2APIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for x2apic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + =item B<nx=BOOLEAN> B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a guest
docs/man/xl.conf.5.pod.in +12 -0
@@ -107,6 +107,18 @@ Sets the default value for the C<max_grant_version> domain config value. Default: maximum grant version supported by the hypervisor. +=item B<assisted_xAPIC=BOOLEAN> + +If enabled, domains will use xAPIC hardware assisted virtualization by default. + +Default: enabled if supported. + +=item B<assisted_x2APIC=BOOLEAN> + +If enabled, domains will use x2APIC hardware assisted virtualization by default. + +Default: enabled if supported. + =item B<vif.default.script="PATH"> Configures the default hotplug script used by virtual network devices.
tools/golang/xenlight/helpers.gen.go +12 -0
@@ -636,6 +636,12 @@ x.Passthrough = Passthrough(xc.passthrough) if err := x.XendSuspendEvtchnCompat.fromC(&xc.xend_suspend_evtchn_compat);err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.fromC(&xc.arch_x86.assisted_xapic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.fromC(&xc.arch_x86.assisted_x2apic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil}
@@ -679,6 +685,12 @@ xc.passthrough = C.libxl_passthrough(x.Passthrough) if err := x.XendSuspendEvtchnCompat.toC(&xc.xend_suspend_evtchn_compat); err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.toC(&xc.arch_x86.assisted_xapic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.toC(&xc.arch_x86.assisted_x2apic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil }
tools/include/libxl.h +7 -0
@@ -535,6 +535,13 @@#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1/* +* LIBXL_HAVE_ASSISTED_APIC indicates that libxl_domain_build_info has +* assisted_x{2}apic fields, for enabling hardware assisted virtualization for +* x{2}apic per domain. +*/ +#define LIBXL_HAVE_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility
tools/libs/light/libxl_arch.h +3 -2
@@ -71,8 +71,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, libxl_domain_create_info *c_info);_hidden -void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, -libxl_domain_build_info *b_info); +int libxl__arch_domain_build_info_setdefault(libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo); _hidden int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
tools/libs/light/libxl_arm.c +5 -2
@@ -1384,8 +1384,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, } } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {/* ACPI is disabled by default */ libxl_defbool_setdefault (&b_info->acpi, false);
@@ -1399,6 +1400,8 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,memset (&b_info->u,'\0' ,sizeof (b_info->u)); b_info->type =LIBXL_DOMAIN_TYPE_INVALID ;libxl_domain_build_info_init_type (b_info,LIBXL_DOMAIN_TYPE_PVH ); + +return 0 ; } int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
tools/libs/light/libxl_create.c +14 -9
@@ -264,7 +264,20 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,if (!b_info->event_channels) b_info->event_channels =1023 ; -libxl__arch_domain_build_info_setdefault (gc, b_info); +libxl_physinfo info; + + rc =libxl_get_physinfo (CTX , &info); +if (rc) { +LOG (ERROR ,"failed to get hypervisor info" ); +return rc; + } + + rc =libxl__arch_domain_build_info_setdefault (gc, b_info, &info); +if (rc) { +LOG (ERROR ,"unable to set domain arch build info defaults" ); +return rc; + } +libxl_defbool_setdefault (&b_info->dm_restrict, false);if (b_info->iommu_memkb ==LIBXL_MEMKB_DEFAULT )
@@ -457,14 +470,6 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, }if (b_info->max_grant_version ==LIBXL_MAX_GRANT_DEFAULT ) { -libxl_physinfo info; - - rc =libxl_get_physinfo (CTX , &info); -if (rc) { -LOG (ERROR ,"failed to get hypervisor info" ); -return rc; - } -if (info.cap_gnttab_v2) b_info->max_grant_version =2 ;else if (info.cap_gnttab_v1)
tools/libs/light/libxl_types.idl +2 -0
@@ -648,6 +648,8 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("vuart", libxl_vuart_type), ])), ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), + ("assisted_xapic", libxl_defbool), + ("assisted_x2apic", libxl_defbool), ])), # Alternate p2m is not bound to any architecture or guest type, as it is # supported by x86 HVM and ARM support is planned.
tools/libs/light/libxl_x86.c +29 -2
@@ -23,6 +23,14 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,if (libxl_defbool_val (d_config->b_info.arch_x86.msr_relaxed)) config->arch.misc_flags |=XEN_X86_MSR_RELAXED ; +if (d_config->c_info.type !=LIBXL_DOMAIN_TYPE_PV ) + { +if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_xapic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_XAPIC ; + +if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_x2apic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_X2APIC ; + }return 0 ; }
@@ -819,11 +827,30 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, { } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {libxl_defbool_setdefault (&b_info->acpi, true);libxl_defbool_setdefault (&b_info->arch_x86.msr_relaxed, false); + +if (b_info->type !=LIBXL_DOMAIN_TYPE_PV ) + { +libxl_defbool_setdefault (&b_info->arch_x86.assisted_xapic, + physinfo->cap_assisted_xapic); +libxl_defbool_setdefault (&b_info->arch_x86.assisted_x2apic, + physinfo->cap_assisted_x2apic); + } + +if (b_info->type ==LIBXL_DOMAIN_TYPE_PV && + (!libxl_defbool_is_default (b_info->arch_x86.assisted_xapic) || + !libxl_defbool_is_default (b_info->arch_x86.assisted_x2apic))) + { +LOG (ERROR ,"Interrupt Controller Virtualization not supported for PV" ); +return ERROR_INVAL ; + } + +return 0 ; } int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
tools/ocaml/libs/xc/xenctrl.ml +2 -0
@@ -50,6 +50,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = {
tools/ocaml/libs/xc/xenctrl.mli +2 -0
@@ -44,6 +44,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = { emulation_flags: x86_arch_emulation_flags list;
tools/ocaml/libs/xc/xenctrl_stubs.c +1 -1
@@ -239,7 +239,7 @@ CAMLprim value stub_xc_domain_create(value xch, value wanted_domid, value config cfg.arch.misc_flags =ocaml_list_to_c_bitmap /* ! x86_arch_misc_flags X86_ none */ -/* ! XEN_X86_ XEN_X86_MSR_RELAXED all */ +/* ! XEN_X86_ XEN_X86_ASSISTED_X2APIC max */ (VAL_MISC_FLAGS );#undef VAL_MISC_FLAGS
tools/xl/xl.c +8 -0
@@ -57,6 +57,8 @@ int max_grant_frames = -1;int max_maptrack_frames =-1 ;int max_grant_version =LIBXL_MAX_GRANT_DEFAULT ;libxl_domid domid_policy =INVALID_DOMID ; +int assisted_xapic =-1 ; +int assisted_x2apic =-1 ;xentoollog_level minmsglevel = minmsglevel_default;
@@ -201,6 +203,12 @@ static void parse_global_config(const char *configfile,if (!xlu_cfg_get_long (config,"claim_mode" , &l,0 )) claim_mode = l; +if (!xlu_cfg_get_long (config,"assisted_xapic" , &l,0 )) + assisted_xapic = l; + +if (!xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 )) + assisted_x2apic = l; +xlu_cfg_replace_string (config,"remus.default.netbufscript" , &default_remus_netbufscript,0 ); xlu_cfg_replace_string (config, "colo.default.proxyscript",
tools/xl/xl.h +2 -0
@@ -286,6 +286,8 @@ extern libxl_bitmap global_vm_affinity_mask;extern libxl_bitmap global_hvm_affinity_mask;extern libxl_bitmap global_pv_affinity_mask;extern libxl_domid domid_policy; +extern int assisted_xapic; +extern int assisted_x2apic;enum output_format {OUTPUT_FORMAT_JSON ,
tools/xl/xl_parse.c +16 -0
@@ -1681,6 +1681,22 @@ void parse_config_data(const char *config_source,xlu_cfg_get_defbool (config,"vpt_align" , &b_info->u.hvm.vpt_align,0 );xlu_cfg_get_defbool (config,"apic" , &b_info->apic,0 ); + e =xlu_cfg_get_long (config,"assisted_xapic" , &l ,0 ); +if ((e ==ESRCH && assisted_xapic !=-1 ))/* use global default if present */ +libxl_defbool_set (&b_info->arch_x86.assisted_xapic, assisted_xapic); +else if (!e) +libxl_defbool_set (&b_info->arch_x86.assisted_xapic, l); +else +exit (1 ); + + e =xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 ); +if ((e ==ESRCH && assisted_x2apic !=-1 ))/* use global default if present */ +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, assisted_x2apic); +else if (!e) +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, l); +else +exit (1 ); +switch (xlu_cfg_get_list (config,"viridian" , &viridian, &num_viridian,1 )) {
xen/arch/x86/domain.c +27 -1
@@ -619,6 +619,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)bool hvm = config->flags & XEN_DOMCTL_CDF_hvm;bool hap = config->flags & XEN_DOMCTL_CDF_hap;bool nested_virt = config->flags & XEN_DOMCTL_CDF_nested_virt; +bool assisted_xapic = config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; +bool assisted_x2apic = config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ;unsigned int max_vcpus;if ( hvm ? !hvm_enabled : !IS_ENABLED (CONFIG_PV ) )
@@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) } } -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED ) +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED | +XEN_X86_ASSISTED_XAPIC | +XEN_X86_ASSISTED_X2APIC ) ) {dprintk (XENLOG_INFO ,"Invalid arch misc flags %#x\n" , config->arch.misc_flags);return -EINVAL ; } +if ( (assisted_xapic || assisted_x2apic) && !hvm ) + { +dprintk (XENLOG_INFO , +"Interrupt Controller Virtualization not supported for PV\n" ); +return -EINVAL ; + } + +if ( (assisted_xapic && !assisted_xapic_available) || + (assisted_x2apic && !assisted_x2apic_available) ) + { +dprintk (XENLOG_INFO , +"Hardware assisted x%sAPIC requested but not available\n" , + assisted_xapic && !assisted_xapic_available ?"" :"2" ); +return -EINVAL ; + } +return 0 ; }
@@ -863,6 +883,12 @@ int arch_domain_create(struct domain *d, d->arch.msr_relaxed = config->arch.misc_flags &XEN_X86_MSR_RELAXED ; + d->arch.hvm.assisted_xapic = + config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; + + d->arch.hvm.assisted_x2apic = + config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ; +return 0 ; fail:
xen/arch/x86/hvm/vmx/vmcs.c +4 -0
@@ -1157,6 +1157,10 @@ static int construct_vmcs(struct vcpu *v)__vmwrite (PLE_WINDOW , ple_window); } +if ( !v->domain->arch.hvm.assisted_xapic ) + v->arch.hvm.vmx.secondary_exec_control &= + ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +if ( cpu_has_vmx_secondary_exec_control )__vmwrite (SECONDARY_VM_EXEC_CONTROL , v->arch.hvm.vmx.secondary_exec_control);
xen/arch/x86/hvm/vmx/vmx.c +5 -9
@@ -3333,16 +3333,11 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)void vmx_vlapic_msr_changed (struct vcpu *v) { -int virtualize_x2apic_mode;struct vlapic *vlapic =vcpu_vlapic (v);unsigned int msr; - virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt || - cpu_has_vmx_virtual_intr_delivery) && - cpu_has_vmx_virtualize_x2apic_mode ); - -if ( !cpu_has_vmx_virtualize_apic_accesses && - !virtualize_x2apic_mode ) +if ( ! v->domain->arch.hvm.assisted_xapic && + ! v->domain->arch.hvm.assisted_x2apic )return ;vmx_vmcs_enter (v);
@@ -3352,7 +3347,8 @@ void vmx_vlapic_msr_changed(struct vcpu *v)if ( !vlapic_hw_disabled (vlapic) && (vlapic_base_address (vlapic) ==APIC_DEFAULT_PHYS_BASE ) ) { -if ( virtualize_x2apic_mode &&vlapic_x2apic_mode (vlapic) ) +if ( v->domain->arch.hvm.assisted_x2apic + &&vlapic_x2apic_mode (vlapic) ) { v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE ;
@@ -3373,7 +3369,7 @@ void vmx_vlapic_msr_changed(struct vcpu *v)vmx_clear_msr_intercept (v,MSR_X2APIC_SELF ,VMX_MSR_W ); } } -else +else if ( v->domain->arch.hvm.assisted_xapic ) v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; }
xen/arch/x86/include/asm/hvm/domain.h +6 -0
@@ -117,6 +117,12 @@ struct hvm_domain {bool is_s3_suspended; +/* xAPIC hardware assisted virtualization. */ +bool assisted_xapic; + +/* x2APIC hardware assisted virtualization. */ +bool assisted_x2apic; +/* hypervisor intercepted msix table */ struct list_head msixtbl_list;
xen/arch/x86/traps.c +4 -4
@@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,if ( !is_hvm_domain (d) || subleaf !=0 )break ; -if ( cpu_has_vmx_apic_reg_virt ) +if ( cpu_has_vmx_apic_reg_virt && + v->domain->arch.hvm.assisted_xapic ) res->a |=XEN_HVM_CPUID_APIC_ACCESS_VIRT ;/*
@@ -1124,9 +1125,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,* and wrmsr in the guest will run without VMEXITs (see * vmx_vlapic_msr_changed()). */ -if ( cpu_has_vmx_virtualize_x2apic_mode && - cpu_has_vmx_apic_reg_virt && - cpu_has_vmx_virtual_intr_delivery ) +if ( (cpu_has_vmx_apic_reg_virt && cpu_has_vmx_virtual_intr_delivery) && + v->domain->arch.hvm.assisted_x2apic ) res->a |=XEN_HVM_CPUID_X2APIC_VIRT ;/*
xen/include/public/arch-x86/xen.h +2 -0
@@ -317,6 +317,8 @@ struct xen_arch_domainconfig {* doesn't allow the guest to read or write to the underlying MSR. */ #define XEN_X86_MSR_RELAXED (1u << 0) +#define XEN_X86_ASSISTED_XAPIC (1u << 1) +#define XEN_X86_ASSISTED_X2APIC (1u << 2)uint32_t misc_flags; };
Signature · 1 line
2.11.0
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
Quoted 23 lines · Jane Malalane
Introduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
---
docs/man/xl.cfg.5.pod.in | 10 ++++++++++ docs/man/xl.conf.5.pod.in | 12 ++++++++++++ tools/golang/xenlight/helpers.gen.go | 12 ++++++++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl_arch.h | 5 +++-- tools/libs/light/libxl_arm.c | 7 +++++-- tools/libs/light/libxl_create.c | 23 ++++++++++++++--------- tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 31 +++++++++++++++++++++++++++++-- tools/ocaml/libs/xc/xenctrl.ml | 2 ++ tools/ocaml/libs/xc/xenctrl.mli | 2 ++ tools/ocaml/libs/xc/xenctrl_stubs.c | 2 +- tools/xl/xl.c | 8 ++++++++ tools/xl/xl.h | 2 ++
Quoted 8 lines
tools/xl/xl_parse.c | 16 ++++++++++++++++ xen/arch/x86/domain.c | 28 +++++++++++++++++++++++++++- xen/arch/x86/hvm/vmx/vmcs.c | 4 ++++ xen/arch/x86/hvm/vmx/vmx.c | 14 +++++--------- xen/arch/x86/include/asm/hvm/domain.h | 6 ++++++ xen/arch/x86/traps.c | 8 ++++---- xen/include/public/arch-x86/xen.h | 2 ++ 21 files changed, 173 insertions(+), 30 deletions(-)
The changes to the OCaml part are minimal.
— C
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
In reply to Jane MalalaneQuoted 8 lines · Jane Malalane
Introduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
A per-domain option is added to xl in order to select the usage of x{2}APIC hardware assisted vitualization, as well as a global configuration option.
Quoted 6 lines
Having all APIC interaction exit to Xen for emulation is slow and can induce much overhead. Hardware can speed up x{2}APIC by running APIC read/write accesses without taking a VM exit.
Being able to disable x{2}APIC hardware assisted vitualization can be useful for testing and debugging purposes.
I think you have agreed with Jan some changes to the description regarding the purpose of the commit.
Quoted 85 lines
Signed-off-by Jane Malalane <[email protected]>Suggested-by Andrew Cooper <[email protected]>---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Christian Lindig <[email protected]>
CC: David Scott <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: "Roger Pau Monné" <[email protected]>v2:
* Add aLIBXL_HAVE_ASSISTED_APICmacro
* Pass xcpyshinfo as a pointer inlibxl__arch_get_physinfo
* Add a return statement in now "int"libxl__arch_domain_build_info_setdefault
* Preservelibxl__arch_domain_build_info_setdefault's location inlibxl_create.c
* Correct x{2}apic default setting logic inlibxl__arch_domain_prepare_config
* Correct logic for parsing assisted_x{2}apic host/guest options inxl_parse.cand initialize them to -1 inxl.c
* Use guest options directly invmx_vlapic_msr_changed
* Fix indentation of bool assisted_x{2}apic in structhvm_domain
* Add a change inxenctrl_stubs.cto pass xenctrl ABI checks--- docs/man/xl.cfg.5.pod.in | 10 ++++++++++ docs/man/xl.conf.5.pod.in | 12 ++++++++++++ tools/golang/xenlight/helpers.gen.go | 12 ++++++++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl_arch.h | 5 +++-- tools/libs/light/libxl_arm.c | 7 +++++-- tools/libs/light/libxl_create.c | 23 ++++++++++++++--------- tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 31 +++++++++++++++++++++++++++++-- tools/ocaml/libs/xc/xenctrl.ml | 2 ++ tools/ocaml/libs/xc/xenctrl.mli | 2 ++ tools/ocaml/libs/xc/xenctrl_stubs.c | 2 +- tools/xl/xl.c | 8 ++++++++ tools/xl/xl.h | 2 ++ tools/xl/xl_parse.c | 16 ++++++++++++++++ xen/arch/x86/domain.c | 28 +++++++++++++++++++++++++++- xen/arch/x86/hvm/vmx/vmcs.c | 4 ++++ xen/arch/x86/hvm/vmx/vmx.c | 14 +++++--------- xen/arch/x86/include/asm/hvm/domain.h | 6 ++++++ xen/arch/x86/traps.c | 8 ++++---- xen/include/public/arch-x86/xen.h | 2 ++ 21 files changed, 173 insertions(+), 30 deletions(-)docs/man/xl.cfg.5.pod.in +10
-0@@ -1862,6 +1862,16 @@ firmware tables when using certain older guest Operating Systems. These tables have been superseded by newer constructs within the ACPI tables. +=item B<assisted_xAPIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for xapic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + +=item B<assisted_x2APIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for x2apic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + =item B<nx=BOOLEAN> B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a guestdocs/man/xl.conf.5.pod.in +1
-0@@ -107,6 +107,18 @@ Sets the default value for the C<max_grant_version> domain config value. Default: maximum grant version supported by the hypervisor. +=item B<assisted_xAPIC=BOOLEAN>
docs/man/xl.conf.5.pod.in Sets the default value for the C<max_grant_version> domain config value. · line 120
Quoted 10 lines
docs/man/xl.conf.5.pod.in +10
-0+ +If enabled, domains will use xAPIC hardware assisted virtualization by default. + +Default: enabled if supported. + +=item B<assisted_x2APIC=BOOLEAN> + +If enabled, domains will use x2APIC hardware assisted virtualization by default. + +Default: enabled if supported.
We don't capitalize xl options, so I would suggest to lowercase APIC for all the option names.
Quoted 98 lines
docs/man/xl.conf.5.pod.in +1
-0+ =item B<vif.default.script="PATH"> Configures the default hotplug script used by virtual network devices.tools/golang/xenlight/helpers.gen.go +12
-0@@ -636,6 +636,12 @@ x.Passthrough = Passthrough(xc.passthrough) if err := x.XendSuspendEvtchnCompat.fromC(&xc.xend_suspend_evtchn_compat);err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.fromC(&xc.arch_x86.assisted_xapic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.fromC(&xc.arch_x86.assisted_x2apic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil}@@ -679,6 +685,12 @@ xc.passthrough = C.libxl_passthrough(x.Passthrough) if err := x.XendSuspendEvtchnCompat.toC(&xc.xend_suspend_evtchn_compat); err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.toC(&xc.arch_x86.assisted_xapic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.toC(&xc.arch_x86.assisted_x2apic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil }tools/include/libxl.h +7
-0@@ -535,6 +535,13 @@#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1/* +* LIBXL_HAVE_ASSISTED_APIC indicates that libxl_domain_build_info has +* assisted_x{2}apic fields, for enabling hardware assisted virtualization for +* x{2}apic per domain. +*/ +#define LIBXL_HAVE_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility tools/libs/light/libxl_arch.h +3
-2@@ -71,8 +71,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, libxl_domain_create_info *c_info);_hidden -void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, -libxl_domain_build_info *b_info); +int libxl__arch_domain_build_info_setdefault(libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo); _hidden int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/libs/light/libxl_arm.c +5
-2@@ -1384,8 +1384,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, } } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {/* ACPI is disabled by default */ libxl_defbool_setdefault (&b_info->acpi, false);@@ -1399,6 +1400,8 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,memset (&b_info->u,'\0' ,sizeof (b_info->u)); b_info->type =LIBXL_DOMAIN_TYPE_INVALID ;libxl_domain_build_info_init_type (b_info,LIBXL_DOMAIN_TYPE_PVH ); + +return 0 ; } int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/libs/light/libxl_create.c +0
-0
tools/libs/light/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc *gc, · line 267
Quoted 5 lines
tools/libs/light/libxl_create.c +1
-1@@ -264,7 +264,20 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,if (!b_info->event_channels) b_info->event_channels =1023 ; -libxl__arch_domain_build_info_setdefault (gc, b_info); +libxl_physinfo info;
The definition of info needs to be at the top of the function, together with the rest of the variable definitions.
Quoted 49 lines
tools/libs/light/libxl_create.c +13
-8+ + rc =libxl_get_physinfo (CTX , &info); +if (rc) { +LOG (ERROR ,"failed to get hypervisor info" ); +return rc; + } + + rc =libxl__arch_domain_build_info_setdefault (gc, b_info, &info); +if (rc) { +LOG (ERROR ,"unable to set domain arch build info defaults" ); +return rc; + } +libxl_defbool_setdefault (&b_info->dm_restrict, false);if (b_info->iommu_memkb ==LIBXL_MEMKB_DEFAULT )@@ -457,14 +470,6 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, }if (b_info->max_grant_version ==LIBXL_MAX_GRANT_DEFAULT ) { -libxl_physinfo info; - - rc =libxl_get_physinfo (CTX , &info); -if (rc) { -LOG (ERROR ,"failed to get hypervisor info" ); -return rc; - } -if (info.cap_gnttab_v2) b_info->max_grant_version =2 ;else if (info.cap_gnttab_v1)tools/libs/light/libxl_types.idl +2
-0@@ -648,6 +648,8 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("vuart", libxl_vuart_type), ])), ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), + ("assisted_xapic", libxl_defbool), + ("assisted_x2apic", libxl_defbool), ])), # Alternate p2m is not bound to any architecture or guest type, as it is # supported by x86 HVM and ARM support is planned.tools/libs/light/libxl_x86.c +0
-0
tools/libs/light/libxl_x86.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 27
Quoted 5 lines
tools/libs/light/libxl_x86.c +2
-0@@ -23,6 +23,14 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,if (libxl_defbool_val (d_config->b_info.arch_x86.msr_relaxed)) config->arch.misc_flags |=XEN_X86_MSR_RELAXED ; +if (d_config->c_info.type !=LIBXL_DOMAIN_TYPE_PV ) + {
Coding style for libxl is to place the bracket in the same line as the if.
Quoted 23 lines
tools/libs/light/libxl_x86.c +11
-2+if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_xapic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_XAPIC ; + +if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_x2apic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_X2APIC ; + }return 0 ; }@@ -819,11 +827,30 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, { } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {libxl_defbool_setdefault (&b_info->acpi, true);libxl_defbool_setdefault (&b_info->arch_x86.msr_relaxed, false); + +if (b_info->type !=LIBXL_DOMAIN_TYPE_PV )
tools/libs/light/libxl_x86.c void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, · line 847
Quoted 10 lines
tools/libs/light/libxl_x86.c +10
-0+ { +libxl_defbool_setdefault (&b_info->arch_x86.assisted_xapic, + physinfo->cap_assisted_xapic); +libxl_defbool_setdefault (&b_info->arch_x86.assisted_x2apic, + physinfo->cap_assisted_x2apic); + } + +if (b_info->type ==LIBXL_DOMAIN_TYPE_PV && + (!libxl_defbool_is_default (b_info->arch_x86.assisted_xapic) || + !libxl_defbool_is_default (b_info->arch_x86.assisted_x2apic)))
You could just do:
if (b_info->type != LIBXL_DOMAIN_TYPE_PV) {
libxl_defbool_setdefault(&b_info->arch_x86.assisted_xapic,
physinfo->cap_assisted_xapic);
libxl_defbool_setdefault(&b_info->arch_x86.assisted_x2apic,
physinfo->cap_assisted_x2apic);
} else if (!libxl_defbool_is_default(b_info->arch_x86.assisted_xapic) ||
!libxl_defbool_is_default(b_info->arch_x86.assisted_x2apic))
/* ERROR */Quoted 40 lines
tools/libs/light/libxl_x86.c +6
-0+ { +LOG (ERROR ,"Interrupt Controller Virtualization not supported for PV" ); +return ERROR_INVAL ; + } + +return 0 ; } int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/ocaml/libs/xc/xenctrl.ml +2
-0@@ -50,6 +50,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = {tools/ocaml/libs/xc/xenctrl.mli +2
-0@@ -44,6 +44,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = { emulation_flags: x86_arch_emulation_flags list;tools/ocaml/libs/xc/xenctrl_stubs.c +0
-0
tools/ocaml/libs/xc/xenctrl_stubs.c CAMLprim value stub_xc_domain_create(value xch, value wanted_domid, value config · line 242
Quoted 5 lines
tools/ocaml/libs/xc/xenctrl_stubs.c +1
-1@@ -239,7 +239,7 @@ CAMLprim value stub_xc_domain_create(value xch, value wanted_domid, value config cfg.arch.misc_flags = ocaml_list_to_c_bitmap/* ! x86_arch_misc_flags X86_ none */ -/* ! XEN_X86_ XEN_X86_MSR_RELAXED all */ +/* ! XEN_X86_ XEN_X86_ASSISTED_X2APIC max */
We would usually define an XEN_X86_MISC_MAX that would point to XEN_X86_ASSISTED_X2APIC currently.
Quoted 47 lines
tools/ocaml/libs/xc/xenctrl_stubs.c +0
-0(VAL_MISC_FLAGS );#undef VAL_MISC_FLAGStools/xl/xl.c +8
-0@@ -57,6 +57,8 @@ int max_grant_frames = -1;int max_maptrack_frames =-1 ;int max_grant_version =LIBXL_MAX_GRANT_DEFAULT ;libxl_domid domid_policy =INVALID_DOMID ; +int assisted_xapic =-1 ; +int assisted_x2apic =-1 ;xentoollog_level minmsglevel = minmsglevel_default;@@ -201,6 +203,12 @@ static void parse_global_config(const char *configfile,if (!xlu_cfg_get_long (config,"claim_mode" , &l,0 )) claim_mode = l; +if (!xlu_cfg_get_long (config,"assisted_xapic" , &l,0 )) + assisted_xapic = l; + +if (!xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 )) + assisted_x2apic = l; +xlu_cfg_replace_string (config,"remus.default.netbufscript" , &default_remus_netbufscript,0 ); xlu_cfg_replace_string (config, "colo.default.proxyscript",tools/xl/xl.h +2
-0@@ -286,6 +286,8 @@ extern libxl_bitmap global_vm_affinity_mask;extern libxl_bitmap global_hvm_affinity_mask;extern libxl_bitmap global_pv_affinity_mask;extern libxl_domid domid_policy; +extern int assisted_xapic; +extern int assisted_x2apic;enum output_format {OUTPUT_FORMAT_JSON ,tools/xl/xl_parse.c +0
-0
tools/xl/xl_parse.c void parse_config_data(const char *config_source, · line 1685
Quoted 5 lines
tools/xl/xl_parse.c +2
-0@@ -1681,6 +1681,22 @@ void parse_config_data(const char *config_source,xlu_cfg_get_defbool (config,"vpt_align" , &b_info->u.hvm.vpt_align,0 );xlu_cfg_get_defbool (config,"apic" , &b_info->apic,0 ); + e =xlu_cfg_get_long (config,"assisted_xapic" , &l ,0 ); +if ((e ==ESRCH && assisted_xapic !=-1 ))/* use global default if present */
^ no need for the extra parentheses here and below.
Quoted 101 lines
tools/xl/xl_parse.c +14
-0+libxl_defbool_set (&b_info->arch_x86.assisted_xapic, assisted_xapic); + else if (!e) + libxl_defbool_set(&b_info->arch_x86.assisted_xapic, l); + else + exit(1); + + e =xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 ); +if ((e ==ESRCH && assisted_x2apic !=-1 ))/* use global default if present */ +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, assisted_x2apic); +else if (!e) +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, l); +else +exit (1 ); +switch (xlu_cfg_get_list (config,"viridian" , &viridian, &num_viridian,1 )) {xen/arch/x86/domain.c +27
-1@@ -619,6 +619,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)bool hvm = config->flags & XEN_DOMCTL_CDF_hvm;bool hap = config->flags & XEN_DOMCTL_CDF_hap;bool nested_virt = config->flags & XEN_DOMCTL_CDF_nested_virt; +bool assisted_xapic = config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; +bool assisted_x2apic = config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ;unsigned int max_vcpus;if ( hvm ? !hvm_enabled : !IS_ENABLED (CONFIG_PV ) )@@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) } } -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED ) +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED | +XEN_X86_ASSISTED_XAPIC | +XEN_X86_ASSISTED_X2APIC ) ) {dprintk (XENLOG_INFO ,"Invalid arch misc flags %#x\n" , config->arch.misc_flags);return -EINVAL ; } +if ( (assisted_xapic || assisted_x2apic) && !hvm ) + { +dprintk (XENLOG_INFO , +"Interrupt Controller Virtualization not supported for PV\n" ); +return -EINVAL ; + } + +if ( (assisted_xapic && !assisted_xapic_available) || + (assisted_x2apic && !assisted_x2apic_available) ) + { +dprintk (XENLOG_INFO , +"Hardware assisted x%sAPIC requested but not available\n" , + assisted_xapic && !assisted_xapic_available ?"" :"2" ); +return -EINVAL ; + } +return 0 ; }@@ -863,6 +883,12 @@ int arch_domain_create(struct domain *d, d->arch.msr_relaxed = config->arch.misc_flags &XEN_X86_MSR_RELAXED ; + d->arch.hvm.assisted_xapic = + config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; + + d->arch.hvm.assisted_x2apic = + config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ; +return 0 ; fail:xen/arch/x86/hvm/vmx/vmcs.c +4
-0@@ -1157,6 +1157,10 @@ static int construct_vmcs(struct vcpu *v)__vmwrite (PLE_WINDOW , ple_window); } +if ( !v->domain->arch.hvm.assisted_xapic ) + v->arch.hvm.vmx.secondary_exec_control &= + ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +if ( cpu_has_vmx_secondary_exec_control )__vmwrite (SECONDARY_VM_EXEC_CONTROL , v->arch.hvm.vmx.secondary_exec_control);xen/arch/x86/hvm/vmx/vmx.c +0
-1@@ -3333,16 +3333,11 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)void vmx_vlapic_msr_changed (struct vcpu *v) { -int virtualize_x2apic_mode;struct vlapic *vlapic =vcpu_vlapic (v);
xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3340
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +2
-6unsigned int msr; - virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt || - cpu_has_vmx_virtual_intr_delivery) && - cpu_has_vmx_virtualize_x2apic_mode ); - -if ( !cpu_has_vmx_virtualize_apic_accesses && - !virtualize_x2apic_mode ) +if ( ! v->domain->arch.hvm.assisted_xapic && + ! v->domain->arch.hvm.assisted_x2apic )
^ extra space.
Quoted 44 lines
xen/arch/x86/hvm/vmx/vmx.c +3
-2return ;vmx_vmcs_enter (v);@@ -3352,7 +3347,8 @@ void vmx_vlapic_msr_changed(struct vcpu *v)if ( !vlapic_hw_disabled (vlapic) && (vlapic_base_address (vlapic) ==APIC_DEFAULT_PHYS_BASE ) ) { -if ( virtualize_x2apic_mode &&vlapic_x2apic_mode (vlapic) ) +if ( v->domain->arch.hvm.assisted_x2apic + &&vlapic_x2apic_mode (vlapic) ) { v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE ;@@ -3373,7 +3369,7 @@ void vmx_vlapic_msr_changed(struct vcpu *v)vmx_clear_msr_intercept (v,MSR_X2APIC_SELF ,VMX_MSR_W ); } } -else +else if ( v->domain->arch.hvm.assisted_xapic ) v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; }xen/arch/x86/include/asm/hvm/domain.h +6
-0@@ -117,6 +117,12 @@ struct hvm_domain {bool is_s3_suspended; +/* xAPIC hardware assisted virtualization. */ +bool assisted_xapic; + +/* x2APIC hardware assisted virtualization. */ +bool assisted_x2apic; +/* hypervisor intercepted msix table */ struct list_head msixtbl_list;xen/arch/x86/traps.c +0
-0
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1118
Quoted 5 lines
xen/arch/x86/traps.c +1
-1@@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,if ( !is_hvm_domain (d) || subleaf !=0 )break ; -if ( cpu_has_vmx_apic_reg_virt ) + if ( cpu_has_vmx_apic_reg_virt &&
You can drop the cpu_has_vmx_apic_reg_virt check here, if cpu_has_vmx_apic_reg_virt is false assisted_xapic won't be set to true.
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1128
Quoted 12 lines
xen/arch/x86/traps.c +2
-3+ v->domain->arch.hvm.assisted_xapic ) res->a |=XEN_HVM_CPUID_APIC_ACCESS_VIRT ;/* @@ -1124,9 +1125,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,* and wrmsr in the guest will run without VMEXITs (see * vmx_vlapic_msr_changed()). */ -if ( cpu_has_vmx_virtualize_x2apic_mode && - cpu_has_vmx_apic_reg_virt && - cpu_has_vmx_virtual_intr_delivery ) + if ( (cpu_has_vmx_apic_reg_virt && cpu_has_vmx_virtual_intr_delivery) &&
^ unneeded parentheses
Thanks, Roger.
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
Quoted 13 lines · Roger Pau Monné
Quoted 12 lines · Jane Malalane
Introduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
A per-domain option is added to xl in order to select the usage of x{2}APIC hardware assisted vitualization, as well as a global configuration option.
Having all APIC interaction exit to Xen for emulation is slow and can induce much overhead. Hardware can speed up x{2}APIC by running APIC read/write accesses without taking a VM exit.
Quoted 5 lines
Quoted 2 lines
Being able to disable x{2}APIC hardware assisted vitualization can be useful for testing and debugging purposes.
I think you have agreed with Jan some changes to the description regarding the purpose of the commit.
Yes.
Quoted 4 lines
Quoted 4 lines
Signed-off-by Jane Malalane <[email protected]>Suggested-by Andrew Cooper <[email protected]>
Will swap the ordering, forgot to do this previously, apologies.
Quoted 81 lines
Quoted 81 lines
---
CC: Wei Liu <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Julien Grall <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Christian Lindig <[email protected]>
CC: David Scott <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: "Roger Pau Monné" <[email protected]>v2:
* Add aLIBXL_HAVE_ASSISTED_APICmacro
* Pass xcpyshinfo as a pointer inlibxl__arch_get_physinfo
* Add a return statement in now "int"libxl__arch_domain_build_info_setdefault
* Preservelibxl__arch_domain_build_info_setdefault's location inlibxl_create.c
* Correct x{2}apic default setting logic inlibxl__arch_domain_prepare_config
* Correct logic for parsing assisted_x{2}apic host/guest options inxl_parse.cand initialize them to -1 inxl.c
* Use guest options directly invmx_vlapic_msr_changed
* Fix indentation of bool assisted_x{2}apic in structhvm_domain
* Add a change inxenctrl_stubs.cto pass xenctrl ABI checks--- docs/man/xl.cfg.5.pod.in | 10 ++++++++++ docs/man/xl.conf.5.pod.in | 12 ++++++++++++ tools/golang/xenlight/helpers.gen.go | 12 ++++++++++++ tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl_arch.h | 5 +++-- tools/libs/light/libxl_arm.c | 7 +++++-- tools/libs/light/libxl_create.c | 23 ++++++++++++++--------- tools/libs/light/libxl_types.idl | 2 ++ tools/libs/light/libxl_x86.c | 31 +++++++++++++++++++++++++++++-- tools/ocaml/libs/xc/xenctrl.ml | 2 ++ tools/ocaml/libs/xc/xenctrl.mli | 2 ++ tools/ocaml/libs/xc/xenctrl_stubs.c | 2 +- tools/xl/xl.c | 8 ++++++++ tools/xl/xl.h | 2 ++ tools/xl/xl_parse.c | 16 ++++++++++++++++ xen/arch/x86/domain.c | 28 +++++++++++++++++++++++++++- xen/arch/x86/hvm/vmx/vmcs.c | 4 ++++ xen/arch/x86/hvm/vmx/vmx.c | 14 +++++--------- xen/arch/x86/include/asm/hvm/domain.h | 6 ++++++ xen/arch/x86/traps.c | 8 ++++---- xen/include/public/arch-x86/xen.h | 2 ++ 21 files changed, 173 insertions(+), 30 deletions(-)docs/man/xl.cfg.5.pod.in +10
-0@@ -1862,6 +1862,16 @@ firmware tables when using certain older guest Operating Systems. These tables have been superseded by newer constructs within the ACPI tables. +=item B<assisted_xAPIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for xapic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + +=item B<assisted_x2APIC=BOOLEAN> +B<(x86 only)> Enables or disables hardware assisted virtualization for x2apic. +This allows accessing APIC registers without a VM-exit. +The default is settable via L<xl.conf(5)>. + =item B<nx=BOOLEAN> B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a guestdocs/man/xl.conf.5.pod.in +0
-0@@ -107,6 +107,18 @@ Sets the default value for the C<max_grant_version> domain config value. Default: maximum grant version supported by the hypervisor.
Quoted 14 lines
docs/man/xl.conf.5.pod.in Sets the default value for the C<max_grant_version> domain config value. · line 120
Quoted 11 lines
docs/man/xl.conf.5.pod.in +11
-0+=item B<assisted_xAPIC=BOOLEAN> + +If enabled, domains will use xAPIC hardware assisted virtualization by default. + +Default: enabled if supported. + +=item B<assisted_x2APIC=BOOLEAN> + +If enabled, domains will use x2APIC hardware assisted virtualization by default. + +Default: enabled if supported.We don't capitalize xl options, so I would suggest to lowercase APIC for all the option names.
Okay.
Quoted 102 lines
Quoted 102 lines
docs/man/xl.conf.5.pod.in +1
-0+ =item B<vif.default.script="PATH"> Configures the default hotplug script used by virtual network devices.tools/golang/xenlight/helpers.gen.go +12
-0@@ -636,6 +636,12 @@ x.Passthrough = Passthrough(xc.passthrough) if err := x.XendSuspendEvtchnCompat.fromC(&xc.xend_suspend_evtchn_compat);err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.fromC(&xc.arch_x86.assisted_xapic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.fromC(&xc.arch_x86.assisted_x2apic);err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil}@@ -679,6 +685,12 @@ xc.passthrough = C.libxl_passthrough(x.Passthrough) if err := x.XendSuspendEvtchnCompat.toC(&xc.xend_suspend_evtchn_compat); err != nil { return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err) } +if err := x.ArchX86.AssistedXapic.toC(&xc.arch_x86.assisted_xapic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err) +} +if err := x.ArchX86.AssistedX2Apic.toC(&xc.arch_x86.assisted_x2apic); err != nil { +return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err) +} return nil }tools/include/libxl.h +7
-0@@ -535,6 +535,13 @@#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1/* +* LIBXL_HAVE_ASSISTED_APIC indicates that libxl_domain_build_info has +* assisted_x{2}apic fields, for enabling hardware assisted virtualization for +* x{2}apic per domain. +*/ +#define LIBXL_HAVE_ASSISTED_APIC 1 + +/* * libxl ABI compatibility * * The only guarantee which libxl makes regarding ABI compatibility tools/libs/light/libxl_arch.h +3
-2@@ -71,8 +71,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, libxl_domain_create_info *c_info);_hidden -void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, -libxl_domain_build_info *b_info); +int libxl__arch_domain_build_info_setdefault(libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo); _hidden int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/libs/light/libxl_arm.c +5
-2@@ -1384,8 +1384,9 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, } } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {/* ACPI is disabled by default */ libxl_defbool_setdefault (&b_info->acpi, false);@@ -1399,6 +1400,8 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,memset (&b_info->u,'\0' ,sizeof (b_info->u)); b_info->type =LIBXL_DOMAIN_TYPE_INVALID ;libxl_domain_build_info_init_type (b_info,LIBXL_DOMAIN_TYPE_PVH ); + +return 0 ; } int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/libs/light/libxl_create.c +0
-0@@ -264,7 +264,20 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,if (!b_info->event_channels) b_info->event_channels =1023 ;
Quoted 5 lines
tools/libs/light/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc *gc, · line 267
Quoted 2 lines
tools/libs/light/libxl_create.c +1
-1-libxl__arch_domain_build_info_setdefault (gc, b_info); +libxl_physinfo info;The definition of info needs to be at the top of the function, together with the rest of the variable definitions.
Okay.
Quoted 53 lines
Quoted 53 lines
tools/libs/light/libxl_create.c +13
-8+ + rc =libxl_get_physinfo (CTX , &info); +if (rc) { +LOG (ERROR ,"failed to get hypervisor info" ); +return rc; + } + + rc =libxl__arch_domain_build_info_setdefault (gc, b_info, &info); +if (rc) { +LOG (ERROR ,"unable to set domain arch build info defaults" ); +return rc; + } +libxl_defbool_setdefault (&b_info->dm_restrict, false);if (b_info->iommu_memkb ==LIBXL_MEMKB_DEFAULT )@@ -457,14 +470,6 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, }if (b_info->max_grant_version ==LIBXL_MAX_GRANT_DEFAULT ) { -libxl_physinfo info; - - rc =libxl_get_physinfo (CTX , &info); -if (rc) { -LOG (ERROR ,"failed to get hypervisor info" ); -return rc; - } -if (info.cap_gnttab_v2) b_info->max_grant_version =2 ;else if (info.cap_gnttab_v1)tools/libs/light/libxl_types.idl +2
-0@@ -648,6 +648,8 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("vuart", libxl_vuart_type), ])), ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), + ("assisted_xapic", libxl_defbool), + ("assisted_x2apic", libxl_defbool), ])), # Alternate p2m is not bound to any architecture or guest type, as it is # supported by x86 HVM and ARM support is planned.tools/libs/light/libxl_x86.c +0
-0@@ -23,6 +23,14 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,if (libxl_defbool_val (d_config->b_info.arch_x86.msr_relaxed)) config->arch.misc_flags |=XEN_X86_MSR_RELAXED ;
Quoted 5 lines
tools/libs/light/libxl_x86.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 27
Quoted 2 lines
tools/libs/light/libxl_x86.c +2
-0+if (d_config->c_info.type !=LIBXL_DOMAIN_TYPE_PV ) + {Coding style for libxl is to place the bracket in the same line as the if.
Okay.
Quoted 37 lines
Quoted 24 lines
tools/libs/light/libxl_x86.c +11
-2+if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_xapic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_XAPIC ; + +if (libxl_defbool_val (d_config->b_info.arch_x86.assisted_x2apic)) + config->arch.misc_flags |=XEN_X86_ASSISTED_X2APIC ; + }return 0 ; }@@ -819,11 +827,30 @@ void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, { } -void libxl__arch_domain_build_info_setdefault (libxl__gc *gc, -libxl_domain_build_info *b_info) +int libxl__arch_domain_build_info_setdefault (libxl__gc *gc, +libxl_domain_build_info *b_info, +const libxl_physinfo *physinfo) {libxl_defbool_setdefault (&b_info->acpi, true);libxl_defbool_setdefault (&b_info->arch_x86.msr_relaxed, false); + +if (b_info->type !=LIBXL_DOMAIN_TYPE_PV )tools/libs/light/libxl_x86.c void libxl__arch_domain_create_info_setdefault(libxl__gc *gc, · line 847
Quoted 10 lines
tools/libs/light/libxl_x86.c +10
-0+ { +libxl_defbool_setdefault (&b_info->arch_x86.assisted_xapic, + physinfo->cap_assisted_xapic); +libxl_defbool_setdefault (&b_info->arch_x86.assisted_x2apic, + physinfo->cap_assisted_x2apic); + } + +if (b_info->type ==LIBXL_DOMAIN_TYPE_PV && + (!libxl_defbool_is_default (b_info->arch_x86.assisted_xapic) || + !libxl_defbool_is_default (b_info->arch_x86.assisted_x2apic)))You could just do:
Quoted 15 lines
if (b_info->type != LIBXL_DOMAIN_TYPE_PV) { libxl_defbool_setdefault(&b_info->arch_x86.assisted_xapic, physinfo->cap_assisted_xapic); libxl_defbool_setdefault(&b_info->arch_x86.assisted_x2apic, physinfo->cap_assisted_x2apic); } else if (!libxl_defbool_is_default(b_info->arch_x86.assisted_xapic) || !libxl_defbool_is_default(b_info->arch_x86.assisted_x2apic)) /* ERROR */Quoted 6 lines
tools/libs/light/libxl_x86.c +6
-0+ { +LOG (ERROR ,"Interrupt Controller Virtualization not supported for PV" ); +return ERROR_INVAL ; + } + +return 0 ;
True.
Quoted 256 lines
Quoted 34 lines
tools/libs/light/libxl_x86.c +0
-0} int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,tools/ocaml/libs/xc/xenctrl.ml +2
-0@@ -50,6 +50,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = {tools/ocaml/libs/xc/xenctrl.mli +2
-0@@ -44,6 +44,8 @@ type x86_arch_emulation_flags = type x86_arch_misc_flags = | X86_MSR_RELAXED + | X86_ASSISTED_XAPIC + | X86_ASSISTED_X2APIC type xen_x86_arch_domainconfig = { emulation_flags: x86_arch_emulation_flags list;tools/ocaml/libs/xc/xenctrl_stubs.c +0
-0tools/ocaml/libs/xc/xenctrl_stubs.c CAMLprim value stub_xc_domain_create(value xch, value wanted_domid, value config · line 242
Quoted 5 lines
tools/ocaml/libs/xc/xenctrl_stubs.c +1
-1@@ -239,7 +239,7 @@ CAMLprim value stub_xc_domain_create(value xch, value wanted_domid, value config cfg.arch.misc_flags = ocaml_list_to_c_bitmap/* ! x86_arch_misc_flags X86_ none */ -/* ! XEN_X86_ XEN_X86_MSR_RELAXED all */ +/* ! XEN_X86_ XEN_X86_ASSISTED_X2APIC max */ We would usually define an
XEN_X86_MISC_MAXthat would point toXEN_X86_ASSISTED_X2APICcurrently.Quoted 47 lines
tools/ocaml/libs/xc/xenctrl_stubs.c +0
-0(VAL_MISC_FLAGS );#undef VAL_MISC_FLAGStools/xl/xl.c +8
-0@@ -57,6 +57,8 @@ int max_grant_frames = -1;int max_maptrack_frames =-1 ;int max_grant_version =LIBXL_MAX_GRANT_DEFAULT ;libxl_domid domid_policy =INVALID_DOMID ; +int assisted_xapic =-1 ; +int assisted_x2apic =-1 ;xentoollog_level minmsglevel = minmsglevel_default;@@ -201,6 +203,12 @@ static void parse_global_config(const char *configfile,if (!xlu_cfg_get_long (config,"claim_mode" , &l,0 )) claim_mode = l; +if (!xlu_cfg_get_long (config,"assisted_xapic" , &l,0 )) + assisted_xapic = l; + +if (!xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 )) + assisted_x2apic = l; +xlu_cfg_replace_string (config,"remus.default.netbufscript" , &default_remus_netbufscript,0 ); xlu_cfg_replace_string (config, "colo.default.proxyscript",tools/xl/xl.h +2
-0@@ -286,6 +286,8 @@ extern libxl_bitmap global_vm_affinity_mask;extern libxl_bitmap global_hvm_affinity_mask;extern libxl_bitmap global_pv_affinity_mask;extern libxl_domid domid_policy; +extern int assisted_xapic; +extern int assisted_x2apic;enum output_format {OUTPUT_FORMAT_JSON ,tools/xl/xl_parse.c +0
-0tools/xl/xl_parse.c void parse_config_data(const char *config_source, · line 1685
Quoted 5 lines
tools/xl/xl_parse.c +2
-0@@ -1681,6 +1681,22 @@ void parse_config_data(const char *config_source,xlu_cfg_get_defbool (config,"vpt_align" , &b_info->u.hvm.vpt_align,0 );xlu_cfg_get_defbool (config,"apic" , &b_info->apic,0 ); + e =xlu_cfg_get_long (config,"assisted_xapic" , &l ,0 ); +if ((e ==ESRCH && assisted_xapic !=-1 ))/* use global default if present */ ^ no need for the extra parentheses here and below.
Quoted 101 lines
tools/xl/xl_parse.c +14
-0+libxl_defbool_set (&b_info->arch_x86.assisted_xapic, assisted_xapic); + else if (!e) + libxl_defbool_set(&b_info->arch_x86.assisted_xapic, l); + else + exit(1); + + e =xlu_cfg_get_long (config,"assisted_x2apic" , &l,0 ); +if ((e ==ESRCH && assisted_x2apic !=-1 ))/* use global default if present */ +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, assisted_x2apic); +else if (!e) +libxl_defbool_set (&b_info->arch_x86.assisted_x2apic, l); +else +exit (1 ); +switch (xlu_cfg_get_list (config,"viridian" , &viridian, &num_viridian,1 )) {xen/arch/x86/domain.c +27
-1@@ -619,6 +619,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)bool hvm = config->flags & XEN_DOMCTL_CDF_hvm;bool hap = config->flags & XEN_DOMCTL_CDF_hap;bool nested_virt = config->flags & XEN_DOMCTL_CDF_nested_virt; +bool assisted_xapic = config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; +bool assisted_x2apic = config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ;unsigned int max_vcpus;if ( hvm ? !hvm_enabled : !IS_ENABLED (CONFIG_PV ) )@@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) } } -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED ) +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED | +XEN_X86_ASSISTED_XAPIC | +XEN_X86_ASSISTED_X2APIC ) ) {dprintk (XENLOG_INFO ,"Invalid arch misc flags %#x\n" , config->arch.misc_flags);return -EINVAL ; } +if ( (assisted_xapic || assisted_x2apic) && !hvm ) + { +dprintk (XENLOG_INFO , +"Interrupt Controller Virtualization not supported for PV\n" ); +return -EINVAL ; + } + +if ( (assisted_xapic && !assisted_xapic_available) || + (assisted_x2apic && !assisted_x2apic_available) ) + { +dprintk (XENLOG_INFO , +"Hardware assisted x%sAPIC requested but not available\n" , + assisted_xapic && !assisted_xapic_available ?"" :"2" ); +return -EINVAL ; + } +return 0 ; }@@ -863,6 +883,12 @@ int arch_domain_create(struct domain *d, d->arch.msr_relaxed = config->arch.misc_flags &XEN_X86_MSR_RELAXED ; + d->arch.hvm.assisted_xapic = + config->arch.misc_flags &XEN_X86_ASSISTED_XAPIC ; + + d->arch.hvm.assisted_x2apic = + config->arch.misc_flags &XEN_X86_ASSISTED_X2APIC ; +return 0 ; fail:xen/arch/x86/hvm/vmx/vmcs.c +4
-0@@ -1157,6 +1157,10 @@ static int construct_vmcs(struct vcpu *v)__vmwrite (PLE_WINDOW , ple_window); } +if ( !v->domain->arch.hvm.assisted_xapic ) + v->arch.hvm.vmx.secondary_exec_control &= + ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +if ( cpu_has_vmx_secondary_exec_control )__vmwrite (SECONDARY_VM_EXEC_CONTROL , v->arch.hvm.vmx.secondary_exec_control);xen/arch/x86/hvm/vmx/vmx.c +0
-1@@ -3333,16 +3333,11 @@ static void vmx_install_vlapic_mapping(struct vcpu *v)void vmx_vlapic_msr_changed (struct vcpu *v) { -int virtualize_x2apic_mode;struct vlapic *vlapic =vcpu_vlapic (v);xen/arch/x86/hvm/vmx/vmx.c static void vmx_install_vlapic_mapping(struct vcpu *v) · line 3340
Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +2
-6unsigned int msr; - virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt || - cpu_has_vmx_virtual_intr_delivery) && - cpu_has_vmx_virtualize_x2apic_mode ); - -if ( !cpu_has_vmx_virtualize_apic_accesses && - !virtualize_x2apic_mode ) +if ( ! v->domain->arch.hvm.assisted_xapic && + ! v->domain->arch.hvm.assisted_x2apic )^ extra space.
Quoted 47 lines
xen/arch/x86/hvm/vmx/vmx.c +3
-2return ;vmx_vmcs_enter (v);@@ -3352,7 +3347,8 @@ void vmx_vlapic_msr_changed(struct vcpu *v)if ( !vlapic_hw_disabled (vlapic) && (vlapic_base_address (vlapic) ==APIC_DEFAULT_PHYS_BASE ) ) { -if ( virtualize_x2apic_mode &&vlapic_x2apic_mode (vlapic) ) +if ( v->domain->arch.hvm.assisted_x2apic + &&vlapic_x2apic_mode (vlapic) ) { v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE ;@@ -3373,7 +3369,7 @@ void vmx_vlapic_msr_changed(struct vcpu *v)vmx_clear_msr_intercept (v,MSR_X2APIC_SELF ,VMX_MSR_W ); } } -else +else if ( v->domain->arch.hvm.assisted_xapic ) v->arch.hvm.vmx.secondary_exec_control |=SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; }xen/arch/x86/include/asm/hvm/domain.h +6
-0@@ -117,6 +117,12 @@ struct hvm_domain {bool is_s3_suspended; +/* xAPIC hardware assisted virtualization. */ +bool assisted_xapic; + +/* x2APIC hardware assisted virtualization. */ +bool assisted_x2apic; +/* hypervisor intercepted msix table */ struct list_head msixtbl_list;xen/arch/x86/traps.c +0
-0@@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,if ( !is_hvm_domain (d) || subleaf !=0 )break ;
Quoted 5 lines
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1118
Quoted 2 lines
xen/arch/x86/traps.c +1
-1-if ( cpu_has_vmx_apic_reg_virt ) + if ( cpu_has_vmx_apic_reg_virt &&You can drop the
cpu_has_vmx_apic_reg_virtcheck here, ifcpu_has_vmx_apic_reg_virtis falseassisted_xapicwon't be set to true.
Oh, but assisted_xapic_available is only set to depend on cpu_has_vmx_virtualize_apic_accesses, unless I should correct this, but
I might be missing something...
Quoted 4 lines
Quoted 4 lines
xen/arch/x86/traps.c +1
-0+ v->domain->arch.hvm.assisted_xapic ) res->a |=XEN_HVM_CPUID_APIC_ACCESS_VIRT ;
Quoted 11 lines
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1128
Quoted 9 lines
xen/arch/x86/traps.c +1
-3/* @@ -1124,9 +1125,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,* and wrmsr in the guest will run without VMEXITs (see * vmx_vlapic_msr_changed()). */ -if ( cpu_has_vmx_virtualize_x2apic_mode && - cpu_has_vmx_apic_reg_virt && - cpu_has_vmx_virtual_intr_delivery ) + if ( (cpu_has_vmx_apic_reg_virt && cpu_has_vmx_virtual_intr_delivery) &&^ unneeded parentheses
Okay.
Quoted 1 line
Thanks, Roger.
Thank you,
Jane.
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
Quoted 10 lines · Jane Malalane
Quoted 9 lines · Roger Pau Monné
Quoted 8 lines · Jane Malalane
xen/arch/x86/traps.c +0
-0@@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,if ( !is_hvm_domain (d) || subleaf !=0 )break ;
Quoted 8 lines
Quoted 5 lines
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1118
Quoted 2 lines
xen/arch/x86/traps.c +1
-1-if ( cpu_has_vmx_apic_reg_virt ) + if ( cpu_has_vmx_apic_reg_virt &&You can drop the
cpu_has_vmx_apic_reg_virtcheck here, ifcpu_has_vmx_apic_reg_virtis falseassisted_xapicwon't be set to true.Oh, but
assisted_xapic_availableis only set to depend oncpu_has_vmx_virtualize_apic_accesses, unless I should correct this, but
I might be missing something...
No, you are right. We only report hw virtualized xAPIC to guests if both apic_reg_virt and virtualize_apic_accesses are available.
Thanks, Roger.
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
In reply to Roger Pau MonnéQuoted 7 lines · Roger Pau Monné
Quoted 6 lines · Jane Malalane
xen/arch/x86/traps.c +0
-0@@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,if ( !is_hvm_domain (d) || subleaf !=0 )break ;
Quoted 5 lines
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1118
Quoted 2 lines
xen/arch/x86/traps.c +1
-1-if ( cpu_has_vmx_apic_reg_virt ) + if ( cpu_has_vmx_apic_reg_virt &&You can drop the
cpu_has_vmx_apic_reg_virtcheck here, ifcpu_has_vmx_apic_reg_virtis falseassisted_xapicwon't be set to true.
Along these lines ...
Quoted 13 lines
xen/arch/x86/traps.c void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf, · line 1128
Quoted 12 lines
xen/arch/x86/traps.c +2
-3+ v->domain->arch.hvm.assisted_xapic ) res->a |=XEN_HVM_CPUID_APIC_ACCESS_VIRT ;/* @@ -1124,9 +1125,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,* and wrmsr in the guest will run without VMEXITs (see * vmx_vlapic_msr_changed()). */ -if ( cpu_has_vmx_virtualize_x2apic_mode && - cpu_has_vmx_apic_reg_virt && - cpu_has_vmx_virtual_intr_delivery ) + if ( (cpu_has_vmx_apic_reg_virt && cpu_has_vmx_virtual_intr_delivery) &&^ unneeded parentheses
... this also wants simplifying to just v->domain->arch.hvm.assisted_x2apic:
The apparently stray parentheses were, I think, added in reply to me pointing out that the check here isn't in line with that put in place by patch 1 in vmx_init_vmcs_config(). I.e. the inner && really was meant to be || as it looks. Yet once the two are in line, the same simplification as above is possible.
Jan
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
In reply to Jane MalalaneQuoted 8 lines · Jane Malalane
Introduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
A per-domain option is added to xl in order to select the usage of x{2}APIC hardware assisted vitualization, as well as a global configuration option.
Quoted 6 lines
Having all APIC interaction exit to Xen for emulation is slow and can induce much overhead. Hardware can speed up x{2}APIC by running APIC read/write accesses without taking a VM exit.
Being able to disable x{2}APIC hardware assisted vitualization can be useful for testing and debugging purposes.
Might be worth adding a note to the commit log in order to note that vmx_install_vlapic_mapping doesn't require modifications regardless of whether the guest has virtualize_apic_accesses enabled or not.
Setting the APIC_ACCESS_ADDR VMCS field is fine even if virtualize_apic_accesses is not enabled for the guest: as long as the feature is supported by the CPU the field will exist.
Thanks, Roger.
Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC
Quoted 16 lines · Roger Pau Monné
Quoted 8 lines · Jane Malalane
Introduce a new per-domain creation x86 specific flag to select whether hardware assisted virtualization should be used for x{2}APIC.
A per-domain option is added to xl in order to select the usage of x{2}APIC hardware assisted vitualization, as well as a global configuration option.
Quoted 6 lines
Having all APIC interaction exit to Xen for emulation is slow and can induce much overhead. Hardware can speed up x{2}APIC by running APIC read/write accesses without taking a VM exit.
Being able to disable x{2}APIC hardware assisted vitualization can be useful for testing and debugging purposes.
Quoted 7 lines
Might be worth adding a note to the commit log in order to note that
vmx_install_vlapic_mappingdoesn't require modifications regardless of whether the guest hasvirtualize_apic_accessesenabled or not.Setting the
APIC_ACCESS_ADDRVMCS field is fine even ifvirtualize_apic_accessesis not enabled for the guest: as long as the feature is supported by the CPU the field will exist.
Oh right. Will add these two points.
Thanks,
Jane.