Physical address hypercall ABI ("HVMv2")
RFC PATCH 1/4 xen: Introduce physaddr_abi CDF flag
In reply to Teddy AstieIntroduce the physaddr_abi flag to indicate that the guest uses physical addresses (gpaddr) instead of virtual addresses when performing hypercalls.
--- xen/common/domain.c | 10 +++++++++- xen/include/public/domctl.h | 4 +++- xen/include/xen/sched.h | 6 ++++++ 3 files changed, 18 insertions(+), 2 deletions(-)
xen/common/domain.c +9 -1
@@ -716,12 +716,14 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)bool hap = config->flags & XEN_DOMCTL_CDF_hap;bool iommu = config->flags & XEN_DOMCTL_CDF_iommu;bool vpmu = config->flags & XEN_DOMCTL_CDF_vpmu; +bool physaddr_abi = config->flags & XEN_DOMCTL_CDF_physaddr_abi;if ( config->flags & ~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap | XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off | XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu | - XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) ) + XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu | + XEN_DOMCTL_CDF_physaddr_abi) ) {dprintk (XENLOG_INFO ,"Unknown CDF flags %#x\n" , config->flags);return -EINVAL ;
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( physaddr_abi && !hvm ) + { +dprintk (XENLOG_INFO ,"Physical address ABI requested for non-HVM guest" ); +return -EINVAL ; + } +if ( iommu ) {if ( config->iommu_opts & ~XEN_DOMCTL_IOMMU_no_sharept )
xen/include/public/domctl.h +3 -1
@@ -66,9 +66,11 @@ struct xen_domctl_createdomain {#define XEN_DOMCTL_CDF_nested_virt (1U << _XEN_DOMCTL_CDF_nested_virt)/* Should we expose the vPMU to the guest? */ #define XEN_DOMCTL_CDF_vpmu (1U << 7) +/* Do the guest use physical addresses for hypercalls ? */ +#define XEN_DOMCTL_CDF_physaddr_abi (1U << 8)/* Max XEN_DOMCTL_CDF_* constant. Used for ABI checking. */ -#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu +#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_physaddr_abiuint32_t flags;
xen/include/xen/sched.h +6 -0
@@ -1201,6 +1201,12 @@ static always_inline bool hap_enabled(const struct domain *d)evaluate_nospec (d->options & XEN_DOMCTL_CDF_hap); } +static always_inline bool is_hvm_physaddr_abi(const struct domain *d) +{ +return IS_ENABLED (CONFIG_HVM ) && +evaluate_nospec (d->options & XEN_DOMCTL_CDF_physaddr_abi); +} +static inline bool is_hwdom_pinned_vcpu (const struct vcpu *v) { return (is_hardware_domain(v->domain) &&
Signature · 9 lines
2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
RFC PATCH 3/4 x86/public: Expose physaddr_abi through Xen HVM CPUID leaf
In reply to Teddy AstieThe guest needs to know this information before attempting to make a hypercall (especially ones to fetch XENFEAT_*).
--- xen/arch/x86/cpuid.c | 3 +++ xen/include/public/arch-x86/cpuid.h | 2 ++ 2 files changed, 5 insertions(+)
xen/arch/x86/cpuid.c +3 -0
@@ -153,6 +153,9 @@ static void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,*/ res->a |=XEN_HVM_CPUID_UPCALL_VECTOR ; +/* Indicate if the guest uses the physical addresses hypercall ABI. */ +if (is_hvm_physaddr_abi (v->domain) ) + res->a |=XEN_HVM_CPUID_PHYS_ADDR_ABI ;break ;case 5 :/* PV-specific parameters */
xen/include/public/arch-x86/cpuid.h +2 -0
@@ -106,6 +106,8 @@* bound to event channels. */ #define XEN_HVM_CPUID_UPCALL_VECTOR (1u << 6) +/* Hypercalls are using physical addresses instead of linear ones. */ +#define XEN_HVM_CPUID_PHYS_ADDR_ABI (1u << 7)/* * Leaf 6 (0x40000x05)
Signature · 9 lines
2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
cover Physical address hypercall ABI ("HVMv2")
In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).
This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.
This mode is opt-in and must be enabled explicitely by the toolstack.
This is also mandatory for confidential-computing guests (e.g SEV) where the guest pagetable are not visible to the hypervisor.
In a synthetic xtf-based hypercall benchmark (VCPUOP_get_runstate_info loop), it gives broadly a 30% overhead reduction when tested on a AMD EPYC 9124.
This serie only implement support for x86 yet this ABI is also meaningful for other architectures as well.
A separate patch for adding support in Linux is planned.
Teddy Astie (4):
xen: Introduce physaddr_abi CDF flag
x86/hvm: Consider phyaddr_abi when copying from/to guest memory x86/public: Expose physaddr_abi through Xen HVM CPUID leaf
libxl: Add support for enabling physaddr_abi
tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_create.c | 4 ++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 1 + xen/arch/x86/cpuid.c | 3 +++ xen/arch/x86/hvm/hvm.c | 17 ++++++++++++++--- xen/common/domain.c | 10 +++++++++- xen/include/public/arch-x86/cpuid.h | 2 ++ xen/include/public/domctl.h | 4 +++- xen/include/xen/sched.h | 6 ++++++ 10 files changed, 48 insertions(+), 5 deletions(-)
Signature · 9 lines
2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
RFC PATCH 4/4 libxl: Add support for enabling physaddr_abi
--- tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_create.c | 4 ++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 1 + 4 files changed, 11 insertions(+)
tools/include/libxl.h +5 -0
@@ -178,6 +178,11 @@*/ #define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1 +/* +* The libxl_domain_build_info has physaddr_abi field. +*/ +#define LIBXL_HAVE_BUILDINFO_PHYS_ADDR_ABI 1 +/* * libxl_domain_build_info has the u.hvm.ms_vm_genid field. */
tools/libs/light/libxl_create.c +4 -0
@@ -466,6 +466,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,libxl_defbool_setdefault (&b_info->apic, true);libxl_defbool_setdefault (&b_info->nested_hvm, false); +libxl_defbool_setdefault (&b_info->physaddr_abi, false); }if (b_info->max_grant_version ==LIBXL_MAX_GRANT_DEFAULT ) {
@@ -662,6 +663,9 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,if (libxl_defbool_val (b_info->nested_hvm) ) create.flags |= XEN_DOMCTL_CDF_nested_virt; + +if (libxl_defbool_val (b_info->physaddr_abi)) + create.flags |= XEN_DOMCTL_CDF_physaddr_abi; }if (libxl_defbool_val (b_info->vpmu))
tools/libs/light/libxl_types.idl +1 -0
@@ -637,6 +637,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("nested_hvm", libxl_defbool), ("apic", libxl_defbool), ("dm_restrict", libxl_defbool), + ("physaddr_abi", libxl_defbool), ("tee", libxl_tee_type), ("u", KeyedUnion(None, libxl_domain_type, "type", [("hvm", Struct(None, [("firmware", string),
tools/xl/xl_parse.c +1 -0
@@ -1803,6 +1803,7 @@ void parse_config_data(const char *config_source, }xlu_cfg_get_defbool (config,"nestedhvm" , &b_info->nested_hvm,0 ); +xlu_cfg_get_defbool (config,"physaddr_abi" , &b_info->physaddr_abi,0 );switch (b_info->type) {case LIBXL_DOMAIN_TYPE_HVM :
Signature · 9 lines
2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
RFC PATCH 2/4 x86/hvm: Consider phyaddr_abi when copying from/to guest memory
In reply to Teddy Astie--- xen/arch/x86/hvm/hvm.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
xen/arch/x86/hvm/hvm.c +14 -3
@@ -3499,7 +3499,11 @@ unsigned int copy_to_user_hvm(void *to, const void *from, unsigned int len)return 0 ; } - rc =hvm_copy_to_guest_linear ((unsigned long )to, from, len,0 ,NULL ); +if (is_hvm_physaddr_abi (current->domain) ) + rc =hvm_copy_to_guest_phys ((unsigned long )to, from, len, current); +else + rc =hvm_copy_to_guest_linear ((unsigned long )to, from, len,0 ,NULL ); +return rc ? len :0 ;/* fake a copy_to_user() return code */ }
@@ -3513,7 +3517,10 @@ unsigned int clear_user_hvm(void *to, unsigned int len)return 0 ; } - rc =hvm_copy_to_guest_linear ((unsigned long )to,NULL , len,0 ,NULL ); +if (is_hvm_physaddr_abi (current->domain) ) + rc =hvm_copy_to_guest_phys ((unsigned long )to,NULL , len, current); +else + rc =hvm_copy_to_guest_linear ((unsigned long )to,NULL , len,0 ,NULL );return rc ? len :0 ;/* fake a clear_user() return code */ }
@@ -3528,7 +3535,11 @@ unsigned int copy_from_user_hvm(void *to, const void *from, unsigned int len)return 0 ; } - rc =hvm_copy_from_guest_linear (to, (unsigned long )from, len,0 ,NULL ); +if (is_hvm_physaddr_abi (current->domain) ) + rc =hvm_copy_from_guest_phys (to, (unsigned long )from, len); +else + rc =hvm_copy_from_guest_linear (to, (unsigned long )from, len,0 ,NULL ); +return rc ? len :0 ;/* fake a copy_from_user() return code */ }
Signature · 9 lines
2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Quoted 9 lines · Teddy Astie
In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).
Quoted 5 lines
This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.
This mode is opt-in and must be enabled explicitely by the toolstack.
Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way a guest needs to be converted all in one go. While doable, it'll be increasingly risky with the size of the guest kernel code base.
A prior proposal of mine was to add an indicator to hypercall numbers (e.g. to set the top bit there), to indicate which of the two models a particular hypercall invocation uses.
Aiui Andrew had yet different (albeit also never spelled out) plans.
Jan
Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag
In reply to Teddy Astiexen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754
Quoted 9 lines · Teddy Astie
xen/common/domain.c +5
-0@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( physaddr_abi && !hvm ) + { +dprintk (XENLOG_INFO ,"Physical address ABI requested for non-HVM guest" ); +return -EINVAL ; + }
Why this restriction?
Jan
Hi,
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Teddy Astie
In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.
Quoted 5 lines
Quoted 1 line
This mode is opt-in and must be enabled explicitely by the toolstack.
Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way a guest needs to be converted all in one go. While doable, it'll be increasingly risky with the size of the guest kernel code base.
+1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).
Furthermore, I don't think this can be easily adopted in public cloud where the admin for Xen and the guest will be different. So any indication of the ABI would have to come from the guest itself rather than the configuration.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag
In reply to Jan BeulichLe 30/04/2025 à 17:59, Jan Beulich a écrit :
Quoted 5 lines · Teddy Astie
Quoted 4 lines
xen/common/domain.c +0
-0@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; }
Quoted 9 lines
xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754
Quoted 5 lines
xen/common/domain.c +5
-0+if ( physaddr_abi && !hvm ) + { +dprintk (XENLOG_INFO ,"Physical address ABI requested for non-HVM guest" ); +return -EINVAL ; + }Why this restriction?
physaddr_abi changes how copy_from/to_guest works to make it use GPA instead of GVA. As non-HVM probably means PV guest, it would mean something like PV guest hypercalls uses physical addresses (derived from MFN?) instead of virtual addresses, which would not really be practical for both the guest and the hypervisor.
Quoted 2 lines
Jan
Teddy
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Le 01/05/2025 à 13:14, Julien Grall a écrit :
Quoted 34 lines
Hi,
Quoted 23 lines · Jan Beulich
Quoted 22 lines · Teddy Astie
In current HVM mode, when a hypercall references a structure in guest memory,
it is passed to the hypervisor as its "linear address" (e.g virtual address for
the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly
usable by the Xen and needs to be translated from GVA to GPA then HPA. This
implies a complex and potentially expensive lookup of the pagetables inside the
guest. This can be significant, especially if the P2M cannot use efficiently
superpages (or with e.g XSA-304).This proposal introduce a new mode where all addresses used for hypercalls are
GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA
only needs a P2M lookup and not looking through the inside-guest pagetables.Quoted 7 lines
Quoted 1 line
This mode is opt-in and must be enabled explicitely by the toolstack.
Which I view as a severe downside (leaving aside the PVH Dom0 aspect):
This way
a guest needs to be converted all in one go. While doable, it'll be increasingly
risky with the size of the guest kernel code base.
Quoted 7 lines
+1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).
Furthermore, I don't think this can be easily adopted in public cloud where the admin for Xen and the guest will be different. So any indication of the ABI would have to come from the guest itself rather than the configuration.
Makes sense, I am experimenting with a alternative design which requires setting the 30th bit (31th is already used for viridian) of the eax register to indicate the use of this new ABI. Also keeping a CPUID to indicate that the feature is supported by the hypervisor (thus no need to enable it in advance for a guest).
Quoted 2 lines
Cheers,
Teddy
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag
In reply to Teddy AstieQuoted 16 lines · Teddy Astie
Le 30/04/2025 à 17:59, Jan Beulich a écrit :
Quoted 5 lines · Teddy Astie
Quoted 4 lines
xen/common/domain.c +0
-0@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; }Quoted 9 lines
xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754
Quoted 5 lines
xen/common/domain.c +5
-0+if ( physaddr_abi && !hvm ) + { +dprintk (XENLOG_INFO ,"Physical address ABI requested for non-HVM guest" ); +return -EINVAL ; + }Why this restriction?
Quoted 4 lines
physaddr_abichanges how copy_from/to_guest works to make it use GPA instead of GVA. As non-HVM probably means PV guest, it would mean something like PV guest hypercalls uses physical addresses (derived from MFN?)
Machine addresses, yes (hence MFN). If it was PFNs / (pseudo-)physical addresses, ...
Quoted 2 lines
instead of virtual addresses, which would not really be practical for both the guest and the hypervisor.
... I'd maybe agree here.
Jan