Skip to contentSkip to conversation

Physical address hypercall ABI ("HVMv2")

RFC PATCH In review Teddy Astie 3 participants 11 messages Apr 18, 2025 – May 2, 2025

lore
Share & tools
lore
#1 lore raw

RFC PATCH 1/4 xen: Introduce physaddr_abi CDF flag

In reply to Teddy Astie

Introduce the physaddr_abi flag to indicate that the guest uses physical addresses (gpaddr) instead of virtual addresses when performing hypercalls.

---
 xen/common/domain.c         | 10 +++++++++-
 xen/include/public/domctl.h |  4 +++-
 xen/include/xen/sched.h     |  6 ++++++
 3 files changed, 18 insertions(+), 2 deletions(-)
xen/common/domain.c +9 -1
index 585fd726a9..1a1e51c32d 100644
@@ -716,12 +716,14 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
     bool hap = config->flags & XEN_DOMCTL_CDF_hap;
     bool iommu = config->flags & XEN_DOMCTL_CDF_iommu;
     bool vpmu = config->flags & XEN_DOMCTL_CDF_vpmu;
+    bool physaddr_abi = config->flags & XEN_DOMCTL_CDF_physaddr_abi;
 
     if ( config->flags &
          ~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
            XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
            XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
-           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
+           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
+           XEN_DOMCTL_CDF_physaddr_abi) )
     {
         dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
         return -EINVAL;
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( physaddr_abi && !hvm )
+    {
+        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
+        return -EINVAL;
+    }
+
     if ( iommu )
     {
         if ( config->iommu_opts & ~XEN_DOMCTL_IOMMU_no_sharept )
xen/include/public/domctl.h +3 -1
index 5b2063eed9..3a77efe673 100644
@@ -66,9 +66,11 @@ struct xen_domctl_createdomain {
 #define XEN_DOMCTL_CDF_nested_virt    (1U << _XEN_DOMCTL_CDF_nested_virt)
 /* Should we expose the vPMU to the guest? */
 #define XEN_DOMCTL_CDF_vpmu           (1U << 7)
+/* Do the guest use physical addresses for hypercalls ? */
+#define XEN_DOMCTL_CDF_physaddr_abi   (1U << 8)
 
 /* Max XEN_DOMCTL_CDF_* constant.  Used for ABI checking. */
-#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
+#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_physaddr_abi
 
     uint32_t flags;
 
xen/include/xen/sched.h +6 -0
index 559d201e0c..a29c63c737 100644
@@ -1201,6 +1201,12 @@ static always_inline bool hap_enabled(const struct domain *d)
         evaluate_nospec(d->options & XEN_DOMCTL_CDF_hap);
 }
 
+static always_inline bool is_hvm_physaddr_abi(const struct domain *d)
+{
+    return IS_ENABLED(CONFIG_HVM) &&
+        evaluate_nospec(d->options & XEN_DOMCTL_CDF_physaddr_abi);
+}
+
 static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v)
 {
     return (is_hardware_domain(v->domain) &&
Signature · 9 lines
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#6 lore raw

RFC PATCH 3/4 x86/public: Expose physaddr_abi through Xen HVM CPUID leaf

In reply to Teddy Astie

The guest needs to know this information before attempting to make a hypercall (especially ones to fetch XENFEAT_*).

---
 xen/arch/x86/cpuid.c                | 3 +++
 xen/include/public/arch-x86/cpuid.h | 2 ++
 2 files changed, 5 insertions(+)
xen/arch/x86/cpuid.c +3 -0
index 8dc68945f7..a721499920 100644
@@ -153,6 +153,9 @@ static void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
          */
         res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
 
+        /* Indicate if the guest uses the physical addresses hypercall ABI. */
+        if ( is_hvm_physaddr_abi(v->domain) )
+            res->a |= XEN_HVM_CPUID_PHYS_ADDR_ABI;
         break;
 
     case 5: /* PV-specific parameters */
xen/include/public/arch-x86/cpuid.h +2 -0
index 3bb0dd249f..b8ceb833d7 100644
@@ -106,6 +106,8 @@
  * bound to event channels.
  */
 #define XEN_HVM_CPUID_UPCALL_VECTOR    (1u << 6)
+/* Hypercalls are using physical addresses instead of linear ones. */
+#define XEN_HVM_CPUID_PHYS_ADDR_ABI    (1u << 7)
 
 /*
  * Leaf 6 (0x40000x05)
Signature · 9 lines
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#0 lore raw

cover Physical address hypercall ABI ("HVMv2")

In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).

This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.

This mode is opt-in and must be enabled explicitely by the toolstack.

This is also mandatory for confidential-computing guests (e.g SEV) where the guest pagetable are not visible to the hypervisor.

In a synthetic xtf-based hypercall benchmark (VCPUOP_get_runstate_info loop), it gives broadly a 30% overhead reduction when tested on a AMD EPYC 9124.

This serie only implement support for x86 yet this ABI is also meaningful for other architectures as well.
A separate patch for adding support in Linux is planned.

Teddy Astie (4):
xen: Introduce physaddr_abi CDF flag
x86/hvm: Consider phyaddr_abi when copying from/to guest memory x86/public: Expose physaddr_abi through Xen HVM CPUID leaf
libxl: Add support for enabling physaddr_abi

 tools/include/libxl.h               |  5 +++++
 tools/libs/light/libxl_create.c     |  4 ++++
 tools/libs/light/libxl_types.idl    |  1 +
 tools/xl/xl_parse.c                 |  1 +
 xen/arch/x86/cpuid.c                |  3 +++
 xen/arch/x86/hvm/hvm.c              | 17 ++++++++++++++---
 xen/common/domain.c                 | 10 +++++++++-
 xen/include/public/arch-x86/cpuid.h |  2 ++
 xen/include/public/domctl.h         |  4 +++-
 xen/include/xen/sched.h             |  6 ++++++
 10 files changed, 48 insertions(+), 5 deletions(-)
Signature · 9 lines
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#7 lore raw

RFC PATCH 4/4 libxl: Add support for enabling physaddr_abi

---
 tools/include/libxl.h            | 5 +++++
 tools/libs/light/libxl_create.c  | 4 ++++
 tools/libs/light/libxl_types.idl | 1 +
 tools/xl/xl_parse.c              | 1 +
 4 files changed, 11 insertions(+)
tools/include/libxl.h +5 -0
index b7ad7735ca..6af9a658c9 100644
@@ -178,6 +178,11 @@
  */
 #define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1
 
+/*
+ * The libxl_domain_build_info has physaddr_abi field.
+ */
+#define LIBXL_HAVE_BUILDINFO_PHYS_ADDR_ABI 1
+
 /*
  * libxl_domain_build_info has the u.hvm.ms_vm_genid field.
  */
tools/libs/light/libxl_create.c +4 -0
index e03599ea99..43e8c18768 100644
@@ -466,6 +466,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
 
         libxl_defbool_setdefault(&b_info->apic,                     true);
         libxl_defbool_setdefault(&b_info->nested_hvm,               false);
+        libxl_defbool_setdefault(&b_info->physaddr_abi,             false);
     }
 
     if (b_info->max_grant_version == LIBXL_MAX_GRANT_DEFAULT) {
@@ -662,6 +663,9 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
 
             if ( libxl_defbool_val(b_info->nested_hvm) )
                 create.flags |= XEN_DOMCTL_CDF_nested_virt;
+
+            if (libxl_defbool_val(b_info->physaddr_abi))
+                create.flags |= XEN_DOMCTL_CDF_physaddr_abi;
         }
 
         if (libxl_defbool_val(b_info->vpmu))
tools/libs/light/libxl_types.idl +1 -0
index 9bb2969931..d02e84471b 100644
@@ -637,6 +637,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("nested_hvm",       libxl_defbool),
     ("apic",             libxl_defbool),
     ("dm_restrict",      libxl_defbool),
+    ("physaddr_abi",     libxl_defbool),
     ("tee",              libxl_tee_type),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
tools/xl/xl_parse.c +1 -0
index 089a88935a..bd674684fa 100644
@@ -1803,6 +1803,7 @@ void parse_config_data(const char *config_source,
     }
 
     xlu_cfg_get_defbool(config, "nestedhvm", &b_info->nested_hvm, 0);
+    xlu_cfg_get_defbool(config, "physaddr_abi", &b_info->physaddr_abi, 0);
 
     switch(b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
Signature · 9 lines
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#5 lore raw

RFC PATCH 2/4 x86/hvm: Consider phyaddr_abi when copying from/to guest memory

In reply to Teddy Astie
---
 xen/arch/x86/hvm/hvm.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
xen/arch/x86/hvm/hvm.c +14 -3
index 6f1174c512..5923166b9b 100644
@@ -3499,7 +3499,11 @@ unsigned int copy_to_user_hvm(void *to, const void *from, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_to_guest_linear((unsigned long)to, from, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )    
+        rc = hvm_copy_to_guest_phys((unsigned long)to, from, len, current);
+    else
+        rc = hvm_copy_to_guest_linear((unsigned long)to, from, len, 0, NULL);
+
     return rc ? len : 0; /* fake a copy_to_user() return code */
 }
 
@@ -3513,7 +3517,10 @@ unsigned int clear_user_hvm(void *to, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_to_guest_linear((unsigned long)to, NULL, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )
+        rc = hvm_copy_to_guest_phys((unsigned long)to, NULL, len, current);
+    else
+        rc = hvm_copy_to_guest_linear((unsigned long)to, NULL, len, 0, NULL);
 
     return rc ? len : 0; /* fake a clear_user() return code */
 }
@@ -3528,7 +3535,11 @@ unsigned int copy_from_user_hvm(void *to, const void *from, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_from_guest_linear(to, (unsigned long)from, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )
+        rc = hvm_copy_from_guest_phys(to, (unsigned long)from, len);
+    else
+        rc = hvm_copy_from_guest_linear(to, (unsigned long)from, len, 0, NULL);
+
     return rc ? len : 0; /* fake a copy_from_user() return code */
 }
 
Signature · 9 lines
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#8 lore raw
In reply to Teddy Astie
Quoted 9 lines · Teddy Astie

In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).

Quoted 5 lines

This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.

This mode is opt-in and must be enabled explicitely by the toolstack.

Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way a guest needs to be converted all in one go. While doable, it'll be increasingly risky with the size of the guest kernel code base.

A prior proposal of mine was to add an indicator to hypercall numbers (e.g. to set the top bit there), to indicate which of the two models a particular hypercall invocation uses.

Aiui Andrew had yet different (albeit also never spelled out) plans.

Jan

#2 lore raw

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

In reply to Teddy Astie

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754

Quoted 9 lines · Teddy Astie
xen/common/domain.c +5 -0
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( physaddr_abi && !hvm )
+    {
+        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
+        return -EINVAL;
+    }

Why this restriction?

Jan

#9 lore raw
In reply to Jan Beulich

Hi,

Quoted 14 lines · Jan Beulich
Quoted 13 lines · Teddy Astie

In current HVM mode, when a hypercall references a structure in guest memory, it is passed to the hypervisor as its "linear address" (e.g virtual address for the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly usable by the Xen and needs to be translated from GVA to GPA then HPA. This implies a complex and potentially expensive lookup of the pagetables inside the guest. This can be significant, especially if the P2M cannot use efficiently superpages (or with e.g XSA-304).

This proposal introduce a new mode where all addresses used for hypercalls are GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA only needs a P2M lookup and not looking through the inside-guest pagetables.

Quoted 5 lines
Quoted 1 line

This mode is opt-in and must be enabled explicitely by the toolstack.

Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way a guest needs to be converted all in one go. While doable, it'll be increasingly risky with the size of the guest kernel code base.

+1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).

Furthermore, I don't think this can be easily adopted in public cloud where the admin for Xen and the guest will be different. So any indication of the ABI would have to come from the guest itself rather than the configuration.

Cheers,

Signature · 1 line
Julien Grall
#3 lore raw

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

In reply to Jan Beulich

Le 30/04/2025 à 17:59, Jan Beulich a écrit :

Quoted 5 lines · Teddy Astie
Quoted 4 lines
xen/common/domain.c +0 -0
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
          return -EINVAL;
      }
 
Quoted 9 lines

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754

Quoted 5 lines
xen/common/domain.c +5 -0
+    if ( physaddr_abi && !hvm )
+    {
+        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
+        return -EINVAL;
+    }

Why this restriction?

physaddr_abi changes how copy_from/to_guest works to make it use GPA instead of GVA. As non-HVM probably means PV guest, it would mean something like PV guest hypercalls uses physical addresses (derived from MFN?) instead of virtual addresses, which would not really be practical for both the guest and the hypervisor.

Quoted 2 lines

Jan

Teddy

Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#10 lore raw
In reply to Julien Grall

Le 01/05/2025 à 13:14, Julien Grall a écrit :

Quoted 34 lines

Hi,

Quoted 23 lines · Jan Beulich
Quoted 22 lines · Teddy Astie

In current HVM mode, when a hypercall references a structure in guest memory,
it is passed to the hypervisor as its "linear address" (e.g virtual address for
the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly
usable by the Xen and needs to be translated from GVA to GPA then HPA. This
implies a complex and potentially expensive lookup of the pagetables inside the
guest. This can be significant, especially if the P2M cannot use efficiently
superpages (or with e.g XSA-304).

This proposal introduce a new mode where all addresses used for hypercalls are
GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA
only needs a P2M lookup and not looking through the inside-guest pagetables.

Quoted 7 lines
Quoted 1 line

This mode is opt-in and must be enabled explicitely by the toolstack.

Which I view as a severe downside (leaving aside the PVH Dom0 aspect):
This way
a guest needs to be converted all in one go. While doable, it'll be increasingly
risky with the size of the guest kernel code base.

Quoted 7 lines

+1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).

Furthermore, I don't think this can be easily adopted in public cloud where the admin for Xen and the guest will be different. So any indication of the ABI would have to come from the guest itself rather than the configuration.

Makes sense, I am experimenting with a alternative design which requires setting the 30th bit (31th is already used for viridian) of the eax register to indicate the use of this new ABI. Also keeping a CPUID to indicate that the feature is supported by the hypervisor (thus no need to enable it in advance for a guest).

Quoted 2 lines

Cheers,

Teddy

Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#4 lore raw

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

In reply to Teddy Astie
Quoted 16 lines · Teddy Astie

Le 30/04/2025 à 17:59, Jan Beulich a écrit :

Quoted 5 lines · Teddy Astie
Quoted 4 lines
xen/common/domain.c +0 -0
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
          return -EINVAL;
      }
 
Quoted 9 lines

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 754

Quoted 5 lines
xen/common/domain.c +5 -0
+    if ( physaddr_abi && !hvm )
+    {
+        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
+        return -EINVAL;
+    }

Why this restriction?

Quoted 4 lines

physaddr_abi changes how copy_from/to_guest works to make it use GPA instead of GVA. As non-HVM probably means PV guest, it would mean something like PV guest hypercalls uses physical addresses (derived from MFN?)

Machine addresses, yes (hence MFN). If it was PFNs / (pseudo-)physical addresses, ...

Quoted 2 lines

instead of virtual addresses, which would not really be practical for both the guest and the hypervisor.

... I'd maybe agree here.

Jan