Skip to contentSkip to conversation

x86: Make MAX_ALTP2M configurable

PATCH Superseded Petr Beneš 2 participants 18 messages Apr 28, 2024 – May 2, 2024

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover x86: Make MAX_ALTP2M configurable

From: Petr Beneš <[email protected]>

This series introduces the ability to configure the maximum number of altp2m tables during domain creation. Previously, the limits were hardcoded to a maximum of 10. This change allows for greater flexibility in environments that require more or fewer altp2m views.

Adjustments include:
- "Prepare" commits with style changes.
- Adding a new max_altp2m parameter to xl.
- Adding a new max_altp2m parameter to the OCaml bindings.
- Adding a new max_altp2m parameter to the xl.cfg manual.
- Adding a new max_altp2m field into the xen_domctl_createdomain, which, after sanity checks, is stored in newly introduced max_altp2m field of struct domain - leaving room for other architectures to implement the altp2m feature.
- Replacing MAX_ALTP2M macro occurrences with domain->max_altp2m.
- Finally, adjusting the initial allocation of pages in hap_enable from 256 to 1024 pages to accommodate potentially larger max_altp2m values (i.e., maximum of 512).

This enhancement is particularly relevant for users leveraging Xen's features for virtual machine introspection.

Petr Beneš (7):
x86/p2m: Add braces for better code clarity
tools/xl: Add max_altp2m parameter
docs/man: Add max_altp2m parameter to the xl.cfg manual
x86: Make the maximum number of altp2m views configurable tools/libxl: Activate the max_altp2m feature tools/ocaml: Add max_altp2m parameter
x86/hap: Increase the number of initial mempool_size to 1024 pages

 docs/man/xl.cfg.5.pod.in                      | 14 +++++
 tools/golang/xenlight/helpers.gen.go          |  2 +
 tools/golang/xenlight/types.gen.go            |  1 +
 tools/include/libxl.h                         |  8 +++
 tools/libs/light/libxl_create.c               |  9 ++++
 tools/libs/light/libxl_types.idl              |  1 +
 tools/ocaml/libs/xc/xenctrl.ml                |  1 +
 tools/ocaml/libs/xc/xenctrl.mli               |  1 +
 tools/ocaml/libs/xc/xenctrl_stubs.c           | 17 +++---
 .../paging-mempool/test-paging-mempool.c      |  2 +-
 tools/xl/xl_parse.c                           |  4 ++
 xen/arch/x86/domain.c                         |  6 +++
 xen/arch/x86/hvm/hvm.c                        |  8 ++-
 xen/arch/x86/hvm/vmx/vmx.c                    |  2 +-
 xen/arch/x86/include/asm/domain.h             |  7 ++-
 xen/arch/x86/include/asm/p2m.h                |  4 +-
 xen/arch/x86/mm/altp2m.c                      | 27 +++++++++-
 xen/arch/x86/mm/hap/hap.c                     |  8 +--
 xen/arch/x86/mm/mem_access.c                  | 14 ++---
 xen/arch/x86/mm/mem_sharing.c                 |  2 +-
 xen/arch/x86/mm/p2m-ept.c                     |  6 +--
 xen/arch/x86/mm/p2m.c                         | 54 ++++++++++---------
 xen/common/domain.c                           |  7 +++
 xen/include/public/domctl.h                   |  3 +-
 xen/include/xen/sched.h                       |  2 +
 25 files changed, 151 insertions(+), 59 deletions(-)

--
2.34.1

#1 lore raw

PATCH v2 1/7 x86/p2m: Add braces for better code clarity

From: Petr Beneš <[email protected]>

No functional change.

---
 xen/arch/x86/mm/p2m.c | 4 ++++
 1 file changed, 4 insertions(+)
xen/arch/x86/mm/p2m.c +4 -0
index ce742c12e0..eb7996170d 100644
@@ -106,6 +106,7 @@ void p2m_change_entry_type_global(struct domain *d,
         unsigned int i;
 
         for ( i = 0; i < MAX_ALTP2M; i++ )
+        {
             if ( d->arch.altp2m_eptp[i] != mfn_x(INVALID_MFN) )
             {
                 struct p2m_domain *altp2m = d->arch.altp2m_p2m[i];
@@ -114,6 +115,7 @@ void p2m_change_entry_type_global(struct domain *d,
                 change_entry_type_global(altp2m, ot, nt);
                 p2m_unlock(altp2m);
             }
+        }
     }
 
     p2m_unlock(hostp2m);
@@ -139,6 +141,7 @@ void p2m_memory_type_changed(struct domain *d)
         unsigned int i;
 
         for ( i = 0; i < MAX_ALTP2M; i++ )
+        {
             if ( d->arch.altp2m_eptp[i] != mfn_x(INVALID_MFN) )
             {
                 struct p2m_domain *altp2m = d->arch.altp2m_p2m[i];
@@ -147,6 +150,7 @@ void p2m_memory_type_changed(struct domain *d)
                 _memory_type_changed(altp2m);
                 p2m_unlock(altp2m);
             }
+        }
     }
 
     p2m_unlock(hostp2m);
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v2 1/7] x86/p2m: Add braces for better code clarity

Quoted 11 lines · Jan Beulich
Quoted 5 lines · Petr Beneš

From: Petr Beneš <[email protected]>

No functional change.

Where did Stefano's R-b go?

Jan

Oh no, I missed that one. Should I do v3?

#4 lore raw

Re: [PATCH v2 1/7] x86/p2m: Add braces for better code clarity

Quoted 5 lines · Petr Beneš
Quoted 4 lines · Jan Beulich
Quoted 2 lines · Petr Beneš

From: Petr Beneš <[email protected]>

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

No functional change.

Where did Stefano's R-b go?

Oh no, I missed that one. Should I do v3?

Not just for this, I'd say. Just don't forget it again, if the patch needs re-posting.

Jan

#5 lore raw

PATCH v2 2/7 tools/xl: Add max_altp2m parameter

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

Introduce a new max_altp2m parameter to control the maximum number of altp2m views a domain can use. By default, if max_altp2m is unspecified and altp2m is enabled, the value is set to 10, reflecting the legacy behavior.

This change is preparatory; it establishes the groundwork for the feature but does not activate it.

---
Changed since v1:
* Removed the change of xen/include/public/domctl.h (moved into future commit)
* Removed corresponding assignment of xen_domctl_createdomain::max_altp2m in libxl_create.c (moved into future commit)
* Clarify in commit message that this change is preparatory

 tools/golang/xenlight/helpers.gen.go | 2 ++
 tools/golang/xenlight/types.gen.go   | 1 +
 tools/include/libxl.h                | 8 ++++++++
 tools/libs/light/libxl_create.c      | 8 ++++++++
 tools/libs/light/libxl_types.idl     | 1 +
 tools/xl/xl_parse.c                  | 4 ++++
 6 files changed, 24 insertions(+)
tools/golang/xenlight/helpers.gen.go +2 -0
index 78bdb08b15..4458d5bcbb 100644
@@ -1158,6 +1158,7 @@ if err := x.ArchX86.MsrRelaxed.fromC(&xc.arch_x86.msr_relaxed);err != nil {
 return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err)
 }
 x.Altp2M = Altp2MMode(xc.altp2m)
+x.MaxAltp2M = uint32(xc.max_altp2m)
 x.VmtraceBufKb = int(xc.vmtrace_buf_kb)
 if err := x.Vpmu.fromC(&xc.vpmu);err != nil {
 return fmt.Errorf("converting field Vpmu: %v", err)
@@ -1674,6 +1675,7 @@ if err := x.ArchX86.MsrRelaxed.toC(&xc.arch_x86.msr_relaxed); err != nil {
 return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err)
 }
 xc.altp2m = C.libxl_altp2m_mode(x.Altp2M)
+xc.max_altp2m = C.uint32_t(x.MaxAltp2M)
 xc.vmtrace_buf_kb = C.int(x.VmtraceBufKb)
 if err := x.Vpmu.toC(&xc.vpmu); err != nil {
 return fmt.Errorf("converting field Vpmu: %v", err)
tools/golang/xenlight/types.gen.go +1 -0
index ccfe18019e..7139bcf324 100644
@@ -602,6 +602,7 @@ ArchX86 struct {
 MsrRelaxed Defbool
 }
 Altp2M Altp2MMode
+MaxAltp2M uint32
 VmtraceBufKb int
 Vpmu Defbool
 }
tools/include/libxl.h +8 -0
index 62cb07dea6..c73d9f2ff1 100644
@@ -1239,6 +1239,14 @@ typedef struct libxl__ctx libxl_ctx;
  */
 #define LIBXL_HAVE_ALTP2M 1
 
+/*
+ * LIBXL_HAVE_MAX_ALTP2M
+ * If this is defined, then libxl supports setting the maximum number of
+ * alternate p2m tables.
+ */
+#define LIBXL_HAVE_MAX_ALTP2M 1
+#define LIBXL_MAX_ALTP2M_DEFAULT (~(uint32_t)0)
+
 /*
  * LIBXL_HAVE_REMUS
  * If this is defined, then libxl supports remus.
tools/libs/light/libxl_create.c +8 -0
index 41252ec553..801f704a02 100644
@@ -483,6 +483,14 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
         return -ERROR_INVAL;
     }
 
+    if (b_info->max_altp2m == LIBXL_MAX_ALTP2M_DEFAULT) {
+        if ((libxl_defbool_val(b_info->u.hvm.altp2m) ||
+            b_info->altp2m != LIBXL_ALTP2M_MODE_DISABLED))
+            b_info->max_altp2m = 10;
+        else
+            b_info->max_altp2m = 0;
+    }
+
     /* Assume that providing a bootloader user implies enabling restrict. */
     libxl_defbool_setdefault(&b_info->bootloader_restrict,
                              !!b_info->bootloader_user);
tools/libs/light/libxl_types.idl +1 -0
index 470122e768..c887d8ea8c 100644
@@ -728,6 +728,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     # Alternate p2m is not bound to any architecture or guest type, as it is
     # supported by x86 HVM and ARM support is planned.
     ("altp2m", libxl_altp2m_mode),
+    ("max_altp2m", uint32, {'init_val': 'LIBXL_MAX_ALTP2M_DEFAULT'}),
 
     # Size of preallocated vmtrace trace buffers (in KBYTES).
     # Use zero value to disable this feature.
tools/xl/xl_parse.c +4 -0
index ab09d0288b..741668e10a 100644
@@ -2061,6 +2061,10 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "max_altp2m", &l, 1)) {
+        b_info->max_altp2m = l;
+    }
+
     if (!xlu_cfg_get_long(config, "vmtrace_buf_kb", &l, 1) && l) {
         b_info->vmtrace_buf_kb = l;
     }

--
2.34.1

#6 lore raw

PATCH v2 3/7 docs/man: Add max_altp2m parameter to the xl.cfg manual

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

Update manual pages to include detailed information about the max_altp2m configuration parameter.

---
 docs/man/xl.cfg.5.pod.in | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
docs/man/xl.cfg.5.pod.in +14 -0
index 8f2b375ce9..2d4ea35736 100644
@@ -2039,6 +2039,20 @@ a single guest HVM domain. B<This option is deprecated, use the option
 B<Note>: While the option "altp2mhvm" is deprecated, legacy applications for
 x86 systems will continue to work using it.
 
+=item B<max_altp2m=NUMBER>
+
+Specifies the maximum number of alternate-p2m views available to the guest.
+This setting is crucial in domain introspection scenarios that require
+multiple physical-to-machine (p2m) memory mappings to be established
+simultaneously.
+
+Enabling multiple p2m views may increase memory usage. It is advisable to
+review and adjust the B<shadow_memory> setting as necessary to accommodate
+the additional memory requirements.
+
+B<Note>: This option is ignored if B<altp2m> is disabled. The default value
+is 10.
+
 =item B<nestedhvm=BOOLEAN>
 
 Enable or disables guest access to hardware virtualisation features,
Signature · 1 line
2.34.1
#7 lore raw

PATCH v2 4/7 x86: Make the maximum number of altp2m views configurable

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

This commit introduces the ability to configure the maximum number of altp2m tables during domain creation. Previously, the limits were hardcoded to a maximum of 10. This change allows for greater flexibility in environments that require more or fewer altp2m views.

The maximum configurable limit for max_altp2m on x86 is now set to MAX_EPTP (512). This cap is linked to the architectural limit of the EPTP-switching VMFUNC, which supports up to 512 entries. Despite there being no inherent need for limiting max_altp2m in scenarios not utilizing VMFUNC, decoupling these components would necessitate substantial code changes.

---
Changed since v1:
* Added xen_domctl_createdomain::max_altp2m field to xen/include/public/domctl.h
* Bumped the XEN_DOMCTL_INTERFACE_VERSION
* More elaborate commit message

 xen/arch/x86/domain.c             |  6 ++++
 xen/arch/x86/hvm/hvm.c            |  8 ++++-
 xen/arch/x86/hvm/vmx/vmx.c        |  2 +-
 xen/arch/x86/include/asm/domain.h |  7 ++---
 xen/arch/x86/include/asm/p2m.h    |  4 +--
 xen/arch/x86/mm/altp2m.c          | 27 +++++++++++++++--
 xen/arch/x86/mm/hap/hap.c         |  6 ++--
 xen/arch/x86/mm/mem_access.c      | 14 ++++-----
 xen/arch/x86/mm/mem_sharing.c     |  2 +-
 xen/arch/x86/mm/p2m-ept.c         |  6 ++--
 xen/arch/x86/mm/p2m.c             | 50 +++++++++++++++----------------
 xen/common/domain.c               |  7 +++++
 xen/include/public/domctl.h       |  3 +-
 xen/include/xen/sched.h           |  2 ++
 14 files changed, 94 insertions(+), 50 deletions(-)
xen/arch/x86/domain.c +6 -0
index 20e83cf38b..6b458f330c 100644
@@ -685,6 +685,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->max_altp2m > MAX_EPTP )
+    {
+        dprintk(XENLOG_INFO, "max_altp2m must be <= %u\n", MAX_EPTP);
+        return -EINVAL;
+    }
+
     if ( config->vmtrace_size )
     {
         unsigned int size = config->vmtrace_size;
xen/arch/x86/hvm/hvm.c +7 -1
index 0ce45b177c..9b70fe7cfc 100644
@@ -4633,6 +4633,12 @@ static int do_altp2m_op(
         goto out;
     }
 
+    if ( d->max_altp2m == 0 )
+    {
+        rc = -EINVAL;
+        goto out;
+    }
+
     if ( (rc = xsm_hvm_altp2mhvm_op(XSM_OTHER, d, mode, a.cmd)) )
         goto out;
 
@@ -5222,7 +5228,7 @@ void hvm_fast_singlestep(struct vcpu *v, uint16_t p2midx)
     if ( !hvm_is_singlestep_supported() )
         return;
 
-    if ( p2midx >= MAX_ALTP2M )
+    if ( p2midx >= v->domain->max_altp2m )
         return;
 
     v->arch.hvm.single_step = true;
xen/arch/x86/hvm/vmx/vmx.c +1 -1
index 5f67a48592..8f57f3a7f5 100644
@@ -4888,7 +4888,7 @@ bool asmlinkage vmx_vmenter_helper(const struct cpu_user_regs *regs)
         {
             unsigned int i;
 
-            for ( i = 0; i < MAX_ALTP2M; ++i )
+            for ( i = 0; i < currd->max_altp2m; ++i )
             {
                 if ( currd->arch.altp2m_eptp[i] == mfn_x(INVALID_MFN) )
                     continue;
xen/arch/x86/include/asm/domain.h +3 -4
index f5daeb182b..5bb0bcae81 100644
@@ -258,11 +258,10 @@ struct paging_vcpu {
     struct shadow_vcpu shadow;
 };
 
+#define INVALID_ALTP2M  0xffff
+#define MAX_EPTP        ((unsigned int)(PAGE_SIZE / sizeof(uint64_t)))
 #define MAX_NESTEDP2M 10
 
-#define MAX_ALTP2M      10 /* arbitrary */
-#define INVALID_ALTP2M  0xffff
-#define MAX_EPTP        (PAGE_SIZE / sizeof(uint64_t))
 struct p2m_domain;
 struct time_scale {
     int shift;
@@ -353,7 +352,7 @@ struct arch_domain
 
     /* altp2m: allow multiple copies of host p2m */
     bool altp2m_active;
-    struct p2m_domain *altp2m_p2m[MAX_ALTP2M];
+    struct p2m_domain **altp2m_p2m;
     mm_lock_t altp2m_list_lock;
     uint64_t *altp2m_eptp;
     uint64_t *altp2m_visible_eptp;
xen/arch/x86/include/asm/p2m.h +2 -2
index 111badf89a..2086bcb633 100644
@@ -881,7 +881,7 @@ static inline struct p2m_domain *p2m_get_altp2m(struct vcpu *v)
     if ( index == INVALID_ALTP2M )
         return NULL;
 
-    BUG_ON(index >= MAX_ALTP2M);
+    BUG_ON(index >= v->domain->max_altp2m);
 
     return v->domain->arch.altp2m_p2m[index];
 }
@@ -891,7 +891,7 @@ static inline bool p2m_set_altp2m(struct vcpu *v, unsigned int idx)
 {
     struct p2m_domain *orig;
 
-    BUG_ON(idx >= MAX_ALTP2M);
+    BUG_ON(idx >= v->domain->max_altp2m);
 
     if ( idx == vcpu_altp2m(v).p2midx )
         return false;
xen/arch/x86/mm/altp2m.c +25 -2
index a04297b646..c91e0fbfd1 100644
@@ -13,6 +13,12 @@
 void
 altp2m_vcpu_initialise(struct vcpu *v)
 {
+    struct domain *d = v->domain;
+
+    /* Skip initialisation if no altp2m will be used. */
+    if ( d->max_altp2m == 0 )
+        return;
+
     if ( v != current )
         vcpu_pause(v);
 
@@ -28,8 +34,13 @@ altp2m_vcpu_initialise(struct vcpu *v)
 void
 altp2m_vcpu_destroy(struct vcpu *v)
 {
+    struct domain *d = v->domain;
     struct p2m_domain *p2m;
 
+    /* Skip destruction if no altp2m was used. */
+    if ( d->max_altp2m == 0 )
+        return;
+
     if ( v != current )
         vcpu_pause(v);
 
@@ -120,7 +131,13 @@ int p2m_init_altp2m(struct domain *d)
     struct p2m_domain *hostp2m = p2m_get_hostp2m(d);
 
     mm_lock_init(&d->arch.altp2m_list_lock);
-    for ( i = 0; i < MAX_ALTP2M; i++ )
+
+    if ( (d->arch.altp2m_p2m = xzalloc_array(struct p2m_domain *, d->max_altp2m)) == NULL )
+    {
+        return -ENOMEM;
+    }
+
+    for ( i = 0; i < d->max_altp2m; i++ )
     {
         d->arch.altp2m_p2m[i] = p2m = p2m_init_one(d);
         if ( p2m == NULL )
@@ -141,7 +158,10 @@ void p2m_teardown_altp2m(struct domain *d)
     unsigned int i;
     struct p2m_domain *p2m;
 
-    for ( i = 0; i < MAX_ALTP2M; i++ )
+    if ( !d->arch.altp2m_p2m )
+        return;
+
+    for ( i = 0; i < d->max_altp2m; i++ )
     {
         if ( !d->arch.altp2m_p2m[i] )
             continue;
@@ -149,6 +169,9 @@ void p2m_teardown_altp2m(struct domain *d)
         d->arch.altp2m_p2m[i] = NULL;
         p2m_free_one(p2m);
     }
+
+    xfree(d->arch.altp2m_p2m);
+    d->arch.altp2m_p2m = NULL;
 }
 
 /*
xen/arch/x86/mm/hap/hap.c +3 -3
index d2011fde24..7aff5fa664 100644
@@ -515,7 +515,7 @@ int hap_enable(struct domain *d, u32 mode)
             d->arch.altp2m_visible_eptp[i] = mfn_x(INVALID_MFN);
         }
 
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
         {
             rv = p2m_alloc_table(d->arch.altp2m_p2m[i]);
             if ( rv != 0 )
@@ -538,7 +538,7 @@ void hap_final_teardown(struct domain *d)
     unsigned int i;
 
     if ( hvm_altp2m_supported() )
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
             p2m_teardown(d->arch.altp2m_p2m[i], true, NULL);
 
     /* Destroy nestedp2m's first */
@@ -590,7 +590,7 @@ void hap_teardown(struct domain *d, bool *preempted)
         FREE_XENHEAP_PAGE(d->arch.altp2m_eptp);
         FREE_XENHEAP_PAGE(d->arch.altp2m_visible_eptp);
 
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
         {
             p2m_teardown(d->arch.altp2m_p2m[i], false, preempted);
             if ( preempted && *preempted )
xen/arch/x86/mm/mem_access.c +7 -7
index 60a0cce68a..1bf40cb746 100644
@@ -347,12 +347,12 @@ long p2m_set_mem_access(struct domain *d, gfn_t gfn, uint32_t nr,
     /* altp2m view 0 is treated as the hostp2m */
     if ( altp2m_idx )
     {
-        if ( altp2m_idx >= min(ARRAY_SIZE(d->arch.altp2m_p2m), MAX_EPTP) ||
+        if ( altp2m_idx >= min(d->max_altp2m, MAX_EPTP) ||
              d->arch.altp2m_eptp[array_index_nospec(altp2m_idx, MAX_EPTP)] ==
              mfn_x(INVALID_MFN) )
             return -EINVAL;
 
-        ap2m = array_access_nospec(d->arch.altp2m_p2m, altp2m_idx);
+        ap2m = d->arch.altp2m_p2m[array_index_nospec(altp2m_idx, d->max_altp2m)];
     }
 
     if ( !xenmem_access_to_p2m_access(p2m, access, &a) )
@@ -403,12 +403,12 @@ long p2m_set_mem_access_multi(struct domain *d,
     /* altp2m view 0 is treated as the hostp2m */
     if ( altp2m_idx )
     {
-        if ( altp2m_idx >= min(ARRAY_SIZE(d->arch.altp2m_p2m), MAX_EPTP) ||
+        if ( altp2m_idx >= min(d->max_altp2m, MAX_EPTP) ||
              d->arch.altp2m_eptp[array_index_nospec(altp2m_idx, MAX_EPTP)] ==
              mfn_x(INVALID_MFN) )
             return -EINVAL;
 
-        ap2m = array_access_nospec(d->arch.altp2m_p2m, altp2m_idx);
+        ap2m = d->arch.altp2m_p2m[array_index_nospec(altp2m_idx, d->max_altp2m)];
     }
 
     p2m_lock(p2m);
@@ -466,12 +466,12 @@ int p2m_get_mem_access(struct domain *d, gfn_t gfn, xenmem_access_t *access,
     }
     else if ( altp2m_idx ) /* altp2m view 0 is treated as the hostp2m */
     {
-        if ( altp2m_idx >= min(ARRAY_SIZE(d->arch.altp2m_p2m), MAX_EPTP) ||
+        if ( altp2m_idx >= min(d->max_altp2m, MAX_EPTP) ||
              d->arch.altp2m_eptp[array_index_nospec(altp2m_idx, MAX_EPTP)] ==
              mfn_x(INVALID_MFN) )
             return -EINVAL;
 
-        p2m = array_access_nospec(d->arch.altp2m_p2m, altp2m_idx);
+        p2m = d->arch.altp2m_p2m[array_index_nospec(altp2m_idx, d->max_altp2m)];
     }
 
     return _p2m_get_mem_access(p2m, gfn, access);
@@ -486,7 +486,7 @@ void arch_p2m_set_access_required(struct domain *d, bool access_required)
     if ( altp2m_active(d) )
     {
         unsigned int i;
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
         {
             struct p2m_domain *p2m = d->arch.altp2m_p2m[i];
 
xen/arch/x86/mm/mem_sharing.c +1 -1
index da28266ef0..3aaf1a3b8d 100644
@@ -912,7 +912,7 @@ static int nominate_page(struct domain *d, gfn_t gfn,
 
         altp2m_list_lock(d);
 
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
         {
             ap2m = d->arch.altp2m_p2m[i];
             if ( !ap2m )
xen/arch/x86/mm/p2m-ept.c +3 -3
index f83610cb8c..6b75fafd49 100644
@@ -1293,7 +1293,7 @@ static void ept_set_ad_sync(struct domain *d, bool value)
     {
         unsigned int i;
 
-        for ( i = 0; i < MAX_ALTP2M; i++ )
+        for ( i = 0; i < d->max_altp2m; i++ )
         {
             struct p2m_domain *p2m;
 
@@ -1500,7 +1500,7 @@ void setup_ept_dump(void)
 
 void p2m_init_altp2m_ept(struct domain *d, unsigned int i)
 {
-    struct p2m_domain *p2m = array_access_nospec(d->arch.altp2m_p2m, i);
+    struct p2m_domain *p2m = d->arch.altp2m_p2m[array_index_nospec(i, d->max_altp2m)];
     struct p2m_domain *hostp2m = p2m_get_hostp2m(d);
     struct ept_data *ept;
 
@@ -1519,7 +1519,7 @@ unsigned int p2m_find_altp2m_by_eptp(struct domain *d, uint64_t eptp)
 
     altp2m_list_lock(d);
 
-    for ( i = 0; i < MAX_ALTP2M; i++ )
+    for ( i = 0; i < d->max_altp2m; i++ )
     {
         if ( d->arch.altp2m_eptp[i] == mfn_x(INVALID_MFN) )
             continue;
xen/arch/x86/mm/p2m.c +25 -25

179 lines not on this page · raw message

xen/common/domain.c +7 -0
index 6773f7fb90..18785cc22a 100644
@@ -568,6 +568,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         }
     }
 
+    if ( config->max_altp2m && !hvm_altp2m_supported() )
+    {
+        dprintk(XENLOG_INFO, "altp2m requested but not available\n");
+        return -EINVAL;
+    }
+
     if ( config->vmtrace_size && !vmtrace_available )
     {
         dprintk(XENLOG_INFO, "vmtrace requested but not available\n");
@@ -610,6 +616,7 @@ struct domain *domain_create(domid_t domid,
     if ( config )
     {
         d->options = config->flags;
+        d->max_altp2m = config->max_altp2m;
         d->vmtrace_size = config->vmtrace_size;
     }
 
xen/include/public/domctl.h +2 -1
index a33f9ec32b..f9bd0ffe1a 100644
@@ -21,7 +21,7 @@
 #include "hvm/save.h"
 #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000016
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000017
 
 /*
  * NB. xen_domctl.domain is an IN/OUT parameter for this operation.
@@ -77,6 +77,7 @@ struct xen_domctl_createdomain {
      */
     uint32_t max_vcpus;
     uint32_t max_evtchn_port;
+    uint32_t max_altp2m;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
 
xen/include/xen/sched.h +2 -0
index 132b841995..46436fcb0b 100644
@@ -602,6 +602,8 @@ struct domain
         unsigned int guest_request_sync          : 1;
     } monitor;
 
+    unsigned int max_altp2m; /* Maximum number of altp2m tables */
+
     unsigned int vmtrace_size; /* Buffer size in bytes, or 0 to disable. */
 
 #ifdef CONFIG_ARGO

--
2.34.1

#8 lore raw

Re: [PATCH v2 4/7] x86: Make the maximum number of altp2m views configurable

Quoted 7 lines · Petr Beneš

From: Petr Beneš <[email protected]>

This commit introduces the ability to configure the maximum number of altp2m tables during domain creation. Previously, the limits were hardcoded to a maximum of 10. This change allows for greater flexibility in environments that require more or fewer altp2m views.

Quoted 5 lines

The maximum configurable limit for max_altp2m on x86 is now set to MAX_EPTP (512). This cap is linked to the architectural limit of the EPTP-switching VMFUNC, which supports up to 512 entries. Despite there being no inherent need for limiting max_altp2m in scenarios not utilizing VMFUNC, decoupling these components would necessitate substantial code changes.

While I don't mind this connection, ...

xen/arch/x86/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 692

Quoted 11 lines
xen/arch/x86/domain.c +5 -0
@@ -685,6 +685,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->max_altp2m > MAX_EPTP )
+    {
+        dprintk(XENLOG_INFO, "max_altp2m must be <= %u\n", MAX_EPTP);
+        return -EINVAL;
+    }

... using MAX_EPTP here feels like a layering violation to me. Imo there want to be separate constants, tied together with a suitably placed BUILD_BUG_ON().

Furthermore comparisons like this (there are further ones elsewhere) suggest there is a (continued) naming issue: A max_ or MAX_ prefix ought to name a "maximum valid value", not "number of permitted values". This is not a request to alter MAX_EPTP, but one to make sure the new struct fields really have matching names and purposes.

xen/arch/x86/include/asm/domain.h struct paging_vcpu { · line 262

Quoted 8 lines
xen/arch/x86/include/asm/domain.h +2 -0
@@ -258,11 +258,10 @@ struct paging_vcpu {
     struct shadow_vcpu shadow;
 };
 
+#define INVALID_ALTP2M  0xffff
+#define MAX_EPTP        ((unsigned int)(PAGE_SIZE / sizeof(uint64_t)))

Aiui you add this cast because of the various min() uses. However, besides wanting to avoid such casts (or in fact any, whenever possible), I don't see why you need to retain all those min(): You bound d->max_altp2m against MAX_EPTP during domain creation anyway.

xen/arch/x86/mm/altp2m.c line 20

Quoted 11 lines
xen/arch/x86/mm/altp2m.c +5 -0
@@ -13,6 +13,12 @@
 void
 altp2m_vcpu_initialise(struct vcpu *v)
 {
+    struct domain *d = v->domain;
+
+    /* Skip initialisation if no altp2m will be used. */
+    if ( d->max_altp2m == 0 )
+        return;

While I'm fine with this comment, ...

xen/arch/x86/mm/altp2m.c altp2m_vcpu_initialise(struct vcpu *v) · line 42

Quoted 10 lines
xen/arch/x86/mm/altp2m.c +4 -0
@@ -28,8 +34,13 @@ altp2m_vcpu_initialise(struct vcpu *v)
 void
 altp2m_vcpu_destroy(struct vcpu *v)
 {
+    struct domain *d = v->domain;
     struct p2m_domain *p2m;
 
+    /* Skip destruction if no altp2m was used. */
+    if ( d->max_altp2m == 0 )
+        return;

... this one doesn't look quite right: Even if altp2m-s were initialized, none may have been used (yet).

xen/arch/x86/mm/altp2m.c int p2m_init_altp2m(struct domain *d) · line 138

Quoted 10 lines
xen/arch/x86/mm/altp2m.c +5 -1
@@ -120,7 +131,13 @@ int p2m_init_altp2m(struct domain *d)
     struct p2m_domain *hostp2m = p2m_get_hostp2m(d);
 
     mm_lock_init(&d->arch.altp2m_list_lock);
-    for ( i = 0; i < MAX_ALTP2M; i++ )
+
+    if ( (d->arch.altp2m_p2m = xzalloc_array(struct p2m_domain *, d->max_altp2m)) == NULL )
+    {
+        return -ENOMEM;
+    }

Nit: Overlong line and no need for the braces.

xen/arch/x86/mm/altp2m.c int p2m_init_altp2m(struct domain *d) · line 142

Quoted 3 lines
xen/arch/x86/mm/altp2m.c +1 -0
+    for ( i = 0; i < d->max_altp2m; i++ )
     {
         d->arch.altp2m_p2m[i] = p2m = p2m_init_one(d);

This loop, btw, also demonstrates that "maximum" isn't true here. The domain gets all of them allocated in one go.

Quoted 13 lines
xen/arch/x86/mm/altp2m.c +4 -1
@@ -141,7 +158,10 @@ void p2m_teardown_altp2m(struct domain *d)
     unsigned int i;
     struct p2m_domain *p2m;
 
-    for ( i = 0; i < MAX_ALTP2M; i++ )
+    if ( !d->arch.altp2m_p2m )
+        return;
+
+    for ( i = 0; i < d->max_altp2m; i++ )
     {
         if ( !d->arch.altp2m_p2m[i] )
             continue;

xen/arch/x86/mm/altp2m.c void p2m_teardown_altp2m(struct domain *d) · line 175

Quoted 7 lines
xen/arch/x86/mm/altp2m.c +3 -0
@@ -149,6 +169,9 @@ void p2m_teardown_altp2m(struct domain *d)
         d->arch.altp2m_p2m[i] = NULL;
         p2m_free_one(p2m);
     }
+
+    xfree(d->arch.altp2m_p2m);
+    d->arch.altp2m_p2m = NULL;
 }

Please don't (wrongly) open-code XFREE().

xen/arch/x86/mm/altp2m.c int p2m_change_altp2m_gfn(struct domain *d, unsigned int idx, · line 2093

Quoted 6 lines
xen/arch/x86/mm/altp2m.c +1 -1
@@ -2090,13 +2090,13 @@ int p2m_change_altp2m_gfn(struct domain *d, unsigned int idx,
     mfn_t mfn;
     int rc = -EINVAL;
 
-    if ( idx >=  min(ARRAY_SIZE(d->arch.altp2m_p2m), MAX_EPTP) ||
+    if ( idx >=  min(d->max_altp2m, MAX_EPTP) ||

Nit: Please take the opportunity and remove the excess blank.

xen/arch/x86/mm/altp2m.c int p2m_change_altp2m_gfn(struct domain *d, unsigned int idx, · line 2094

Quoted 1 line
xen/arch/x86/mm/altp2m.c +0 -0
          d->arch.altp2m_eptp[array_index_nospec(idx, MAX_EPTP)] ==

This use of MAX_EPTP also needs replacing, to avoid speculatively overrunning the allocated array. At least one more instance elsewhere.

xen/arch/x86/mm/altp2m.c int p2m_set_suppress_ve_multi(struct domain *d, · line 2583

Quoted 12 lines
xen/arch/x86/mm/altp2m.c +2 -2
@@ -2575,12 +2575,12 @@ int p2m_set_suppress_ve_multi(struct domain *d,
 
     if ( sve->view > 0 )
     {
-        if ( sve->view >= min(ARRAY_SIZE(d->arch.altp2m_p2m), MAX_EPTP) ||
+        if ( sve->view >= min(d->max_altp2m, MAX_EPTP) ||
              d->arch.altp2m_eptp[array_index_nospec(sve->view, MAX_EPTP)] ==
              mfn_x(INVALID_MFN) )
             return -EINVAL;
 
-        p2m = ap2m = array_access_nospec(d->arch.altp2m_p2m, sve->view);
+        p2m = ap2m = d->arch.altp2m_p2m[array_index_nospec(sve->view, d->max_altp2m)];

Nit: Overlong line (more elsewhere).

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 571

Quoted 7 lines
xen/common/domain.c +1 -0
@@ -568,6 +568,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         }
     }
 
+    if ( config->max_altp2m && !hvm_altp2m_supported() )

This looks like it'll break the build on non-x86.

Quoted 12 lines
xen/include/public/domctl.h +1 -1
@@ -21,7 +21,7 @@
 #include "hvm/save.h"
 #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000016
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000017
 
 /*
  * NB. xen_domctl.domain is an IN/OUT parameter for this operation.

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 82

Quoted 6 lines
xen/include/public/domctl.h +1 -0
@@ -77,6 +77,7 @@ struct xen_domctl_createdomain {
      */
     uint32_t max_vcpus;
     uint32_t max_evtchn_port;
+    uint32_t max_altp2m;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;

Both this and ...

xen/include/xen/sched.h struct domain · line 607

Quoted 9 lines
xen/include/xen/sched.h +2 -0
@@ -602,6 +602,8 @@ struct domain
         unsigned int guest_request_sync          : 1;
     } monitor;
 
+    unsigned int max_altp2m; /* Maximum number of altp2m tables */
+
     unsigned int vmtrace_size; /* Buffer size in bytes, or 0 to disable. */

... this suggest you're confident other architectures will also want to support altp2m. Yet nothing like that is said in the description. In the absence of that common code should not require touching (much).

Jan

#9 lore raw

Re: [PATCH v2 4/7] x86: Make the maximum number of altp2m views configurable

Quoted 16 lines · Jan Beulich

xen/arch/x86/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 692

Quoted 12 lines
xen/arch/x86/domain.c +5 -0
@@ -685,6 +685,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->max_altp2m > MAX_EPTP )
+    {
+        dprintk(XENLOG_INFO, "max_altp2m must be <= %u\n", MAX_EPTP);
+        return -EINVAL;
+    }

... using MAX_EPTP here feels like a layering violation to me. Imo there want to be separate constants, tied together with a suitably placed BUILD_BUG_ON().

Quoted 5 lines

Furthermore comparisons like this (there are further ones elsewhere) suggest there is a (continued) naming issue: A max_ or MAX_ prefix ought to name a "maximum valid value", not "number of permitted values". This is not a request to alter MAX_EPTP, but one to make sure the new struct fields really have matching names and purposes.

Do you have any proposals? I was considering nr_altp2m. Another question is what it should be named in xl.cfg - also nr_altp2m? I was too hesitant to name it like that, since there aren't any nr_* fields currently.

Quoted 6 lines
Quoted 6 lines
xen/arch/x86/include/asm/domain.h +0 -0
@@ -258,11 +258,10 @@ struct paging_vcpu {
     struct shadow_vcpu shadow;
 };
 
Quoted 7 lines

xen/arch/x86/include/asm/domain.h struct paging_vcpu { · line 262

Quoted 2 lines
xen/arch/x86/include/asm/domain.h +2 -0
+#define INVALID_ALTP2M  0xffff
+#define MAX_EPTP        ((unsigned int)(PAGE_SIZE / sizeof(uint64_t)))

Aiui you add this cast because of the various min() uses. However, besides wanting to avoid such casts (or in fact any, whenever possible), I don't see why you need to retain all those min(): You bound d->max_altp2m against MAX_EPTP during domain creation anyway.

Fair. I considered removing the min()s altogether. I left them in place because I was too paranoid.

Quoted 8 lines
Quoted 8 lines
xen/arch/x86/include/asm/domain.h +1 -0
@@ -28,8 +34,13 @@ altp2m_vcpu_initialise(struct vcpu *v)
 void
 altp2m_vcpu_destroy(struct vcpu *v)
 {
+    struct domain *d = v->domain;
     struct p2m_domain *p2m;
 
Quoted 6 lines

xen/arch/x86/include/asm/domain.h altp2m_vcpu_initialise(struct vcpu *v) · line 42

Quoted 3 lines
xen/arch/x86/include/asm/domain.h +3 -0
+    /* Skip destruction if no altp2m was used. */
+    if ( d->max_altp2m == 0 )
+        return;

... this one doesn't look quite right: Even if altp2m-s were initialized, none may have been used (yet).

Fair. Bad choice of words.

Quoted 7 lines
Quoted 7 lines
xen/include/xen/sched.h +0 -0
@@ -602,6 +602,8 @@ struct domain
         unsigned int guest_request_sync          : 1;
     } monitor;
 
Quoted 7 lines

xen/include/xen/sched.h struct domain · line 607

Quoted 3 lines
xen/include/xen/sched.h +2 -0
+    unsigned int max_altp2m; /* Maximum number of altp2m tables */
+
     unsigned int vmtrace_size; /* Buffer size in bytes, or 0 to disable. */

... this suggest you're confident other architectures will also want to support altp2m. Yet nothing like that is said in the description. In the absence of that common code should not require touching (much).

Depends on the definition of "want". altp2m patches for arm/aarch64 were sent to Xen some 6 years ago but were unfortunately rejected. I would very much welcome them.

I have added the field to domain instead of arch_domain simply because it is not an arch-bound feature - similarly to vmtrace below, which also doesn't have an equivalent implementation on other archs than x86 (yet).

As far as other comments/nits go - understood.

P.

#10 lore raw

Re: [PATCH v2 4/7] x86: Make the maximum number of altp2m views configurable

Quoted 22 lines · Petr Beneš
Quoted 15 lines · Jan Beulich

xen/arch/x86/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 692

Quoted 11 lines
xen/arch/x86/domain.c +5 -0
@@ -685,6 +685,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->max_altp2m > MAX_EPTP )
+    {
+        dprintk(XENLOG_INFO, "max_altp2m must be <= %u\n", MAX_EPTP);
+        return -EINVAL;
+    }

... using MAX_EPTP here feels like a layering violation to me. Imo there want to be separate constants, tied together with a suitably placed BUILD_BUG_ON().

Quoted 5 lines

Furthermore comparisons like this (there are further ones elsewhere) suggest there is a (continued) naming issue: A max_ or MAX_ prefix ought to name a "maximum valid value", not "number of permitted values". This is not a request to alter MAX_EPTP, but one to make sure the new struct fields really have matching names and purposes.

Quoted 4 lines

Do you have any proposals? I was considering nr_altp2m. Another question is what it should be named in xl.cfg - also nr_altp2m? I was too hesitant to name it like that, since there aren't any nr_* fields currently.

Internally nr_ or num_ are going to be fine. For xl whether either of those would be, or maybe altp2ms= (along the lines of e.g. vcpus=), or altp2m_count, or yet something else I simply don't know. That'll be the maintainers there to help with.

Jan

#11 lore raw

PATCH v2 5/7 tools/libxl: Activate the max_altp2m feature

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

This commit activates the previously introduced max_altp2m parameter, establishing the connection between libxl and Xen.

---
Changed since v1:
* This is a new commit in the series

 tools/libs/light/libxl_create.c | 1 +
 1 file changed, 1 insertion(+)
tools/libs/light/libxl_create.c +1 -0
index 801f704a02..6ccc1fa158 100644
@@ -653,6 +653,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
             .ssidref = info->ssidref,
             .max_vcpus = b_info->max_vcpus,
             .max_evtchn_port = b_info->event_channels,
+            .max_altp2m = b_info->max_altp2m,
             .max_grant_frames = b_info->max_grant_frames,
             .max_maptrack_frames = b_info->max_maptrack_frames,
             .grant_opts = XEN_DOMCTL_GRANT_version(b_info->max_grant_version),

--
2.34.1

#12 lore raw

PATCH v2 6/7 tools/ocaml: Add max_altp2m parameter

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

Allow developers using the OCaml bindings to set the max_altp2m parameter.

---
Changed since v1:
* Moved this commit AFTER Xen changes (where xen_domctl_createdomain::max_altp2m field is introduced) to avoid breaking the build

 tools/ocaml/libs/xc/xenctrl.ml      |  1 +
 tools/ocaml/libs/xc/xenctrl.mli     |  1 +
 tools/ocaml/libs/xc/xenctrl_stubs.c | 17 ++++++++++-------
 3 files changed, 12 insertions(+), 7 deletions(-)
tools/ocaml/libs/xc/xenctrl.ml +1 -0
index 55923857ec..ed851bb071 100644
@@ -82,6 +82,7 @@ type domctl_create_config =
     iommu_opts: domain_create_iommu_opts list;
     max_vcpus: int;
     max_evtchn_port: int;
+    max_altp2m: int;
     max_grant_frames: int;
     max_maptrack_frames: int;
     max_grant_version: int;
tools/ocaml/libs/xc/xenctrl.mli +1 -0
index 9b4b45db3a..971b269d85 100644
@@ -74,6 +74,7 @@ type domctl_create_config = {
   iommu_opts: domain_create_iommu_opts list;
   max_vcpus: int;
   max_evtchn_port: int;
+  max_altp2m: int;
   max_grant_frames: int;
   max_maptrack_frames: int;
   max_grant_version: int;
tools/ocaml/libs/xc/xenctrl_stubs.c +10 -7
index 2b6d3c09df..0b70cc9b08 100644
@@ -207,12 +207,13 @@ CAMLprim value stub_xc_domain_create(value xch_val, value wanted_domid, value co
 #define VAL_IOMMU_OPTS          Field(config, 3)
 #define VAL_MAX_VCPUS           Field(config, 4)
 #define VAL_MAX_EVTCHN_PORT     Field(config, 5)
-#define VAL_MAX_GRANT_FRAMES    Field(config, 6)
-#define VAL_MAX_MAPTRACK_FRAMES Field(config, 7)
-#define VAL_MAX_GRANT_VERSION   Field(config, 8)
-#define VAL_VMTRACE_BUF_KB      Field(config, 9)
-#define VAL_CPUPOOL_ID          Field(config, 10)
-#define VAL_ARCH                Field(config, 11)
+#define VAL_MAX_ALTP2M          Field(config, 6)
+#define VAL_MAX_GRANT_FRAMES    Field(config, 7)
+#define VAL_MAX_MAPTRACK_FRAMES Field(config, 8)
+#define VAL_MAX_GRANT_VERSION   Field(config, 9)
+#define VAL_VMTRACE_BUF_KB      Field(config, 10)
+#define VAL_CPUPOOL_ID          Field(config, 11)
+#define VAL_ARCH                Field(config, 12)
 
 	uint32_t domid = Int_val(wanted_domid);
 	uint64_t vmtrace_size = Int32_val(VAL_VMTRACE_BUF_KB);
@@ -226,6 +227,7 @@ CAMLprim value stub_xc_domain_create(value xch_val, value wanted_domid, value co
 		.ssidref = Int32_val(VAL_SSIDREF),
 		.max_vcpus = Int_val(VAL_MAX_VCPUS),
 		.max_evtchn_port = Int_val(VAL_MAX_EVTCHN_PORT),
+		.max_altp2m = Int_val(VAL_MAX_ALTP2M),
 		.max_grant_frames = Int_val(VAL_MAX_GRANT_FRAMES),
 		.max_maptrack_frames = Int_val(VAL_MAX_MAPTRACK_FRAMES),
 		.grant_opts =
@@ -257,7 +259,7 @@ CAMLprim value stub_xc_domain_create(value xch_val, value wanted_domid, value co
 #if defined(__i386__) || defined(__x86_64__)
 
 		/* Quick & dirty check for ABI changes. */
-		BUILD_BUG_ON(sizeof(cfg) != 64);
+		BUILD_BUG_ON(sizeof(cfg) != 68);
 
         /* Mnemonics for the named fields inside xen_x86_arch_domainconfig */
 #define VAL_EMUL_FLAGS          Field(arch_domconfig, 0)
@@ -291,6 +293,7 @@ CAMLprim value stub_xc_domain_create(value xch_val, value wanted_domid, value co
 #undef VAL_MAX_GRANT_VERSION
 #undef VAL_MAX_MAPTRACK_FRAMES
 #undef VAL_MAX_GRANT_FRAMES
+#undef VAL_MAX_ALTP2M
 #undef VAL_MAX_EVTCHN_PORT
 #undef VAL_MAX_VCPUS
 #undef VAL_IOMMU_OPTS

--
2.34.1

#13 lore raw

PATCH v2 7/7 x86/hap: Increase the number of initial mempool_size to 1024 pages

In reply to Petr Beneš

From: Petr Beneš <[email protected]>

This change anticipates scenarios where max_altp2m is set to its maximum supported value (i.e., 512), ensuring sufficient memory is allocated upfront to accommodate all altp2m tables without initialization failure.

The necessity for this increase arises from the current mechanism where altp2m tables are allocated at initialization, requiring one page from the mempool for each altp2m view.

---
 tools/tests/paging-mempool/test-paging-mempool.c | 2 +-
 xen/arch/x86/mm/hap/hap.c                        | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/tests/paging-mempool/test-paging-mempool.c +1 -1
index 1ebc13455a..91b06fa0cf 100644
@@ -35,7 +35,7 @@ static struct xen_domctl_createdomain create = {
 
 static uint64_t default_mempool_size_bytes =
 #if defined(__x86_64__) || defined(__i386__)
-    256 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */
+    1024 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */
 #elif defined (__arm__) || defined(__aarch64__)
     16 << 12;
 #endif
xen/arch/x86/mm/hap/hap.c +1 -1
index 7aff5fa664..fab7e256a4 100644
@@ -468,7 +468,7 @@ int hap_enable(struct domain *d, u32 mode)
     if ( old_pages == 0 )
     {
         paging_lock(d);
-        rv = hap_set_allocation(d, 256, NULL);
+        rv = hap_set_allocation(d, 1024, NULL);
         if ( rv != 0 )
         {
             hap_set_allocation(d, 0, NULL);
Signature · 1 line
2.34.1
#14 lore raw

Re: [PATCH v2 7/7] x86/hap: Increase the number of initial mempool_size to 1024 pages

Quoted 5 lines · Petr Beneš

From: Petr Beneš <[email protected]>

This change anticipates scenarios where max_altp2m is set to its maximum supported value (i.e., 512), ensuring sufficient memory is allocated upfront to accommodate all altp2m tables without initialization failure.

And guests with fewer or even no altp2m-s still need the same bump? You know the number of altp2m-s upon domain creation, so why bump by any more than what's strictly needed for that?

Quoted 3 lines

The necessity for this increase arises from the current mechanism where altp2m tables are allocated at initialization, requiring one page from the mempool for each altp2m view.

So that's the p2m_alloc_table() out of hap_enable()? If you're permitting up to 512 altp2m-s, I think it needs considering to not waste up to 2Mb without knowing how many of the altp2m-s are actually going to be used. How complicate on-demand allocation would be I can't tell though, I have to admit.

tools/tests/paging-mempool/test-paging-mempool.c static struct xen_domctl_createdomain create = { · line 38

Quoted 8 lines
tools/tests/paging-mempool/test-paging-mempool.c +1 -1
@@ -35,7 +35,7 @@ static struct xen_domctl_createdomain create = {
 
 static uint64_t default_mempool_size_bytes =
 #if defined(__x86_64__) || defined(__i386__)
-    256 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */
+    1024 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */

I also can't derive from the description why we'd need to go from 256 to 1024 here and ...

xen/arch/x86/mm/hap/hap.c int hap_enable(struct domain *d, u32 mode) · line 471

Quoted 8 lines
xen/arch/x86/mm/hap/hap.c +1 -1
@@ -468,7 +468,7 @@ int hap_enable(struct domain *d, u32 mode)
     if ( old_pages == 0 )
     {
         paging_lock(d);
-        rv = hap_set_allocation(d, 256, NULL);
+        rv = hap_set_allocation(d, 1024, NULL);

... here. You talk of (up to) 512 pages there only.

Also isn't there at least one more place where the tool stack (libxl I think) would need changing, where Dom0 ballooning needs are calculated? And/or doesn't the pool size have a default calculation in the tool stack, too?

Jan

#15 lore raw

Re: [PATCH v2 7/7] x86/hap: Increase the number of initial mempool_size to 1024 pages

Quoted 11 lines · Jan Beulich
Quoted 5 lines · Petr Beneš

From: Petr Beneš <[email protected]>

This change anticipates scenarios where max_altp2m is set to its maximum supported value (i.e., 512), ensuring sufficient memory is allocated upfront to accommodate all altp2m tables without initialization failure.

And guests with fewer or even no altp2m-s still need the same bump? You know the number of altp2m-s upon domain creation, so why bump by any more than what's strictly needed for that?

I have to admit I've considered computing the value which goes to hap_set_allocation
by simply adding 256 + max_altp2m, but that felt so arbitrary - the 256 value itself
feels arbitrary, as I haven't found any reasoning for it anywhere.

I have also tried to make code changes to make the initial allocation size configurable
via libxl (possibly reusing the shadow_memkb) - which seemed to me like the "correct"
solution, but those changes were more complicated than I had anticipated and I would
definitely not make it till the 4.19 deadline.

Question is, what to do now? Should I change it to 256 + max_altp2m?

Quoted 7 lines
Quoted 3 lines

The necessity for this increase arises from the current mechanism where altp2m tables are allocated at initialization, requiring one page from the mempool for each altp2m view.

So that's the p2m_alloc_table() out of hap_enable()? If you're permitting up to 512 altp2m-s, I think it needs considering to not waste up to 2Mb without knowing how many of the altp2m-s are actually going to be used.

Yes and I ultimately agree.

Quoted 2 lines

How complicate on-demand allocation would be I can't tell though, I have to admit.

That's also a fix I've been trying to work on - to make whole altp2m allocations on-demand. Unfortunately, I didn't make it in time.

Quoted 11 lines

tools/tests/paging-mempool/test-paging-mempool.c static struct xen_domctl_createdomain create = { · line 38

Quoted 8 lines
tools/tests/paging-mempool/test-paging-mempool.c +1 -1
@@ -35,7 +35,7 @@ static struct xen_domctl_createdomain create = {
 
 static uint64_t default_mempool_size_bytes =
 #if defined(__x86_64__) || defined(__i386__)
-    256 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */
+    1024 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */

I also can't derive from the description why we'd need to go from 256 to 1024 here and ...

It's explained in the code few lines below:

/*
* Check that the domain has the expected default allocation size. This
* will fail if the logic in Xen is altered without an equivalent
* adjustment here.
*/

I have verified that the default_mempool_size_bytes must reflect the number of pages in the initial hap_set_allocation() call.

Is it something I should include in the commit message, too?

Quoted 11 lines

xen/arch/x86/mm/hap/hap.c int hap_enable(struct domain *d, u32 mode) · line 471

Quoted 8 lines
xen/arch/x86/mm/hap/hap.c +1 -1
@@ -468,7 +468,7 @@ int hap_enable(struct domain *d, u32 mode)
     if ( old_pages == 0 )
     {
         paging_lock(d);
-        rv = hap_set_allocation(d, 256, NULL);
+        rv = hap_set_allocation(d, 1024, NULL);

... here. You talk of (up to) 512 pages there only.

Quoted 4 lines

Also isn't there at least one more place where the tool stack (libxl I think) would need changing, where Dom0 ballooning needs are calculated? And/or doesn't the pool size have a default calculation in the tool stack, too?

I have found places in libxl where the mempool_size is calculated, but that mempool
size is then set AFTER the domain is created via xc_set_paging_mempool_size.

In my opinion it doesn't necessarily require change, since it's expected by the user
to manually set it via shadow_memkb. The only current problem is (which this commit is trying to fix) that setting shadow_memkb doesn't help when max_altp2m > (256 - 1 + vcpus + MAX_NESTEDP2M), since the initial mempool size is hardcoded.

I didn't find any other places that would require reflecting the current "256" value.

P.

#16 lore raw

Re: [PATCH v2 7/7] x86/hap: Increase the number of initial mempool_size to 1024 pages

Quoted 27 lines · Petr Beneš
Quoted 11 lines · Jan Beulich
Quoted 5 lines · Petr Beneš

From: Petr Beneš <[email protected]>

This change anticipates scenarios where max_altp2m is set to its maximum supported value (i.e., 512), ensuring sufficient memory is allocated upfront to accommodate all altp2m tables without initialization failure.

And guests with fewer or even no altp2m-s still need the same bump? You know the number of altp2m-s upon domain creation, so why bump by any more than what's strictly needed for that?

I have to admit I've considered computing the value which goes to hap_set_allocation
by simply adding 256 + max_altp2m, but that felt so arbitrary - the 256 value itself
feels arbitrary, as I haven't found any reasoning for it anywhere.

I have also tried to make code changes to make the initial allocation size configurable
via libxl (possibly reusing the shadow_memkb) - which seemed to me like the "correct"
solution, but those changes were more complicated than I had anticipated and I would
definitely not make it till the 4.19 deadline.

Quoted 1 line

Question is, what to do now? Should I change it to 256 + max_altp2m?

Counter question: Is accounting for just the root page table really enough? Meaning to say: I'm not convinced that minimum would really be appropriate for altp2m use even before your changes. You growing the number of root page tables _always_ required just makes things worse even without considering how (many) altp2m-s are then going to be used. Such an issue, if I'm right with this, would imo want addressing up front, in a separate patch.

You may also want to take a look at the shadow side of things, even if for altp2m shadow mode may not be relevant. Things like sh_min_allocation() and shadow_min_acceptable_pages() may well need proper counterparts in HAP now.

Quoted 20 lines
Quoted 11 lines

tools/tests/paging-mempool/test-paging-mempool.c static struct xen_domctl_createdomain create = { · line 38

Quoted 8 lines
tools/tests/paging-mempool/test-paging-mempool.c +1 -1
@@ -35,7 +35,7 @@ static struct xen_domctl_createdomain create = {
 
 static uint64_t default_mempool_size_bytes =
 #if defined(__x86_64__) || defined(__i386__)
-    256 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */
+    1024 << 12; /* Only x86 HAP for now.  x86 Shadow needs more work. */

I also can't derive from the description why we'd need to go from 256 to 1024 here and ...

It's explained in the code few lines below:

/*
* Check that the domain has the expected default allocation size. This
* will fail if the logic in Xen is altered without an equivalent
* adjustment here.
*/

Quoted 4 lines

I have verified that the default_mempool_size_bytes must reflect the number of pages in the initial hap_set_allocation() call.

Is it something I should include in the commit message, too?

Well, yes and no. My question wasn't about the "why", but ...

Quoted 10 lines
Quoted 10 lines

xen/arch/x86/mm/hap/hap.c int hap_enable(struct domain *d, u32 mode) · line 471

Quoted 8 lines
xen/arch/x86/mm/hap/hap.c +1 -1
@@ -468,7 +468,7 @@ int hap_enable(struct domain *d, u32 mode)
     if ( old_pages == 0 )
     {
         paging_lock(d);
-        rv = hap_set_allocation(d, 256, NULL);
+        rv = hap_set_allocation(d, 1024, NULL);

... here. You talk of (up to) 512 pages there only.

... about the "by how many".

Quoted 9 lines
Quoted 4 lines

Also isn't there at least one more place where the tool stack (libxl I think) would need changing, where Dom0 ballooning needs are calculated? And/or doesn't the pool size have a default calculation in the tool stack, too?

I have found places in libxl where the mempool_size is calculated, but that mempool
size is then set AFTER the domain is created via xc_set_paging_mempool_size.

Quoted 6 lines

In my opinion it doesn't necessarily require change, since it's expected by the user
to manually set it via shadow_memkb. The only current problem is (which this commit is trying to fix) that setting shadow_memkb doesn't help when max_altp2m > (256 - 1 + vcpus + MAX_NESTEDP2M), since the initial mempool size is hardcoded.

Wait - are you saying the guest config value isn't respected in certain cases? That would be another thing wanting to be fixed separately, up front.

Jan

#17 lore raw

Re: [PATCH v2 7/7] x86/hap: Increase the number of initial mempool_size to 1024 pages

Quoted 31 lines · Jan Beulich
Quoted 27 lines · Petr Beneš
Quoted 11 lines · Jan Beulich
Quoted 5 lines · Petr Beneš

From: Petr Beneš <[email protected]>

This change anticipates scenarios where max_altp2m is set to its maximum supported value (i.e., 512), ensuring sufficient memory is allocated upfront to accommodate all altp2m tables without initialization failure.

And guests with fewer or even no altp2m-s still need the same bump? You know the number of altp2m-s upon domain creation, so why bump by any more than what's strictly needed for that?

I have to admit I've considered computing the value which goes to hap_set_allocation
by simply adding 256 + max_altp2m, but that felt so arbitrary - the 256 value itself
feels arbitrary, as I haven't found any reasoning for it anywhere.

I have also tried to make code changes to make the initial allocation size configurable
via libxl (possibly reusing the shadow_memkb) - which seemed to me like the "correct"
solution, but those changes were more complicated than I had anticipated and I would
definitely not make it till the 4.19 deadline.

Quoted 1 line

Question is, what to do now? Should I change it to 256 + max_altp2m?

Quoted 7 lines

Counter question: Is accounting for just the root page table really enough? Meaning to say: I'm not convinced that minimum would really be appropriate for altp2m use even before your changes. You growing the number of root page tables _always_ required just makes things worse even without considering how (many) altp2m-s are then going to be used. Such an issue, if I'm right with this, would imo want addressing up front, in a separate patch.

It is enough - at least based on my experiments. I'll try to explain it below.

Quoted 16 lines
Quoted 9 lines
Quoted 4 lines

Also isn't there at least one more place where the tool stack (libxl I think) would need changing, where Dom0 ballooning needs are calculated? And/or doesn't the pool size have a default calculation in the tool stack, too?

I have found places in libxl where the mempool_size is calculated, but that mempool
size is then set AFTER the domain is created via xc_set_paging_mempool_size.

Quoted 6 lines

In my opinion it doesn't necessarily require change, since it's expected by the user
to manually set it via shadow_memkb. The only current problem is (which this commit is trying to fix) that setting shadow_memkb doesn't help when max_altp2m > (256 - 1 + vcpus + MAX_NESTEDP2M), since the initial mempool size is hardcoded.

Quoted 3 lines

Wait - are you saying the guest config value isn't respected in certain cases? That would be another thing wanting to be fixed separately, up front.

The xc_set_paging_mempool_size is still called within domain_create. So the value of shadow_memkb is respected before any of the guest code is run. My point was that shadow_memkb isn't respected here:

Quoted 8 lines
Quoted 8 lines

xen/arch/x86/mm/hap/hap.c int hap_enable(struct domain *d, u32 mode) · line 471

Quoted 8 lines
xen/arch/x86/mm/hap/hap.c +1 -1
@@ -468,7 +468,7 @@ int hap_enable(struct domain *d, u32 mode)
     if ( old_pages == 0 )
     {
         paging_lock(d);
-        rv = hap_set_allocation(d, 256, NULL);
+        rv = hap_set_allocation(d, 1024, NULL);

This code (+ the root altp2ms allocation) is executed before the libxl sends the shadow_memkb.

In another words, the sequence is following:

libxl:
------

do_domain_create

    initiate_domain_create
        libxl__domain_make
            xc_domain_create // MAX_ALTP2M is passed here
                             // and hap_enable is called

        dcs->bl.callback = domcreate_bootloader_done

domcreate_bootloader_done
libxl__domain_build
libxl__build_pre
libxl__arch_domain_create
libxl__domain_set_paging_mempool_size
xc_set_paging_mempool_size(shadow_mem)

xen (xc_domain_create cont.):
-----------------------------
domain_create

    arch_domain_create
        hvm_domain_initialise
            paging_enable
                hap_enable
                    // note that we shadow_mem (from config) hasn't been
                    // provided yet
                    hap_set_allocation(d, 1024, NULL);
                    p2m_alloc_table(p2m_get_hostp2m(d));
                    p2m_alloc_table(d->arch.nested_p2m[i..MAX_NESTEDP2M]);
                    p2m_alloc_table(d->arch.altp2m_p2m[i..MAX_ALTP2M]);

(I hope the email will preserve the spacing...)

Based on this, I would argue that shadow_memkb should be also part of xc_domain_create/xen_domctl_createdomain. Which is why I've said in previous email:

Quoted 7 lines
Quoted 7 lines

I have also tried to make code changes to make the initial allocation size configurable
via libxl (possibly reusing the shadow_memkb) - which seemed to me like the "correct"
solution, but those changes were more complicated than I had anticipated and I would
definitely not make it till the 4.19 deadline.

P.