Skip to contentSkip to conversation

xenguest optimisations

PATCH In review Frediano Ziglio 4 participants 26 messages Aug 10 – Aug 17

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v10 cover xenguest optimisations

Reduce number of allocations sending memory state.

Implement and use new Xen and Linux kernel ABI to copy foreign memory. This new ABI allows to replace the expensive map/copy/unmap sequence with a single call.

Changes since v1:
- add commit to cache up to 4 pages in hypercall;
- add other 2 commits reducing chunks passed to write/writev.

Changes since v2:
- update patches commit prefixes;
- add other 2 optisations.

Changes since v3:
- address some comments;
- add patches for foreign copy optimisation.

Changes since v4:
- added Reviewed-by;
- improved commit messages;
- other minor fixes, see individual commits.

Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc;
- do not overwrite errno if xenforeignmemory_map fails;
- lot of changes to "implement new foreign copy hypercall", see specific commit.

Changes since v6:
- removed merged patch;
- keep only optimization commits for now;
- improve comments;
- merged "fill directly iov structure collapsing them" and moved it;
- split "allocate various migration arrays just once";
- add a commit for memory checks using Valgrind.

Changes since v7:
- removed merged commits;
- minor style fixes.

Changes since v8:
- added Reviewed-by;
- remove useless check;
- remove useless memset;
- initialize variables while declaring them.

Changes since v9:
- add back commits for foreign copy;
- rework page permissions check;
- do not limit domain for new hypercall;
- add back some memory check using Valgrind and sanitizers;
- fixed compatibility for ARM;
- minor fixes.

Edwin Török (3):
libs/call: cache up to 4 pages in hypercall bounce buffers libs/guest: allocate various migration arrays just once libs/guest: use foreign copy API during migration

Frediano Ziglio (6):
libs/guest: move batch_pfns into a separate structure libs/guest: use Valgrind or sanitizers to detect various buffer
overflows
libs/guest: add xg_foreignmemory_copy_{from,to}
xen: implement new foreign copy hypercall
privcmd: Add definition for new Linux privcmd to access new Xen
hypercall
libs/guest: use new hypercall if available

 tools/config.h.in                     |   6 ++
 tools/configure                       |  12 +++
 tools/configure.ac                    |   3 +-
 tools/include/xen-sys/Linux/privcmd.h |  10 ++
 tools/libs/call/buffer.c              |  34 ++++--
 tools/libs/call/core.c                |   3 +-
 tools/libs/call/private.h             |   8 +-
 tools/libs/ctrl/xc_private.h          |  61 ++++++++++-
 tools/libs/guest/xg_sr_common.c       |  86 +++++++++++++++
 tools/libs/guest/xg_sr_common.h       |  25 ++++-
 tools/libs/guest/xg_sr_restore.c      |  78 +++++++-------
 tools/libs/guest/xg_sr_save.c         | 134 +++++++++++------------
 xen/common/memory.c                   | 149 ++++++++++++++++++++++++++
 xen/include/public/memory.h           |  45 +++++++-
 xen/include/xsm/dummy.h               |  14 +++
 xen/include/xsm/hooks.h               |   2 +
 xen/xsm/flask/hooks.c                 |  10 ++
 17 files changed, 552 insertions(+), 128 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v10 1/10 libs/call: cache up to 4 pages in hypercall bounce buffers

From: Edwin Török <[email protected]>

During migration there are a lot of mmap/munmap calls, because xc_get_pfn_type_batch() exceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.

munmap() is slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.

Eventually it'd be good if the memory pool from xmalloc_tlsf.c was reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).

---
Changes since v2:
- change prefix in subject.

Changes since v4:
- fix off-by-one bug.

Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc.

Changes since v6:
- align changes made to cache_alloc to cache_free.

Changes since v7:
- use "unsigned int" instead of "unsigned".

Changes since v8:
- added Reviewed-by.

---
 tools/libs/call/buffer.c  | 34 +++++++++++++++++++++++-----------
 tools/libs/call/core.c    |  3 ++-
 tools/libs/call/private.h |  8 +++++---
 3 files changed, 30 insertions(+), 15 deletions(-)
tools/libs/call/buffer.c +23 -11
index 155e4f9d43..b7d00185c4 100644
@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)
 {
     void *p = NULL;
 
+    if ( nr_pages == 0 )
+        return NULL;
+
     cache_lock(xcall);
 
     xcall->buffer_total_allocations++;
@@ -56,13 +59,13 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)
     if ( xcall->buffer_current_allocations > xcall->buffer_maximum_allocations )
         xcall->buffer_maximum_allocations = xcall->buffer_current_allocations;
 
-    if ( nr_pages > 1 )
+    if ( nr_pages > ARRAY_SIZE(xcall->buffer_cache) )
     {
         xcall->buffer_cache_toobig++;
     }
-    else if ( xcall->buffer_cache_nr > 0 )
+    else if ( xcall->buffer_cache_nr[nr_pages-1] > 0 )
     {
-        p = xcall->buffer_cache[--xcall->buffer_cache_nr];
+        p = xcall->buffer_cache[nr_pages-1][--xcall->buffer_cache_nr[nr_pages-1]];
         xcall->buffer_cache_hits++;
     }
     else
@@ -79,15 +82,18 @@ static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages)
 {
     int rc = 0;
 
+    if ( nr_pages == 0 )
+        return 0;
+
     cache_lock(xcall);
 
     xcall->buffer_total_releases++;
     xcall->buffer_current_allocations--;
 
-    if ( nr_pages == 1 &&
-         xcall->buffer_cache_nr < BUFFER_CACHE_SIZE )
+    if ( nr_pages && nr_pages <= ARRAY_SIZE(xcall->buffer_cache) &&
+         xcall->buffer_cache_nr[nr_pages-1] < BUFFER_CACHE_SIZE )
     {
-        xcall->buffer_cache[xcall->buffer_cache_nr++] = p;
+        xcall->buffer_cache[nr_pages-1][xcall->buffer_cache_nr[nr_pages-1]++] = p;
         rc = 1;
     }
 
@@ -108,17 +114,23 @@ void buffer_release_cache(xencall_handle *xcall)
     DBGPRINTF("current allocations:%d maximum allocations:%d",
               xcall->buffer_current_allocations,
               xcall->buffer_maximum_allocations);
-    DBGPRINTF("cache current size:%d",
-              xcall->buffer_cache_nr);
+    for ( unsigned int i = 0; i < ARRAY_SIZE(xcall->buffer_cache_nr); ++i )
+    {
+        DBGPRINTF("cache current size[%u pages]:%d", i+1,
+                xcall->buffer_cache_nr[i]);
+    }
     DBGPRINTF("cache hits:%d misses:%d toobig:%d",
               xcall->buffer_cache_hits,
               xcall->buffer_cache_misses,
               xcall->buffer_cache_toobig);
 
-    while ( xcall->buffer_cache_nr > 0 )
+    for ( unsigned int i = 0; i < ARRAY_SIZE(xcall->buffer_cache_nr); ++i )
     {
-        p = xcall->buffer_cache[--xcall->buffer_cache_nr];
-        osdep_free_pages(xcall, p, 1);
+        while ( xcall->buffer_cache_nr[i] > 0 )
+        {
+            p = xcall->buffer_cache[i][--xcall->buffer_cache_nr[i]];
+            osdep_free_pages(xcall, p, i + 1);
+        }
     }
 
     cache_unlock(xcall);
tools/libs/call/core.c +2 -1
index 02c4f8e1ae..dd8877c1a0 100644
@@ -14,6 +14,7 @@
  */
 
 #include <stdlib.h>
+#include <string.h>
 
 #include "private.h"
 
@@ -44,7 +45,7 @@ xencall_handle *xencall_open(xentoollog_logger *logger, unsigned open_flags)
     xentoolcore__register_active_handle(&xcall->tc_ah);
 
     xcall->flags = open_flags;
-    xcall->buffer_cache_nr = 0;
+    memset(xcall->buffer_cache_nr, 0, sizeof(xcall->buffer_cache_nr));
 
     xcall->buffer_total_allocations = 0;
     xcall->buffer_total_releases = 0;
tools/libs/call/private.h +5 -3
index 9c3aa432ef..8e6a208975 100644
@@ -31,13 +31,15 @@ struct xencall_handle {
     Xentoolcore__Active_Handle tc_ah;
 
     /*
-     * A simple cache of unused, single page, hypercall buffers
+     * A simple cache of unused, small, hypercall buffers
+     * buffer_cache[i]'s size is (i+1) pages
      *
      * Protected by a global lock.
      */
 #define BUFFER_CACHE_SIZE 4
-    int buffer_cache_nr;
-    void *buffer_cache[BUFFER_CACHE_SIZE];
+#define BUFFER_CACHE_NRPAGES 4
+    int buffer_cache_nr[BUFFER_CACHE_NRPAGES];
+    void *buffer_cache[BUFFER_CACHE_NRPAGES][BUFFER_CACHE_SIZE];
 
     /*
      * Hypercall buffer statistics. All protected by the global
Signature · 1 line
2.43.0
#2 lore raw

PATCH v10 2/10 libs/guest: move batch_pfns into a separate structure

In reply to Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

---
Changes since v6:
- split from "libs/guest: allocate various migration arrays just once".

Changes since v7:
- initialize "batch_pfns" on declaration.

Changes since v8:
- remove useless check;
- added Reviewed-by.

---
 tools/libs/guest/xg_sr_common.h |  5 ++++-
 tools/libs/guest/xg_sr_save.c   | 28 ++++++++++++++--------------
 2 files changed, 18 insertions(+), 15 deletions(-)
tools/libs/guest/xg_sr_common.h +4 -1
index f1573aefcb..7574c9f5b6 100644
@@ -239,11 +239,14 @@ struct xc_sr_context
 
             struct precopy_stats stats;
 
-            xen_pfn_t *batch_pfns;
             unsigned int nr_batch_pfns;
             unsigned long *deferred_pages;
             unsigned long nr_deferred_pages;
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
+            struct xc_sr_context_save_buffers
+            {
+                xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+            } *buffers;
         } save;
 
         struct /* Restore data. */
tools/libs/guest/xg_sr_save.c +14 -14
index 84fdbe4140..22348db445 100644
@@ -75,7 +75,7 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)
 
 /*
  * Writes a batch of memory as a PAGE_DATA record into the stream.  The batch
- * is constructed in ctx->save.batch_pfns.
+ * is constructed in ctx->save.buffers->batch_pfns.
  *
  * This function:
  * - gets the types for each pfn in the batch.
@@ -95,6 +95,7 @@ static int write_batch(struct xc_sr_context *ctx)
     void *page, *orig_page;
     uint64_t *rec_pfns = NULL;
     struct iovec *iov = NULL; int iovcnt = 0;
+    xen_pfn_t *const batch_pfns = ctx->save.buffers->batch_pfns;
     struct {
         struct xc_sr_rhdr rec;
         struct xc_sr_rec_page_data_header page_data;
@@ -110,6 +111,7 @@ static int write_batch(struct xc_sr_context *ctx)
     };
 
     assert(nr_pfns != 0);
+    assert(nr_pfns <= MAX_BATCH_SIZE);
 
     /* Mfns of the batch pfns. */
     mfns = malloc(nr_pfns * sizeof(*mfns));
@@ -141,13 +143,12 @@ static int write_batch(struct xc_sr_context *ctx)
 
     for ( i = 0; i < nr_pfns; ++i )
     {
-        types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx,
-                                                      ctx->save.batch_pfns[i]);
+        types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx, batch_pfns[i]);
 
         /* Likely a ballooned page. */
         if ( mfns[i] == INVALID_MFN )
         {
-            set_bit(ctx->save.batch_pfns[i], ctx->save.deferred_pages);
+            set_bit(batch_pfns[i], ctx->save.deferred_pages);
             ++ctx->save.nr_deferred_pages;
         }
     }
@@ -193,7 +194,7 @@ static int write_batch(struct xc_sr_context *ctx)
             if ( errors[p] )
             {
                 ERROR("Mapping of pfn %#"PRIpfn" (mfn %#"PRIpfn") failed %d",
-                      ctx->save.batch_pfns[i], mfns[p], errors[p]);
+                      batch_pfns[i], mfns[p], errors[p]);
                 goto err;
             }
 
@@ -207,7 +208,7 @@ static int write_batch(struct xc_sr_context *ctx)
             {
                 if ( rc == -1 && errno == EAGAIN )
                 {
-                    set_bit(ctx->save.batch_pfns[i], ctx->save.deferred_pages);
+                    set_bit(batch_pfns[i], ctx->save.deferred_pages);
                     ++ctx->save.nr_deferred_pages;
                     types[i] = XEN_DOMCTL_PFINFO_XTAB;
                     --nr_pages;
@@ -235,7 +236,7 @@ static int write_batch(struct xc_sr_context *ctx)
     hdrs.rec.length += nr_pages * PAGE_SIZE;
 
     for ( i = 0; i < nr_pfns; ++i )
-        rec_pfns[i] = ((uint64_t)(types[i]) << 32) | ctx->save.batch_pfns[i];
+        rec_pfns[i] = ((uint64_t)(types[i]) << 32) | batch_pfns[i];
 
     if ( writev_exact(ctx->fd, iov, iovcnt) )
     {
@@ -274,9 +275,9 @@ static int flush_batch(struct xc_sr_context *ctx)
 
     if ( !rc )
     {
-        VALGRIND_MAKE_MEM_UNDEFINED(ctx->save.batch_pfns,
+        VALGRIND_MAKE_MEM_UNDEFINED(ctx->save.buffers->batch_pfns,
                                     MAX_BATCH_SIZE *
-                                    sizeof(*ctx->save.batch_pfns));
+                                    sizeof(*ctx->save.buffers->batch_pfns));
     }
 
     return rc;
@@ -293,7 +294,7 @@ static int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn)
         rc = flush_batch(ctx);
 
     if ( rc == 0 )
-        ctx->save.batch_pfns[ctx->save.nr_batch_pfns++] = pfn;
+        ctx->save.buffers->batch_pfns[ctx->save.nr_batch_pfns++] = pfn;
 
     return rc;
 }
@@ -784,11 +785,10 @@ static int setup(struct xc_sr_context *ctx)
 
     dirty_bitmap = xc_hypercall_buffer_alloc_pages(
         xch, dirty_bitmap, NRPAGES(bitmap_size(ctx->save.p2m_size)));
-    ctx->save.batch_pfns = malloc(MAX_BATCH_SIZE *
-                                  sizeof(*ctx->save.batch_pfns));
     ctx->save.deferred_pages = bitmap_alloc(ctx->save.p2m_size);
+    ctx->save.buffers = calloc(1, sizeof(*ctx->save.buffers));
 
-    if ( !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages )
+    if ( !ctx->save.buffers || !dirty_bitmap || !ctx->save.deferred_pages )
     {
         ERROR("Unable to allocate memory for dirty bitmaps, batch pfns and"
               " deferred pages");
@@ -819,7 +819,7 @@ static void cleanup(struct xc_sr_context *ctx)
     xc_hypercall_buffer_free_pages(xch, dirty_bitmap,
                                    NRPAGES(bitmap_size(ctx->save.p2m_size)));
     free(ctx->save.deferred_pages);
-    free(ctx->save.batch_pfns);
+    free(ctx->save.buffers);
 }
 
 /*
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH v10 2/10] libs/guest: move batch_pfns into a separate structure

Quoted 5 lines · Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

Coverity thinks this change has memory corruption.  I have to admit that I'm not completely sure why it's noticed now; possibly because now it can see the size of batch_pfns[] where previously it couldn't

** CID 1700057:       Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284           in add_to_batch()
_____________________________________________________________________________________________
*** CID 1700057:         Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284             in add_to_batch() 278         int rc = 0;
279
280         if ( ctx->save.nr_batch_pfns == MAX_BATCH_SIZE ) 281             rc = flush_batch(ctx);
282
283         if ( rc == 0 )

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

     CID 1700057:         Memory - corruptions  (OVERRUN)
     Overrunning array "(*ctx).save.buffers->batch_pfns" of 1024 8-byte elements at element index 1024 (byte offset 8199) using index "(*ctx).save.nr_batch_pfns++" (which evaluates to 1024).

284             ctx->save.buffers->batch_pfns[ctx->save.nr_batch_pfns++] = pfn; 285
286         return rc;
287     }
288
289     /*

~Andrew

#4 lore raw

Re: [PATCH v10 2/10] libs/guest: move batch_pfns into a separate structure

Quoted 10 lines · Andrew Cooper
Quoted 5 lines · Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

Coverity thinks this change has memory corruption.  I have to admit that I'm not completely sure why it's noticed now; possibly because now it can see the size of batch_pfns[] where previously it couldn't

I had looked into that too, and I'm puzzled that ...

Quoted 9 lines

** CID 1700057:       Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284           in add_to_batch()
_____________________________________________________________________________________________
*** CID 1700057:         Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284             in add_to_batch() 278         int rc = 0;
279
280         if ( ctx->save.nr_batch_pfns == MAX_BATCH_SIZE ) 281             rc = flush_batch(ctx);

... the tool can't spot that flush_batch() resets ctx->save.nr_batch_pfns to 0 in the success case. And ...

Quoted 1 line

283         if ( rc == 0 )

... only the success case is what matters.

Jan

Quoted 12 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

     CID 1700057:         Memory - corruptions  (OVERRUN)
     Overrunning array "(*ctx).save.buffers->batch_pfns" of 1024 8-byte elements at element index 1024 (byte offset 8199) using index "(*ctx).save.nr_batch_pfns++" (which evaluates to 1024).

284             ctx->save.buffers->batch_pfns[ctx->save.nr_batch_pfns++] = pfn; 285
286         return rc;
287     }
288
289     /*

~Andrew

#5 lore raw

Re: [PATCH v10 2/10] libs/guest: move batch_pfns into a separate structure

Quoted 21 lines · Jan Beulich
Quoted 9 lines · Andrew Cooper
Quoted 5 lines · Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

Coverity thinks this change has memory corruption.  I have to admit that I'm not completely sure why it's noticed now; possibly because now it can see the size of batch_pfns[] where previously it couldn't

I had looked into that too, and I'm puzzled that ...

Quoted 9 lines

** CID 1700057:       Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284           in add_to_batch()
_____________________________________________________________________________________________
*** CID 1700057:         Memory - corruptions  (OVERRUN) /tools/libs/guest/xg_sr_save.c: 284             in add_to_batch() 278         int rc = 0;
279
280         if ( ctx->save.nr_batch_pfns == MAX_BATCH_SIZE ) 281             rc = flush_batch(ctx);

Quoted 5 lines

... the tool can't spot that flush_batch() resets ctx->save.nr_batch_pfns to 0 in the success case. And ...

Quoted 1 line

283         if ( rc == 0 )

... only the success case is what matters.

Hmm.  Both flush_batch() and write_batch() are static, so fully visible to Coverity.

I guess this means that Coverity failed to figure out the properties of write_batch(); it is a complicated function, even if it has become less complicated recently.

I'm still advocating to remove the Valgrind logic rather than extend it in patch 3, and that will remove flush_batch() which might simplify things.

~Andrew

#6 lore raw

Re: [PATCH v10 2/10] libs/guest: move batch_pfns into a separate structure

Quoted 32 lines · Andrew Cooper
Quoted 21 lines · Jan Beulich
Quoted 9 lines · Andrew Cooper
Quoted 5 lines · Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

Coverity thinks this change has memory corruption. I have to admit that I'm not completely sure why it's noticed now; possibly because now it can see the size of batch_pfns[] where previously it couldn't

I had looked into that too, and I'm puzzled that ...

Quoted 9 lines

** CID 1700057: Memory - corruptions (OVERRUN)
/tools/libs/guest/xg_sr_save.c: 284 in add_to_batch()
_____________________________________________________________________________________________
*** CID 1700057: Memory - corruptions (OVERRUN)
/tools/libs/guest/xg_sr_save.c: 284 in add_to_batch()
278 int rc = 0;
279
280 if ( ctx->save.nr_batch_pfns == MAX_BATCH_SIZE )
281 rc = flush_batch(ctx);

Quoted 5 lines

... the tool can't spot that flush_batch() resets ctx->save.nr_batch_pfns to 0 in the success case. And ...

Quoted 1 line

283 if ( rc == 0 )

... only the success case is what matters.

Hmm. Both flush_batch() and write_batch() are static, so fully visible to Coverity.

Quoted 8 lines

I guess this means that Coverity failed to figure out the properties of write_batch(); it is a complicated function, even if it has become less complicated recently.

I'm still advocating to remove the Valgrind logic rather than extend it in patch 3, and that will remove flush_batch() which might simplify things.

~Andrew

Hi,
as said by Jan this is a false positive from Coverity.

It's not guaranteed that Coverity will detect that it's not an issue if either valgrind macro is not called and/or flush_batch calling write_batch, I would either test the change or simply add an explicit Coverity comment.
If we decide to remove support for valgrind then what about the valgrind patch on the series (4/10) ?

Frediano

#7 lore raw

PATCH v10 3/10 libs/guest: allocate various migration arrays just once

In reply to Frediano Ziglio

From: Edwin Török <[email protected]>

Allocate these array just once at the start of migration, using the maximum batch size, and free them at the end.

---
Changes since v2:
- change prefix in subject.

Changes since v3:
- fix comment style

Changes since v4:
- change order of fields in structure.

Changes since v6:
- split preparation commit.

Changes since v8:
- remove useless memset;
- initialize variables while declaring them.

Changes since v9:
- added Reviewed-by.

---
 tools/libs/guest/xg_sr_common.h |  6 +++++
 tools/libs/guest/xg_sr_save.c   | 45 ++++++++++++---------------------
 2 files changed, 22 insertions(+), 29 deletions(-)
tools/libs/guest/xg_sr_common.h +6 -0
index 7574c9f5b6..c07c6db59e 100644
@@ -246,6 +246,12 @@ struct xc_sr_context
             struct xc_sr_context_save_buffers
             {
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                xen_pfn_t mfns[MAX_BATCH_SIZE];
+                xen_pfn_t types[MAX_BATCH_SIZE];
+                void *local_pages[MAX_BATCH_SIZE];
+                struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                uint64_t rec_pfns[MAX_BATCH_SIZE];
+                int errors[MAX_BATCH_SIZE];
             } *buffers;
         } save;
 
tools/libs/guest/xg_sr_save.c +16 -29

81 lines not on this page · raw message

Signature · 1 line
2.43.0
#8 lore raw

PATCH v10 4/10 libs/guest: use Valgrind or sanitizers to detect various buffer overflows

In reply to Frediano Ziglio

Previously this was done as buffers were allocated separately.

---
Changes since v9:
- add support for sanitizers also;
- remove some unneeded check buffers.

---
 tools/config.h.in               |  6 ++++
 tools/configure                 | 12 +++++++
 tools/configure.ac              |  3 +-
 tools/libs/ctrl/xc_private.h    | 61 +++++++++++++++++++++++++++++++--
 tools/libs/guest/xg_sr_common.h |  6 ++++
 tools/libs/guest/xg_sr_save.c   | 11 ++++++
 6 files changed, 96 insertions(+), 3 deletions(-)
tools/config.h.in +6 -0
index ed0042018d..d51816453b 100644
@@ -48,6 +48,12 @@
 /* ROMBIOS enabled */
 #undef HAVE_ROMBIOS
 
+/* Define to 1 if you have the <sanitizer/asan_interface.h> header file. */
+#undef HAVE_SANITIZER_ASAN_INTERFACE_H
+
+/* Define to 1 if you have the <sanitizer/msan_interface.h> header file. */
+#undef HAVE_SANITIZER_MSAN_INTERFACE_H
+
 /* Define to 1 if you have the <stdint.h> header file. */
 #undef HAVE_STDINT_H
 
tools/configure +12 -0
index cd989925ed..94e630665f 100755
@@ -10203,6 +10203,18 @@ then :
   printf "%s\n" "#define HAVE_UTMP_H 1" >>confdefs.h
 
 fi
+ac_fn_c_check_header_compile "$LINENO" "sanitizer/asan_interface.h" "ac_cv_header_sanitizer_asan_interface_h" "$ac_includes_default"
+if test "x$ac_cv_header_sanitizer_asan_interface_h" = xyes
+then :
+  printf "%s\n" "#define HAVE_SANITIZER_ASAN_INTERFACE_H 1" >>confdefs.h
+
+fi
+ac_fn_c_check_header_compile "$LINENO" "sanitizer/msan_interface.h" "ac_cv_header_sanitizer_msan_interface_h" "$ac_includes_default"
+if test "x$ac_cv_header_sanitizer_msan_interface_h" = xyes
+then :
+  printf "%s\n" "#define HAVE_SANITIZER_MSAN_INTERFACE_H 1" >>confdefs.h
+
+fi
 
 
 # Check for libnl3 >=3.2.8. If present enable remus network buffering.
tools/configure.ac +2 -1
index 74b9f56025..5346ff6129 100644
@@ -454,7 +454,8 @@ AC_CHECK_DECLS([fdt_property_u32],,,[#include <libfdt.h>])
 esac
 
 # Checks for header files.
-AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h valgrind/memcheck.h utmp.h])
+AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h valgrind/memcheck.h \
+                  utmp.h sanitizer/asan_interface.h sanitizer/msan_interface.h])
 
 # Check for libnl3 >=3.2.8. If present enable remus network buffering.
 PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8],
tools/libs/ctrl/xc_private.h +59 -2

72 lines not on this page · raw message

tools/libs/guest/xg_sr_common.h +6 -0
index c07c6db59e..020b1a5272 100644
@@ -246,11 +246,17 @@ struct xc_sr_context
             struct xc_sr_context_save_buffers
             {
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na0, 64);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 64);
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 64);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 64);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na4, 64);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na5, 64);
                 int errors[MAX_BATCH_SIZE];
             } *buffers;
         } save;
tools/libs/guest/xg_sr_save.c +11 -0
index 6a77e33a47..96d7e9e2f8 100644
@@ -123,6 +123,11 @@ static int write_batch(struct xc_sr_context *ctx)
     assert(nr_pfns != 0);
     assert(nr_pfns <= MAX_BATCH_SIZE);
 
+    MEM_UNDEFINED_INIT(ctx->save.buffers->mfns);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->types);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->iov);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->rec_pfns);
+
     iov[0].iov_base = &hdrs;
     iov[0].iov_len = sizeof(hdrs);
 
@@ -783,6 +788,12 @@ static int setup(struct xc_sr_context *ctx)
         errno = ENOMEM;
         goto err;
     }
+    MEM_NOACCESS_INIT(ctx->save.buffers->na0);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na1);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na2);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na3);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na4);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na5);
 
     rc = 0;
 
Signature · 1 line
2.43.0
#9 lore raw

PATCH v10 5/10 libs/guest: add xg_foreignmemory_copy_{from,to}

In reply to Frediano Ziglio

This change prepare code to use a new "foreign copy" hypercall. The new hypercall will copy memory from/to a foreign domain. The new hypercall can be emulated with a sequence of:
- map foreign memory;
- copy memory;
- unmap foreign memory.

The reason to introduce the emulation first is that you can refactor on the emulation without having to introduce the new hypercall. Introducing the hypercall first would make testing more complicated as bugs on the hypercall have to be taken into account and considered. Also it is easier that way to enable or disable new code. For instance you want to test for performance regression (in this case the code emulated should not perform worse).

---
Changes since v5:
- Do not overwrite errno if xenforeignmemory_map fails.

Changes since v6:
- improve commit message, explain order and changes.

---
 tools/libs/guest/xg_sr_common.c | 57 +++++++++++++++++++++++++++++++++
 tools/libs/guest/xg_sr_common.h |  8 +++++
 2 files changed, 65 insertions(+)
tools/libs/guest/xg_sr_common.c +57 -0
index 9b2782b5cf..90da21c35f 100644
@@ -156,6 +156,63 @@ static void __attribute__((unused)) build_assertions(void)
     BUILD_BUG_ON(sizeof(struct xc_sr_rec_hvm_params)        != 8);
 }
 
+enum {
+    foreigncopy_from,
+    foreigncopy_to
+};
+
+static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,
+                                 int dir, size_t nr_pages, void *buffer,
+                                 const xen_pfn_t foreign_pfns[nr_pages])
+{
+    if ( nr_pages == 0 )
+        return 0;
+
+    if ( !buffer || !foreign_pfns )
+    {
+        errno = EINVAL;
+        return -1;
+    }
+
+    int err[nr_pages];
+    const int prot = (dir == foreigncopy_from) ? PROT_READ : PROT_READ|PROT_WRITE;
+
+    void *p = xenforeignmemory_map(xch->fmem, domid, prot, nr_pages, foreign_pfns, err);
+    if ( !p )
+        return -1;
+
+    for ( size_t n = 0; n < nr_pages; ++n )
+        if ( err[n] )
+        {
+            xenforeignmemory_unmap(xch->fmem, p, nr_pages);
+            errno = -err[n];
+            return -1;
+        }
+
+    if ( dir == foreigncopy_from )
+        memcpy(buffer, p, nr_pages * XC_PAGE_SIZE);
+    else
+        memcpy(p, buffer, nr_pages * XC_PAGE_SIZE);
+
+    return xenforeignmemory_unmap(xch->fmem, p, nr_pages);
+}
+
+int xg_foreignmemory_copy_from(xc_interface *xch, domid_t dom,
+                               size_t nr_pages, void *dest,
+                               const xen_pfn_t source[nr_pages])
+{
+    return xg_foreignmemory_copy(xch, dom, foreigncopy_from,
+                                 nr_pages, dest, source);
+}
+
+int xg_foreignmemory_copy_to(xc_interface *xch, domid_t dom,
+                             size_t nr_pages, const xen_pfn_t dest[nr_pages],
+                             const void *source)
+{
+    return xg_foreignmemory_copy(xch, dom, foreigncopy_to,
+                                 nr_pages, (void *) source, dest);
+}
+
 /*
  * Local variables:
  * mode: C
tools/libs/guest/xg_sr_common.h +8 -0
index 020b1a5272..50f235ba87 100644
@@ -556,6 +556,14 @@ static inline bool page_type_has_stream_data(uint32_t type)
     }
 }
 
+int xg_foreignmemory_copy_from(xc_interface *xch, domid_t dom,
+                               size_t nr_pages, void *dest,
+                               const xen_pfn_t source[nr_pages]);
+
+int xg_foreignmemory_copy_to(xc_interface *xch, domid_t dom,
+                             size_t nr_pages, const xen_pfn_t dest[nr_pages],
+                             const void *source);
+
 #endif
 /*
  * Local variables:
Signature · 1 line
2.43.0
#10 lore raw

PATCH v10 6/10 libs/guest: use foreign copy API during migration

In reply to Frediano Ziglio

From: Edwin Török <[email protected]>

Use foreign code emulation code provided by previous commit to prepare to use new hypercall.
This to make sure there are no regression in both functionality and performance.

In particular tested:
- HVM VM;
- PV VM;
- verification code.

Migration times did not change.

---
Changes since v6:
- merge with "finalize PoC" to remove the PoC;
- remove statistics, old and not clear at all how they were made;
- describe tests made.

---
 tools/libs/guest/xg_sr_common.h  |  4 +-
 tools/libs/guest/xg_sr_restore.c | 78 +++++++++++++++++---------------
 tools/libs/guest/xg_sr_save.c    | 62 +++++++++++--------------
 3 files changed, 71 insertions(+), 73 deletions(-)
tools/libs/guest/xg_sr_common.h +2 -2
index 50f235ba87..ec3435790a 100644
@@ -243,6 +243,7 @@ struct xc_sr_context
             unsigned long *deferred_pages;
             unsigned long nr_deferred_pages;
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
+            xc_hypercall_buffer_t dest_buf;
             struct xc_sr_context_save_buffers
             {
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
@@ -256,8 +257,6 @@ struct xc_sr_context
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
                 MEM_NOACCESS_BUFFER(na4, 64);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
-                MEM_NOACCESS_BUFFER(na5, 64);
-                int errors[MAX_BATCH_SIZE];
             } *buffers;
         } save;
 
@@ -269,6 +268,7 @@ struct xc_sr_context
             int send_back_fd;
             unsigned long p2m_size;
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
+            xc_hypercall_buffer_t verify_buf;
 
             /* From Image Header. */
             uint32_t format_version;
tools/libs/guest/xg_sr_restore.c +42 -36

134 lines not on this page · raw message

tools/libs/guest/xg_sr_save.c +27 -35

150 lines not on this page · raw message

Signature · 1 line
2.43.0
#11 lore raw

PATCH v10 7/10 xen: implement new foreign copy hypercall

In reply to Frediano Ziglio

Add a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.

Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.

---
Changes since v4:
- Fix typo in comment.

Changes since v5:
- update xen_foreigncopy structure comments;
- move check for no frames after checking the domain;
- use mnemonic instead of 1U;
- fix page type checks;
- do not overwrite error copying back structure;
- latch MFN value;
- improved commit message.

Changes since v6:
- check permissions before nr_frames;
- different flag for read or write;
- print error as negative for coherence;
- update some comments;
- different page types for different architectures.

Changes since v9:
- page permission checks like MMU_UPDATE;
- new XSM settings;
- do not restrict domain;
- different explanation why HVM guests are not supported.

---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
xen/common/memory.c +149 -0
index 9443e35a7f..29a70d99b1 100644
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;
+    xen_foreigncopy_t copy;
+    int rc, direction;
+
+    if ( copy_from_guest(&copy, arg, 1) )
+        return -EFAULT;
+
+    if ( copy.flags & ~XENMEM_foreigncopy_direction )
+        return -EINVAL;
+
+    direction = copy.flags & XENMEM_foreigncopy_direction;
+
+    d = rcu_lock_domain_by_any_id(copy.domid);
+    if ( !d )
+        return -ESRCH;
+
+    /*
+     * Check we are allowed to map and access these foreign pages.
+     */
+    if ( direction == XENMEM_foreigncopy_from )
+        rc = xsm_foreigncopy_from(XSM_TARGET, currd, d);
+    else
+        rc = xsm_foreigncopy_to(XSM_TARGET, currd, d);
+    if ( rc )
+        goto out;
+
+    while ( copy.nr_frames )
+    {
+        /*
+         * Arbitrary size.  Not too much stack space, and a reasonable stride
+         * for continuation checks.
+         */
+        xen_pfn_t gfn_list[32];
+        unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames);
+
+        rc = -EFAULT;
+        if ( copy_from_guest(gfn_list, copy.frame_list, todo) )
+            goto out;
+
+        for ( unsigned int i = 0; i < todo; i++ )
+        {
+            struct page_info *foreign_page;
+            mfn_t foreign_mfn;
+            void *foreign;
+            p2m_type_t p2mt;
+            p2m_query_t q = (direction == XENMEM_foreigncopy_to) ?
+                            P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;
+
+            foreign_page = get_page_from_gfn(d, gfn_list[i], &p2mt, q);
+
+            if ( unlikely(p2m_is_paged(p2mt)) )
+            {
+                if ( foreign_page )
+                    put_page(foreign_page);
+                p2m_mem_paging_populate(d, _gfn(gfn_list[i]));
+                p2mt = p2m_ram_paging_in;
+                foreign_page = NULL;
+            }
+
+            if ( unlikely(!foreign_page) )
+            {
+                rc = -ENOENT;
+                if ( p2mt != p2m_ram_paging_in )
+                {
+                    gdprintk(XENLOG_WARNING,
+                             "Error accessing foreign gfn %" PRI_gfn "\n",
+                             gfn_list[i]);
+                    rc = -EINVAL;
+                }
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;
+            }
+
+            foreign_mfn = page_to_mfn(foreign_page);
+
+            /* A page is dirtied when it's being copied to. */
+            if ( direction == XENMEM_foreigncopy_to )
+                paging_mark_dirty(d, foreign_mfn);
+
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);
+            unmap_domain_page(foreign);
+            put_page(foreign_page);
+
+            if ( unlikely(rc) )
+            {
+                gdprintk(XENLOG_WARNING,
+                         "Error %d copying gfn %" PRI_gfn "\n",
+                         rc, gfn_list[i]);
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;
+            }
+
+            guest_handle_add_offset(copy.buffer, PAGE_SIZE);
+        }
+
+        copy.nr_frames -= todo;
+        guest_handle_add_offset(copy.frame_list, todo);
+
+        if ( copy.nr_frames && hypercall_preempt_check() )
+        {
+            rc = hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh", XENMEM_foreigncopy, arg);
+            goto out;
+        }
+    }
+
+    rc = 0;
+
+ out:
+    rcu_unlock_domain(d);
+
+    /*
+     * Update in all cases, it allows the caller to know how many
+     * frames were successfully copied and the continuation to
+     * continue correctly.
+     */
+    if ( __copy_to_guest(arg, &copy, 1) && rc >= 0 )
+        rc = -EFAULT;
+
+    return rc;
+}
+
 long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
     struct domain *d, *curr_d = current->domain;
@@ -2027,6 +2162,20 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
             start_extent);
         break;
 
+    case XENMEM_foreigncopy:
+        /*
+         * Instead of using "start_extent" for the continuation, we update
+         * the xen_foreigncopy structure back, so we are not constrained by
+         * MEMOP_EXTENT_SHIFT.
+         * We copy it back also to tell the caller where the copy stopped
+         * (either for error or because all frames were copied).
+         */
+        if ( unlikely(start_extent) )
+            return -EINVAL;
+
+        rc = mem_foreigncopy(guest_handle_cast(arg, xen_foreigncopy_t));
+        break;
+
     default:
         rc = arch_memory_op(cmd, arg);
         break;
xen/include/public/memory.h +44 -1

51 lines not on this page · raw message

xen/include/xsm/dummy.h +14 -0
index 131631cb27..dcdb7f5396 100644
@@ -569,6 +569,20 @@ static XSM_INLINE int cf_check xsm_map_gmfn_foreign(
     return xsm_default_action(action, d, t);
 }
 
+static XSM_INLINE int cf_check xsm_foreigncopy_from(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
+static XSM_INLINE int cf_check xsm_foreigncopy_to(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
 #ifdef CONFIG_HVM
 
 static XSM_INLINE int cf_check xsm_hvm_param(
xen/include/xsm/hooks.h +2 -0
index 5bdb23f26d..63e2831d31 100644
@@ -58,6 +58,8 @@ XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, remove_from_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, map_gmfn_foreign, struct domain *, struct domain *)
 XSM_HOOK(int, claim_pages, struct domain *)
+XSM_HOOK(int, foreigncopy_from, struct domain *, struct domain *);
+XSM_HOOK(int, foreigncopy_to, struct domain *, struct domain *);
 
 XSM_HOOK(int, console_io, struct domain *, int)
 
xen/xsm/flask/hooks.c +10 -0
index 3cfdf6bf08..281800e176 100644
@@ -1368,6 +1368,16 @@ static int cf_check flask_map_gmfn_foreign(struct domain *d, struct domain *t)
     return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);
 }
 
+static int cf_check flask_foreigncopy_from(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ);
+}
+
+static int cf_check flask_foreigncopy_to(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);
+}
+
 #ifdef CONFIG_HVM
 
 static int cf_check flask_hvm_param(struct domain *d, unsigned long op)
Signature · 1 line
2.43.0
#12 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 8 lines · Frediano Ziglio

Add a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.

Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.

First: Can you please update your recipient list before sending a new version? Roger's move pre-dates this submission by quite a few days.

Then: The asymmetry of the new sub-op also continues to have no justification at all. If you insist on not using two (domid,list-of-gfns) tuples to describe the buffers, despite multiple maintainers having asked you to do so, please put down a word explaining that decision.

Quoted 7 lines
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 11 lines
xen/common/memory.c +5 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

xen/common/memory.c static int acquire_resource( · line 1560

Quoted 2 lines
xen/common/memory.c +2 -0
+    xen_foreigncopy_t copy;
+    int rc, direction;

Plain int for rc is fine of course, but direction can't go negative, can it?

Quoted 5 lines
xen/common/memory.c +5 -0
+    if ( copy_from_guest(&copy, arg, 1) )
+        return -EFAULT;
+
+    if ( copy.flags & ~XENMEM_foreigncopy_direction )
+        return -EINVAL;

xen/common/memory.c static int acquire_resource( · line 1576

Quoted 10 lines
xen/common/memory.c +10 -0
+
+    direction = copy.flags & XENMEM_foreigncopy_direction;
+
+    d = rcu_lock_domain_by_any_id(copy.domid);
+    if ( !d )
+        return -ESRCH;
+
+    /*
+     * Check we are allowed to map and access these foreign pages.
+     */

This really means to be a single-line comment.

Quoted 20 lines
xen/common/memory.c +20 -0
+    if ( direction == XENMEM_foreigncopy_from )
+        rc = xsm_foreigncopy_from(XSM_TARGET, currd, d);
+    else
+        rc = xsm_foreigncopy_to(XSM_TARGET, currd, d);
+    if ( rc )
+        goto out;
+
+    while ( copy.nr_frames )
+    {
+        /*
+         * Arbitrary size.  Not too much stack space, and a reasonable stride
+         * for continuation checks.
+         */
+        xen_pfn_t gfn_list[32];
+        unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames);
+
+        rc = -EFAULT;
+        if ( copy_from_guest(gfn_list, copy.frame_list, todo) )
+            goto out;
+

xen/common/memory.c static int acquire_resource( · line 1606

Quoted 10 lines
xen/common/memory.c +10 -0
+        for ( unsigned int i = 0; i < todo; i++ )
+        {
+            struct page_info *foreign_page;
+            mfn_t foreign_mfn;
+            void *foreign;
+            p2m_type_t p2mt;
+            p2m_query_t q = (direction == XENMEM_foreigncopy_to) ?
+                            P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;
+
+            foreign_page = get_page_from_gfn(d, gfn_list[i], &p2mt, q);

Is there a reason check_get_page_from_gfn() can't or shouldn't be used here? That would then also make this code properly deal with shared pages, and refuse use against MMIO ones.

Quoted 19 lines
xen/common/memory.c +19 -0
+            if ( unlikely(p2m_is_paged(p2mt)) )
+            {
+                if ( foreign_page )
+                    put_page(foreign_page);
+                p2m_mem_paging_populate(d, _gfn(gfn_list[i]));
+                p2mt = p2m_ram_paging_in;
+                foreign_page = NULL;
+            }
+
+            if ( unlikely(!foreign_page) )
+            {
+                rc = -ENOENT;
+                if ( p2mt != p2m_ram_paging_in )
+                {
+                    gdprintk(XENLOG_WARNING,
+                             "Error accessing foreign gfn %" PRI_gfn "\n",
+                             gfn_list[i]);
+                    rc = -EINVAL;
+                }

xen/common/memory.c static int acquire_resource( · line 1636

Quoted 10 lines
xen/common/memory.c +10 -0
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;
+            }
+
+            foreign_mfn = page_to_mfn(foreign_page);
+
+            /* A page is dirtied when it's being copied to. */
+            if ( direction == XENMEM_foreigncopy_to )
+                paging_mark_dirty(d, foreign_mfn);

This may better be folded into the "else" below.

xen/common/memory.c static int acquire_resource( · line 1642

Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

xen/common/memory.c static int acquire_resource( · line 1653

Quoted 11 lines
xen/common/memory.c +11 -0
+            unmap_domain_page(foreign);
+            put_page(foreign_page);
+
+            if ( unlikely(rc) )
+            {
+                gdprintk(XENLOG_WARNING,
+                         "Error %d copying gfn %" PRI_gfn "\n",
+                         rc, gfn_list[i]);
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;

rc at this point holds the number of bytes which could not be copied. That's not suitable as a return value from this function. (Strictly speaking this is also an abuse of rc, as copy_{to,from}_*() return unsigned quantities.)

xen/common/memory.c static int acquire_resource( · line 1665

Quoted 12 lines
xen/common/memory.c +12 -0
+            }
+
+            guest_handle_add_offset(copy.buffer, PAGE_SIZE);
+        }
+
+        copy.nr_frames -= todo;
+        guest_handle_add_offset(copy.frame_list, todo);
+
+        if ( copy.nr_frames && hypercall_preempt_check() )
+        {
+            rc = hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh", XENMEM_foreigncopy, arg);

Nit: Indentation (the anchor point is the start of the function name, not the start of the statement).

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Quoted 6 lines
xen/include/public/memory.h +6 -0
+ */
+#define XENMEM_foreigncopy 29
+struct xen_foreigncopy {
+    /* IN - The domain whose memory is to be copied. */
+    domid_t domid;
+

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 762

Quoted 10 lines
xen/include/public/memory.h +10 -0
+    /* IN - Flags. */
+#define XENMEM_foreigncopy_from 0
+#define XENMEM_foreigncopy_to 1
+#define XENMEM_foreigncopy_direction 1
+    uint16_t flags;
+
+    /*
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the domain to be copied.

I think there's a comma wanted after "parameter".

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 763

Quoted 1 line
xen/include/public/memory.h +1 -0
+     * On output updated number of frames left (0 if success).

I further think that adding "to" after "updated" would help here.

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 573

Quoted 8 lines
xen/include/xsm/dummy.h +2 -0
@@ -569,6 +569,20 @@ static XSM_INLINE int cf_check xsm_map_gmfn_foreign(
     return xsm_default_action(action, d, t);
 }
 
+static XSM_INLINE int cf_check xsm_foreigncopy_from(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

I think these and ...

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 580

Quoted 7 lines
xen/include/xsm/dummy.h +7 -0
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
+static XSM_INLINE int cf_check xsm_foreigncopy_to(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

... want to be pointer-to-const right away, requiring to re-base over "XSM:
make Argo hooks well-formed ones" (or alternatively requiring to split out the change there to xsm_default_action()). We really should avoid gaining
...

xen/include/xsm/hooks.h XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *) · line 62

Quoted 8 lines
xen/include/xsm/hooks.h +2 -0
@@ -58,6 +58,8 @@ XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, remove_from_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, map_gmfn_foreign, struct domain *, struct domain *)
 XSM_HOOK(int, claim_pages, struct domain *)
+XSM_HOOK(int, foreigncopy_from, struct domain *, struct domain *);
+XSM_HOOK(int, foreigncopy_to, struct domain *, struct domain *);

... new hooks with non-const-correct parameters.

Further: Why two new hooks? See e.g. "XSM: fold xsm_{,un}map_domain_pirq() hooks", "XSM: fold xsm_{,un}map_domain_irq() hooks", or "XSM: fold xsm_{,un}bind_pt_irq() hooks": We'd like to reduce the number of hooks, to reduce (when non-dummy XSM is in use) the number of cf_check entry points.

Quoted 4 lines
xen/xsm/flask/hooks.c +0 -0
@@ -1368,6 +1368,16 @@ static int cf_check flask_map_gmfn_foreign(struct domain *d, struct domain *t)
     return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);

xen/xsm/flask/hooks.c static int cf_check flask_map_gmfn_foreign(struct domain *d, struct domain *t) · line 1378

Quoted 10 lines
xen/xsm/flask/hooks.c +8 -0
 }
 
+static int cf_check flask_foreigncopy_from(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ);
+}
+
+static int cf_check flask_foreigncopy_to(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);

Why both READ and WRITE?

Jan

#13 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 5 lines · Jan Beulich
Quoted 3 lines · Frediano Ziglio

Add a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.

Quoted 9 lines
Quoted 5 lines

Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.

First: Can you please update your recipient list before sending a new version? Roger's move pre-dates this submission by quite a few days.

Yes, updated, I realized it after sending it.

Quoted 5 lines

Then: The asymmetry of the new sub-op also continues to have no justification at all. If you insist on not using two (domid,list-of-gfns) tuples to describe the buffers, despite multiple maintainers having asked you to do so, please put down a word explaining that decision.

All other maintainers, after replying to their comments didn't disagree with me, and this was different versions before, so stop counting them, it's only you that don't like this at the moment. I already explained why this is consistent with all other current hypercall. I asked for a practical change to the API and nobody (including you) has a proposal. I made a proposal and nobody replied to it.
If the issue is just adding a comment in the commit message I can do it.

Quoted 8 lines
Quoted 7 lines
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
Quoted 4 lines

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

Yes, I was just going to reply.
I spent multiple days trying to implement the compat case or simply HVM support with an issue after the other:
- multiple distributions removed the 32 bit support so it was hard to have a setup;
- the original hypercall this PR is trying to optimise is supported only in PV (so no HVM or compat guests);
- migration and other operations can work only on PV (like dm_op operation) due to the usage of userspace handles used.
I tried to bypass the above limitations to have the new hypercall tested and I manage to have 64 bit HVM working but the result is a collection of nasty hacks and supporting them properly is a different bigger task.
Surely a comment in the commit message on this is worth it.

Quoted 6 lines
Quoted 6 lines
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Quoted 5 lines

xen/common/memory.c static int acquire_resource( · line 1560

Quoted 2 lines
xen/common/memory.c +2 -0
+    xen_foreigncopy_t copy;
+    int rc, direction;

Plain int for rc is fine of course, but direction can't go negative, can it?

No, but the default type for constants is int and "flags" is promoted to int.

Quoted 9 lines
Quoted 9 lines
xen/common/memory.c +9 -0
+    if ( copy_from_guest(&copy, arg, 1) )
+        return -EFAULT;
+
+    if ( copy.flags & ~XENMEM_foreigncopy_direction )
+        return -EINVAL;
+
+    direction = copy.flags & XENMEM_foreigncopy_direction;
+
+    d = rcu_lock_domain_by_any_id(copy.domid);
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1576

Quoted 6 lines
xen/common/memory.c +6 -0
+    if ( !d )
+        return -ESRCH;
+
+    /*
+     * Check we are allowed to map and access these foreign pages.
+     */

This really means to be a single-line comment.

Changed.

Quoted 26 lines
Quoted 26 lines
xen/common/memory.c +26 -0
+    if ( direction == XENMEM_foreigncopy_from )
+        rc = xsm_foreigncopy_from(XSM_TARGET, currd, d);
+    else
+        rc = xsm_foreigncopy_to(XSM_TARGET, currd, d);
+    if ( rc )
+        goto out;
+
+    while ( copy.nr_frames )
+    {
+        /*
+         * Arbitrary size.  Not too much stack space, and a reasonable stride
+         * for continuation checks.
+         */
+        xen_pfn_t gfn_list[32];
+        unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames);
+
+        rc = -EFAULT;
+        if ( copy_from_guest(gfn_list, copy.frame_list, todo) )
+            goto out;
+
+        for ( unsigned int i = 0; i < todo; i++ )
+        {
+            struct page_info *foreign_page;
+            mfn_t foreign_mfn;
+            void *foreign;
+            p2m_type_t p2mt;
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1606

Quoted 4 lines
xen/common/memory.c +4 -0
+            p2m_query_t q = (direction == XENMEM_foreigncopy_to) ?
+                            P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;
+
+            foreign_page = get_page_from_gfn(d, gfn_list[i], &p2mt, q);

Is there a reason check_get_page_from_gfn() can't or shouldn't be used here? That would then also make this code properly deal with shared pages, and refuse use against MMIO ones.

It seems sensible, I didn't know it, I'll try it.

Quoted 23 lines
Quoted 23 lines
xen/common/memory.c +23 -0
+            if ( unlikely(p2m_is_paged(p2mt)) )
+            {
+                if ( foreign_page )
+                    put_page(foreign_page);
+                p2m_mem_paging_populate(d, _gfn(gfn_list[i]));
+                p2mt = p2m_ram_paging_in;
+                foreign_page = NULL;
+            }
+
+            if ( unlikely(!foreign_page) )
+            {
+                rc = -ENOENT;
+                if ( p2mt != p2m_ram_paging_in )
+                {
+                    gdprintk(XENLOG_WARNING,
+                             "Error accessing foreign gfn %" PRI_gfn "\n",
+                             gfn_list[i]);
+                    rc = -EINVAL;
+                }
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;
+            }
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1636

Quoted 6 lines
xen/common/memory.c +6 -0
+
+            foreign_mfn = page_to_mfn(foreign_page);
+
+            /* A page is dirtied when it's being copied to. */
+            if ( direction == XENMEM_foreigncopy_to )
+                paging_mark_dirty(d, foreign_mfn);

This may better be folded into the "else" below.

Changed

Quoted 11 lines

xen/common/memory.c static int acquire_resource( · line 1642

Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

Unfortunately that does not work.
The code is coherent with MMU_UPDATE.
I'll write a comment.

Quoted 12 lines

xen/common/memory.c static int acquire_resource( · line 1653

Quoted 11 lines
xen/common/memory.c +11 -0
+            unmap_domain_page(foreign);
+            put_page(foreign_page);
+
+            if ( unlikely(rc) )
+            {
+                gdprintk(XENLOG_WARNING,
+                         "Error %d copying gfn %" PRI_gfn "\n",
+                         rc, gfn_list[i]);
+                copy.nr_frames -= i;
+                guest_handle_add_offset(copy.frame_list, i);
+                goto out;
Quoted 4 lines

rc at this point holds the number of bytes which could not be copied. That's not suitable as a return value from this function. (Strictly speaking this is also an abuse of rc, as copy_{to,from}_*() return unsigned quantities.)

Yes, add an "uncopied" variable and return -EFAULT.

Quoted 7 lines
Quoted 7 lines
xen/common/memory.c +7 -0
+            }
+
+            guest_handle_add_offset(copy.buffer, PAGE_SIZE);
+        }
+
+        copy.nr_frames -= todo;
+        guest_handle_add_offset(copy.frame_list, todo);
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1665

Quoted 5 lines
xen/common/memory.c +5 -0
+
+        if ( copy.nr_frames && hypercall_preempt_check() )
+        {
+            rc = hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh", XENMEM_foreigncopy, arg);

Nit: Indentation (the anchor point is the start of the function name, not the start of the statement).

Changed.

Quoted 13 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

Quoted 9 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Maybe:
This call is meant to replace expensive operations (mmap/copy/munmap) during migration which can only be issued from PV guests.

You can migrate any domain. Just from a PV guest (this is not a regression).

Quoted 19 lines
Quoted 6 lines
xen/include/public/memory.h +6 -0
+ */
+#define XENMEM_foreigncopy 29
+struct xen_foreigncopy {
+    /* IN - The domain whose memory is to be copied. */
+    domid_t domid;
+

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 762

Quoted 10 lines
xen/include/public/memory.h +10 -0
+    /* IN - Flags. */
+#define XENMEM_foreigncopy_from 0
+#define XENMEM_foreigncopy_to 1
+#define XENMEM_foreigncopy_direction 1
+    uint16_t flags;
+
+    /*
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the domain to be copied.

I think there's a comma wanted after "parameter".

Quoted 4 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 763

Quoted 1 line
xen/include/public/memory.h +1 -0
+     * On output updated number of frames left (0 if success).

I further think that adding "to" after "updated" would help here.

Updated to

    /*
     * IN/OUT
     *
     * As an IN parameter, number of frames of the domain to be copied.
     * On output updated to the number of frames left (0 if successful).
     */
    uint32_t nr_frames;

    /*
     * IN/OUT
     *
     * Frames to be copied.
     * On output updated to the point to the first frame unhandled, if any.
     */
    XEN_GUEST_HANDLE(xen_pfn_t) frame_list;

    /*
     * IN/OUT
     *
     * Guest buffer to read/write from.
     * On output updated to point to the first page pointer unhandled.
     */
    XEN_GUEST_HANDLE(uint8) buffer;
Quoted 24 lines

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 573

Quoted 8 lines
xen/include/xsm/dummy.h +2 -0
@@ -569,6 +569,20 @@ static XSM_INLINE int cf_check xsm_map_gmfn_foreign(
     return xsm_default_action(action, d, t);
 }
 
+static XSM_INLINE int cf_check xsm_foreigncopy_from(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

I think these and ...

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 580

Quoted 7 lines
xen/include/xsm/dummy.h +7 -0
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
+static XSM_INLINE int cf_check xsm_foreigncopy_to(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

... want to be pointer-to-const right away, requiring to re-base over "XSM:
make Argo hooks well-formed ones" (or alternatively requiring to split out the change there to xsm_default_action()). We really should avoid gaining
...

Quoted 11 lines

xen/include/xsm/hooks.h XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *) · line 62

Quoted 8 lines
xen/include/xsm/hooks.h +2 -0
@@ -58,6 +58,8 @@ XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, remove_from_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, map_gmfn_foreign, struct domain *, struct domain *)
 XSM_HOOK(int, claim_pages, struct domain *)
+XSM_HOOK(int, foreigncopy_from, struct domain *, struct domain *);
+XSM_HOOK(int, foreigncopy_to, struct domain *, struct domain *);

... new hooks with non-const-correct parameters.

It would honestly make sense if they were all const.
However at the moment this would not be coherent with the current code. For instance these functions call "xsm_default_action" which does not accept constant domains (xen/include/xsm/dummy.h).
I think the most clean thing would be first to change xsm arguments to const first.
But it looks a bit out of scope with this PR.

Quoted 5 lines

Further: Why two new hooks? See e.g. "XSM: fold xsm_{,un}map_domain_pirq() hooks", "XSM: fold xsm_{,un}map_domain_irq() hooks", or "XSM: fold xsm_{,un}bind_pt_irq() hooks": We'd like to reduce the number of hooks, to reduce (when non-dummy XSM is in use) the number of cf_check entry points.

It was a comment from Daniel.

Quoted 6 lines
Quoted 6 lines
xen/xsm/flask/hooks.c +0 -0
@@ -1368,6 +1368,16 @@ static int cf_check flask_map_gmfn_foreign(struct domain *d, struct domain *t)
     return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);
 }
 
Quoted 11 lines

xen/xsm/flask/hooks.c static int cf_check flask_map_gmfn_foreign(struct domain *d, struct domain *t) · line 1378

Quoted 8 lines
xen/xsm/flask/hooks.c +8 -0
+static int cf_check flask_foreigncopy_from(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ);
+}
+
+static int cf_check flask_foreigncopy_to(struct domain *d, struct domain *t)
+{
+    return domain_has_perm(d, t, SECCLASS_MMU, MMU__MAP_READ | MMU__MAP_WRITE);

Why both READ and WRITE?

Usually for memory you don't have write-only due to cache reasons. But I suppose in this case only writint can work.
I'll change and test.

Quoted 1 line

Jan

Frediano

#14 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 17 lines · Frediano Ziglio
Quoted 4 lines · Jan Beulich
Quoted 3 lines · Frediano Ziglio

Add a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.

Quoted 9 lines
Quoted 5 lines

Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.

First: Can you please update your recipient list before sending a new version? Roger's move pre-dates this submission by quite a few days.

Yes, updated, I realized it after sending it.

Quoted 8 lines
Quoted 4 lines

Then: The asymmetry of the new sub-op also continues to have no justification at all. If you insist on not using two (domid,list-of-gfns) tuples to describe the buffers, despite multiple maintainers having asked you to do so, please put down a word explaining that decision.

All other maintainers, after replying to their comments didn't disagree with me, and this was different versions before, so stop counting them, it's only you that don't like this at the moment.

None of this happened in public, though? In which case, how would I know?

Quoted 13 lines
Quoted 8 lines
Quoted 7 lines
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
Quoted 4 lines

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

Quoted 9 lines

Yes, I was just going to reply.
I spent multiple days trying to implement the compat case or simply HVM support with an issue after the other:
- multiple distributions removed the 32 bit support so it was hard to have a setup;
- the original hypercall this PR is trying to optimise is supported only in PV (so no HVM or compat guests);
- migration and other operations can work only on PV (like dm_op operation) due to the usage of userspace handles used.

I don't understand how use of guest (not userspace) handles would get in the way of anything.

Quoted 5 lines

I tried to bypass the above limitations to have the new hypercall tested and I manage to have 64 bit HVM working but the result is a collection of nasty hacks and supporting them properly is a different bigger task.
Surely a comment in the commit message on this is worth it.

Thanks.

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 4 lines
Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;
Quoted 6 lines
Quoted 3 lines

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Quoted 6 lines
Quoted 4 lines

xen/common/memory.c static int acquire_resource( · line 1560

Quoted 2 lines
xen/common/memory.c +2 -0
+    xen_foreigncopy_t copy;
+    int rc, direction;

Plain int for rc is fine of course, but direction can't go negative, can it?

No, but the default type for constants is int and "flags" is promoted to int.

Doesn't really matter. Unsigned types want using for values which can only be non-negative. IF nothing else, then for consistency and doc purposes.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);
Quoted 7 lines
Quoted 4 lines

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

Unfortunately that does not work.
The code is coherent with MMU_UPDATE.

How's that relevant? That's operating on page tables, when here we want to _prevent_ to copy into page tables (or descriptor ones, for that matter).

Quoted 23 lines
Quoted 13 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

Quoted 9 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Quoted 5 lines

Maybe:
This call is meant to replace expensive operations (mmap/copy/munmap) during migration which can only be issued from PV guests.

You can migrate any domain. Just from a PV guest (this is not a regression).

Both Andrew and Roger confirm that this is supposed to work also from PVH Dom0 (not sure why you keep saying "guest"), and also used to work. If it doesn't, it would be a regression, and it would help if you supplied more detail on the observed failure.

Quoted 34 lines
Quoted 19 lines
Quoted 6 lines
xen/include/public/memory.h +6 -0
+ */
+#define XENMEM_foreigncopy 29
+struct xen_foreigncopy {
+    /* IN - The domain whose memory is to be copied. */
+    domid_t domid;
+

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 762

Quoted 10 lines
xen/include/public/memory.h +10 -0
+    /* IN - Flags. */
+#define XENMEM_foreigncopy_from 0
+#define XENMEM_foreigncopy_to 1
+#define XENMEM_foreigncopy_direction 1
+    uint16_t flags;
+
+    /*
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the domain to be copied.

I think there's a comma wanted after "parameter".

Quoted 4 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 763

Quoted 1 line
xen/include/public/memory.h +1 -0
+     * On output updated number of frames left (0 if success).

I further think that adding "to" after "updated" would help here.

Updated to

/*
* IN/OUT
*
* As an IN parameter, number of frames of the domain to be copied.
* On output updated to the number of frames left (0 if successful). */
uint32_t nr_frames;

Quoted 5 lines

/*
* IN/OUT
*
* Frames to be copied.
* On output updated to the point to the first frame unhandled, if any.

Nit: There's now a stray "the".

Quoted 35 lines
Quoted 24 lines

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 573

Quoted 8 lines
xen/include/xsm/dummy.h +2 -0
@@ -569,6 +569,20 @@ static XSM_INLINE int cf_check xsm_map_gmfn_foreign(
     return xsm_default_action(action, d, t);
 }
 
+static XSM_INLINE int cf_check xsm_foreigncopy_from(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

I think these and ...

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 580

Quoted 7 lines
xen/include/xsm/dummy.h +7 -0
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
+static XSM_INLINE int cf_check xsm_foreigncopy_to(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

... want to be pointer-to-const right away, requiring to re-base over "XSM:
make Argo hooks well-formed ones" (or alternatively requiring to split out the change there to xsm_default_action()). We really should avoid gaining
...

Quoted 10 lines

xen/include/xsm/hooks.h XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *) · line 62

Quoted 8 lines
xen/include/xsm/hooks.h +2 -0
@@ -58,6 +58,8 @@ XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, remove_from_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, map_gmfn_foreign, struct domain *, struct domain *)
 XSM_HOOK(int, claim_pages, struct domain *)
+XSM_HOOK(int, foreigncopy_from, struct domain *, struct domain *);
+XSM_HOOK(int, foreigncopy_to, struct domain *, struct domain *);

... new hooks with non-const-correct parameters.

Quoted 7 lines

It would honestly make sense if they were all const.
However at the moment this would not be coherent with the current code. For instance these functions call "xsm_default_action" which does not accept constant domains (xen/include/xsm/dummy.h).
I think the most clean thing would be first to change xsm arguments to const first.
But it looks a bit out of scope with this PR.

I'm not asking that you tidy up other hooks. What I'm asking is that new hooks please be const-correct. Yet of course I'm not a maintainer of XSM, so Daniel may tell you otherwise.

Quoted 6 lines
Quoted 4 lines

Further: Why two new hooks? See e.g. "XSM: fold xsm_{,un}map_domain_pirq() hooks", "XSM: fold xsm_{,un}map_domain_irq() hooks", or "XSM: fold xsm_{,un}bind_pt_irq() hooks": We'd like to reduce the number of hooks, to reduce (when non-dummy XSM is in use) the number of cf_check entry points.

It was a comment from Daniel.

Daniel, can you clarify this please?

Jan

#15 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 24 lines · Jan Beulich
Quoted 22 lines · Frediano Ziglio
Quoted 4 lines · Jan Beulich
Quoted 3 lines · Frediano Ziglio

Add a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.

Quoted 9 lines
Quoted 5 lines

Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.

First: Can you please update your recipient list before sending a new version? Roger's move pre-dates this submission by quite a few days.

Yes, updated, I realized it after sending it.

Quoted 4 lines

Then: The asymmetry of the new sub-op also continues to have no justification at all. If you insist on not using two (domid,list-of-gfns) tuples to describe the buffers, despite multiple maintainers having asked you to do so, please put down a word explaining that decision.

Quoted 6 lines
Quoted 3 lines

All other maintainers, after replying to their comments didn't disagree with me, and this was different versions before, so stop counting them, it's only you that don't like this at the moment.

None of this happened in public, though? In which case, how would I know?

The discussion with Terry happened in public. The ones with Roger probably not, but as you knew it means that whoever reported to you reported only half of it (or maybe at the time it was reported he didn't know).

Quoted 23 lines
Quoted 13 lines
Quoted 8 lines
Quoted 7 lines
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
Quoted 4 lines

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

Quoted 9 lines

Yes, I was just going to reply.
I spent multiple days trying to implement the compat case or simply HVM support with an issue after the other:
- multiple distributions removed the 32 bit support so it was hard to have a setup;
- the original hypercall this PR is trying to optimise is supported only in PV (so no HVM or compat guests);
- migration and other operations can work only on PV (like dm_op operation) due to the usage of userspace handles used.

Quoted 3 lines

I don't understand how use of guest (not userspace) handles would get in the way of anything.

In this case userspace is not a typo. For HVM copy_from_user_hvm/copy_to_user_hvm are used and these functions accept only kernel space pointers.

Quoted 34 lines
Quoted 5 lines

I tried to bypass the above limitations to have the new hypercall tested and I manage to have 64 bit HVM working but the result is a collection of nasty hacks and supporting them properly is a different bigger task.
Surely a comment in the commit message on this is worth it.

Thanks.

Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Quoted 6 lines
Quoted 2 lines

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Yes, but my changes would have to have casts from const pointers to no-const pointers to avoid warnings and the patch you are pointing to would have to remove these casts. I find this less clean than having one patch using the current code style (that is no-const pointers) and another that changes the style entirely.
But obviously this is just my opinion.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

xen/common/memory.c static int acquire_resource( · line 1560

Quoted 2 lines
xen/common/memory.c +2 -0
+    xen_foreigncopy_t copy;
+    int rc, direction;

Plain int for rc is fine of course, but direction can't go negative, can it?

No, but the default type for constants is int and "flags" is promoted to int.

Doesn't really matter. Unsigned types want using for values which can only be non-negative. IF nothing else, then for consistency and doc purposes.

Changed.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

xen/common/memory.c static int acquire_resource( · line 1642

Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

Quoted 6 lines
Quoted 2 lines

Unfortunately that does not work.
The code is coherent with MMU_UPDATE.

How's that relevant? That's operating on page tables, when here we want to _prevent_ to copy into page tables (or descriptor ones, for that matter).

This new ABI is to better support migration.
We are migrating all the VM status including page tables... how can we not be able to write them but migrate them from one host to another ? You are basically explaining why changing the check the migration fails.

Quoted 27 lines
Quoted 27 lines
Quoted 13 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

Quoted 9 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Maybe:
This call is meant to replace expensive operations (mmap/copy/munmap) during migration which can only be issued from PV guests.

Quoted 7 lines
Quoted 1 line

You can migrate any domain. Just from a PV guest (this is not a regression).

Both Andrew and Roger confirm that this is supposed to work also from PVH Dom0 (not sure why you keep saying "guest"), and also used to work. If it doesn't, it would be a regression, and it would help if you supplied more detail on the observed failure.

Indeed I tested the migration of various domains (PV, HVM, PV-in-PVH), but only access to added hypercall from PV and HVM. I should add a test from a PVH guest.
I say guest because to test HVM I used a hack to allow all guests (not only dom0).

Quoted 34 lines
Quoted 34 lines
Quoted 19 lines
Quoted 6 lines
xen/include/public/memory.h +6 -0
+ */
+#define XENMEM_foreigncopy 29
+struct xen_foreigncopy {
+    /* IN - The domain whose memory is to be copied. */
+    domid_t domid;
+

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 762

Quoted 10 lines
xen/include/public/memory.h +10 -0
+    /* IN - Flags. */
+#define XENMEM_foreigncopy_from 0
+#define XENMEM_foreigncopy_to 1
+#define XENMEM_foreigncopy_direction 1
+    uint16_t flags;
+
+    /*
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the domain to be copied.

I think there's a comma wanted after "parameter".

Quoted 4 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 763

Quoted 1 line
xen/include/public/memory.h +1 -0
+     * On output updated number of frames left (0 if success).

I further think that adding "to" after "updated" would help here.

Updated to

    /*
     * IN/OUT
     *
     * As an IN parameter, number of frames of the domain to be copied.
     * On output updated to the number of frames left (0 if successful).
     */
    uint32_t nr_frames;
Quoted 8 lines
Quoted 5 lines

/*
* IN/OUT
*
* Frames to be copied.
* On output updated to the point to the first frame unhandled, if any.

Nit: There's now a stray "the".

Removed, thanks

Quoted 52 lines
Quoted 35 lines
Quoted 24 lines

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 573

Quoted 8 lines
xen/include/xsm/dummy.h +2 -0
@@ -569,6 +569,20 @@ static XSM_INLINE int cf_check xsm_map_gmfn_foreign(
     return xsm_default_action(action, d, t);
 }
 
+static XSM_INLINE int cf_check xsm_foreigncopy_from(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

I think these and ...

xen/include/xsm/dummy.h static XSM_INLINE int cf_check xsm_map_gmfn_foreign( · line 580

Quoted 7 lines
xen/include/xsm/dummy.h +7 -0
+{
+    XSM_ASSERT_ACTION(XSM_TARGET);
+    return xsm_default_action(action, d, t);
+}
+
+static XSM_INLINE int cf_check xsm_foreigncopy_to(
+    XSM_DEFAULT_ARG struct domain *d, struct domain *t)

... want to be pointer-to-const right away, requiring to re-base over "XSM:
make Argo hooks well-formed ones" (or alternatively requiring to split out the change there to xsm_default_action()). We really should avoid gaining
...

Quoted 10 lines

xen/include/xsm/hooks.h XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *) · line 62

Quoted 8 lines
xen/include/xsm/hooks.h +2 -0
@@ -58,6 +58,8 @@ XSM_HOOK(int, add_to_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, remove_from_physmap, struct domain *, struct domain *)
 XSM_HOOK(int, map_gmfn_foreign, struct domain *, struct domain *)
 XSM_HOOK(int, claim_pages, struct domain *)
+XSM_HOOK(int, foreigncopy_from, struct domain *, struct domain *);
+XSM_HOOK(int, foreigncopy_to, struct domain *, struct domain *);

... new hooks with non-const-correct parameters.

Quoted 7 lines

It would honestly make sense if they were all const.
However at the moment this would not be coherent with the current code. For instance these functions call "xsm_default_action" which does not accept constant domains (xen/include/xsm/dummy.h).
I think the most clean thing would be first to change xsm arguments to const first.
But it looks a bit out of scope with this PR.

I'm not asking that you tidy up other hooks. What I'm asking is that new hooks please be const-correct. Yet of course I'm not a maintainer of XSM, so Daniel may tell you otherwise.

Quoted 5 lines
Quoted 4 lines

Further: Why two new hooks? See e.g. "XSM: fold xsm_{,un}map_domain_pirq() hooks", "XSM: fold xsm_{,un}map_domain_irq() hooks", or "XSM: fold xsm_{,un}bind_pt_irq() hooks": We'd like to reduce the number of hooks, to reduce (when non-dummy XSM is in use) the number of cf_check entry points.

Quoted 5 lines
Quoted 1 line

It was a comment from Daniel.

Daniel, can you clarify this please?

Jan

Frediano

#16 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 27 lines · Frediano Ziglio
Quoted 26 lines · Jan Beulich
Quoted 24 lines · Frediano Ziglio
Quoted 9 lines · Jan Beulich
Quoted 7 lines · Frediano Ziglio
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
Quoted 4 lines

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

Yes, I was just going to reply.
I spent multiple days trying to implement the compat case or simply HVM support with an issue after the other:
- multiple distributions removed the 32 bit support so it was hard to have a setup;
- the original hypercall this PR is trying to optimise is supported only in PV (so no HVM or compat guests);
- migration and other operations can work only on PV (like dm_op operation) due to the usage of userspace handles used.

Quoted 6 lines
Quoted 2 lines

I don't understand how use of guest (not userspace) handles would get in the way of anything.

In this case userspace is not a typo. For HVM copy_from_user_hvm/copy_to_user_hvm are used and these functions accept only kernel space pointers.

I fear you've now completely lost me.

Quoted 32 lines
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Quoted 5 lines
Quoted 2 lines

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Quoted 6 lines

Yes, but my changes would have to have casts from const pointers to no-const pointers to avoid warnings and the patch you are pointing to would have to remove these casts. I find this less clean than having one patch using the current code style (that is no-const pointers) and another that changes the style entirely.
But obviously this is just my opinion.

Such casts would be unacceptable. What instead I have been trying to convey:
Your patch wants to gain a dependency on my patch. And if my patch would take too long to make it in, that one hunk could be broken out into a separate, easy to get in patch.

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 10 lines

xen/common/memory.c static int acquire_resource( · line 1642

Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

Unfortunately that does not work.
The code is coherent with MMU_UPDATE.

Quoted 7 lines
Quoted 2 lines

How's that relevant? That's operating on page tables, when here we want to _prevent_ to copy into page tables (or descriptor ones, for that matter).

This new ABI is to better support migration.
We are migrating all the VM status including page tables... how can we not be able to write them but migrate them from one host to another ? You are basically explaining why changing the check the migration fails.

No, what I'm trying to explain is that without such a check, you introduce a security issue (of privilege escalation kind). I hope you agree that we cannot knowingly allow such code to be committed.

To migrate-in page tables, you'd need to copy their contents before they obtain their PGT_l<N>_page_table type, so that upon being converted to page tables, they can be properly audited by the mm.c functions we have for that exact purpose.

Quoted 34 lines
Quoted 27 lines
Quoted 27 lines
Quoted 13 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

Quoted 9 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Maybe:
This call is meant to replace expensive operations (mmap/copy/munmap) during migration which can only be issued from PV guests.

Quoted 6 lines
Quoted 1 line

You can migrate any domain. Just from a PV guest (this is not a regression).

Both Andrew and Roger confirm that this is supposed to work also from PVH Dom0 (not sure why you keep saying "guest"), and also used to work. If it doesn't, it would be a regression, and it would help if you supplied more detail on the observed failure.

Quoted 5 lines

Indeed I tested the migration of various domains (PV, HVM, PV-in-PVH), but only access to added hypercall from PV and HVM. I should add a test from a PVH guest.
I say guest because to test HVM I used a hack to allow all guests (not only dom0).

And why would testing from PVH Dom0 not do?

Jan

#17 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 29 lines · Jan Beulich
Quoted 27 lines · Frediano Ziglio
Quoted 26 lines · Jan Beulich
Quoted 24 lines · Frediano Ziglio
Quoted 9 lines · Jan Beulich
Quoted 7 lines · Frediano Ziglio
---
 xen/common/memory.c         | 149 ++++++++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  45 ++++++++++-
 xen/include/xsm/dummy.h     |  14 ++++
 xen/include/xsm/hooks.h     |   2 +
 xen/xsm/flask/hooks.c       |  10 +++
 5 files changed, 219 insertions(+), 1 deletion(-)
Quoted 4 lines

As before: If you insist on not implementing the compat case, that decision wants justifying in the description. Without that it'll look like an oversight.

Yes, I was just going to reply.
I spent multiple days trying to implement the compat case or simply HVM support with an issue after the other:
- multiple distributions removed the 32 bit support so it was hard to have a setup;
- the original hypercall this PR is trying to optimise is supported only in PV (so no HVM or compat guests);
- migration and other operations can work only on PV (like dm_op operation) due to the usage of userspace handles used.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I don't understand how use of guest (not userspace) handles would get in the way of anything.

In this case userspace is not a typo. For HVM copy_from_user_hvm/copy_to_user_hvm are used and these functions accept only kernel space pointers.

I fear you've now completely lost me.

Try to pass a handle to a userspace page and the functions above will fail because they won't accept userspace pages.

In guest_walk_tables you have:

    if ( walk & PFEC_user_mode ) /* Requested a user access. */
    {
        if ( !(ar & _PAGE_USER) )
            /* Got a supervisor walk?  Unconditional fail. */
            goto out;

        if ( (walk & PFEC_write_access) && !(ar & _PAGE_RW) )
            /* Requested a write and only got a read? Fail. */
            goto out;
    }
    else /* Requested a supervisor access. */
    {
        if ( ar & _PAGE_USER ) /* Got a user walk. */
        {
            if ( (walk & PFEC_insn_fetch) && guest_smep_enabled(v) )
                /* User insn fetch and smep? Fail. */
                goto out;

            if ( !(walk & PFEC_insn_fetch) && guest_smap_enabled(v) &&
                 ((walk & PFEC_implicit) ||
                  !(guest_cpu_user_regs()->eflags & X86_EFLAGS_AC)) )
                /* User data access and smap? Fail. */
                goto out;
        }

        if ( (walk & PFEC_write_access) && !(ar & _PAGE_RW) &&
             guest_wp_enabled(v) )
            /* Requested a write, got a read, and CR0.WP is set? Fail. */
            goto out;
    }

and we don't have a PFEC_user_mode set.

Quoted 39 lines
Quoted 32 lines
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Quoted 5 lines
Quoted 2 lines

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Quoted 6 lines

Yes, but my changes would have to have casts from const pointers to no-const pointers to avoid warnings and the patch you are pointing to would have to remove these casts. I find this less clean than having one patch using the current code style (that is no-const pointers) and another that changes the style entirely.
But obviously this is just my opinion.

Quoted 5 lines

Such casts would be unacceptable. What instead I have been trying to convey:
Your patch wants to gain a dependency on my patch. And if my patch would take too long to make it in, that one hunk could be broken out into a separate, easy to get in patch.

Okay, then the only choice that's left is the code producing warnings as const pointers are passed to functions requiring no-const pointers. Is this acceptable? Apparently as you are suggesting it it is.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
xen/common/memory.c +5 -0
+            foreign = map_domain_page(foreign_mfn);
+            if ( direction == XENMEM_foreigncopy_from )
+                rc = copy_to_guest(copy.buffer, foreign, PAGE_SIZE);
+            else
+                rc = copy_from_guest(foreign, copy.buffer, PAGE_SIZE);
Quoted 7 lines
Quoted 4 lines

What I continue to be missing prior to this is the obtaining of a writable page ref. That's, as previously said, imperative for PV guests and at the very least advisable for HVM ones. (I really wonder how many more times I need to comment on this.)

Unfortunately that does not work.
The code is coherent with MMU_UPDATE.

How's that relevant? That's operating on page tables, when here we want to _prevent_ to copy into page tables (or descriptor ones, for that matter).

Quoted 9 lines
Quoted 4 lines

This new ABI is to better support migration.
We are migrating all the VM status including page tables... how can we not be able to write them but migrate them from one host to another ? You are basically explaining why changing the check the migration fails.

No, what I'm trying to explain is that without such a check, you introduce a security issue (of privilege escalation kind). I hope you agree that we cannot knowingly allow such code to be committed.

Then the security issue is already present in the code without my changes.

Quoted 5 lines

To migrate-in page tables, you'd need to copy their contents before they obtain their PGT_l<N>_page_table type, so that upon being converted to page tables, they can be properly audited by the mm.c functions we have for that exact purpose.

That makes sense.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 23 lines
Quoted 13 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 745

Quoted 10 lines
xen/include/public/memory.h +3 -1
@@ -740,7 +740,50 @@ struct xen_vnuma_topology_info {
 typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(xen_vnuma_topology_info_t);
 
-/* Next available subop number is 29 */
+/*
+ * Copy memory from/to a given domain.
+ * This calls is meant to replace expensive operations during migration which

Nit: "This call is ..." However, is ...

Quoted 9 lines

xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746

Quoted 1 line
xen/include/public/memory.h +1 -0
+ * are only supported for PV guests.

... this entire sentence really worth to have here (it looks more like something to have in the description)? For it to be possible to find if someone considered using those "expensive operations", I think it would need to be less vague and name those operations. Furthermore, if those other operations were supported only for PV guests, how would migration work for non-PV ones?

Quoted 5 lines

Maybe:
This call is meant to replace expensive operations (mmap/copy/munmap) during migration which can only be issued from PV guests.

You can migrate any domain. Just from a PV guest (this is not a regression).

Both Andrew and Roger confirm that this is supposed to work also from PVH Dom0 (not sure why you keep saying "guest"), and also used to work. If it doesn't, it would be a regression, and it would help if you supplied more detail on the observed failure.

Quoted 8 lines
Quoted 5 lines

Indeed I tested the migration of various domains (PV, HVM, PV-in-PVH), but only access to added hypercall from PV and HVM. I should add a test from a PVH guest.
I say guest because to test HVM I used a hack to allow all guests (not only dom0).

And why would testing from PVH Dom0 not do?

Just that it's easier for me testing from a different guest.

Quoted 1 line

Jan

Frediano

#18 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 45 lines · Frediano Ziglio
Quoted 44 lines · Jan Beulich
Quoted 42 lines · Frediano Ziglio
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Frediano Ziglio
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Frediano Ziglio
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Quoted 5 lines
Quoted 2 lines

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Yes, but my changes would have to have casts from const pointers to no-const pointers to avoid warnings and the patch you are pointing to would have to remove these casts. I find this less clean than having one patch using the current code style (that is no-const pointers) and another that changes the style entirely.
But obviously this is just my opinion.

Quoted 8 lines
Quoted 4 lines

Such casts would be unacceptable. What instead I have been trying to convey:
Your patch wants to gain a dependency on my patch. And if my patch would take too long to make it in, that one hunk could be broken out into a separate, easy to get in patch.

Okay, then the only choice that's left is the code producing warnings as const pointers are passed to functions requiring no-const pointers. Is this acceptable? Apparently as you are suggesting it it is.

That's not acceptable, the more that due to -Werror this would break the build. But that's also not what I said, and I'm having a hard time seeing how what I said can be mis-interpreted. What exactly is not clear in "Your patch wants to gain a dependency on my patch"?

Btw, I'm about to submit v2 of that XSM series, where I've broken out that hunk (for the change to then hopefully go in quickly, allowing you to simply re-base rather than carrying a prereq patch).

Jan

#19 lore raw

Re: [PATCH v10 7/10] xen: implement new foreign copy hypercall

Quoted 61 lines · Jan Beulich
Quoted 45 lines · Frediano Ziglio
Quoted 44 lines · Jan Beulich
Quoted 42 lines · Frediano Ziglio
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Frediano Ziglio
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Frediano Ziglio
xen/common/memory.c +0 -0
@@ -1548,6 +1548,141 @@ static int acquire_resource(
     return rc;
 }
 
Quoted 9 lines

xen/common/memory.c static int acquire_resource( · line 1555

Quoted 5 lines
xen/common/memory.c +5 -0
+/*
+ * The "noinline" qualifier avoids the compiler to create a large function
+ * consuming quite a lot of stack.
+ */
+static int noinline mem_foreigncopy(

I'm wondering: Is the "mem" prefix really meaningful for a static function in a file named memory.c?

Changed

Quoted 7 lines

xen/common/memory.c static int acquire_resource( · line 1558

Quoted 3 lines
xen/common/memory.c +3 -0
+    XEN_GUEST_HANDLE_PARAM(xen_foreigncopy_t) arg)
+{
+    struct domain *d, *const currd = current->domain;

With the comment on the new XSM hooks (below) in mind: currd wants to be pointer-to-const.

Quoted 5 lines
Quoted 2 lines

Just rebased on master, all XSM hooks accept no-const pointers to domains. So the suggested change would create warnings.

Well, as per below, I pointed you at a particular pending patch, a single hunk of which could be broken out.

Yes, but my changes would have to have casts from const pointers to no-const pointers to avoid warnings and the patch you are pointing to would have to remove these casts. I find this less clean than having one patch using the current code style (that is no-const pointers) and another that changes the style entirely.
But obviously this is just my opinion.

Quoted 8 lines
Quoted 4 lines

Such casts would be unacceptable. What instead I have been trying to convey:
Your patch wants to gain a dependency on my patch. And if my patch would take too long to make it in, that one hunk could be broken out into a separate, easy to get in patch.

Okay, then the only choice that's left is the code producing warnings as const pointers are passed to functions requiring no-const pointers. Is this acceptable? Apparently as you are suggesting it it is.

That's not acceptable, the more that due to -Werror this would break the build. But that's also not what I said, and I'm having a hard time seeing how what I said can be mis-interpreted. What exactly is not clear in "Your patch wants to gain a dependency on my patch"?

Quoted 5 lines

Btw, I'm about to submit v2 of that XSM series, where I've broken out that hunk (for the change to then hopefully go in quickly, allowing you to simply re-base rather than carrying a prereq patch).

Jan

Okay, being dependent on
https://lists.xenproject.org/archives/html/xen-devel/2026-08/msg00768.html
would be acceptable.
What was not was being dependent on other large series. That is more in line to what I proposed multiple times, that is having a preparation patch for constification and code using const pointer on my patch.

Frediano

#20 lore raw

PATCH v10 8/10 privcmd: Add definition for new Linux privcmd to access new Xen hypercall

In reply to Frediano Ziglio

Userspace should use new ioctl to access new hypercall.

---
Changes since v4:
- update comment.

---
 tools/include/xen-sys/Linux/privcmd.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
tools/include/xen-sys/Linux/privcmd.h +10 -0
index 607dfa2287..7a3c41308b 100644
@@ -100,6 +100,14 @@ typedef struct privcmd_pcidev_get_gsi {
 	__u32 gsi;
 } privcmd_pcidev_get_gsi_t;
 
+typedef struct privcmd_foreigncopy {
+	domid_t dom;          /* Foreign domain. */
+	__u16 dir;            /* Direction,  0 from, 1 to. */
+	__u32 num;            /* Number of pages to copy. */
+	const xen_pfn_t __user *pfns; /* Array of pfns. */
+	void __user *buffer;  /* Buffer to copy to/from. */
+} privcmd_foreigncopy_t;
+
 /*
  * @cmd: IOCTL_PRIVCMD_HYPERCALL
  * @arg: &privcmd_hypercall_t
@@ -121,6 +129,8 @@ typedef struct privcmd_pcidev_get_gsi {
 	_IOC(_IOC_NONE, 'P', 7, sizeof(privcmd_mmap_resource_t))
 #define IOCTL_PRIVCMD_PCIDEV_GET_GSI			\
 	_IOC(_IOC_NONE, 'P', 10, sizeof(privcmd_pcidev_get_gsi_t))
+#define IOCTL_PRIVCMD_FOREIGNCOPY				\
+	_IOWR('P', 11, privcmd_foreigncopy_t)
 #define IOCTL_PRIVCMD_UNIMPLEMENTED				\
 	_IOC(_IOC_NONE, 'P', 0xFF, 0)
 
Signature · 1 line
2.43.0
#21 lore raw

PATCH v10 9/10 libs/guest: use new hypercall if available

In reply to Frediano Ziglio

Use new hypercall if available, otherwise fall back to map+copy+unmap sequence.

I took some statistics while migrating some machines instrumenting the code to use new and old code and doing it 5 times in a row for each and the raw operation takes at least 4 (from) or 5 (to) times less.

Specifically for a test done with a machine with Intel Xeon Sapphire Rapids CPUs and migrating a Windows 10 machine with 12 GB of RAM the ratios were:
- 4.9 times faster copying from guest to dom0;
- 5.3 times faster copying to guest from dom0.
The test was repeated multiple times resulting consistent in all rans.

---
Changes since v4:
- use int8_t instead of char for signed type.

Changes since v6:
- add some statistics.

Changes since v9:
- fixed a pointer initialization.

---
 tools/libs/guest/xg_sr_common.c | 47 ++++++++++++++++++++++++++-------
 1 file changed, 38 insertions(+), 9 deletions(-)
tools/libs/guest/xg_sr_common.c +38 -9
index 90da21c35f..ce5026c707 100644
@@ -156,11 +156,6 @@ static void __attribute__((unused)) build_assertions(void)
     BUILD_BUG_ON(sizeof(struct xc_sr_rec_hvm_params)        != 8);
 }
 
-enum {
-    foreigncopy_from,
-    foreigncopy_to
-};
-
 static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,
                                  int dir, size_t nr_pages, void *buffer,
                                  const xen_pfn_t foreign_pfns[nr_pages])
@@ -174,8 +169,42 @@ static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,
         return -1;
     }
 
+    /*
+     * If foreign copy is supported, -1 not initialized, 0 not supported,
+     * 1 supported.
+     */
+    static int8_t foreign_copy_supported = -1;
+
+    if ( foreign_copy_supported )
+    {
+        int rc;
+        privcmd_foreigncopy_t copy = {
+            .dom = domid,
+            .dir = dir,
+            .num = nr_pages,
+            .buffer = buffer,
+        };
+        DECLARE_HYPERCALL_BOUNCE_IN(foreign_pfns, nr_pages * sizeof(xen_pfn_t));
+
+        if ( xc_hypercall_bounce_pre(xch, foreign_pfns) )
+            return -1;
+
+        copy.pfns = (xen_pfn_t *)HYPERCALL_BUFFER_AS_ARG(foreign_pfns);
+
+        rc = ioctl(xencall_fd(xch->xcall), IOCTL_PRIVCMD_FOREIGNCOPY, &copy);
+        if ( foreign_copy_supported < 0 )
+            foreign_copy_supported =
+                (!rc || (errno != ENOTTY && errno != ENOSYS));
+
+        xc_hypercall_bounce_post(xch, foreign_pfns);
+
+        if ( foreign_copy_supported )
+            return rc;
+    }
+
+    /* Fallback, emulate. */
     int err[nr_pages];
-    const int prot = (dir == foreigncopy_from) ? PROT_READ : PROT_READ|PROT_WRITE;
+    const int prot = (dir == XENMEM_foreigncopy_from) ? PROT_READ : PROT_READ|PROT_WRITE;
 
     void *p = xenforeignmemory_map(xch->fmem, domid, prot, nr_pages, foreign_pfns, err);
     if ( !p )
@@ -189,7 +218,7 @@ static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,
             return -1;
         }
 
-    if ( dir == foreigncopy_from )
+    if ( dir == XENMEM_foreigncopy_from )
         memcpy(buffer, p, nr_pages * XC_PAGE_SIZE);
     else
         memcpy(p, buffer, nr_pages * XC_PAGE_SIZE);
@@ -201,7 +230,7 @@ int xg_foreignmemory_copy_from(xc_interface *xch, domid_t dom,
                                size_t nr_pages, void *dest,
                                const xen_pfn_t source[nr_pages])
 {
-    return xg_foreignmemory_copy(xch, dom, foreigncopy_from,
+    return xg_foreignmemory_copy(xch, dom, XENMEM_foreigncopy_from,
                                  nr_pages, dest, source);
 }
 
@@ -209,7 +238,7 @@ int xg_foreignmemory_copy_to(xc_interface *xch, domid_t dom,
                              size_t nr_pages, const xen_pfn_t dest[nr_pages],
                              const void *source)
 {
-    return xg_foreignmemory_copy(xch, dom, foreigncopy_to,
+    return xg_foreignmemory_copy(xch, dom, XENMEM_foreigncopy_to,
                                  nr_pages, (void *) source, dest);
 }
 
Signature · 1 line
2.43.0
#22 lore raw

PATCH v6 10/10 xen/privcmd: Add new ABI to allow copying foreign memory

In reply to Frediano Ziglio

This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.

--
Changes since v4:
- fix wrong assign;
- use set_xen_guest_handle to set handle;
- wrap slow hypercall with xen_preemptible_hcall_{begin,end};
- use _IOWR for ioctl code to be more specific;
- use __copy_to_user if buffer already checked.

Changes since v6:
- compatibility ARM/x86.

---
 arch/x86/include/asm/xen/interface.h |  3 ++
 drivers/xen/privcmd.c                | 49 ++++++++++++++++++++++++++++
 include/uapi/xen/privcmd.h           | 10 ++++++
 include/xen/arm/interface.h          |  2 ++
 include/xen/interface/memory.h       | 37 +++++++++++++++++++++
 5 files changed, 101 insertions(+)
arch/x86/include/asm/xen/interface.h +3 -0
index a078a2b0f032..fc76fac8fb16 100644
@@ -59,6 +59,7 @@
 #elif defined(__x86_64__)
 #define set_xen_guest_handle(hnd, val)	do { (hnd).p = val; } while (0)
 #endif
+#define get_xen_guest_handle(hnd) ((hnd).p)
 #else
 #if defined(__i386__)
 #define set_xen_guest_handle(hnd, val)			\
@@ -70,6 +71,7 @@
 #elif defined(__x86_64__)
 #define set_xen_guest_handle(hnd, val)	do { (hnd) = val; } while (0)
 #endif
+#define get_xen_guest_handle(hnd) (hnd)
 #endif
 
 #ifndef __ASSEMBLER__
@@ -91,6 +93,7 @@ DEFINE_GUEST_HANDLE(int);
 DEFINE_GUEST_HANDLE(void);
 DEFINE_GUEST_HANDLE(uint64_t);
 DEFINE_GUEST_HANDLE(uint32_t);
+DEFINE_GUEST_HANDLE(uint8_t);
 DEFINE_GUEST_HANDLE(xen_pfn_t);
 DEFINE_GUEST_HANDLE(xen_ulong_t);
 #endif
drivers/xen/privcmd.c +49 -0

61 lines not on this page · raw message

include/uapi/xen/privcmd.h +10 -0
index 8e2c8fd44764..993b501e35bf 100644
@@ -131,6 +131,14 @@ struct privcmd_pcidev_get_gsi {
 	__u32 gsi;
 };
 
+struct privcmd_foreigncopy {
+	domid_t dom;		/* foreign domain */
+	__u16 dir;		/* direction,  0 from, 1 to */
+	__u32 num;		/* number of pages to copy */
+	const xen_pfn_t __user *pfns;	/* array of pfns */
+	void __user *buffer;	/* buffer to copy to/from */
+};
+
 /*
  * @cmd: IOCTL_PRIVCMD_HYPERCALL
  * @arg: &privcmd_hypercall_t
@@ -164,5 +172,7 @@ struct privcmd_pcidev_get_gsi {
 	_IOW('P', 9, struct privcmd_ioeventfd)
 #define IOCTL_PRIVCMD_PCIDEV_GET_GSI				\
 	_IOC(_IOC_NONE, 'P', 10, sizeof(struct privcmd_pcidev_get_gsi))
+#define IOCTL_PRIVCMD_FOREIGNCOPY				\
+	_IOWR('P', 11, struct privcmd_foreigncopy)
 
 #endif /* __LINUX_PUBLIC_PRIVCMD_H__ */
include/xen/arm/interface.h +2 -0
index 61360b89da40..20ee1ed44436 100644
@@ -27,6 +27,7 @@
 			*(uint64_t *)&(hnd) = 0;	\
 		(hnd).p = val;				\
 	} while (0)
+#define get_xen_guest_handle(hnd) ((hnd).p)
 
 #define __HYPERVISOR_platform_op_raw __HYPERVISOR_platform_op
 
@@ -53,6 +54,7 @@ DEFINE_GUEST_HANDLE(int);
 DEFINE_GUEST_HANDLE(void);
 DEFINE_GUEST_HANDLE(uint64_t);
 DEFINE_GUEST_HANDLE(uint32_t);
+DEFINE_GUEST_HANDLE(uint8_t);
 DEFINE_GUEST_HANDLE(xen_pfn_t);
 DEFINE_GUEST_HANDLE(xen_ulong_t);
 
include/xen/interface/memory.h +37 -0
index 1a371a825c55..5981402fccde 100644
@@ -325,4 +325,41 @@ struct xen_mem_acquire_resource {
 };
 DEFINE_GUEST_HANDLE_STRUCT(xen_mem_acquire_resource);
 
+/*
+ * Copy memory from/to a given domain.
+ */
+#define XENMEM_foreigncopy 29
+struct xen_foreigncopy {
+    /* IN - The domain whose resource is to be copied */
+    domid_t domid;
+
+    /* IN - Flags */
+#define XENMEM_foreigncopy_from 0
+#define XENMEM_foreigncopy_to 1
+#define XENMEM_foreigncopy_direction 1
+    uint16_t flags;
+
+    /*
+     * IN
+     *
+     * As an IN parameter number of frames of the domain to be copied.
+     */
+    uint32_t nr_frames;
+
+    /*
+     * IN
+     *
+     * Frames to be copied.
+     */
+    GUEST_HANDLE(xen_pfn_t) frame_list;
+
+    /*
+     * IN/OUT
+     *
+     * Userspace buffer to read/write from.
+     */
+    GUEST_HANDLE(uint8_t) buffer;
+};
+DEFINE_GUEST_HANDLE_STRUCT(xen_foreigncopy);
+
 #endif /* __XEN_PUBLIC_MEMORY_H__ */
Signature · 1 line
2.54.0
#23 lore raw

Re: [PATCH Linux v6 10/10] xen/privcmd: Add new ABI to allow copying foreign memory

Quoted 7 lines · Frediano Ziglio

This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.

Typo in subject line, it's v10, not v6.

Frediano

#24 lore raw
In reply to Frediano Ziglio
Quoted 5 lines · Frediano Ziglio

Edwin Török (3):
libs/call: cache up to 4 pages in hypercall bounce buffers libs/guest: allocate various migration arrays just once
Frediano Ziglio (6):
libs/guest: move batch_pfns into a separate structure

I've committed these first three patches.

I've notice that git am will put the wrong author on your patch, it would use the gmail addr instead of the citrix one. Could you fix your config so that git does the right thing? I think it is just a matter of changing the config:

git config sendemail.from 'Frediano Ziglio <[email protected]>'

(with maybe --global or with an identity added to the config option if you use that).

This will tell git that the email is send from a different email address and it will format the patch appropriately. Otherwise, gmail just overwrite the "From:" without telling git. The original "From:" is still available in "X-Google-Original-From:" but it's not something git should care about.

Cheers,

--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#25 lore raw
Quoted 10 lines · Anthony PERARD
Quoted 5 lines · Frediano Ziglio

Edwin Török (3):
libs/call: cache up to 4 pages in hypercall bounce buffers libs/guest: allocate various migration arrays just once
Frediano Ziglio (6):
libs/guest: move batch_pfns into a separate structure

I've committed these first three patches.

Thanks.

Quoted 7 lines

I've notice that git am will put the wrong author on your patch, it would use the gmail addr instead of the citrix one. Could you fix your config so that git does the right thing? I think it is just a matter of changing the config:

git config sendemail.from 'Frediano Ziglio <[email protected]>'

Quoted 9 lines

(with maybe --global or with an identity added to the config option if you use that).

This will tell git that the email is send from a different email address and it will format the patch appropriately. Otherwise, gmail just overwrite the "From:" without telling git. The original "From:" is still available in "X-Google-Original-From:" but it's not something git should care about.

Done. Yes, it looks like it is working. An additional "From; xxx" line is added at the beginning of the body.

Quoted 3 lines

Cheers,

Frediano