Skip to contentSkip to conversation

xenguest optimisations

PATCH Superseded Frediano Ziglio 3 participants 10 messages Jul 13 – Jul 20

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v9 cover xenguest optimisations

Reduce number of allocations sending memory state.

Changes since v1:
- add commit to cache up to 4 pages in hypercall;
- add other 2 commits reducing chunks passed to write/writev.

Changes since v2:
- update patches commit prefixes;
- add other 2 optisations.

Changes since v3:
- address some comments;
- add patches for foreign copy optimisation.

Changes since v4:
- added Reviewed-by;
- improved commit messages;
- other minor fixes, see individual commits.

Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc;
- do not overwrite errno if xenforeignmemory_map fails;
- lot of changes to "implement new foreign copy hypercall", see specific commit.

Changes since v6:
- removed merged patch;
- keep only optimization commits for now;
- improve comments;
- merged "fill directly iov structure collapsing them" and moved it;
- split "allocate various migration arrays just once";
- add a commit for memory checks using Valgrind.

Changes since v7:
- removed merged commits;
- minor style fixes.

Changes since v8:
- added Reviewed-by;
- remove useless check;
- remove useless memset;
- initialize variables while declaring them.

Edwin Török (2):
libs/call: cache up to 4 pages in hypercall bounce buffers libs/guest: allocate various migration arrays just once

Frediano Ziglio (2):
libs/guest: move batch_pfns into a separate structure libs/guest: use Valgrind to detect various buffer overflows

 tools/libs/call/buffer.c        | 34 ++++++++-----
 tools/libs/call/core.c          |  3 +-
 tools/libs/call/private.h       |  8 ++--
 tools/libs/ctrl/xc_private.h    |  8 ++++
 tools/libs/guest/xg_sr_common.h | 19 +++++++-
 tools/libs/guest/xg_sr_save.c   | 84 ++++++++++++++++-----------------
 6 files changed, 98 insertions(+), 58 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v9 1/4 libs/call: cache up to 4 pages in hypercall bounce buffers

From: Edwin Török <[email protected]>

During migration there are a lot of mmap/munmap calls, because xc_get_pfn_type_batch() exceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.

munmap() is slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.

Eventually it'd be good if the memory pool from xmalloc_tlsf.c was reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).

---
Changes since v2:
- change prefix in subject.

Changes since v4:
- fix off-by-one bug.

Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc.

Changes since v6:
- align changes made to cache_alloc to cache_free.

Changes since v7:
- use "unsigned int" instead of "unsigned".

Changes since v8:
- added Reviewed-by.

---
 tools/libs/call/buffer.c  | 34 +++++++++++++++++++++++-----------
 tools/libs/call/core.c    |  3 ++-
 tools/libs/call/private.h |  8 +++++---
 3 files changed, 30 insertions(+), 15 deletions(-)
tools/libs/call/buffer.c +23 -11
index 155e4f9d43..b7d00185c4 100644
@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)
 {
     void *p = NULL;
 
+    if ( nr_pages == 0 )
+        return NULL;
+
     cache_lock(xcall);
 
     xcall->buffer_total_allocations++;
@@ -56,13 +59,13 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)
     if ( xcall->buffer_current_allocations > xcall->buffer_maximum_allocations )
         xcall->buffer_maximum_allocations = xcall->buffer_current_allocations;
 
-    if ( nr_pages > 1 )
+    if ( nr_pages > ARRAY_SIZE(xcall->buffer_cache) )
     {
         xcall->buffer_cache_toobig++;
     }
-    else if ( xcall->buffer_cache_nr > 0 )
+    else if ( xcall->buffer_cache_nr[nr_pages-1] > 0 )
     {
-        p = xcall->buffer_cache[--xcall->buffer_cache_nr];
+        p = xcall->buffer_cache[nr_pages-1][--xcall->buffer_cache_nr[nr_pages-1]];
         xcall->buffer_cache_hits++;
     }
     else
@@ -79,15 +82,18 @@ static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages)
 {
     int rc = 0;
 
+    if ( nr_pages == 0 )
+        return 0;
+
     cache_lock(xcall);
 
     xcall->buffer_total_releases++;
     xcall->buffer_current_allocations--;
 
-    if ( nr_pages == 1 &&
-         xcall->buffer_cache_nr < BUFFER_CACHE_SIZE )
+    if ( nr_pages && nr_pages <= ARRAY_SIZE(xcall->buffer_cache) &&
+         xcall->buffer_cache_nr[nr_pages-1] < BUFFER_CACHE_SIZE )
     {
-        xcall->buffer_cache[xcall->buffer_cache_nr++] = p;
+        xcall->buffer_cache[nr_pages-1][xcall->buffer_cache_nr[nr_pages-1]++] = p;
         rc = 1;
     }
 
@@ -108,17 +114,23 @@ void buffer_release_cache(xencall_handle *xcall)
     DBGPRINTF("current allocations:%d maximum allocations:%d",
               xcall->buffer_current_allocations,
               xcall->buffer_maximum_allocations);
-    DBGPRINTF("cache current size:%d",
-              xcall->buffer_cache_nr);
+    for ( unsigned int i = 0; i < ARRAY_SIZE(xcall->buffer_cache_nr); ++i )
+    {
+        DBGPRINTF("cache current size[%u pages]:%d", i+1,
+                xcall->buffer_cache_nr[i]);
+    }
     DBGPRINTF("cache hits:%d misses:%d toobig:%d",
               xcall->buffer_cache_hits,
               xcall->buffer_cache_misses,
               xcall->buffer_cache_toobig);
 
-    while ( xcall->buffer_cache_nr > 0 )
+    for ( unsigned int i = 0; i < ARRAY_SIZE(xcall->buffer_cache_nr); ++i )
     {
-        p = xcall->buffer_cache[--xcall->buffer_cache_nr];
-        osdep_free_pages(xcall, p, 1);
+        while ( xcall->buffer_cache_nr[i] > 0 )
+        {
+            p = xcall->buffer_cache[i][--xcall->buffer_cache_nr[i]];
+            osdep_free_pages(xcall, p, i + 1);
+        }
     }
 
     cache_unlock(xcall);
tools/libs/call/core.c +2 -1
index 02c4f8e1ae..dd8877c1a0 100644
@@ -14,6 +14,7 @@
  */
 
 #include <stdlib.h>
+#include <string.h>
 
 #include "private.h"
 
@@ -44,7 +45,7 @@ xencall_handle *xencall_open(xentoollog_logger *logger, unsigned open_flags)
     xentoolcore__register_active_handle(&xcall->tc_ah);
 
     xcall->flags = open_flags;
-    xcall->buffer_cache_nr = 0;
+    memset(xcall->buffer_cache_nr, 0, sizeof(xcall->buffer_cache_nr));
 
     xcall->buffer_total_allocations = 0;
     xcall->buffer_total_releases = 0;
tools/libs/call/private.h +5 -3
index 9c3aa432ef..8e6a208975 100644
@@ -31,13 +31,15 @@ struct xencall_handle {
     Xentoolcore__Active_Handle tc_ah;
 
     /*
-     * A simple cache of unused, single page, hypercall buffers
+     * A simple cache of unused, small, hypercall buffers
+     * buffer_cache[i]'s size is (i+1) pages
      *
      * Protected by a global lock.
      */
 #define BUFFER_CACHE_SIZE 4
-    int buffer_cache_nr;
-    void *buffer_cache[BUFFER_CACHE_SIZE];
+#define BUFFER_CACHE_NRPAGES 4
+    int buffer_cache_nr[BUFFER_CACHE_NRPAGES];
+    void *buffer_cache[BUFFER_CACHE_NRPAGES][BUFFER_CACHE_SIZE];
 
     /*
      * Hypercall buffer statistics. All protected by the global
Signature · 1 line
2.43.0
#2 lore raw

PATCH v9 2/4 libs/guest: move batch_pfns into a separate structure

In reply to Frediano Ziglio

Preparation for a followup patch "libs/guest: allocate various migration arrays just once".

---
Changes since v6:
- split from "libs/guest: allocate various migration arrays just once".

Changes since v7:
- initialize "batch_pfns" on declaration.

Changes since v8:
- remove useless check;
- added Reviewed-by.

---
 tools/libs/guest/xg_sr_common.h |  5 ++++-
 tools/libs/guest/xg_sr_save.c   | 28 ++++++++++++++--------------
 2 files changed, 18 insertions(+), 15 deletions(-)
tools/libs/guest/xg_sr_common.h +4 -1
index f1573aefcb..7574c9f5b6 100644
@@ -239,11 +239,14 @@ struct xc_sr_context
 
             struct precopy_stats stats;
 
-            xen_pfn_t *batch_pfns;
             unsigned int nr_batch_pfns;
             unsigned long *deferred_pages;
             unsigned long nr_deferred_pages;
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
+            struct xc_sr_context_save_buffers
+            {
+                xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+            } *buffers;
         } save;
 
         struct /* Restore data. */
tools/libs/guest/xg_sr_save.c +14 -14
index 84fdbe4140..22348db445 100644
@@ -75,7 +75,7 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)
 
 /*
  * Writes a batch of memory as a PAGE_DATA record into the stream.  The batch
- * is constructed in ctx->save.batch_pfns.
+ * is constructed in ctx->save.buffers->batch_pfns.
  *
  * This function:
  * - gets the types for each pfn in the batch.
@@ -95,6 +95,7 @@ static int write_batch(struct xc_sr_context *ctx)
     void *page, *orig_page;
     uint64_t *rec_pfns = NULL;
     struct iovec *iov = NULL; int iovcnt = 0;
+    xen_pfn_t *const batch_pfns = ctx->save.buffers->batch_pfns;
     struct {
         struct xc_sr_rhdr rec;
         struct xc_sr_rec_page_data_header page_data;
@@ -110,6 +111,7 @@ static int write_batch(struct xc_sr_context *ctx)
     };
 
     assert(nr_pfns != 0);
+    assert(nr_pfns <= MAX_BATCH_SIZE);
 
     /* Mfns of the batch pfns. */
     mfns = malloc(nr_pfns * sizeof(*mfns));
@@ -141,13 +143,12 @@ static int write_batch(struct xc_sr_context *ctx)
 
     for ( i = 0; i < nr_pfns; ++i )
     {
-        types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx,
-                                                      ctx->save.batch_pfns[i]);
+        types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx, batch_pfns[i]);
 
         /* Likely a ballooned page. */
         if ( mfns[i] == INVALID_MFN )
         {
-            set_bit(ctx->save.batch_pfns[i], ctx->save.deferred_pages);
+            set_bit(batch_pfns[i], ctx->save.deferred_pages);
             ++ctx->save.nr_deferred_pages;
         }
     }
@@ -193,7 +194,7 @@ static int write_batch(struct xc_sr_context *ctx)
             if ( errors[p] )
             {
                 ERROR("Mapping of pfn %#"PRIpfn" (mfn %#"PRIpfn") failed %d",
-                      ctx->save.batch_pfns[i], mfns[p], errors[p]);
+                      batch_pfns[i], mfns[p], errors[p]);
                 goto err;
             }
 
@@ -207,7 +208,7 @@ static int write_batch(struct xc_sr_context *ctx)
             {
                 if ( rc == -1 && errno == EAGAIN )
                 {
-                    set_bit(ctx->save.batch_pfns[i], ctx->save.deferred_pages);
+                    set_bit(batch_pfns[i], ctx->save.deferred_pages);
                     ++ctx->save.nr_deferred_pages;
                     types[i] = XEN_DOMCTL_PFINFO_XTAB;
                     --nr_pages;
@@ -235,7 +236,7 @@ static int write_batch(struct xc_sr_context *ctx)
     hdrs.rec.length += nr_pages * PAGE_SIZE;
 
     for ( i = 0; i < nr_pfns; ++i )
-        rec_pfns[i] = ((uint64_t)(types[i]) << 32) | ctx->save.batch_pfns[i];
+        rec_pfns[i] = ((uint64_t)(types[i]) << 32) | batch_pfns[i];
 
     if ( writev_exact(ctx->fd, iov, iovcnt) )
     {
@@ -274,9 +275,9 @@ static int flush_batch(struct xc_sr_context *ctx)
 
     if ( !rc )
     {
-        VALGRIND_MAKE_MEM_UNDEFINED(ctx->save.batch_pfns,
+        VALGRIND_MAKE_MEM_UNDEFINED(ctx->save.buffers->batch_pfns,
                                     MAX_BATCH_SIZE *
-                                    sizeof(*ctx->save.batch_pfns));
+                                    sizeof(*ctx->save.buffers->batch_pfns));
     }
 
     return rc;
@@ -293,7 +294,7 @@ static int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn)
         rc = flush_batch(ctx);
 
     if ( rc == 0 )
-        ctx->save.batch_pfns[ctx->save.nr_batch_pfns++] = pfn;
+        ctx->save.buffers->batch_pfns[ctx->save.nr_batch_pfns++] = pfn;
 
     return rc;
 }
@@ -784,11 +785,10 @@ static int setup(struct xc_sr_context *ctx)
 
     dirty_bitmap = xc_hypercall_buffer_alloc_pages(
         xch, dirty_bitmap, NRPAGES(bitmap_size(ctx->save.p2m_size)));
-    ctx->save.batch_pfns = malloc(MAX_BATCH_SIZE *
-                                  sizeof(*ctx->save.batch_pfns));
     ctx->save.deferred_pages = bitmap_alloc(ctx->save.p2m_size);
+    ctx->save.buffers = calloc(1, sizeof(*ctx->save.buffers));
 
-    if ( !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages )
+    if ( !ctx->save.buffers || !dirty_bitmap || !ctx->save.deferred_pages )
     {
         ERROR("Unable to allocate memory for dirty bitmaps, batch pfns and"
               " deferred pages");
@@ -819,7 +819,7 @@ static void cleanup(struct xc_sr_context *ctx)
     xc_hypercall_buffer_free_pages(xch, dirty_bitmap,
                                    NRPAGES(bitmap_size(ctx->save.p2m_size)));
     free(ctx->save.deferred_pages);
-    free(ctx->save.batch_pfns);
+    free(ctx->save.buffers);
 }
 
 /*
Signature · 1 line
2.43.0
#3 lore raw

PATCH v9 3/4 libs/guest: allocate various migration arrays just once

In reply to Frediano Ziglio

From: Edwin Török <[email protected]>

Allocate these array just once at the start of migration, using the maximum batch size, and free them at the end.

---
Changes since v2:
- change prefix in subject.

Changes since v3:
- fix comment style

Changes since v4:
- change order of fields in structure.

Changes since v6:
- split preparation commit.

Changes since v8:
- remove useless memset;
- initialize variables while declaring them.

---
 tools/libs/guest/xg_sr_common.h |  6 +++++
 tools/libs/guest/xg_sr_save.c   | 45 ++++++++++++---------------------
 2 files changed, 22 insertions(+), 29 deletions(-)
tools/libs/guest/xg_sr_common.h +6 -0
index 7574c9f5b6..c07c6db59e 100644
@@ -246,6 +246,12 @@ struct xc_sr_context
             struct xc_sr_context_save_buffers
             {
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                xen_pfn_t mfns[MAX_BATCH_SIZE];
+                xen_pfn_t types[MAX_BATCH_SIZE];
+                void *local_pages[MAX_BATCH_SIZE];
+                struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                uint64_t rec_pfns[MAX_BATCH_SIZE];
+                int errors[MAX_BATCH_SIZE];
             } *buffers;
         } save;
 
tools/libs/guest/xg_sr_save.c +16 -29

81 lines not on this page · raw message

Signature · 1 line
2.43.0
Reviewed-by on v9 3/4
#4 lore raw

Re: [PATCH v9 3/4] libs/guest: allocate various migration arrays just once

Quoted 7 lines · Frediano Ziglio

From: Edwin Török <[email protected]>

Allocate these array just once at the start of migration, using the maximum batch size, and free them at the end.

Thanks,

--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#5 lore raw

PATCH v9 4/4 libs/guest: use Valgrind to detect various buffer overflows

In reply to Frediano Ziglio

Previously this was done as buffers were allocated separately.

---
 tools/libs/ctrl/xc_private.h    |  8 ++++++++
 tools/libs/guest/xg_sr_common.h |  8 ++++++++
 tools/libs/guest/xg_sr_save.c   | 13 +++++++++++++
 3 files changed, 29 insertions(+)
tools/libs/ctrl/xc_private.h +8 -0
index 8a325c17b0..bef96521c3 100644
@@ -45,8 +45,16 @@
 #if defined(HAVE_VALGRIND_MEMCHECK_H) && !defined(NDEBUG) && !defined(__MINIOS__)
 /* Compile in Valgrind client requests? */
 #include <valgrind/memcheck.h>
+#define MEM_NOACCESS_BUFFER(name, size) uint8_t name[size];
+#define MEM_NOACCESS_INIT(field) \
+    VALGRIND_MAKE_MEM_NOACCESS(field, sizeof(field))
+#define MEM_UNDEFINED_INIT(field) \
+    VALGRIND_MAKE_MEM_UNDEFINED(field, sizeof(field))
 #else
 #define VALGRIND_MAKE_MEM_UNDEFINED(addr, len) /* addr, len */
+#define MEM_NOACCESS_BUFFER(name, size)
+#define MEM_NOACCESS_INIT(field) do {} while(0)
+#define MEM_UNDEFINED_INIT(field) do {} while(0)
 #endif
 
 #if defined(__MINIOS__)
tools/libs/guest/xg_sr_common.h +8 -0
index c07c6db59e..d3fc7f363e 100644
@@ -245,13 +245,21 @@ struct xc_sr_context
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
             struct xc_sr_context_save_buffers
             {
+                MEM_NOACCESS_BUFFER(na0, 16);
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 16);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 16);
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 16);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na4, 16);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na5, 16);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na6, 16);
                 int errors[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na7, 16);
             } *buffers;
         } save;
 
tools/libs/guest/xg_sr_save.c +13 -0
index 6a77e33a47..25561e369f 100644
@@ -123,6 +123,11 @@ static int write_batch(struct xc_sr_context *ctx)
     assert(nr_pfns != 0);
     assert(nr_pfns <= MAX_BATCH_SIZE);
 
+    MEM_UNDEFINED_INIT(ctx->save.buffers->mfns);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->types);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->iov);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->rec_pfns);
+
     iov[0].iov_base = &hdrs;
     iov[0].iov_len = sizeof(hdrs);
 
@@ -783,6 +788,14 @@ static int setup(struct xc_sr_context *ctx)
         errno = ENOMEM;
         goto err;
     }
+    MEM_NOACCESS_INIT(ctx->save.buffers->na0);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na1);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na2);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na3);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na4);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na5);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na6);
+    MEM_NOACCESS_INIT(ctx->save.buffers->na7);
 
     rc = 0;
 
Signature · 1 line
2.43.0
#6 lore raw

Re: [PATCH v9 4/4] libs/guest: use Valgrind to detect various buffer overflows

Quoted 6 lines · Frediano Ziglio
tools/libs/ctrl/xc_private.h +0 -0
@@ -45,8 +45,16 @@
 #if defined(HAVE_VALGRIND_MEMCHECK_H) && !defined(NDEBUG) && !defined(__MINIOS__)
 /* Compile in Valgrind client requests? */
 #include <valgrind/memcheck.h>

tools/libs/ctrl/xc_private.h line 57

Quoted 10 lines
tools/libs/ctrl/xc_private.h +8 -0
+#define MEM_NOACCESS_BUFFER(name, size) uint8_t name[size];
+#define MEM_NOACCESS_INIT(field) \
+    VALGRIND_MAKE_MEM_NOACCESS(field, sizeof(field))
+#define MEM_UNDEFINED_INIT(field) \
+    VALGRIND_MAKE_MEM_UNDEFINED(field, sizeof(field))
 #else
 #define VALGRIND_MAKE_MEM_UNDEFINED(addr, len) /* addr, len */
+#define MEM_NOACCESS_BUFFER(name, size)
+#define MEM_NOACCESS_INIT(field) do {} while(0)
+#define MEM_UNDEFINED_INIT(field) do {} while(0)

Why the _INIT suffix in the macros? It looks like something is initialised, but that's not the case.

We could follow the valgrind naming and do, while using something different than "mem" to say we don't need a size:
make_buffer_noaccess and make_buffer_undefined

Or
mark_buffer_as_noaccess
mark_buffer_as_undefined

tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 248

Quoted 12 lines
tools/libs/ctrl/xc_private.h +0 -0
 #endif
 
 #if defined(__MINIOS__)
tools/libs/guest/xg_sr_common.h +1 -0
index c07c6db59e..d3fc7f363e 100644
@@ -245,13 +245,21 @@ struct xc_sr_context
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
             struct xc_sr_context_save_buffers
             {
+                MEM_NOACCESS_BUFFER(na0, 16);

This first redzone buffer and the last (na7) one looks unnecessary, as they are before the beginning of the buffer, and after the end.

Quoted 22 lines
tools/libs/guest/xg_sr_common.h +7 -0
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 16);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 16);
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 16);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na4, 16);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na5, 16);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na6, 16);
                 int errors[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na7, 16);
             } *buffers;
         } save;
 
tools/libs/guest/xg_sr_save.c +0 -0
index 6a77e33a47..25561e369f 100644

tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 129

Quoted 7 lines
tools/libs/guest/xg_sr_save.c +4 -0
@@ -123,6 +123,11 @@ static int write_batch(struct xc_sr_context *ctx)
     assert(nr_pfns != 0);
     assert(nr_pfns <= MAX_BATCH_SIZE);
 
+    MEM_UNDEFINED_INIT(ctx->save.buffers->mfns);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->types);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->iov);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->rec_pfns);

Why is errors not also marked as undefined?

To bad we can't really test this patch, beside check that it build.

Thanks,

--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#7 lore raw

Re: [PATCH v9 4/4] libs/guest: use Valgrind to detect various buffer overflows

Quoted 13 lines · Anthony PERARD
Quoted 11 lines · Frediano Ziglio
tools/libs/ctrl/xc_private.h +5 -0
@@ -45,8 +45,16 @@
 #if defined(HAVE_VALGRIND_MEMCHECK_H) && !defined(NDEBUG) && !defined(__MINIOS__)
 /* Compile in Valgrind client requests? */
 #include <valgrind/memcheck.h>
+#define MEM_NOACCESS_BUFFER(name, size) uint8_t name[size];
+#define MEM_NOACCESS_INIT(field) \
+    VALGRIND_MAKE_MEM_NOACCESS(field, sizeof(field))
+#define MEM_UNDEFINED_INIT(field) \
+    VALGRIND_MAKE_MEM_UNDEFINED(field, sizeof(field))
Quoted 9 lines

tools/libs/ctrl/xc_private.h line 57

Quoted 5 lines
tools/libs/ctrl/xc_private.h +3 -0
 #else
 #define VALGRIND_MAKE_MEM_UNDEFINED(addr, len) /* addr, len */
+#define MEM_NOACCESS_BUFFER(name, size)
+#define MEM_NOACCESS_INIT(field) do {} while(0)
+#define MEM_UNDEFINED_INIT(field) do {} while(0)

Why the _INIT suffix in the macros? It looks like something is initialised, but that's not the case.

It initializes the memory to be not accessible.

Quoted 8 lines

We could follow the valgrind naming and do, while using something different than "mem" to say we don't need a size:
make_buffer_noaccess and make_buffer_undefined

Or
mark_buffer_as_noaccess
mark_buffer_as_undefined

It makes sense, but I wanted to be able to remove the buffers entirely if debug is not enabled.

Quoted 17 lines

tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 248

Quoted 13 lines
tools/libs/ctrl/xc_private.h +0 -0
 #endif
 
 #if defined(__MINIOS__)
tools/libs/guest/xg_sr_common.h +1 -0
index c07c6db59e..d3fc7f363e 100644
@@ -245,13 +245,21 @@ struct xc_sr_context
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
             struct xc_sr_context_save_buffers
             {
+                MEM_NOACCESS_BUFFER(na0, 16);

This first redzone buffer and the last (na7) one looks unnecessary, as they are before the beginning of the buffer, and after the end.

Before for buffer underflow, after for buffer overflow, but I suppose you mean that Valgrind/ASAN already add them anyway so it's not really necessary.
Yes, I'll remove.

Quoted 25 lines
Quoted 25 lines
tools/libs/guest/xg_sr_common.h +7 -0
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 16);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 16);
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 16);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na4, 16);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na5, 16);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na6, 16);
                 int errors[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na7, 16);
             } *buffers;
         } save;
 
tools/libs/guest/xg_sr_save.c +0 -0
index 6a77e33a47..25561e369f 100644
@@ -123,6 +123,11 @@ static int write_batch(struct xc_sr_context *ctx)
     assert(nr_pfns != 0);
     assert(nr_pfns <= MAX_BATCH_SIZE);
 
Quoted 7 lines

tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 129

Quoted 4 lines
tools/libs/guest/xg_sr_save.c +4 -0
+    MEM_UNDEFINED_INIT(ctx->save.buffers->mfns);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->types);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->iov);
+    MEM_UNDEFINED_INIT(ctx->save.buffers->rec_pfns);

Why is errors not also marked as undefined?

I suppose a rebase mistake, later in the series errors field is removed.

Quoted 2 lines

To bad we can't really test this patch, beside check that it build.

Indeed, I realize Valgrind support is not that updated at the moment.

I'll add ASAN support as Andrew suggested.

Quoted 9 lines

Thanks,

--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Frediano

#8 lore raw

Re: [PATCH v9 4/4] libs/guest: use Valgrind to detect various buffer overflows

In reply to Frediano Ziglio
Quoted 13 lines · Frediano Ziglio
tools/libs/guest/xg_sr_common.h +3 -0
index c07c6db59e..d3fc7f363e 100644
@@ -245,13 +245,21 @@ struct xc_sr_context
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
             struct xc_sr_context_save_buffers
             {
+                MEM_NOACCESS_BUFFER(na0, 16);
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 16);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 16);

tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 262

Quoted 10 lines
tools/libs/guest/xg_sr_common.h +5 -0
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 16);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na4, 16);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na5, 16);
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na6, 16);
                 int errors[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na7, 16);

If you're going to do this, use 64 not 16, and put iov[] at the end.  It is good practice to misalign large buffers by one cacheline, not that I suspect the access pattern in this algorithm is relevant.

But, I'm not sure of the utility of doing this.  By removing the dynamic allocations you also prevent ASAN from working.

We either care about them all (and keep the dynamic allocations), or don't care, and get rid of it all.  This half-way-house serves no good purpose.

~Andrew

#9 lore raw

Re: [PATCH v9 4/4] libs/guest: use Valgrind to detect various buffer overflows

Quoted 21 lines · Andrew Cooper
Quoted 19 lines · Frediano Ziglio
tools/libs/guest/xg_sr_common.h +6 -0
index c07c6db59e..d3fc7f363e 100644
@@ -245,13 +245,21 @@ struct xc_sr_context
             xc_hypercall_buffer_t dirty_bitmap_hbuf;
             struct xc_sr_context_save_buffers
             {
+                MEM_NOACCESS_BUFFER(na0, 16);
                 xen_pfn_t batch_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na1, 16);
                 xen_pfn_t mfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na2, 16);
                 xen_pfn_t types[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na3, 16);
                 void *local_pages[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na4, 16);
                 struct iovec iov[MAX_BATCH_SIZE + 2]; /* Headers + data. */
+                MEM_NOACCESS_BUFFER(na5, 16);
Quoted 9 lines

tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 262

Quoted 4 lines
tools/libs/guest/xg_sr_common.h +2 -0
                 uint64_t rec_pfns[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na6, 16);
                 int errors[MAX_BATCH_SIZE];
+                MEM_NOACCESS_BUFFER(na7, 16);

If you're going to do this, use 64 not 16, and put iov[] at the end. It is good practice to misalign large buffers by one cacheline, not that I suspect the access pattern in this algorithm is relevant.

Easily doable. Why put "iov" at the end?

Quoted 3 lines

But, I'm not sure of the utility of doing this. By removing the dynamic allocations you also prevent ASAN from working.

It can be extended to ASAN using "sanitizer/asan_interface.h", specifically "ASAN_POISON_MEMORY_REGION" macro.

Quoted 5 lines

We either care about them all (and keep the dynamic allocations), or don't care, and get rid of it all. This half-way-house serves no good purpose.

~Andrew

Why is a half-way-house ?

Frediano