Skip to contentSkip to conversation

tools/migration: Py3 fixes, and page type helpers

PATCH In review Andrew Cooper 4 participants 10 messages Jul 5, 2021 – Jul 6, 2021

lore
Share & tools
lore
#0 lore raw

cover tools/migration: Py3 fixes, and page type helpers

FAOD, this is what I'm planning to push, unless anyone shouts urgently.

Olaf Hering (5):
tools/python: handle libxl__physmap_info.name properly in convert-legacy-stream tools/python: fix Python3.4 TypeError in format string tools/migration: unify known page type checking
tools/migration: unify type checking for data pfns in the VM tools/migration: unify type checking for data pfns in migration stream

 tools/libs/guest/xg_sr_common.h            | 85 ++++++++++++++++++++++++++++++
 tools/libs/guest/xg_sr_restore.c           | 38 +++----------
 tools/libs/guest/xg_sr_save.c              | 18 +++----
 tools/python/scripts/convert-legacy-stream |  8 +--
 4 files changed, 102 insertions(+), 47 deletions(-)
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/5 tools/python: handle libxl__physmap_info.name properly in convert-legacy-stream

From: Olaf Hering <[email protected]>

The trailing member name[] in libxl__physmap_info is written as a cstring into the stream. The current code does a sanity check if the last byte is zero. This attempt fails with python3 because name[-1] returns a type int. As a result the comparison with byte(\00) fails:

File "/usr/lib/xen/bin/convert-legacy-stream", line 347, in read_libxl_toolstack
raise StreamError("physmap name not NUL terminated")
StreamError: physmap name not NUL terminated

To handle both python variants, cast to bytearray().

---
 tools/python/scripts/convert-legacy-stream | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/python/scripts/convert-legacy-stream +1 -1
index 66ee3d2f5d29..227e1b5c3fd3 100755
@@ -343,7 +343,7 @@ def read_libxl_toolstack(vm, data):
         if twidth == 64:
             name = name[:-4]
 
-        if name[-1] != b'\x00':
+        if bytearray(name)[-1] != 0:
             raise StreamError("physmap name not NUL terminated")
 
         root = b"physmap/%x" % (phys, )
Signature · 1 line
2.11.0
Acked-by on v1 1/5
#2 lore raw

Re: [PATCH 1/5] tools/python: handle libxl__physmap_info.name properly in convert-legacy-stream

Quoted 7 lines · Andrew Cooper

From: Olaf Hering <[email protected]>

The trailing member name[] in libxl__physmap_info is written as a cstring into the stream. The current code does a sanity check if the last byte is zero. This attempt fails with python3 because name[-1] returns a type int. As a result the comparison with byte(\00) fails:

Quoted 8 lines

File "/usr/lib/xen/bin/convert-legacy-stream", line 347, in read_libxl_toolstack
raise StreamError("physmap name not NUL terminated")
StreamError: physmap name not NUL terminated

To handle both python variants, cast to bytearray().

Quoted 20 lines
---
 tools/python/scripts/convert-legacy-stream | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/python/scripts/convert-legacy-stream +1 -1
index 66ee3d2f5d29..227e1b5c3fd3 100755
@@ -343,7 +343,7 @@ def read_libxl_toolstack(vm, data):
         if twidth == 64:
             name = name[:-4]
 
-        if name[-1] != b'\x00':
+        if bytearray(name)[-1] != 0:
             raise StreamError("physmap name not NUL terminated")
 
         root = b"physmap/%x" % (phys, )
Signature · 1 line
2.11.0
Signature · 3 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
#3 lore raw

PATCH 2/5 tools/python: fix Python3.4 TypeError in format string

In reply to Andrew Cooper

From: Olaf Hering <[email protected]>

Using the first element of a tuple for a format specifier fails with python3.4 as included in SLE12:

    b = b"string/%x" % (i, )

TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple'

It happens to work with python 2.7 and 3.6.
To support older Py3, format as strings and explicitly encode as ASCII.

---
 tools/python/scripts/convert-legacy-stream | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/python/scripts/convert-legacy-stream +3 -3
index 227e1b5c3fd3..7fe375a66810 100755
@@ -346,9 +346,9 @@ def read_libxl_toolstack(vm, data):
         if bytearray(name)[-1] != 0:
             raise StreamError("physmap name not NUL terminated")
 
-        root = b"physmap/%x" % (phys, )
-        kv = [root + b"/start_addr", b"%x" % (start, ),
-              root + b"/size",       b"%x" % (size, ),
+        root = ("physmap/%x" % (phys, )).encode('ascii')
+        kv = [root + b"/start_addr", ("%x" % (start, )).encode('ascii'),
+              root + b"/size",       ("%x" % (size, )).encode('ascii'),
               root + b"/name",       name[:-1]]
 
         for key, val in zip(kv[0::2], kv[1::2]):
Signature · 1 line
2.11.0
Acked-by on v1 2/5
#4 lore raw

Re: [PATCH 2/5] tools/python: fix Python3.4 TypeError in format string

Quoted 4 lines · Andrew Cooper

From: Olaf Hering <[email protected]>

Using the first element of a tuple for a format specifier fails with python3.4 as included in SLE12:

Quoted 8 lines
    b = b"string/%x" % (i, )

TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple'

It happens to work with python 2.7 and 3.6.
To support older Py3, format as strings and explicitly encode as ASCII.

Quoted 24 lines
---
 tools/python/scripts/convert-legacy-stream | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/python/scripts/convert-legacy-stream +3 -3
index 227e1b5c3fd3..7fe375a66810 100755
@@ -346,9 +346,9 @@ def read_libxl_toolstack(vm, data):
         if bytearray(name)[-1] != 0:
             raise StreamError("physmap name not NUL terminated")
 
-        root = b"physmap/%x" % (phys, )
-        kv = [root + b"/start_addr", b"%x" % (start, ),
-              root + b"/size",       b"%x" % (size, ),
+        root = ("physmap/%x" % (phys, )).encode('ascii')
+        kv = [root + b"/start_addr", ("%x" % (start, )).encode('ascii'),
+              root + b"/size",       ("%x" % (size, )).encode('ascii'),
               root + b"/name",       name[:-1]]
 
         for key, val in zip(kv[0::2], kv[1::2]):
Signature · 1 line
2.11.0
Signature · 3 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
#5 lore raw

PATCH 3/5 tools/migration: unify known page type checking

In reply to Andrew Cooper

From: Olaf Hering <[email protected]>

Users of xc_get_pfn_type_batch may want to sanity check the data returned by Xen. Add helpers for this purpose:

is_known_page_type verifies the type returned by Xen on the saving side, or the incoming type for a page on the restoring side, is known by the save/restore code.

---
 tools/libs/guest/xg_sr_common.h  | 29 +++++++++++++++++++++++++++++
 tools/libs/guest/xg_sr_restore.c |  5 ++---
 tools/libs/guest/xg_sr_save.c    |  6 ++++++
 3 files changed, 37 insertions(+), 3 deletions(-)
tools/libs/guest/xg_sr_common.h +29 -0
index e2994e18acf1..e8436d1abb14 100644
@@ -455,6 +455,35 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,
 /* Handle a STATIC_DATA_END record. */
 int handle_static_data_end(struct xc_sr_context *ctx);
 
+/* Page type known to the migration logic? */
+static inline bool is_known_page_type(uint32_t type)
+{
+    switch ( type )
+    {
+    case XEN_DOMCTL_PFINFO_NOTAB:
+
+    case XEN_DOMCTL_PFINFO_L1TAB:
+    case XEN_DOMCTL_PFINFO_L1TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L2TAB:
+    case XEN_DOMCTL_PFINFO_L2TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L3TAB:
+    case XEN_DOMCTL_PFINFO_L3TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L4TAB:
+    case XEN_DOMCTL_PFINFO_L4TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_XTAB:
+    case XEN_DOMCTL_PFINFO_XALLOC: /* Synthetic type in Xen 4.2 - 4.5 */
+    case XEN_DOMCTL_PFINFO_BROKEN:
+        return true;
+
+    default:
+        return false;
+    }
+}
+
 #endif
 /*
  * Local variables:
tools/libs/guest/xg_sr_restore.c +2 -3
index b57a7875192f..508953fd3cf7 100644
@@ -406,10 +406,9 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
         }
 
         type = (pages->pfn[i] & PAGE_DATA_TYPE_MASK) >> 32;
-        if ( ((type >> XEN_DOMCTL_PFINFO_LTAB_SHIFT) >= 5) &&
-             ((type >> XEN_DOMCTL_PFINFO_LTAB_SHIFT) <= 8) )
+        if ( !is_known_page_type(type) )
         {
-            ERROR("Invalid type %#"PRIx32" for pfn %#"PRIpfn" (index %u)",
+            ERROR("Unknown type %#"PRIx32" for pfn %#"PRIpfn" (index %u)",
                   type, pfn, i);
             goto err;
         }
tools/libs/guest/xg_sr_save.c +6 -0
index f0e2bd048d37..c84a073e86f4 100644
@@ -147,6 +147,12 @@ static int write_batch(struct xc_sr_context *ctx)
 
     for ( i = 0; i < nr_pfns; ++i )
     {
+        if ( !is_known_page_type(types[i]) )
+        {
+            ERROR("Unknown type %#"PRIpfn" for pfn %#"PRIpfn, types[i], mfns[i]);
+            goto err;
+        }
+
         switch ( types[i] )
         {
         case XEN_DOMCTL_PFINFO_BROKEN:
Signature · 1 line
2.11.0
#6 lore raw

PATCH 4/5 tools/migration: unify type checking for data pfns in the VM

In reply to Andrew Cooper

From: Olaf Hering <[email protected]>

Introduce a helper which decides if a given pfn in the migration stream is backed by memory.

This highlights more clearly that type XEN_DOMCTL_PFINFO_XALLOC (a synthetic toolstack-only type used between Xen 4.2 to 4.5 which indicated a dirty page on the sending side for which no data will be send in the initial iteration) does get populated in the VM.

No change in behaviour intended, except for invalid page types which now have a safer default.

---
 tools/libs/guest/xg_sr_common.h  | 29 +++++++++++++++++++++++++++++
 tools/libs/guest/xg_sr_restore.c |  4 +---
 2 files changed, 30 insertions(+), 3 deletions(-)
tools/libs/guest/xg_sr_common.h +29 -0
index e8436d1abb14..7f4b0439f6bd 100644
@@ -484,6 +484,35 @@ static inline bool is_known_page_type(uint32_t type)
     }
 }
 
+/* Page type backed by RAM in the guest? */
+static inline bool page_type_to_populate(uint32_t type)
+{
+    switch ( type )
+    {
+    case XEN_DOMCTL_PFINFO_NOTAB:
+
+    case XEN_DOMCTL_PFINFO_L1TAB:
+    case XEN_DOMCTL_PFINFO_L1TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L2TAB:
+    case XEN_DOMCTL_PFINFO_L2TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L3TAB:
+    case XEN_DOMCTL_PFINFO_L3TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L4TAB:
+    case XEN_DOMCTL_PFINFO_L4TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_XALLOC:
+        return true;
+
+    case XEN_DOMCTL_PFINFO_XTAB:
+    case XEN_DOMCTL_PFINFO_BROKEN:
+    default:
+        return false;
+    }
+}
+
 #endif
 /*
  * Local variables:
tools/libs/guest/xg_sr_restore.c +1 -3
index 508953fd3cf7..3d63442d4537 100644
@@ -152,9 +152,7 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,
 
     for ( i = 0; i < count; ++i )
     {
-        if ( (!types || (types &&
-                         (types[i] != XEN_DOMCTL_PFINFO_XTAB &&
-                          types[i] != XEN_DOMCTL_PFINFO_BROKEN))) &&
+        if ( (!types || page_type_to_populate(types[i])) &&
              !pfn_is_populated(ctx, original_pfns[i]) )
         {
             rc = pfn_set_populated(ctx, original_pfns[i]);
Signature · 1 line
2.11.0
#7 lore raw

PATCH 5/5 tools/migration: unify type checking for data pfns in migration stream

In reply to Andrew Cooper

From: Olaf Hering <[email protected]>

Introduce a helper which decides if a given pfn type has data in the migration stream.

No change in behaviour intended, except for invalid page types which now have a safer default.

---
 tools/libs/guest/xg_sr_common.h  | 27 +++++++++++++++++++++++++++
 tools/libs/guest/xg_sr_restore.c | 29 +++--------------------------
 tools/libs/guest/xg_sr_save.c    | 14 ++------------
 3 files changed, 32 insertions(+), 38 deletions(-)
tools/libs/guest/xg_sr_common.h +27 -0
index 7f4b0439f6bd..36d45ef56fb8 100644
@@ -513,6 +513,33 @@ static inline bool page_type_to_populate(uint32_t type)
     }
 }
 
+static inline bool page_type_has_stream_data(uint32_t type)
+{
+    switch ( type )
+    {
+    case XEN_DOMCTL_PFINFO_NOTAB:
+
+    case XEN_DOMCTL_PFINFO_L1TAB:
+    case XEN_DOMCTL_PFINFO_L1TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L2TAB:
+    case XEN_DOMCTL_PFINFO_L2TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L3TAB:
+    case XEN_DOMCTL_PFINFO_L3TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+
+    case XEN_DOMCTL_PFINFO_L4TAB:
+    case XEN_DOMCTL_PFINFO_L4TAB | XEN_DOMCTL_PFINFO_LPINTAB:
+        return true;
+
+    case XEN_DOMCTL_PFINFO_XTAB:
+    case XEN_DOMCTL_PFINFO_BROKEN:
+    case XEN_DOMCTL_PFINFO_XALLOC:
+    default:
+        return false;
+    }
+}
+
 #endif
 /*
  * Local variables:
tools/libs/guest/xg_sr_restore.c +3 -26
index 3d63442d4537..aa4113d7f679 100644
@@ -231,25 +231,8 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,
     {
         ctx->restore.ops.set_page_type(ctx, pfns[i], types[i]);
 
-        switch ( types[i] )
-        {
-        case XEN_DOMCTL_PFINFO_NOTAB:
-
-        case XEN_DOMCTL_PFINFO_L1TAB:
-        case XEN_DOMCTL_PFINFO_L1TAB | XEN_DOMCTL_PFINFO_LPINTAB:
-
-        case XEN_DOMCTL_PFINFO_L2TAB:
-        case XEN_DOMCTL_PFINFO_L2TAB | XEN_DOMCTL_PFINFO_LPINTAB:
-
-        case XEN_DOMCTL_PFINFO_L3TAB:
-        case XEN_DOMCTL_PFINFO_L3TAB | XEN_DOMCTL_PFINFO_LPINTAB:
-
-        case XEN_DOMCTL_PFINFO_L4TAB:
-        case XEN_DOMCTL_PFINFO_L4TAB | XEN_DOMCTL_PFINFO_LPINTAB:
-
+        if ( page_type_has_stream_data(types[i]) )
             mfns[nr_pages++] = ctx->restore.ops.pfn_to_gfn(ctx, pfns[i]);
-            break;
-        }
     }
 
     /* Nothing to do? */
@@ -269,14 +252,8 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,
 
     for ( i = 0, j = 0; i < count; ++i )
     {
-        switch ( types[i] )
-        {
-        case XEN_DOMCTL_PFINFO_XTAB:
-        case XEN_DOMCTL_PFINFO_BROKEN:
-        case XEN_DOMCTL_PFINFO_XALLOC:
-            /* No page data to deal with. */
+        if ( !page_type_has_stream_data(types[i]) )
             continue;
-        }
 
         if ( map_errs[j] )
         {
@@ -411,7 +388,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
             goto err;
         }
 
-        if ( type < XEN_DOMCTL_PFINFO_BROKEN )
+        if ( page_type_has_stream_data(type) )
             /* NOTAB and all L1 through L4 tables (including pinned) should
              * have a page worth of data in the record. */
             pages_of_data++;
tools/libs/guest/xg_sr_save.c +2 -12
index c84a073e86f4..3fce41b6cfff 100644
@@ -153,13 +153,8 @@ static int write_batch(struct xc_sr_context *ctx)
             goto err;
         }
 
-        switch ( types[i] )
-        {
-        case XEN_DOMCTL_PFINFO_BROKEN:
-        case XEN_DOMCTL_PFINFO_XALLOC:
-        case XEN_DOMCTL_PFINFO_XTAB:
+        if ( !page_type_has_stream_data(types[i]) )
             continue;
-        }
 
         mfns[nr_pages++] = mfns[i];
     }
@@ -177,13 +172,8 @@ static int write_batch(struct xc_sr_context *ctx)
 
         for ( i = 0, p = 0; i < nr_pfns; ++i )
         {
-            switch ( types[i] )
-            {
-            case XEN_DOMCTL_PFINFO_BROKEN:
-            case XEN_DOMCTL_PFINFO_XALLOC:
-            case XEN_DOMCTL_PFINFO_XTAB:
+            if ( !page_type_has_stream_data(types[i]) )
                 continue;
-            }
 
             if ( errors[p] )
             {
Signature · 1 line
2.11.0
Reviewed-by on v1 5/5
#8 lore raw

Re: [PATCH 5/5] tools/migration: unify type checking for data pfns in migration stream

Quoted 9 lines · Andrew Cooper

From: Olaf Hering <[email protected]>

Introduce a helper which decides if a given pfn type has data in the migration stream.

No change in behaviour intended, except for invalid page types which now have a safer default.

Juergen

Tested-by on the v1 series
#9 lore raw
In reply to Andrew Cooper

Am Mon, 5 Jul 2021 20:10:22 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 5 lines

tools/python: handle libxl__physmap_info.name properly in convert-legacy-stream tools/python: fix Python3.4 TypeError in format string tools/migration: unify known page type checking
tools/migration: unify type checking for data pfns in the VM tools/migration: unify type checking for data pfns in migration stream

Olaf