Skip to contentSkip to conversation

domain context infrastructure

PATCH Superseded Paul Durrant 8 participants 41 messages Sep 24, 2020 – Oct 13, 2020

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v9 cover domain context infrastructure

From: Paul Durrant <[email protected]>

Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to include DOMAIN_CONTEXT common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

 .gitignore                               |   1 +
 docs/specs/libxc-migration-stream.pandoc |  69 ++++-
 tools/flask/policy/modules/xen.if        |   4 +-
 tools/libs/ctrl/include/xenctrl.h        |   5 +
 tools/libs/ctrl/xc_domain.c              |  56 ++++
 tools/libs/guest/xg_sr_common.c          |   1 +
 tools/libs/guest/xg_sr_common.h          |   3 +
 tools/libs/guest/xg_sr_common_x86.c      |  20 --
 tools/libs/guest/xg_sr_common_x86.h      |   6 -
 tools/libs/guest/xg_sr_restore.c         |  45 +++-
 tools/libs/guest/xg_sr_save.c            |  52 +++-
 tools/libs/guest/xg_sr_save_x86_hvm.c    |   5 -
 tools/libs/guest/xg_sr_save_x86_pv.c     |  22 --
 tools/libs/guest/xg_sr_stream_format.h   |   1 +
 tools/misc/Makefile                      |   4 +
 tools/misc/xen-domctx.c                  | 289 +++++++++++++++++++++
 tools/python/xen/migration/libxc.py      |   2 +
 xen/arch/x86/time.c                      |  34 ++-
 xen/common/Makefile                      |   1 +
 xen/common/domain.c                      | 105 ++++++++
 xen/common/domctl.c                      | 173 +++++++++++++
 xen/common/save.c                        | 315 +++++++++++++++++++++++
 xen/include/asm-x86/time.h               |   5 +-
 xen/include/public/arch-arm/hvm/save.h   |   5 +
 xen/include/public/arch-x86/hvm/save.h   |   5 +
 xen/include/public/domctl.h              |  41 +++
 xen/include/public/save.h                | 111 ++++++++
 xen/include/xen/save.h                   | 170 ++++++++++++
 xen/xsm/flask/hooks.c                    |   6 +
 xen/xsm/flask/policy/access_vectors      |   4 +
 30 files changed, 1485 insertions(+), 75 deletions(-)
 create mode 100644 tools/misc/xen-domctx.c
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/public/save.h
 create mode 100644 xen/include/xen/save.h

---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>

Signature · 1 line
2.20.1
#1 lore raw

PATCH v9 1/8 xen/common: introduce a new framework for save/restore of 'domain' context

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v7:
- Add an option to domain_load_end() to ignore unconsumed data, which will be needed by a subsequent patch
- Kept acks since the modification is very small

v4:
- Addressed further comments from Jan

v3:
- Addressed comments from Julien and Jan
- Save handlers no longer need to state entry length up-front
- Save handlers expected to deal with multiple instances internally
- Entries are now auto-padded to 8 byte boundary

v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework

---
 xen/common/Makefile                    |   1 +
 xen/common/save.c                      | 315 +++++++++++++++++++++++++
 xen/include/public/arch-arm/hvm/save.h |   5 +
 xen/include/public/arch-x86/hvm/save.h |   5 +
 xen/include/public/save.h              |  89 +++++++
 xen/include/xen/save.h                 | 170 +++++++++++++
 6 files changed, 585 insertions(+)
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/public/save.h
 create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
index b3b60a1ba2..3e6f21714a 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
 obj-y += rbtree.o
 obj-y += rcupdate.o
 obj-y += rwlock.o
+obj-y += save.o
 obj-y += shutdown.o
 obj-y += softirq.o
 obj-y += sort.o
xen/common/save.c new file +315 -0

315 lines not on this page · raw message

xen/include/public/arch-arm/hvm/save.h +5 -0
index 75b8e65bcb..d5b0c15203 100644
@@ -26,6 +26,11 @@
 #ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__
 #define __XEN_PUBLIC_HVM_SAVE_ARM_H__
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif
 
 /*
xen/include/public/arch-x86/hvm/save.h +5 -0
index 773a380bc2..e61e2dbcd7 100644
@@ -648,6 +648,11 @@ struct hvm_msr {
  */
 #define HVM_SAVE_CODE_MAX 20
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */
 
 /*
xen/include/public/save.h new file +89 -0

89 lines not on this page · raw message

xen/include/xen/save.h new file +170 -0

170 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

xen/common/save.c line 2

Quoted 8 lines · Paul Durrant
xen/common/save.c new file +2 -0
new file mode 100644
index 0000000000..841c4d0e4e
@@ -0,0 +1,315 @@
+/*
+ * save.c: Save and restore PV guest state common to all domain types.

This description will be stale by the time your work is complete.

xen/common/save.c line 85

Quoted 11 lines
xen/common/save.c new file +11 -0
+int domain_save_data(struct domain_context *c, const void *src, size_t len)
+{
+    int rc = c->ops.save->append(c->priv, src, len);
+
+    if ( !rc )
+        c->len += len;
+
+    return rc;
+}
+
+#define DOMAIN_SAVE_ALIGN 8

This is part of the stream ABI.

xen/common/save.c line 90

Quoted 5 lines
xen/common/save.c new file +5 -0
+
+int domain_save_end(struct domain_context *c)
+{
+    struct domain *d = c->domain;
+    size_t len = ROUNDUP(c->len, DOMAIN_SAVE_ALIGN) - c->len; /* padding */

DOMAIN_SAVE_ALIGN - (c->len & (DOMAIN_SAVE_ALIGN - 1))

isn't vulnerable to overflow.

Quoted 6 lines
xen/common/save.c new file +6 -0
+    int rc;
+
+    if ( len )
+    {
+        static const uint8_t pad[DOMAIN_SAVE_ALIGN] = {};
+

xen/common/save.c line 106

Quoted 10 lines
xen/common/save.c new file +10 -0
+        rc = domain_save_data(c, pad, len);
+
+        if ( rc )
+            return rc;
+    }
+    ASSERT(IS_ALIGNED(c->len, DOMAIN_SAVE_ALIGN));
+
+    if ( c->name )
+        gdprintk(XENLOG_INFO, "%pd save: %s[%u] +%zu (-%zu)\n", d, c->name,
+                 c->desc.instance, c->len, len);

IMO, this is unhelpful to print out.  It also appears to be the only use of the c->name field.

It also creates obscure and hard to follow logic based on dry_run.

Quoted 56 lines
xen/include/public/save.h new file +50 -0
new file mode 100644
index 0000000000..551dbbddb8
@@ -0,0 +1,89 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef XEN_PUBLIC_SAVE_H
+#define XEN_PUBLIC_SAVE_H
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+#include "xen.h"
+
+/* Entry data is preceded by a descriptor */
+struct domain_save_descriptor {
+    uint16_t typecode;
+
+    /*
+     * Instance number of the entry (since there may be multiple of some
+     * types of entries).
+     */
+    uint16_t instance;
+
+    /* Entry length not including this descriptor */
+    uint32_t length;
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE

xen/include/public/save.h line 60

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+ * binds these things together, although it is not intended that the
+ * resulting type is ever instantiated.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->t)

I realise this is going to make me very unpopular, but NACK.

This is straight up obfuscation with no redeeming properties.  I know you've copied it from the exist HVMCONTEXT infrastructure, but it is obnoxious to use there (particularly in the domain builder) and not an example wanting copying.

Furthermore, the code will be simpler and easier to follow without it.

Secondly, and more importantly, I do not see anything in docs/specs/ describing the binary format of this stream,  and I'm going to insist that one appears, ahead of this patch in the series.

In doing so, you're hopefully going to discover the bug with the older HVMCONTEXT stream which makes the version field fairly pointless (more below).

It should describe how to forward compatibly extend the stream, and under what circumstances the version number can/should change.  It also needs to describe the alignment and extending rules which ...

xen/include/public/save.h line 68

Quoted 8 lines
xen/include/public/save.h new file +8 -0
+
+/*
+ * All entries will be zero-padded to the next 64-bit boundary when saved,
+ * so there is no need to include trailing pad fields in structure
+ * definitions.
+ * When loading, entries will be zero-extended if the load handler reads
+ * beyond the length specified in the descriptor.
+ */

... shouldn't be this.

The current zero extending property was an emergency hack to fix an ABI breakage which had gone unnoticed for a couple of releases.  The work to implement it created several very hard to debug breakages in Xen.

A properly designed stream shouldn't need auto-extending behaviour, and the legibility of the code is improved by not having it.

It is a trick which can stay up your sleeve for an emergency, in the hope you'll never have to use it.

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+
+/* Terminating entry */
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);
+

xen/include/public/save.h line 83

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+#define DOMAIN_SAVE_MAGIC   0x53415645
+#define DOMAIN_SAVE_VERSION 0x00000001
+
+/* Initial entry */
+struct domain_save_header {
+    uint32_t magic;                /* Must be DOMAIN_SAVE_MAGIC */
+    uint16_t xen_major, xen_minor; /* Xen version */
+    uint32_t version;              /* Save format version */
+};
+DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);

The layout problem with the stream is the fact that this header doesn't come first.

In the eventual future where uint16_t won't be sufficient for instance, and uint32_t might not be sufficient for len, the version number is going to have to be bumped, in order to change the descriptor layout.

Overall, this patch needs to be a minimum of two.  First a written document which is the authoritative stream ABI, and the second which is this implementation.  The header describing the stream format should not be substantively different from xg_sr_stream_format.h

~Andrew

P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.

#3 lore raw

Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

On Fri, Oct 02, 2020 at 10:20:18PM +0100, Andrew Cooper wrote:
[...]

Quoted 3 lines

P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.

For Rust the header is going to be generated by a tool called bindgen. It doesn't like nested macros, so I would be all for a simpler C header file if we can help it.

Wei.

#4 lore raw

RE: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

In reply to Andrew Cooper
Quoted 13 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:20
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

On 24/09/2020 14:10, Paul Durrant wrote:

Quoted 11 lines

xen/common/save.c line 2

Quoted 8 lines
xen/common/save.c new file +2 -0
new file mode 100644
index 0000000000..841c4d0e4e
@@ -0,0 +1,315 @@
+/*
+ * save.c: Save and restore PV guest state common to all domain types.

This description will be stale by the time your work is complete.

True now, I'll just drop the 'PV'

Quoted 5 lines
Quoted 5 lines
xen/common/save.c new file +5 -0
+int domain_save_data(struct domain_context *c, const void *src, size_t len)
+{
+    int rc = c->ops.save->append(c->priv, src, len);
+
+    if ( !rc )
Quoted 9 lines

xen/common/save.c line 85

Quoted 6 lines
xen/common/save.c new file +6 -0
+        c->len += len;
+
+    return rc;
+}
+
+#define DOMAIN_SAVE_ALIGN 8

This is part of the stream ABI.

And what's actually the problem with defining it here?

Quoted 10 lines

xen/common/save.c line 90

Quoted 5 lines
xen/common/save.c new file +5 -0
+
+int domain_save_end(struct domain_context *c)
+{
+    struct domain *d = c->domain;
+    size_t len = ROUNDUP(c->len, DOMAIN_SAVE_ALIGN) - c->len; /* padding */

DOMAIN_SAVE_ALIGN - (c->len & (DOMAIN_SAVE_ALIGN - 1))

isn't vulnerable to overflow.

...and significantly uglier code. What's actually wrong with what I wrote?

Quoted 20 lines
Quoted 6 lines
xen/common/save.c new file +6 -0
+    int rc;
+
+    if ( len )
+    {
+        static const uint8_t pad[DOMAIN_SAVE_ALIGN] = {};
+

xen/common/save.c line 106

Quoted 10 lines
xen/common/save.c new file +10 -0
+        rc = domain_save_data(c, pad, len);
+
+        if ( rc )
+            return rc;
+    }
+    ASSERT(IS_ALIGNED(c->len, DOMAIN_SAVE_ALIGN));
+
+    if ( c->name )
+        gdprintk(XENLOG_INFO, "%pd save: %s[%u] +%zu (-%zu)\n", d, c->name,
+                 c->desc.instance, c->len, len);

IMO, this is unhelpful to print out. It also appears to be the only use of the c->name field.

Quoted 2 lines

It also creates obscure and hard to follow logic based on dry_run.

I'll drop it to debug. I personally find it helpful and would prefer to keep it.

Quoted 69 lines
Quoted 56 lines
xen/include/public/save.h new file +50 -0
new file mode 100644
index 0000000000..551dbbddb8
@@ -0,0 +1,89 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef XEN_PUBLIC_SAVE_H
+#define XEN_PUBLIC_SAVE_H
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+#include "xen.h"
+
+/* Entry data is preceded by a descriptor */
+struct domain_save_descriptor {
+    uint16_t typecode;
+
+    /*
+     * Instance number of the entry (since there may be multiple of some
+     * types of entries).
+     */
+    uint16_t instance;
+
+    /* Entry length not including this descriptor */
+    uint32_t length;
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE

xen/include/public/save.h line 60

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+ * binds these things together, although it is not intended that the
+ * resulting type is ever instantiated.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->t)

I realise this is going to make me very unpopular, but NACK.

Quoted 7 lines

This is straight up obfuscation with no redeeming properties. I know you've copied it from the exist HVMCONTEXT infrastructure, but it is obnoxious to use there (particularly in the domain builder) and not an example wanting copying.

Furthermore, the code will be simpler and easier to follow without it.

OK, I can drop it if you so vehemently object.

Quoted 4 lines

Secondly, and more importantly, I do not see anything in docs/specs/ describing the binary format of this stream, and I'm going to insist that one appears, ahead of this patch in the series.

I can certainly put something there if you wish.

Quoted 8 lines

In doing so, you're hopefully going to discover the bug with the older HVMCONTEXT stream which makes the version field fairly pointless (more below).

It should describe how to forward compatibly extend the stream, and under what circumstances the version number can/should change. It also needs to describe the alignment and extending rules which ...

Quoted 11 lines

xen/include/public/save.h line 68

Quoted 8 lines
xen/include/public/save.h new file +8 -0
+
+/*
+ * All entries will be zero-padded to the next 64-bit boundary when saved,
+ * so there is no need to include trailing pad fields in structure
+ * definitions.
+ * When loading, entries will be zero-extended if the load handler reads
+ * beyond the length specified in the descriptor.
+ */

... shouldn't be this.

Auto-padding was explicitly requested by Julien and extending (with zeroes or otherwise) is the necessary corollary (since the save handlers are not explicitly padding to the alignment boundary).

Quoted 10 lines

The current zero extending property was an emergency hack to fix an ABI breakage which had gone unnoticed for a couple of releases. The work to implement it created several very hard to debug breakages in Xen.

A properly designed stream shouldn't need auto-extending behaviour, and the legibility of the code is improved by not having it.

It is a trick which can stay up your sleeve for an emergency, in the hope you'll never have to use it.

The zero-extending here is different; it does not form part of the record. It is merely there to make sure the alignment constraint is met.

Quoted 10 lines
Quoted 10 lines
xen/include/public/save.h new file +10 -0
+
+/* Terminating entry */
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);
+
+#define DOMAIN_SAVE_MAGIC   0x53415645
+#define DOMAIN_SAVE_VERSION 0x00000001
+
+/* Initial entry */
+struct domain_save_header {
Quoted 9 lines

xen/include/public/save.h line 83

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+    uint32_t magic;                /* Must be DOMAIN_SAVE_MAGIC */
+    uint16_t xen_major, xen_minor; /* Xen version */
+    uint32_t version;              /* Save format version */
+};
+DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);

The layout problem with the stream is the fact that this header doesn't come first.

? It most certainly does some first as is evident from the load and save functions. But I will add a document that states it, as requested.

Quoted 10 lines

In the eventual future where uint16_t won't be sufficient for instance, and uint32_t might not be sufficient for len, the version number is going to have to be bumped, in order to change the descriptor layout.

Overall, this patch needs to be a minimum of two. First a written document which is the authoritative stream ABI, and the second which is this implementation. The header describing the stream format should not be substantively different from xg_sr_stream_format.h

Ok.

Quoted 5 lines

~Andrew

P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.

Fine. I'm happy to drop the macro/type magic if no-one feels it is necessary.

Paul

#5 lore raw

Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 11 lines · Paul Durrant
Quoted 11 lines

From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:20

xen/common/save.c line 90

Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4 -0
+int domain_save_end(struct domain_context *c)
+{
+    struct domain *d = c->domain;
+    size_t len = ROUNDUP(c->len, DOMAIN_SAVE_ALIGN) - c->len; /* padding */

DOMAIN_SAVE_ALIGN - (c->len & (DOMAIN_SAVE_ALIGN - 1))

Quoted 4 lines
Quoted 2 lines

isn't vulnerable to overflow.

...and significantly uglier code. What's actually wrong with what I wrote?

I don't think there's anything "wrong" or "vulnerable" here, but I still can see Andrew's point. The "vulnerable" aspect applies only in the (highly hypothetical I think) cases of either sizeof(size_t) < sizeof(int) or size_t being a signed type, afaict. But since it's easy (and imo not "significantly uglier") to write code that is free of any wrapping or overflowing behavior, I think it is sensible to actually write it that way.

Jan

#6 lore raw

Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context

In reply to Paul Durrant

xen/include/xen/save.h line 99

Quoted 10 lines · Paul Durrant
xen/include/xen/save.h new file +10 -0
+/*
+ * The 'dry_run' flag indicates that the caller of domain_save() (see below)
+ * is not trying to actually acquire the data, only the size of the data.
+ * The save handler can therefore limit work to only that which is necessary
+ * to call domain_save_data() the correct number of times with accurate values
+ * for 'len'.
+ */
+typedef int (*domain_save_handler)(const struct domain *d,
+                                   struct domain_context *c,
+                                   bool dry_run);

Sorry - missed this the first time around.  This cannot take a const domain.

Doing so prevents putting (amongst other things), event channel details into the stream, because you won't be able to take the domain's event lock, and having the domain paused isn't good enough protection.

Removing this const will reduce the churn in subsequent patches somewhat.

~Andrew

#7 lore raw

PATCH v9 2/8 xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

In reply to Paul Durrant

These domctls provide a mechanism to get and set domain context from the toolstack.

---
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v4:
- Add missing zero pad checks

v3:
- Addressed comments from Julien and Jan
- Use vmalloc() rather than xmalloc_bytes()

v2:
- drop mask parameter
- const-ify some more buffers

---
 tools/flask/policy/modules/xen.if   |   4 +-
 tools/libs/ctrl/include/xenctrl.h   |   5 +
 tools/libs/ctrl/xc_domain.c         |  56 +++++++++
 xen/common/domctl.c                 | 173 ++++++++++++++++++++++++++++
 xen/include/public/domctl.h         |  41 +++++++
 xen/xsm/flask/hooks.c               |   6 +
 xen/xsm/flask/policy/access_vectors |   4 +
 7 files changed, 287 insertions(+), 2 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
index 8eb2293a52..2bc9db4f64 100644
@@ -53,7 +53,7 @@ define(`create_domain_common', `
 	allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim
 			set_vnumainfo get_vnumainfo cacheflush
 			psr_cmt_op psr_alloc soft_reset
-			resource_map get_cpu_policy };
+			resource_map get_cpu_policy setcontext };
 	allow $1 $2:security check_context;
 	allow $1 $2:shadow enable;
 	allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', `
 	allow $1 $2:hvm { gethvmc getparam };
 	allow $1 $2:mmu { stat pageinfo map_read };
 	allow $1 $2:domain { getaddrsize getvcpucontext pause destroy };
-	allow $1 $2:domain2 gettsc;
+	allow $1 $2:domain2 { gettsc getcontext };
 	allow $1 $2:shadow { enable disable logdirty };
 ')
 
tools/libs/ctrl/include/xenctrl.h +5 -0
index 73e9535fc8..f18b9b25d1 100644
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,
                              uint8_t *hvm_ctxt,
                              uint32_t size);
 
+int xc_domain_getcontext(xc_interface *xch, uint32_t domid,
+                         void *ctxt_buf, size_t *size);
+int xc_domain_setcontext(xc_interface *xch, uint32_t domid,
+                         const void *ctxt_buf, size_t size);
+
 /**
  * This function will return guest IO ABI protocol
  *
tools/libs/ctrl/xc_domain.c +56 -0

62 lines not on this page · raw message

xen/common/domctl.c +173 -0

191 lines not on this page · raw message

xen/include/public/domctl.h +41 -0

59 lines not on this page · raw message

xen/xsm/flask/hooks.c +6 -0
index fab5d30c3a..18a9f11c74 100644
@@ -744,6 +744,12 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_get_cpu_policy:
         return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GET_CPU_POLICY);
 
+    case XEN_DOMCTL_setdomaincontext:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SETCONTEXT);
+
+    case XEN_DOMCTL_getdomaincontext:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GETCONTEXT);
+
     default:
         return avc_unknown_permission("domctl", cmd);
     }
xen/xsm/flask/policy/access_vectors +4 -0
index fde5162c7e..a7f6f96ed3 100644
@@ -245,6 +245,10 @@ class domain2
     resource_map
 # XEN_DOMCTL_get_cpu_policy
     get_cpu_policy
+# XEN_DOMCTL_setdomaincontext
+    setcontext
+# XEN_DOMCTL_getdomaincontext
+    getcontext
 }
 
 # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
#9 lore raw

Re: [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

In reply to Paul Durrant
Quoted 14 lines · Paul Durrant
xen/include/public/domctl.h +6 -0
index 791f0a2592..743105181f 100644
@@ -1130,6 +1130,43 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * XEN_DOMCTL_getdomaincontext
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1148

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.
+ *                The value passed out will be the size of data written.
+ */
+struct xen_domctl_getdomaincontext {
+    uint32_t size;

This series is full of mismatched 32/64bit sizes, with several truncation bugs in the previous patch.

Just use a 64bit size here.  Life is too short to go searching for all the other truncation bug when this stream tips over 4G, and its not like there is a shortage of space in this structure.

Quoted 25 lines
xen/include/public/domctl.h +21 -0
+    uint32_t pad;
+    XEN_GUEST_HANDLE_64(void) buffer;
+};
+
+/* XEN_DOMCTL_setdomaincontext
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer from which the context data should be
+ *                copied.
+ * size (IN):     The size of the buffer from which data may be copied.
+ *                This data must include DOMAIN_SAVE_CODE_HEADER at the
+ *                start and terminate with a DOMAIN_SAVE_CODE_END record.
+ *                Any data beyond the DOMAIN_SAVE_CODE_END record will be
+ *                ignored.
+ */
+struct xen_domctl_setdomaincontext {
+    uint32_t size;
+    uint32_t pad;
+    XEN_GUEST_HANDLE_64(const_void) buffer;
+};
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1

xen/include/public/domctl.h struct xen_domctl { · line 1255

Quoted 5 lines
xen/include/public/domctl.h +2 -0
@@ -1214,6 +1251,8 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_getdomaincontext              84
+#define XEN_DOMCTL_setdomaincontext              85

So, we've currently got:

#define XEN_DOMCTL_setvcpucontext                12 #define XEN_DOMCTL_getvcpucontext                13 #define XEN_DOMCTL_gethvmcontext                 33 #define XEN_DOMCTL_sethvmcontext                 34 #define XEN_DOMCTL_set_ext_vcpucontext           42 #define XEN_DOMCTL_get_ext_vcpucontext           43 #define XEN_DOMCTL_gethvmcontext_partial         55 #define XEN_DOMCTL_setvcpuextstate               62 #define XEN_DOMCTL_getvcpuextstate               63

which are doing alarmingly related things for vcpus.  (As an amusing exercise to the reader, figure out which are PV specific and which are HVM specific.  Hint: they're not disjoint sets.)

I know breaking with tradition is sacrilege, but at the very minimum, can we get some underscores in that name so you can at least read the words which make it up more easily.

~Andrew

#10 lore raw

RE: [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Quoted 44 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:58
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; Julien Grall <[email protected]>; Daniel De Graaf <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 14 lines · Paul Durrant
xen/include/public/domctl.h +6 -0
index 791f0a2592..743105181f 100644
@@ -1130,6 +1130,43 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * XEN_DOMCTL_getdomaincontext
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1148

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.
+ *                The value passed out will be the size of data written.
+ */
+struct xen_domctl_getdomaincontext {
+    uint32_t size;

This series is full of mismatched 32/64bit sizes, with several truncation bugs in the previous patch.

Quoted 4 lines

Just use a 64bit size here. Life is too short to go searching for all the other truncation bug when this stream tips over 4G, and its not like there is a shortage of space in this structure.

Ok.

Quoted 43 lines
Quoted 25 lines
xen/include/public/domctl.h +21 -0
+    uint32_t pad;
+    XEN_GUEST_HANDLE_64(void) buffer;
+};
+
+/* XEN_DOMCTL_setdomaincontext
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer from which the context data should be
+ *                copied.
+ * size (IN):     The size of the buffer from which data may be copied.
+ *                This data must include DOMAIN_SAVE_CODE_HEADER at the
+ *                start and terminate with a DOMAIN_SAVE_CODE_END record.
+ *                Any data beyond the DOMAIN_SAVE_CODE_END record will be
+ *                ignored.
+ */
+struct xen_domctl_setdomaincontext {
+    uint32_t size;
+    uint32_t pad;
+    XEN_GUEST_HANDLE_64(const_void) buffer;
+};
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1

xen/include/public/domctl.h struct xen_domctl { · line 1255

Quoted 5 lines
xen/include/public/domctl.h +2 -0
@@ -1214,6 +1251,8 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_getdomaincontext              84
+#define XEN_DOMCTL_setdomaincontext              85

So, we've currently got:

#define XEN_DOMCTL_setvcpucontext                12
#define XEN_DOMCTL_getvcpucontext                13
#define XEN_DOMCTL_gethvmcontext                 33
#define XEN_DOMCTL_sethvmcontext                 34
#define XEN_DOMCTL_set_ext_vcpucontext           42
#define XEN_DOMCTL_get_ext_vcpucontext           43
#define XEN_DOMCTL_gethvmcontext_partial         55
#define XEN_DOMCTL_setvcpuextstate               62
#define XEN_DOMCTL_getvcpuextstate               63
Quoted 4 lines

which are doing alarmingly related things for vcpus. (As an amusing exercise to the reader, figure out which are PV specific and which are HVM specific. Hint: they're not disjoint sets.)

Yes, hence the desire to come up with something common.

Quoted 5 lines

I know breaking with tradition is sacrilege, but at the very minimum, can we get some underscores in that name so you can at least read the words which make it up more easily.

Sure.

Paul

Quoted 1 line

~Andrew

#11 lore raw

PATCH v9 3/8 tools/misc: add xen-domctx to present domain context

In reply to Paul Durrant

This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.

---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>

NOTE: Ian requested ack from Andrew

v3:
- Re-worked to avoid copying onto stack
- Added optional typecode and instance arguments

v2:
- Change name from 'xen-ctx' to 'xen-domctx'

---
 .gitignore              |   1 +
 tools/misc/Makefile     |   4 +
 tools/misc/xen-domctx.c | 200 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 205 insertions(+)
 create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
index 5e8c47e2db..efbbd46fa9 100644
@@ -233,6 +233,7 @@ tools/misc/xen_cpuperf
 tools/misc/xen-cpuid
 tools/misc/xen-detect
 tools/misc/xen-diag
+tools/misc/xen-domctx
 tools/misc/xen-tmem-list-parse
 tools/misc/xen-livepatch
 tools/misc/xenperf
tools/misc/Makefile +4 -0
index 7d37f297a9..fb673d0ff6 100644
@@ -32,6 +32,7 @@ INSTALL_SBIN                   += xenpm
 INSTALL_SBIN                   += xenwatchdogd
 INSTALL_SBIN                   += xen-livepatch
 INSTALL_SBIN                   += xen-diag
+INSTALL_SBIN                   += xen-domctx
 INSTALL_SBIN += $(INSTALL_SBIN-y)
 
 # Everything to be installed in a private bin/
@@ -111,6 +112,9 @@ xen-livepatch: xen-livepatch.o
 xen-diag: xen-diag.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-domctx: xen-domctx.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 xen-lowmemd: xen-lowmemd.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
 
tools/misc/xen-domctx.c new file +200 -0

200 lines not on this page · raw message

Signature · 1 line
2.20.1
#13 lore raw

Re: [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context

In reply to Paul Durrant
Quoted 11 lines · Paul Durrant

This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.

---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>

NOTE: Ian requested ack from Andrew

I mean - its a fairly throwaway piece of misc userspace, so ack.

However, it is largely superseded by the changes you need to make to verify-stream-v2 so you might want to bear that in mind.

Also, I wonder if it is wise in general that we're throwing so many misc debugging tools into sbin.

Quoted 10 lines
tools/misc/xen-domctx.c new file +10 -0
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+#include <xenctrl.h>
+#include <xen/xen.h>
+#include <xen/domctl.h>
+#include <xen/save.h>

tools/misc/xen-domctx.c line 46

Quoted 10 lines
tools/misc/xen-domctx.c new file +10 -0
+
+static void *buf = NULL;
+static size_t len, off;
+
+#define GET_PTR(_x)                                                        \
+    do {                                                                   \
+        if ( len - off < sizeof(*(_x)) )                                   \
+        {                                                                  \
+            fprintf(stderr,                                                \
+                    "error: need another %lu bytes, only %lu available\n", \

%zu is the correct formatter for size_t.

tools/misc/xen-domctx.c line 48

Quoted 2 lines
tools/misc/xen-domctx.c new file +2 -0
+                    sizeof(*(_x)), len - off);                             \
+            exit(1);                                                       \

Your error handling will be far more simple by using err() instead of opencoding it.

~Andrew

#14 lore raw

RE: [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context

Quoted 26 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 23:40
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 11 lines · Paul Durrant

This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.

---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>

NOTE: Ian requested ack from Andrew

Quoted 8 lines

I mean - its a fairly throwaway piece of misc userspace, so ack.

However, it is largely superseded by the changes you need to make to verify-stream-v2 so you might want to bear that in mind.

Also, I wonder if it is wise in general that we're throwing so many misc debugging tools into sbin.

The intention is to eventually replace hvm context so we'll need a parallel tool to replace xen-hvmctx, unless we want to retire it.

Quoted 14 lines
Quoted 14 lines
tools/misc/xen-domctx.c new file +14 -0
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+#include <xenctrl.h>
+#include <xen/xen.h>
+#include <xen/domctl.h>
+#include <xen/save.h>
+
+static void *buf = NULL;
+static size_t len, off;
+
Quoted 9 lines

tools/misc/xen-domctx.c line 46

Quoted 6 lines
tools/misc/xen-domctx.c new file +6 -0
+#define GET_PTR(_x)                                                        \
+    do {                                                                   \
+        if ( len - off < sizeof(*(_x)) )                                   \
+        {                                                                  \
+            fprintf(stderr,                                                \
+                    "error: need another %lu bytes, only %lu available\n", \

%zu is the correct formatter for size_t.

True, missed that.

Quoted 6 lines

tools/misc/xen-domctx.c line 48

Quoted 2 lines
tools/misc/xen-domctx.c new file +2 -0
+                    sizeof(*(_x)), len - off);                             \
+            exit(1);                                                       \

Your error handling will be far more simple by using err() instead of opencoding it.

Ok.

Paul

Quoted 1 line

~Andrew

#15 lore raw

PATCH v9 4/8 docs/specs: add missing definitions to libxc-migration-stream

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

v7:
- New in v7

---
 docs/specs/libxc-migration-stream.pandoc | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
docs/specs/libxc-migration-stream.pandoc +9 -2
index 6b0c49e97a..8aeab3b11b 100644
@@ -3,7 +3,7 @@
   Andrew Cooper <<[email protected]>>
   Wen Congyang <<[email protected]>>
   Yang Hongyang <<[email protected]>>
-% Revision 2
+% Revision 3
 
 Introduction
 ============
@@ -227,7 +227,13 @@ type         0x00000000: END
 
              0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary)
 
-             0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_
+             0x00000010: STATIC_DATA_END
+
+             0x00000011: X86_CPUID_POLICY
+
+             0x00000012: X86_MSR_POLICY
+
+             0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_
              records.
 
              0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -732,6 +738,7 @@ A typical save record for an x86 HVM guest image would look like:
 * X86_TSC_INFO
 * HVM_PARAMS
 * HVM_CONTEXT
+* END record
 
 HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect
 the validity of architectural state in the context.
Signature · 1 line
2.20.1
Acked-by on v9 4/8
#16 lore raw

Re: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream

Quoted 6 lines · Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Quoted 7 lines

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")
#17 lore raw

Re: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream

In reply to Paul Durrant
Quoted 6 lines · Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Quoted 7 lines

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")

Oops sorry.  I swear I had these at one point.  I can only presume it got swallowed by a rebase at some point.

~Andrew

#18 lore raw

RE: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream

Quoted 25 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 23:42
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 10 lines · Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Quoted 7 lines
Quoted 3 lines
Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")

Oops sorry. I swear I had these at one point. I can only presume it got swallowed by a rebase at some point.

Can I take that as an R-b?

Paul

Quoted 1 line

~Andrew

#19 lore raw

PATCH v9 5/8 docs / tools: specific migration v4 to include DOMAIN_CONTEXT

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>

v7:
- New in v7

---
 docs/specs/libxc-migration-stream.pandoc | 62 ++++++++++++++++++------
 tools/libs/guest/xg_sr_common.c          |  1 +
 tools/libs/guest/xg_sr_stream_format.h   |  1 +
 tools/python/xen/migration/libxc.py      |  2 +
 4 files changed, 52 insertions(+), 14 deletions(-)
docs/specs/libxc-migration-stream.pandoc +48 -14
index 8aeab3b11b..989f2a0cb6 100644
@@ -3,7 +3,7 @@
   Andrew Cooper <<[email protected]>>
   Wen Congyang <<[email protected]>>
   Yang Hongyang <<[email protected]>>
-% Revision 3
+% Revision 4
 
 Introduction
 ============
@@ -127,7 +127,7 @@ marker      0xFFFFFFFFFFFFFFFF.
 
 id          0x58454E46 ("XENF" in ASCII).
 
-version     0x00000003.  The version of this specification.
+version     0x00000004.  The version of this specification.
 
 options     bit 0: Endianness.  0 = little-endian, 1 = big-endian.
 
@@ -209,9 +209,9 @@ type         0x00000000: END
 
              0x00000006: X86_PV_VCPU_XSAVE
 
-             0x00000007: SHARED_INFO
+             0x00000007: SHARED_INFO (deprecated)
 
-             0x00000008: X86_TSC_INFO
+             0x00000008: X86_TSC_INFO (deprecated)
 
              0x00000009: HVM_CONTEXT
 
@@ -233,7 +233,9 @@ type         0x00000000: END
 
              0x00000012: X86_MSR_POLICY
 
-             0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_
+             0x00000013: DOMAIN_CONTEXT
+
+             0x00000014 - 0x7FFFFFFF: Reserved for future _mandatory_
              records.
 
              0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS             XEN_DOMCTL_{get,set}\_vcpu_msrs
 
 \clearpage
 
-SHARED_INFO
------------
+SHARED_INFO (deprecated)
+------------------------
 
-The content of the Shared Info page.
+The content of the Shared Info page. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.
 
      0     1     2     3     4     5     6     7 octet
     +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info      Contents of the shared info page.  This record
 
 \clearpage
 
-X86_TSC_INFO
-------------
+X86_TSC_INFO (deprecated)
+-------------------------
 
 Domain TSC information, as accessed by the
-XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops.
+XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.
 
      0     1     2     3     4     5     6     7 octet
     +------------------------+------------------------+
@@ -680,6 +684,25 @@ MSR_policy       Array of xen_msr_entry_t[]'s
 
 \clearpage
 
+DOMAIN_CONTEXT
+--------------
+
+Domain context, as accessed by the
+XEN_DOMCTL_{get,set}domaincontext hypercall sub-ops.
+
+     0     1     2     3     4     5     6     7 octet
+    +-------------------------------------------------+
+    | dom_ctx                                         |
+    ...
+    +-------------------------------------------------+
+
+--------------------------------------------------------------------
+Field            Description
+-----------      ---------------------------------------------------
+dom_ctx          The Domain Context blob from Xen.
+--------------------------------------------------------------------
+
+\clearpage
 
 Layout
 ======
@@ -706,8 +729,7 @@ A typical save record for an x86 PV guest image would look like:
     * STATIC_DATA_END
 * X86_PV_P2M_FRAMES record
 * Many PAGE_DATA records
-* X86_TSC_INFO
-* SHARED_INFO record
+* DOMAIN_CONTEXT
 * VCPU context records for each online VCPU
     * X86_PV_VCPU_BASIC record
     * X86_PV_VCPU_EXTENDED record
@@ -735,7 +757,7 @@ A typical save record for an x86 HVM guest image would look like:
     * X86_{CPUID,MSR}_POLICY
     * STATIC_DATA_END
 * Many PAGE_DATA records
-* X86_TSC_INFO
+* DOMAIN_CONTEXT
 * HVM_PARAMS
 * HVM_CONTEXT
 * END record
@@ -746,6 +768,18 @@ the validity of architectural state in the context.
 Compatibility with older versions
 =================================
 
+v4 compat with v3
+-----------------
+
+A v4 stream is compatible with a v3 stream, but mandates the presence of a
+DOMAIN_CONTEXT record. This incorporates context such as the content of
+the domain's Shared Info page and the TSC information, hence the SHARED_INFO
+and TSC_INFO records are deprecated.
+It also supercedes HVM_CONTEXT and, over time, data that is currently part of
+the HVM_CONTEXT blob will move to the DOMAIN_CONTEXT blob. Xen, however, will
+continue to accept all defined HVM_CONTEXT records so a v4-compatible
+receiver can still accept an unmodified v3 stream.
+
 v3 compat with v2
 -----------------
 
tools/libs/guest/xg_sr_common.c +1 -0
index 17567ab133..f813320202 100644
@@ -39,6 +39,7 @@ static const char *const mandatory_rec_types[] =
     [REC_TYPE_STATIC_DATA_END]              = "Static data end",
     [REC_TYPE_X86_CPUID_POLICY]             = "x86 CPUID policy",
     [REC_TYPE_X86_MSR_POLICY]               = "x86 MSR policy",
+    [REC_TYPE_DOMAIN_CONTEXT]               = "Domain context",
 };
 
 const char *rec_type_to_str(uint32_t type)
tools/libs/guest/xg_sr_stream_format.h +1 -0
index 8a0da26f75..bc538bc192 100644
@@ -76,6 +76,7 @@ struct xc_sr_rhdr
 #define REC_TYPE_STATIC_DATA_END            0x00000010U
 #define REC_TYPE_X86_CPUID_POLICY           0x00000011U
 #define REC_TYPE_X86_MSR_POLICY             0x00000012U
+#define REC_TYPE_DOMAIN_CONTEXT             0x00000013U
 
 #define REC_TYPE_OPTIONAL             0x80000000U
 
tools/python/xen/migration/libxc.py +2 -0
index 9881f5ced4..08ac81344f 100644
@@ -59,6 +59,7 @@ REC_TYPE_checkpoint_dirty_pfn_list  = 0x0000000f
 REC_TYPE_static_data_end            = 0x00000010
 REC_TYPE_x86_cpuid_policy           = 0x00000011
 REC_TYPE_x86_msr_policy             = 0x00000012
+REC_TYPE_domain_context             = 0x00000013
 
 rec_type_to_str = {
     REC_TYPE_end                        : "End",
@@ -80,6 +81,7 @@ rec_type_to_str = {
     REC_TYPE_static_data_end            : "Static data end",
     REC_TYPE_x86_cpuid_policy           : "x86 CPUID policy",
     REC_TYPE_x86_msr_policy             : "x86 MSR policy",
+    REC_TYPE_domain_context             : "Domain context",
 }
 
 # page_data
Signature · 1 line
2.20.1
Acked-by on v9 5/8
#20 lore raw

Re: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT

Quoted 14 lines · Paul Durrant

From: Paul Durrant <[email protected]>

A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.

Quoted 1 line

I would like Andrew to give an explicit ack on this patch.

#21 lore raw

Re: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT

In reply to Paul Durrant
Quoted 14 lines · Paul Durrant

From: Paul Durrant <[email protected]>

A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.

Quoted 1 line

What's the plan for the remainder of the work?  We don't want to burn multiple version numbers for each bit of incremental work.

One option might be to specify the full extent of the work, and use an environment variable to alter the behaviour of the sending side, while it is still work-in-progress.

~Andrew

#22 lore raw

RE: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT

Quoted 25 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 05 October 2020 11:09
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Marek Marczykowski-Górecki <[email protected]>
Subject: Re: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT

Quoted 14 lines · Paul Durrant

From: Paul Durrant <[email protected]>

A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.

Quoted 9 lines
Quoted 1 line

What's the plan for the remainder of the work? We don't want to burn multiple version numbers for each bit of incremental work.

One option might be to specify the full extent of the work, and use an environment variable to alter the behaviour of the sending side, while it is still work-in-progress.

The other missing part for transparent migration is xenstore content, but I'd expect that to sit at the next level up so I'm not anticipating any more churn at this level.

Paul

#23 lore raw

PATCH v9 6/8 common/domain: add a domain context record for shared_info...

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v9:
- Use macros to make the code less verbose
- Add missing check for allocation failure

v8:
- Incorporate zero-ing out of shared info fields that would be done in processing of SHARED_INFO from older stream versions

v7:
- Only restore vcpu_info and arch sub-structures for PV domains, to match processing of SHARED_INFO in xc_sr_restore_x86_pv.c
- Use additional option to domain_load_end() to ignore the record for HVM domains

v6:
- Only save compat_shared_info buffer if has_32bit_shinfo is set
- Validate flags field in load handler

v5:
- Addressed comments from Julien

v4:
- Addressed comments from Jan

v3:
- Actually dump some of the content of shared_info

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
 tools/misc/xen-domctx.c   |  78 ++++++++++++++++++++++++++++
 xen/common/domain.c       | 105 ++++++++++++++++++++++++++++++++++++++
 xen/include/public/save.h |  13 ++++-
 3 files changed, 195 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +78 -0
index 243325dfce..6ead7ea89d 100644
@@ -31,6 +31,7 @@
 #include <errno.h>
 
 #include <xenctrl.h>
+#include <xen-tools/libs.h>
 #include <xen/xen.h>
 #include <xen/domctl.h>
 #include <xen/save.h>
@@ -61,6 +62,82 @@ static void dump_header(void)
 
 }
 
+static void print_binary(const char *prefix, const void *val, size_t size,
+                         const char *suffix)
+{
+    printf("%s", prefix);
+
+    while ( size-- )
+    {
+        uint8_t octet = *(const uint8_t *)val++;
+        unsigned int i;
+
+        for ( i = 0; i < 8; i++ )
+        {
+            printf("%u", octet & 1);
+            octet >>= 1;
+        }
+    }
+
+    printf("%s", suffix);
+}
+
+static void dump_shared_info(void)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) *s;
+    bool has_32bit_shinfo;
+    shared_info_any_t *info;
+    unsigned int i, n;
+
+    GET_PTR(s);
+    has_32bit_shinfo = s->flags & DOMAIN_SAVE_32BIT_SHINFO;
+
+    printf("    SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n",
+           has_32bit_shinfo ? "true" : "false", s->buffer_size);
+
+    info = (shared_info_any_t *)s->buffer;
+
+#define GET_FIELD_PTR(_f)            \
+    (has_32bit_shinfo ?              \
+     (const void *)&(info->x32._f) : \
+     (const void *)&(info->x64._f))
+#define GET_FIELD_SIZE(_f) \
+    (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f))
+#define GET_FIELD(_f) \
+    (has_32bit_shinfo ? info->x32._f : info->x64._f)
+
+    n = has_32bit_shinfo ?
+        ARRAY_SIZE(info->x32.evtchn_pending) :
+        ARRAY_SIZE(info->x64.evtchn_pending);
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
+            "                 evtchn_pending: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_pending[0]),
+                 GET_FIELD_SIZE(evtchn_pending[0]), "\n");
+    }
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
+            "                    evtchn_mask: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_mask[0]),
+                 GET_FIELD_SIZE(evtchn_mask[0]), "\n");
+    }
+
+    printf("                 wc: version: %u sec: %u nsec: %u\n",
+           GET_FIELD(wc_version), GET_FIELD(wc_sec), GET_FIELD(wc_nsec));
+
+#undef GET_FIELD
+#undef GET_FIELD_SIZE
+#undef GET_FIELD_PTR
+}
+
 static void dump_end(void)
 {
     DOMAIN_SAVE_TYPE(END) *e;
@@ -173,6 +250,7 @@ int main(int argc, char **argv)
             switch (desc->typecode)
             {
             case DOMAIN_SAVE_CODE(HEADER): dump_header(); break;
+            case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(); break;
             case DOMAIN_SAVE_CODE(END): dump_end(); break;
             default:
                 printf("Unknown type %u: skipping\n", desc->typecode);
xen/common/domain.c +105 -0

117 lines not on this page · raw message

xen/include/public/save.h +12 -1
index 551dbbddb8..0e855a4b97 100644
@@ -82,7 +82,18 @@ struct domain_save_header {
 };
 DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint32_t flags;
+
+#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001
+
+    uint32_t buffer_size;
+    uint8_t buffer[XEN_FLEX_ARRAY_DIM]; /* Implementation specific size */
+};
+
+DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context);
+
+#define DOMAIN_SAVE_CODE_MAX 2
 
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
Signature · 1 line
2.20.1
Reviewed-by on v9 6/8
#24 lore raw

Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

Quoted 11 lines · Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.

Acked-by on v9 6/8
#25 lore raw

Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

In reply to Paul Durrant
Quoted 11 lines · Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.

#26 lore raw

Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

In reply to Paul Durrant
Quoted 73 lines · Paul Durrant
tools/misc/xen-domctx.c +58 -0
index 243325dfce..6ead7ea89d 100644
@@ -31,6 +31,7 @@
 #include <errno.h>
 
 #include <xenctrl.h>
+#include <xen-tools/libs.h>
 #include <xen/xen.h>
 #include <xen/domctl.h>
 #include <xen/save.h>
@@ -61,6 +62,82 @@ static void dump_header(void)
 
 }
 
+static void print_binary(const char *prefix, const void *val, size_t size,
+                         const char *suffix)
+{
+    printf("%s", prefix);
+
+    while ( size-- )
+    {
+        uint8_t octet = *(const uint8_t *)val++;
+        unsigned int i;
+
+        for ( i = 0; i < 8; i++ )
+        {
+            printf("%u", octet & 1);
+            octet >>= 1;
+        }
+    }
+
+    printf("%s", suffix);
+}
+
+static void dump_shared_info(void)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) *s;
+    bool has_32bit_shinfo;
+    shared_info_any_t *info;
+    unsigned int i, n;
+
+    GET_PTR(s);
+    has_32bit_shinfo = s->flags & DOMAIN_SAVE_32BIT_SHINFO;
+
+    printf("    SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n",
+           has_32bit_shinfo ? "true" : "false", s->buffer_size);
+
+    info = (shared_info_any_t *)s->buffer;
+
+#define GET_FIELD_PTR(_f)            \
+    (has_32bit_shinfo ?              \
+     (const void *)&(info->x32._f) : \
+     (const void *)&(info->x64._f))
+#define GET_FIELD_SIZE(_f) \
+    (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f))
+#define GET_FIELD(_f) \
+    (has_32bit_shinfo ? info->x32._f : info->x64._f)
+
+    n = has_32bit_shinfo ?
+        ARRAY_SIZE(info->x32.evtchn_pending) :
+        ARRAY_SIZE(info->x64.evtchn_pending);
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
+            "                 evtchn_pending: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_pending[0]),
+                 GET_FIELD_SIZE(evtchn_pending[0]), "\n");
+    }

tools/misc/xen-domctx.c static void dump_header(void) · line 131

Quoted 10 lines
tools/misc/xen-domctx.c +10 -0
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
+            "                    evtchn_mask: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_mask[0]),
+                 GET_FIELD_SIZE(evtchn_mask[0]), "\n");
+    }

What about domains using FIFO?  This is meaningless for them.

tools/misc/xen-domctx.c static void dump_header(void) · line 134

Quoted 3 lines
tools/misc/xen-domctx.c +3 -0
+
+    printf("                 wc: version: %u sec: %u nsec: %u\n",
+           GET_FIELD(wc_version), GET_FIELD(wc_sec), GET_FIELD(wc_nsec));

wc_sec_hi is also a rather critical field in this calculation.

Quoted 80 lines
tools/misc/xen-domctx.c +7 -0
+
+#undef GET_FIELD
+#undef GET_FIELD_SIZE
+#undef GET_FIELD_PTR
+}
+
 static void dump_end(void)
 {
     DOMAIN_SAVE_TYPE(END) *e;
@@ -173,6 +250,7 @@ int main(int argc, char **argv)
             switch (desc->typecode)
             {
             case DOMAIN_SAVE_CODE(HEADER): dump_header(); break;
+            case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(); break;
             case DOMAIN_SAVE_CODE(END): dump_end(); break;
             default:
                 printf("Unknown type %u: skipping\n", desc->typecode);
xen/common/domain.c +48 -0
index 8cfa2e0b6b..6709f9c79e 100644
@@ -33,6 +33,7 @@
 #include <xen/xenoprof.h>
 #include <xen/irq.h>
 #include <xen/argo.h>
+#include <xen/save.h>
 #include <asm/debugger.h>
 #include <asm/p2m.h>
 #include <asm/processor.h>
@@ -1657,6 +1658,110 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+static int save_shared_info(const struct domain *d, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain_shared_info_context ctxt = {
+#ifdef CONFIG_COMPAT
+        .flags = has_32bit_shinfo(d) ? DOMAIN_SAVE_32BIT_SHINFO : 0,
+        .buffer_size = has_32bit_shinfo(d) ?
+                       sizeof(struct compat_shared_info) :
+                       sizeof(struct shared_info),
+#else
+        .buffer_size = sizeof(struct shared_info),
+#endif
+    };
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    int rc;
+
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, 0);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, ctxt.buffer_size);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+static int load_shared_info(struct domain *d, struct domain_context *c)
+{
+    struct domain_shared_info_context ctxt;
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, &i);
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )

xen/common/domain.c int continue_hypercall_on_cpu( · line 1717

Quoted 10 lines
xen/common/domain.c +10 -0
+        return rc;
+
+    if ( ctxt.buffer_size > sizeof(shared_info_t) ||
+         (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO) )
+        return -EINVAL;
+
+    if ( ctxt.flags & DOMAIN_SAVE_32BIT_SHINFO )
+    {
+#ifdef CONFIG_COMPAT
+        has_32bit_shinfo(d) = true;

d->arch.has_32bit_shinfo

Quoted 5 lines
xen/common/domain.c +5 -0
+#else
+        return -EINVAL;
+#endif
+    }
+

xen/common/domain.c int continue_hypercall_on_cpu( · line 1732

Quoted 10 lines
xen/common/domain.c +10 -0
+    if ( is_pv_domain(d) )
+    {
+        shared_info_t *shinfo = xmalloc(shared_info_t);
+
+        if ( !shinfo )
+            return -ENOMEM;
+
+        rc = domain_load_data(c, shinfo, sizeof(*shinfo));
+        if ( rc )
+            goto out;

There's no need for a memory allocation, or to double buffer this data. You can memcpy() straight out of the context record.

Quoted 4 lines
xen/common/domain.c +4 -0
+
+        memcpy(&shared_info(d, vcpu_info), &__shared_info(d, shinfo, vcpu_info),
+               sizeof(shared_info(d, vcpu_info)));
+        memcpy(&shared_info(d, arch), &__shared_info(d, shinfo, arch),

xen/common/domain.c int continue_hypercall_on_cpu( · line 1746

Quoted 10 lines
xen/common/domain.c +10 -0
+               sizeof(shared_info(d, arch)));
+
+        memset(&shared_info(d, evtchn_pending), 0,
+               sizeof(shared_info(d, evtchn_pending)));
+        memset(&shared_info(d, evtchn_mask), 0xff,
+               sizeof(shared_info(d, evtchn_mask)));
+
+        shared_info(d, arch.pfn_to_mfn_frame_list_list) = 0;
+        for ( i = 0; i < XEN_LEGACY_MAX_VCPUS; i++ )
+            shared_info(d, vcpu_info[i].evtchn_pending_sel) = 0;

What is the plan for transparent migrate here?  While this is ok for regular migrate, its definitely not for transparent.

Quoted 26 lines
xen/common/domain.c +18 -0
+
+        rc = domain_load_end(c, false);
+
+    out:
+        xfree(shinfo);
+    }
+    else
+        /*
+         * No modifications to shared_info are required for restoring non-PV
+         * domains.
+         */
+        rc = domain_load_end(c, true);
+
+    return rc;
+}
+
+DOMAIN_REGISTER_SAVE_LOAD(SHARED_INFO, save_shared_info, load_shared_info);
+
 /*
  * Local variables:
  * mode: C
xen/include/public/save.h +0 -0
index 551dbbddb8..0e855a4b97 100644

xen/include/public/save.h struct domain_save_header { · line 90

Quoted 10 lines
xen/include/public/save.h +6 -1
@@ -82,7 +82,18 @@ struct domain_save_header {
 };
 DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint32_t flags;
+
+#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001
+
+    uint32_t buffer_size;

This struct is already wrapped with a header including a size which encompasses buffer.

Multiple overlapping size fields is an easy way to memory corruption, because it causes ambiguity as to which one is right.

~Andrew

#27 lore raw

RE: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

Quoted 87 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 05 October 2020 11:40
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

Quoted 77 lines · Paul Durrant
tools/misc/xen-domctx.c +62 -0
index 243325dfce..6ead7ea89d 100644
@@ -31,6 +31,7 @@
 #include <errno.h>
 
 #include <xenctrl.h>
+#include <xen-tools/libs.h>
 #include <xen/xen.h>
 #include <xen/domctl.h>
 #include <xen/save.h>
@@ -61,6 +62,82 @@ static void dump_header(void)
 
 }
 
+static void print_binary(const char *prefix, const void *val, size_t size,
+                         const char *suffix)
+{
+    printf("%s", prefix);
+
+    while ( size-- )
+    {
+        uint8_t octet = *(const uint8_t *)val++;
+        unsigned int i;
+
+        for ( i = 0; i < 8; i++ )
+        {
+            printf("%u", octet & 1);
+            octet >>= 1;
+        }
+    }
+
+    printf("%s", suffix);
+}
+
+static void dump_shared_info(void)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) *s;
+    bool has_32bit_shinfo;
+    shared_info_any_t *info;
+    unsigned int i, n;
+
+    GET_PTR(s);
+    has_32bit_shinfo = s->flags & DOMAIN_SAVE_32BIT_SHINFO;
+
+    printf("    SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n",
+           has_32bit_shinfo ? "true" : "false", s->buffer_size);
+
+    info = (shared_info_any_t *)s->buffer;
+
+#define GET_FIELD_PTR(_f)            \
+    (has_32bit_shinfo ?              \
+     (const void *)&(info->x32._f) : \
+     (const void *)&(info->x64._f))
+#define GET_FIELD_SIZE(_f) \
+    (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f))
+#define GET_FIELD(_f) \
+    (has_32bit_shinfo ? info->x32._f : info->x64._f)
+
+    n = has_32bit_shinfo ?
+        ARRAY_SIZE(info->x32.evtchn_pending) :
+        ARRAY_SIZE(info->x64.evtchn_pending);
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
+            "                 evtchn_pending: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_pending[0]),
+                 GET_FIELD_SIZE(evtchn_pending[0]), "\n");
+    }
+
+    for ( i = 0; i < n; i++ )
+    {
+        const char *prefix = !i ?
Quoted 9 lines

tools/misc/xen-domctx.c static void dump_header(void) · line 131

Quoted 6 lines
tools/misc/xen-domctx.c +6 -0
+            "                    evtchn_mask: " :
+            "                                 ";
+
+        print_binary(prefix, GET_FIELD_PTR(evtchn_mask[0]),
+                 GET_FIELD_SIZE(evtchn_mask[0]), "\n");
+    }

What about domains using FIFO? This is meaningless for them.

Indeed, but this is essentially a debug tool so I'd rather it just dumped everything that might be useful.

Quoted 6 lines

tools/misc/xen-domctx.c static void dump_header(void) · line 134

Quoted 3 lines
tools/misc/xen-domctx.c +3 -0
+
+    printf("                 wc: version: %u sec: %u nsec: %u\n",
+           GET_FIELD(wc_version), GET_FIELD(wc_sec), GET_FIELD(wc_nsec));

wc_sec_hi is also a rather critical field in this calculation.

Ok.

Quoted 84 lines
Quoted 84 lines
tools/misc/xen-domctx.c +7 -0
+
+#undef GET_FIELD
+#undef GET_FIELD_SIZE
+#undef GET_FIELD_PTR
+}
+
 static void dump_end(void)
 {
     DOMAIN_SAVE_TYPE(END) *e;
@@ -173,6 +250,7 @@ int main(int argc, char **argv)
             switch (desc->typecode)
             {
             case DOMAIN_SAVE_CODE(HEADER): dump_header(); break;
+            case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(); break;
             case DOMAIN_SAVE_CODE(END): dump_end(); break;
             default:
                 printf("Unknown type %u: skipping\n", desc->typecode);
xen/common/domain.c +52 -0
index 8cfa2e0b6b..6709f9c79e 100644
@@ -33,6 +33,7 @@
 #include <xen/xenoprof.h>
 #include <xen/irq.h>
 #include <xen/argo.h>
+#include <xen/save.h>
 #include <asm/debugger.h>
 #include <asm/p2m.h>
 #include <asm/processor.h>
@@ -1657,6 +1658,110 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+static int save_shared_info(const struct domain *d, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain_shared_info_context ctxt = {
+#ifdef CONFIG_COMPAT
+        .flags = has_32bit_shinfo(d) ? DOMAIN_SAVE_32BIT_SHINFO : 0,
+        .buffer_size = has_32bit_shinfo(d) ?
+                       sizeof(struct compat_shared_info) :
+                       sizeof(struct shared_info),
+#else
+        .buffer_size = sizeof(struct shared_info),
+#endif
+    };
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    int rc;
+
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, 0);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, ctxt.buffer_size);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+static int load_shared_info(struct domain *d, struct domain_context *c)
+{
+    struct domain_shared_info_context ctxt;
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, &i);
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    if ( ctxt.buffer_size > sizeof(shared_info_t) ||
+         (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO) )
Quoted 9 lines

xen/common/domain.c int continue_hypercall_on_cpu( · line 1717

Quoted 6 lines
xen/common/domain.c +6 -0
+        return -EINVAL;
+
+    if ( ctxt.flags & DOMAIN_SAVE_32BIT_SHINFO )
+    {
+#ifdef CONFIG_COMPAT
+        has_32bit_shinfo(d) = true;

d->arch.has_32bit_shinfo

If you'd prefer, ok.

Quoted 10 lines
Quoted 10 lines
xen/common/domain.c +10 -0
+#else
+        return -EINVAL;
+#endif
+    }
+
+    if ( is_pv_domain(d) )
+    {
+        shared_info_t *shinfo = xmalloc(shared_info_t);
+
+        if ( !shinfo )
Quoted 9 lines

xen/common/domain.c int continue_hypercall_on_cpu( · line 1732

Quoted 5 lines
xen/common/domain.c +5 -0
+            return -ENOMEM;
+
+        rc = domain_load_data(c, shinfo, sizeof(*shinfo));
+        if ( rc )
+            goto out;

There's no need for a memory allocation, or to double buffer this data. You can memcpy() straight out of the context record.

That would mean re-working the way that domain_load_data() works. I'd really rather not.

Quoted 9 lines
Quoted 9 lines
xen/common/domain.c +9 -0
+
+        memcpy(&shared_info(d, vcpu_info), &__shared_info(d, shinfo, vcpu_info),
+               sizeof(shared_info(d, vcpu_info)));
+        memcpy(&shared_info(d, arch), &__shared_info(d, shinfo, arch),
+               sizeof(shared_info(d, arch)));
+
+        memset(&shared_info(d, evtchn_pending), 0,
+               sizeof(shared_info(d, evtchn_pending)));
+        memset(&shared_info(d, evtchn_mask), 0xff,
Quoted 9 lines

xen/common/domain.c int continue_hypercall_on_cpu( · line 1746

Quoted 5 lines
xen/common/domain.c +5 -0
+               sizeof(shared_info(d, evtchn_mask)));
+
+        shared_info(d, arch.pfn_to_mfn_frame_list_list) = 0;
+        for ( i = 0; i < XEN_LEGACY_MAX_VCPUS; i++ )
+            shared_info(d, vcpu_info[i].evtchn_pending_sel) = 0;

What is the plan for transparent migrate here? While this is ok for regular migrate, its definitely not for transparent.

Quite true, as evidenced that this is inside 'if ( is_pv_domain(d) )'. It is not yet clear how much of the shared info we need for transparent migrate. It may be nothing.

Quoted 40 lines
Quoted 26 lines
xen/common/domain.c +18 -0
+
+        rc = domain_load_end(c, false);
+
+    out:
+        xfree(shinfo);
+    }
+    else
+        /*
+         * No modifications to shared_info are required for restoring non-PV
+         * domains.
+         */
+        rc = domain_load_end(c, true);
+
+    return rc;
+}
+
+DOMAIN_REGISTER_SAVE_LOAD(SHARED_INFO, save_shared_info, load_shared_info);
+
 /*
  * Local variables:
  * mode: C
xen/include/public/save.h +0 -0
index 551dbbddb8..0e855a4b97 100644

xen/include/public/save.h struct domain_save_header { · line 90

Quoted 10 lines
xen/include/public/save.h +6 -1
@@ -82,7 +82,18 @@ struct domain_save_header {
 };
 DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint32_t flags;
+
+#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001
+
+    uint32_t buffer_size;

This struct is already wrapped with a header including a size which encompasses buffer.

Quoted 3 lines

Multiple overlapping size fields is an easy way to memory corruption, because it causes ambiguity as to which one is right.

The record size currently includes padding. I'm re-working that in v10 and so this size can be dropped.

Paul

#28 lore raw

Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...

In reply to Andrew Cooper
Quoted 21 lines · Andrew Cooper
Quoted 20 lines · Paul Durrant
xen/common/domain.c +20 -0
+static int load_shared_info(struct domain *d, struct domain_context *c)
+{
+    struct domain_shared_info_context ctxt;
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, &i);
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    if ( ctxt.buffer_size > sizeof(shared_info_t) ||
+         (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO) )
Quoted 8 lines

xen/common/domain.c int continue_hypercall_on_cpu( · line 1717

Quoted 6 lines
xen/common/domain.c +6 -0
+        return -EINVAL;
+
+    if ( ctxt.flags & DOMAIN_SAVE_32BIT_SHINFO )
+    {
+#ifdef CONFIG_COMPAT
+        has_32bit_shinfo(d) = true;

d->arch.has_32bit_shinfo

But this is common code, i.e. using d->arch directly is a layering violation. I know your dislike of lvalues disguised by function- like macros, but what do you do?

Jan

#29 lore raw

PATCH v9 7/8 x86/time: add a domain context record for tsc_info...

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Whilst the record definition is x86 specific, it is visible directly
in the common header as context record numbers should be unique across all architectures.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v8:
- Removed stray blank line

v7:
- New in v7

---
 tools/misc/xen-domctx.c    | 11 +++++++++++
 xen/arch/x86/time.c        | 34 +++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/time.h |  5 +++--
 xen/include/public/save.h  | 13 ++++++++++++-
 4 files changed, 59 insertions(+), 4 deletions(-)
tools/misc/xen-domctx.c +11 -0
index 6ead7ea89d..e582a79678 100644
@@ -59,7 +59,17 @@ static void dump_header(void)
 
     printf("    HEADER: magic %#x, version %u\n",
            h->magic, h->version);
+}
+
+static void dump_tsc_info(void)
+{
+    DOMAIN_SAVE_TYPE(TSC_INFO) *t;
+
+    GET_PTR(t);
 
+    printf("    TSC_INFO: mode: %u incarnation: %u\n"
+           "              khz %u elapsed_nsec: %"PRIu64"\n",
+           t->mode, t->incarnation, t->khz, t->elapsed_nsec);
 }
 
 static void print_binary(const char *prefix, const void *val, size_t size,
@@ -251,6 +261,7 @@ int main(int argc, char **argv)
             {
             case DOMAIN_SAVE_CODE(HEADER): dump_header(); break;
             case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(); break;
+            case DOMAIN_SAVE_CODE(TSC_INFO): dump_tsc_info(); break;
             case DOMAIN_SAVE_CODE(END): dump_end(); break;
             default:
                 printf("Unknown type %u: skipping\n", desc->typecode);
xen/arch/x86/time.c +33 -1
index 8938c0f435..25731f7df4 100644
@@ -26,6 +26,7 @@
 #include <xen/symbols.h>
 #include <xen/keyhandler.h>
 #include <xen/guest_access.h>
+#include <xen/save.h>
 #include <asm/io.h>
 #include <asm/iocap.h>
 #include <asm/msr.h>
@@ -2334,7 +2335,7 @@ int host_tsc_is_safe(void)
  * called to collect tsc-related data only for save file or live
  * migrate; called after last rdtsc is done on this incarnation
  */
-void tsc_get_info(struct domain *d, uint32_t *tsc_mode,
+void tsc_get_info(const struct domain *d, uint32_t *tsc_mode,
                   uint64_t *elapsed_nsec, uint32_t *gtsc_khz,
                   uint32_t *incarnation)
 {
@@ -2451,6 +2452,37 @@ int tsc_set_info(struct domain *d,
     return 0;
 }
 
+static int save_tsc_info(const struct domain *d, struct domain_context *c,
+                         bool dry_run)
+{
+    struct domain_tsc_info_context ctxt;
+
+    if ( !dry_run )
+        tsc_get_info(d, &ctxt.mode, &ctxt.elapsed_nsec, &ctxt.khz,
+                     &ctxt.incarnation);
+
+    return DOMAIN_SAVE_ENTRY(TSC_INFO, c, 0, &ctxt, sizeof(ctxt));
+}
+
+static int load_tsc_info(struct domain *d, struct domain_context *c)
+{
+    struct domain_tsc_info_context ctxt;
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_ENTRY(TSC_INFO, c, &i, &ctxt, sizeof(ctxt));
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    return tsc_set_info(d, ctxt.mode, ctxt.elapsed_nsec, ctxt.khz,
+                        ctxt.incarnation);
+}
+
+DOMAIN_REGISTER_SAVE_LOAD(TSC_INFO, save_tsc_info, load_tsc_info);
+
 /* vtsc may incur measurable performance degradation, diagnose with this */
 static void dump_softtsc(unsigned char key)
 {
xen/include/asm-x86/time.h +3 -2
index f347311cc4..7f2ce6226a 100644
@@ -59,8 +59,9 @@ u64 gtsc_to_gtime(struct domain *d, u64 tsc);
 int tsc_set_info(struct domain *d, uint32_t tsc_mode, uint64_t elapsed_nsec,
                  uint32_t gtsc_khz, uint32_t incarnation);
 
-void tsc_get_info(struct domain *d, uint32_t *tsc_mode, uint64_t *elapsed_nsec,
-                  uint32_t *gtsc_khz, uint32_t *incarnation);
+void tsc_get_info(const struct domain *d, uint32_t *tsc_mode,
+                  uint64_t *elapsed_nsec, uint32_t *gtsc_khz,
+                  uint32_t *incarnation);
    
 
 void force_update_vcpu_system_time(struct vcpu *v);
xen/include/public/save.h +12 -1
index 0e855a4b97..aeb17298eb 100644
@@ -93,7 +93,18 @@ struct domain_shared_info_context {
 
 DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context);
 
-#define DOMAIN_SAVE_CODE_MAX 2
+#if defined(__i386__) || defined(__x86_64__)
+struct domain_tsc_info_context {
+    uint32_t mode;
+    uint32_t incarnation;
+    uint64_t elapsed_nsec;
+    uint32_t khz;
+};
+
+DECLARE_DOMAIN_SAVE_TYPE(TSC_INFO, 3, struct domain_tsc_info_context);
+#endif
+
+#define DOMAIN_SAVE_CODE_MAX 3
 
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
Signature · 1 line
2.20.1
Acked-by on v9 7/8
#30 lore raw

Re: [PATCH v9 7/8] x86/time: add a domain context record for tsc_info...

Quoted 10 lines · Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Whilst the record definition is x86 specific, it is visible directly
in the common header as context record numbers should be unique across all architectures.

#31 lore raw

PATCH v9 8/8 tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

... and bump the version.

This patch implements version 4 of the migration stream by adding the code necessary to save and restore DOMAIN_CONTEXT records, and removing the code to save the SHARED_INFO and TSC_INFO records (as these are deprecated in version 4).

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v7:
- New in v7

---
 tools/libs/guest/xg_sr_common.h       |  3 ++
 tools/libs/guest/xg_sr_common_x86.c   | 20 -----------
 tools/libs/guest/xg_sr_common_x86.h   |  6 ----
 tools/libs/guest/xg_sr_restore.c      | 45 +++++++++++++++++++++--
 tools/libs/guest/xg_sr_save.c         | 52 ++++++++++++++++++++++++++-
 tools/libs/guest/xg_sr_save_x86_hvm.c |  5 ---
 tools/libs/guest/xg_sr_save_x86_pv.c  | 22 ------------
 7 files changed, 97 insertions(+), 56 deletions(-)
tools/libs/guest/xg_sr_common.h +3 -0
index 13fcc47420..d440281cc1 100644
@@ -298,6 +298,9 @@ struct xc_sr_context
 
             /* Sender has invoked verify mode on the stream. */
             bool verify;
+
+            /* Domain context blob. */
+            struct xc_sr_blob context;
         } restore;
     };
 
tools/libs/guest/xg_sr_common_x86.c +0 -20
index 6f12483907..10a35b998e 100644
@@ -1,25 +1,5 @@
 #include "xg_sr_common_x86.h"
 
-int write_x86_tsc_info(struct xc_sr_context *ctx)
-{
-    xc_interface *xch = ctx->xch;
-    struct xc_sr_rec_x86_tsc_info tsc = {};
-    struct xc_sr_record rec = {
-        .type = REC_TYPE_X86_TSC_INFO,
-        .length = sizeof(tsc),
-        .data = &tsc,
-    };
-
-    if ( xc_domain_get_tsc_info(xch, ctx->domid, &tsc.mode,
-                                &tsc.nsec, &tsc.khz, &tsc.incarnation) < 0 )
-    {
-        PERROR("Unable to obtain TSC information");
-        return -1;
-    }
-
-    return write_record(ctx, &rec);
-}
-
 int handle_x86_tsc_info(struct xc_sr_context *ctx, struct xc_sr_record *rec)
 {
     xc_interface *xch = ctx->xch;
tools/libs/guest/xg_sr_common_x86.h +0 -6
index b55758c96d..e504169705 100644
@@ -3,12 +3,6 @@
 
 #include "xg_sr_common.h"
 
-/*
- * Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record
- * into the stream.
- */
-int write_x86_tsc_info(struct xc_sr_context *ctx);
-
 /*
  * Parses a X86_TSC_INFO record and applies the result to the domain.
  */
tools/libs/guest/xg_sr_restore.c +43 -2

82 lines not on this page · raw message

tools/libs/guest/xg_sr_save.c +51 -1

70 lines not on this page · raw message

tools/libs/guest/xg_sr_save_x86_hvm.c +0 -5
index 1634a7bc43..d44fb3fc4f 100644
@@ -193,11 +193,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)
 {
     int rc;
 
-    /* Write the TSC record. */
-    rc = write_x86_tsc_info(ctx);
-    if ( rc )
-        return rc;
-
     /* Write the HVM_CONTEXT record. */
     rc = write_hvm_context(ctx);
     if ( rc )
tools/libs/guest/xg_sr_save_x86_pv.c +0 -22
index 4964f1f7b8..3de7b19f54 100644
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)
     return rc;
 }
 
-/*
- * Writes an SHARED_INFO record into the stream.
- */
-static int write_shared_info(struct xc_sr_context *ctx)
-{
-    struct xc_sr_record rec = {
-        .type = REC_TYPE_SHARED_INFO,
-        .length = PAGE_SIZE,
-        .data = ctx->x86.pv.shinfo,
-    };
-
-    return write_record(ctx, &rec);
-}
-
 /*
  * Normalise a pagetable for the migration stream.  Performs mfn->pfn
  * conversions on the ptes.
@@ -1093,14 +1079,6 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx)
 {
     int rc;
 
-    rc = write_x86_tsc_info(ctx);
-    if ( rc )
-        return rc;
-
-    rc = write_shared_info(ctx);
-    if ( rc )
-        return rc;
-
     rc = write_all_vcpu_information(ctx);
     if ( rc )
         return rc;
Signature · 1 line
2.20.1
Acked-by on v9 8/8
#32 lore raw

Re: [PATCH v9 8/8] tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

Quoted 10 lines · Paul Durrant

From: Paul Durrant <[email protected]>

... and bump the version.

This patch implements version 4 of the migration stream by adding the code necessary to save and restore DOMAIN_CONTEXT records, and removing the code to save the SHARED_INFO and TSC_INFO records (as these are deprecated in version 4).

Again, I think Andrew should have a look at this as well.

#33 lore raw

Re: [PATCH v9 8/8] tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

In reply to Paul Durrant
Quoted 10 lines · Paul Durrant

From: Paul Durrant <[email protected]>

... and bump the version.

This patch implements version 4 of the migration stream by adding the code necessary to save and restore DOMAIN_CONTEXT records, and removing the code to save the SHARED_INFO and TSC_INFO records (as these are deprecated in version 4).

This really needs to be at least 3 patches.

First to adjust tools/python/scripts/verify-stream-v2 to understand the new changes in the stream.

My testing tends to include running the script over the result of xl save, and using a script in place of libxl-save-helper which tee's the stream through the script, which lets you test in-line migrate.  (I wonder if it would be better to add a pass-through mode to the script and give libxl a way of running it in the same way as it currently runs covert-legacy-stream.)

Next, a patch updating the receive side only to understand the new changes in the stream.  In particular, this makes it far easier to confirm that backwards compatibility is maintained.

Finally, a patch updating the sending side, if applicable.  (I've got an alternative suggestion to avoid burning a load of major version numbers, but will follow up on a different patch with that).

Quoted 21 lines

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v7:
- New in v7

---
 tools/libs/guest/xg_sr_common.h       |  3 ++
 tools/libs/guest/xg_sr_common_x86.c   | 20 -----------
 tools/libs/guest/xg_sr_common_x86.h   |  6 ----
 tools/libs/guest/xg_sr_restore.c      | 45 +++++++++++++++++++++--
 tools/libs/guest/xg_sr_save.c         | 52 ++++++++++++++++++++++++++-
 tools/libs/guest/xg_sr_save_x86_hvm.c |  5 ---
 tools/libs/guest/xg_sr_save_x86_pv.c  | 22 ------------
 7 files changed, 97 insertions(+), 56 deletions(-)
tools/libs/guest/xg_sr_common.h +0 -0
index 13fcc47420..d440281cc1 100644

tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 303

Quoted 6 lines
tools/libs/guest/xg_sr_common.h +3 -0
@@ -298,6 +298,9 @@ struct xc_sr_context
 
             /* Sender has invoked verify mode on the stream. */
             bool verify;
+
+            /* Domain context blob. */
+            struct xc_sr_blob context;

We already have

ctx->x86.hvm.restore.context

and are now gaining

ctx->restore.context

This is concerning close to being ambiguous.  How about dom_context ?

Also, you leak the memory allocation.  Free it in xg_sr_restore.c:cleanup().

Quoted 11 lines
tools/libs/guest/xg_sr_restore.c +3 -0
index b57a787519..453a383ba4 100644
@@ -529,6 +529,20 @@ static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx)
     return rc;
 }
 
+static int stream_complete(struct xc_sr_context *ctx)
+{
+    xc_interface *xch = ctx->xch;

tools/libs/guest/xg_sr_restore.c static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx) · line 544

Quoted 10 lines
tools/libs/guest/xg_sr_restore.c +10 -0
+    int rc;
+
+    rc = xc_domain_setcontext(xch, ctx->domid,
+                              ctx->restore.context.ptr,
+                              ctx->restore.context.size);
+    if ( rc < 0 )
+        PERROR("Unable to restore domain context");
+
+    return rc;
+}

Please put this in the PV and HVM stream_complete() hooks.

This is somewhat of a layering violation and enforcing an order which might not be appropriate in all cases.

~Andrew

#34 lore raw
In reply to Paul Durrant
Quoted 20 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Paul Durrant
Sent: Thursday, September 24, 2020 9:10 AM
To: [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH v9 0/8] domain context infrastructure

From: Paul Durrant <[email protected]>

Quoted 10 lines

Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to include DOMAIN_CONTEXT common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the series.

Thanks,
Tamas

#35 lore raw
Quoted 44 lines

-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 24 September 2020 20:36
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [PATCH v9 0/8] domain context infrastructure

Quoted 18 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Paul Durrant
Sent: Thursday, September 24, 2020 9:10 AM
To: [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH v9 0/8] domain context infrastructure

From: Paul Durrant <[email protected]>

Quoted 10 lines

Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to include DOMAIN_CONTEXT common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... tools/libxc: add DOMAIN_CONTEXT records to the migration stream...

Quoted 5 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the series.

Hi Tamas,

Done. See https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/heads/domain-save14

Cheers,

Paul

#36 lore raw
Quoted 9 lines
Quoted 4 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the

series.

Quoted 1 line

Hi Tamas,

Quoted 7 lines

Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14

Cheers,

Paul

Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function copy_pv_domaincontext for now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.

Thanks,
Tamas

#37 lore raw
Quoted 34 lines

-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 9 lines
Quoted 4 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the

series.

Quoted 1 line

Hi Tamas,

Quoted 7 lines

Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14

Cheers,

Paul

Quoted 8 lines

Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at
https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function copy_pv_domaincontext for now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.

Hi Tamas,

The code structure appears to be ok... just some cosmetic tweaks:

- I think you should call the function simply 'copy_domaincontext' as the idea is that all state (including what is now in hvm context) will be consolidated
- The prevailing style in domctl.c AFAICS is that assignments are mostly not done inside if statements. Personally I think this is a good thing.

Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a v10... but I'm currently hoping that won't be necessary.

Cheers,

Paul

Quoted 3 lines

Thanks,
Tamas

#38 lore raw
Quoted 44 lines · Paul
Quoted 35 lines

-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 9 lines
Quoted 4 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the

series.

Quoted 1 line

Hi Tamas,

Quoted 7 lines

Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14

Cheers,

Paul

Quoted 8 lines

Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at
https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function copy_pv_domaincontext for now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.

Quoted 5 lines

Hi Tamas,

The code structure appears to be ok... just some cosmetic tweaks:

- I think you should call the function simply 'copy_domaincontext' as the idea is that all state (including what is now in hvm context) will be consolidated

Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.

Quoted 1 line

- The prevailing style in domctl.c AFAICS is that assignments are mostly not done inside if statements. Personally I think this is a good thing.

I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.

Quoted 2 lines

Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a v10... but I'm currently hoping that won't be necessary.

I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.

Tamas

#39 lore raw
Quoted 88 lines

-----Original Message-----
From: Tamas K Lengyel <[email protected]>
Sent: 29 September 2020 13:06
To: Durrant, Paul <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected]; [email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 31 lines · Paul
Quoted 31 lines

-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 13 lines
Quoted 4 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the

series.

Quoted 1 line

Hi Tamas,

Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Cheers,

Paul

Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would

be

Quoted 2 lines
Quoted 2 lines

called during VM forking. Please take a look at

https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4

Quoted 1 line
Quoted 1 line

d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function

copy_pv_domaincontext

Quoted 1 line
Quoted 1 line

for now as that seemed like the most appropriate description for it. Please let me know if this

looks

Quoted 1 line
Quoted 1 line

good to you. I'm still testing it but if everything checks out it would be nice to just append

this

Quoted 7 lines
Quoted 1 line

patch to your series.

Hi Tamas,

The code structure appears to be ok... just some cosmetic tweaks:

- I think you should call the function simply 'copy_domaincontext' as the idea is that all state

(including what is now in hvm context) will be consolidated

Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.

Quoted 1 line

- The prevailing style in domctl.c AFAICS is that assignments are mostly not done inside if

statements. Personally I think this is a good thing.

I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a

v10... but I'm currently hoping that won't be necessary.

I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.

It may be a while before I have everything moved over so you may still want to go ahead with this patch if the delay is likely to block things. Also, without this I assume any records I port over from HVM context (and hence remove the save code) are going to cause breakage for VM forking?

Paul

Quoted 1 line

Tamas

#40 lore raw
Quoted 128 lines

-----Original Message-----
From: Durrant, Paul <[email protected]>
Sent: Tuesday, September 29, 2020 8:14 AM
To: Tamas K Lengyel <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected]; xen- [email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [PATCH v9 0/8] domain context infrastructure

Quoted 5 lines

-----Original Message-----
From: Tamas K Lengyel <[email protected]>
Sent: 29 September 2020 13:06
To: Durrant, Paul <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected];

Quoted 8 lines

[email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu

<[email protected]>

Quoted 4 lines

Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and

know the content is safe.

Quoted 4 lines

On Tue, Sep 29, 2020 at 7:54 AM Durrant, Paul <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf'

<[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson'

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

<[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki'

<[email protected]>; 'Roger Pau Monné'

Quoted 1 line
Quoted 1 line
Quoted 1 line

<[email protected]>; 'Stefano Stabellini' <[email protected]>;

'Volodymyr Babchuk'

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender

and know the content is safe.

Quoted 68 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines

Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the

series.

Quoted 1 line

Hi Tamas,

Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortl
og;h=refs/h
eads/domain-save14

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Cheers,

Paul

Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would

be

Quoted 2 lines
Quoted 2 lines

called during VM forking. Please take a look at

https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdif
f;h=1843ca7302e415317fdb9a63b3a4

Quoted 2 lines
Quoted 2 lines

d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function

copy_pv_domaincontext

Quoted 2 lines
Quoted 2 lines

for now as that seemed like the most appropriate description for it. Please let me know if this

looks

Quoted 2 lines
Quoted 2 lines

good to you. I'm still testing it but if everything checks out it would be nice to just append

this

Quoted 8 lines
Quoted 1 line

patch to your series.

Hi Tamas,

The code structure appears to be ok... just some cosmetic tweaks:

- I think you should call the function simply 'copy_domaincontext' as the idea is that all state

(including what is now in hvm context) will be consolidated

Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.

Quoted 2 lines

- The prevailing style in domctl.c AFAICS is that assignments are mostly not done inside if

statements. Personally I think this is a good thing.

I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.

Quoted 3 lines

Once you have something ready to go then I'd be happy to tag it
onto my series if I need to do a

v10... but I'm currently hoping that won't be necessary.

Quoted 8 lines
Quoted 3 lines

I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.

It may be a while before I have everything moved over so you may still want to go ahead with this patch if the delay is likely to block things. Also, without this I assume any records I port over from HVM context (and hence remove the save code) are going to cause breakage for VM forking?

If you do remove the existing hvm_save/hvm_load functions then yes, that would pretty much break VM forking right away. We use those in hvm_copy_context_and_params. So if you do make significant changes in there I would ask that you at least compile test with CONFIG_MEM_SHARING enabled. That would be the point where we would want to start using this new copy_domaincontext function. Right now just for the PV stuff it's not critical as we do copy those internally ourself, it would just be nicer to use this plumbing you add that will be shared with domain save/restore/migrate.

Tamas