Skip to contentSkip to conversation

domain context infrastructure

PATCH In review Paul Durrant 3 participants 30 messages Oct 8, 2020 – Jan 26, 2021

lore
Share & tools
lore
#0 lore raw

v10 cover domain context infrastructure

From: Paul Durrant <[email protected]>

Paul Durrant (11):
docs / include: introduce a new framework for 'domain context' records
xen: introduce implementation of save/restore of 'domain context' xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context tools/misc: add xen-domctx to present domain context common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... docs/specs: add missing definitions to libxc-migration-stream docs / tools: specify migration v4 to include DOMAIN_CONTEXT tools/python: modify libxc.py to verify v4 stream tools/libs/guest: add code to restore a v4 libxc stream tools/libs/guest: add code to save a v4 libxc stream

 .gitignore                               |   1 +
 docs/specs/domain-context.md             | 198 +++++++++++++
 docs/specs/libxc-migration-stream.pandoc |  69 ++++-
 tools/flask/policy/modules/xen.if        |   4 +-
 tools/libs/ctrl/include/xenctrl.h        |   5 +
 tools/libs/ctrl/xc_domain.c              |  56 ++++
 tools/libs/guest/xg_sr_common.c          |   1 +
 tools/libs/guest/xg_sr_common.h          |   3 +
 tools/libs/guest/xg_sr_common_x86.c      |  20 --
 tools/libs/guest/xg_sr_common_x86.h      |   6 -
 tools/libs/guest/xg_sr_restore.c         |  24 +-
 tools/libs/guest/xg_sr_restore_x86_hvm.c |   9 +
 tools/libs/guest/xg_sr_restore_x86_pv.c  |   9 +
 tools/libs/guest/xg_sr_save.c            |  52 +++-
 tools/libs/guest/xg_sr_save_x86_hvm.c    |   5 -
 tools/libs/guest/xg_sr_save_x86_pv.c     |  22 --
 tools/libs/guest/xg_sr_stream_format.h   |   1 +
 tools/misc/Makefile                      |   4 +
 tools/misc/xen-domctx.c                  | 264 ++++++++++++++++++
 tools/python/xen/migration/libxc.py      |  22 +-
 xen/arch/x86/time.c                      |  30 ++
 xen/common/Makefile                      |   1 +
 xen/common/domain.c                      | 113 ++++++++
 xen/common/domctl.c                      | 173 ++++++++++++
 xen/common/save.c                        | 339 +++++++++++++++++++++++
 xen/include/public/arch-arm/hvm/save.h   |   5 +
 xen/include/public/arch-x86/hvm/save.h   |   5 +
 xen/include/public/domctl.h              |  39 +++
 xen/include/public/save.h                |  85 ++++++
 xen/include/xen/save.h                   | 160 +++++++++++
 xen/xsm/flask/hooks.c                    |   6 +
 xen/xsm/flask/policy/access_vectors      |   4 +
 32 files changed, 1661 insertions(+), 74 deletions(-)
 create mode 100644 docs/specs/domain-context.md
 create mode 100644 tools/misc/xen-domctx.c
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/public/save.h
 create mode 100644 xen/include/xen/save.h

---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>

Signature · 1 line
2.20.1
#1 lore raw

PATCH v10 1/11 docs / include: introduce a new framework for 'domain context' records

From: Paul Durrant <[email protected]>

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the specification of a new common 'domain context' image and the supporting public header. Subsequent patches will introduce other parts of the framework, and code that will make use of it.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v10:
- New in v10

---
 docs/specs/domain-context.md           | 137 +++++++++++++++++++++++++
 xen/include/public/arch-arm/hvm/save.h |   5 +
 xen/include/public/arch-x86/hvm/save.h |   5 +
 xen/include/public/save.h              |  66 ++++++++++++
 4 files changed, 213 insertions(+)
 create mode 100644 docs/specs/domain-context.md
 create mode 100644 xen/include/public/save.h
docs/specs/domain-context.md new file +137 -0
new file mode 100644
index 0000000000..f177cf24b3
@@ -0,0 +1,137 @@
+# Domain Context (v1)
+
+## Background
+
+The design for *Non-Cooperative Migration of Guests*[1] explains that extra
+information is required in the migration stream to allow a guest running PV
+drivers to be migrated without its co-operation. This information includes
+hypervisor state such as the set of event channels in operation and the
+content of the grant table.
+
+There already exists in Xen a mechanism to save and restore *HVM context*[2].
+This specification describes a new framework that will replace that
+mechanism and be suitable for transferring additional *PV state* records
+conveying the information mentioned above. There is also on-going work to
+implement *live update* of Xen where hypervisor state must be transferred in
+memory from one incarnation to the next. The framework described is designed
+to also be suitable for this purpose.
+
+## Image format
+
+The format will read from or written to the hypervisor in a single virtually
+contiguous buffer segmented into **context records** specified in the following
+sections.
+
+Fields within the records will always be aligned to their size. Padding and
+reserved fields will always be set to zero when the context buffer is read
+from the hypervisor and will be verified when written.
+The endianness of numerical values will the native endianness of the
+hypervisor. In the case of migration, that endianness is specified in the
+*libxenctrl (libxc) Domain Image Format*[3].
+
+### Record format
+
+All records have the following format:
+
+```
+    0       1       2       3       4       5       6       7    octet
++-------+-------+-------+-------+-------+-------+-------+-------+
+| type                          | instance                      |
++-------------------------------+-------------------------------+
+| length                                                        |
++---------------------------------------------------------------+
+| body
+...
+|       | padding (0 to 7 octets)                               |
++-------+-------------------------------------------------------+
+```
+
+\pagebreak
+The fields are defined as follows:
+
+
+| Field      | Description                                      |
+|------------|--------------------------------------------------|
+| `type`     | A code which determines the layout and semantics |
+|            | of `body`                                        |
+|            |                                                  |
+| `instance` | The instance of the record                       |
+|            |                                                  |
+| `length`   | The length (in octets) of `body`                 |
+|            |                                                  |
+| `body`     | Zero or more octets of record data               |
+|            |                                                  |
+| `padding`  | Zero to seven octets of zero-filled padding to   |
+|            | bring the total record length up to the next     |
+|            | 64-bit boundary                                  |
+
+The `instance` field is present to distinguish multiple occurences of
+a record. E.g. state that is per-vcpu may need to be described in multiple
+records.
+
+The first record in the image is always a **START** record. The version of
+the image format can be inferred from the `type` of this record.
+
+## Image content
+
+The following records are defined for the v1 image format. This set may be
+extended in newer versions of the hypervisor. It is not expected that an image
+saved on a newer version of Xen will need to be restored on an older version.
+Therefore an image containing unrecognized record types should be rejected.
+
+### START
+
+```
+    0       1       2       3       4       5       6       7    octet
++-------+-------+-------+-------+-------+-------+-------+-------+
+| type == 1                     | instance == 0                 |
++-------------------------------+-------------------------------+
+| length == 8                                                   |
++-------------------------------+-------------------------------+
+| xen_major                     | xen_minor                     |
++-------------------------------+-------------------------------+
+```
+
+A type 1 **START** record implies a v1 image. If a new image format version
+is needed in future then this can be indicated by a new type value for this
+(first) record in the image.
+
+\pagebreak
+The record body contains the following fields:
+
+| Field       | Description                                     |
+|-------------|-------------------------------------------------|
+| `xen_major` | The major version of Xen that created this      |
+|             | image                                           |
+|             |                                                 |
+| `xen_minor` | The minor version of Xen that created this      |
+|             | image                                           |
+
+The version of Xen that created the image can be useful to the version that
+is restoring the image to determine whether certain records are expected to
+be present in the image. For example, a version of Xen prior to X.Y may not
+generate a FOO record but Xen X.Y+ can infer its content. But Xen X.Y+1
+**must** generate the FOO record as, from that version onward, its content
+can no longer be safely inferred.
+
+### END
+
+```
+    0       1       2       3       4       5       6       7    octet
++-------+-------+-------+-------+-------+-------+-------+-------+
+| type == 0                     | instance == 0                 |
++-------------------------------+-------------------------------+
+| length == 0                                                   |
++---------------------------------------------------------------+
+```
+
+A record of this type terminates the image. No further data from the buffer
+should be consumed.
+
+* * *
+
+[1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md
+
+[2] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/hvm/save.h
+
+[3] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/specs/libxc-migration-stream.pandoc
xen/include/public/arch-arm/hvm/save.h +5 -0
index 75b8e65bcb..d5b0c15203 100644
@@ -26,6 +26,11 @@
 #ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__
 #define __XEN_PUBLIC_HVM_SAVE_ARM_H__
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif
 
 /*
xen/include/public/arch-x86/hvm/save.h +5 -0
index 773a380bc2..e61e2dbcd7 100644
@@ -648,6 +648,11 @@ struct hvm_msr {
  */
 #define HVM_SAVE_CODE_MAX 20
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */
 
 /*
xen/include/public/save.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records

Quoted 35 lines · Paul Durrant
xen/include/public/save.h new file +32 -0
@@ -0,0 +1,66 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by Xen.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef XEN_PUBLIC_SAVE_H
+#define XEN_PUBLIC_SAVE_H
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+#include "xen.h"

xen/include/public/save.h line 42

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+
+/*
+ * C structures for the Domain Context v1 format.
+ * See docs/specs/domain-context.md
+ */
+
+struct domain_context_record {
+    uint32_t type;
+    uint32_t instance;
+    uint64_t length;

Should this be uint64_aligned_t, such that alignof() will produce consistent values regardless of bitness of the invoking domain?

Jan

#3 lore raw

Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records

Quoted 41 lines · Jan Beulich
Quoted 40 lines · Paul Durrant
xen/include/public/save.h new file +37 -0
@@ -0,0 +1,66 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by Xen.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef XEN_PUBLIC_SAVE_H
+#define XEN_PUBLIC_SAVE_H
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+#include "xen.h"
+
+/*
+ * C structures for the Domain Context v1 format.
+ * See docs/specs/domain-context.md
+ */
Quoted 8 lines

xen/include/public/save.h line 42

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+
+struct domain_context_record {
+    uint32_t type;
+    uint32_t instance;
+    uint64_t length;

Should this be uint64_aligned_t, such that alignof() will produce consistent values regardless of bitness of the invoking domain?

Does it matter?  Its just a bitstream, and can appear in the migration fd at any arbitrary alignment.

What matters is that the structure is aligned appropriately for the bitness of code operating on these fields.

Even with the tools ABI fixed to allow a 32-on-64-on-64  toolstack to function, I'm not sure that excess alignment would be appropriate.  Sure
- it would be more efficient for 32bit code to align to the 8 byte boundary for the benefit of a 64bit Xen's copy_from_user(), but this alignment happens anyway because of how hypercall buffers work.

~Andrew

#4 lore raw

Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records

Quoted 54 lines · Andrew Cooper
Quoted 41 lines · Jan Beulich
Quoted 40 lines · Paul Durrant
xen/include/public/save.h new file +37 -0
@@ -0,0 +1,66 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by Xen.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef XEN_PUBLIC_SAVE_H
+#define XEN_PUBLIC_SAVE_H
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+#include "xen.h"
+
+/*
+ * C structures for the Domain Context v1 format.
+ * See docs/specs/domain-context.md
+ */
Quoted 8 lines

xen/include/public/save.h line 42

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+
+struct domain_context_record {
+    uint32_t type;
+    uint32_t instance;
+    uint64_t length;

Should this be uint64_aligned_t, such that alignof() will produce consistent values regardless of bitness of the invoking domain?

Does it matter?  Its just a bitstream, and can appear in the migration fd at any arbitrary alignment.

Quoted 8 lines

What matters is that the structure is aligned appropriately for the bitness of code operating on these fields.

Even with the tools ABI fixed to allow a 32-on-64-on-64  toolstack to function, I'm not sure that excess alignment would be appropriate.  Sure
- it would be more efficient for 32bit code to align to the 8 byte boundary for the benefit of a 64bit Xen's copy_from_user(), but this alignment happens anyway because of how hypercall buffers work.

It _probably_ doesn't matter (hence me having put this as a question), but a struct in the ABI doesn't mandate how it is going to be used. I don't expect this to become the (or part of an) argument to a hypercall. I also don't expect anyone needing to use alignof() on it. But by not following the common model for the tools-only parts of the public interface we'd outright exclude such uses down the road.

Jan

#5 lore raw

Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records

In reply to Paul Durrant

xen/include/public/save.h line 4

Quoted 10 lines · Paul Durrant
xen/include/public/save.h new file +4 -0
new file mode 100644
index 0000000000..c4be9f570c
@@ -0,0 +1,66 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by Xen.

What exactly is, and is not in scope, for this new stream?  The PV above I think refers to "paravirtual state", not PV guests.

xen/include/public/save.h line 47

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+#define _DOMAIN_CONTEXT_RECORD_ALIGN 3
+#define DOMAIN_CONTEXT_RECORD_ALIGN (1U << _DOMAIN_CONTEXT_RECORD_ALIGN)

Do we need the logarithm version?

xen/include/public/save.h line 54

Quoted 7 lines
xen/include/public/save.h new file +7 -0
+
+enum {
+    DOMAIN_CONTEXT_END,
+    DOMAIN_CONTEXT_START,
+    /* New types go here */
+    DOMAIN_CONTEXT_NR_TYPES
+};

Does this enum ever end up in an API?

We might be ok as we're inside __XEN_TOOLS__, but enums normally cannot be used in ABI's because their size is implementation defined, and not always 4 bytes.

~Andrew

#6 lore raw

Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records

Quoted 9 lines · Andrew Cooper

xen/include/public/save.h line 54

Quoted 6 lines
xen/include/public/save.h new file +6 -0
+enum {
+    DOMAIN_CONTEXT_END,
+    DOMAIN_CONTEXT_START,
+    /* New types go here */
+    DOMAIN_CONTEXT_NR_TYPES
+};

Does this enum ever end up in an API?

Quoted 3 lines

We might be ok as we're inside __XEN_TOOLS__, but enums normally cannot be used in ABI's because their size is implementation defined, and not always 4 bytes.

The only way to use this as a type (e.g. to declare a struct field) would be via typeof(), which isn't something we'd allow in the public interface (for being an extension). Hence without a tag I would think an enum is fine to have here?

Jan

#7 lore raw

PATCH v10 2/11 xen: introduce implementation of save/restore of 'domain context'

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

A previous patch introduced the specification of the 'domain context' image and the supporting public header. This patch introduces the code necessary to generate and consume such an image. The entry points to the code are domain_save_ctxt() and domain_load_ctxt(). Code to call these functions will be introduced in a subsequent patch.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

v10:
- New in v10
- Largely derived from patch #1 of the v9 series, but the code has been re-worked both functionally and cosmetically

---
 xen/common/Makefile    |   1 +
 xen/common/save.c      | 339 +++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/save.h | 160 +++++++++++++++++++
 3 files changed, 500 insertions(+)
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
index b3b60a1ba2..3e6f21714a 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
 obj-y += rbtree.o
 obj-y += rcupdate.o
 obj-y += rwlock.o
+obj-y += save.o
 obj-y += shutdown.o
 obj-y += softirq.o
 obj-y += sort.o
xen/common/save.c new file +339 -0

339 lines not on this page · raw message

xen/include/xen/save.h new file +160 -0

160 lines not on this page · raw message

Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH v10 02/11] xen: introduce implementation of save/restore of 'domain context'

xen/common/save.c line 49

Quoted 10 lines · Paul Durrant
xen/common/save.c new file +10 -0
+void __init domain_register_ctxt_type(unsigned int type, const char *name,
+                                      domain_save_ctxt_type save,
+                                      domain_load_ctxt_type load)
+{
+    BUG_ON(type >= ARRAY_SIZE(fns));
+
+    ASSERT(!fns[type].save);
+    ASSERT(!fns[type].load);
+
+    fns[type].name = name;

I expect I merely didn't spot this (perhaps just latent) issue in earlier versions: If the caller lives in code getting built into *.init.o, the string passed in will live in .init.rodata. That's a general shortcoming (if you like) of the .o -> .init.o tranformation, but I see no good alternative (or else all format strings passed to printk() and alike won't get moved either). Therefore I wonder whether it wouldn't be safer to have the struct field be e.g. char[16], assuming 15 characters will allow for meaningful names.

xen/common/save.c line 173

Quoted 7 lines
xen/common/save.c new file +7 -0
+int domain_load_ctxt_rec_data(struct domain_ctxt_state *c, void *dst,
+                              size_t len)
+{
+    int rc = 0;
+
+    c->len += len;
+    if (c->len > c->rec.length)

Nit: Missing blanks.

Quoted 13 lines
xen/common/save.c new file +13 -0
+int domain_load_ctxt(struct domain *d, const struct domain_load_ctxt_ops *ops,
+                     void *priv)
+{
+    struct domain_ctxt_state c = { .d = d, .ops.load = ops, .priv = priv, };
+    domain_load_ctxt_type load;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    rc = c.ops.load->read(c.priv, &c.rec, sizeof(c.rec));
+    if ( rc )
+        return rc;
+

xen/common/save.c line 235

Quoted 10 lines
xen/common/save.c new file +10 -0
+    load = fns[DOMAIN_CONTEXT_START].load;
+    BUG_ON(!load);
+
+    rc = load(d, &c);
+    if ( rc )
+        return rc;
+
+    domain_pause(d);
+
+    for (;;)

Nit: Missing blanks again.

Quoted 6 lines
xen/common/save.c new file +6 -0
+    {
+        unsigned int type;
+
+        rc = c.ops.load->read(c.priv, &c.rec, sizeof(c.rec));
+        if ( rc )
+            break;

xen/common/save.c line 251

Quoted 10 lines
xen/common/save.c new file +10 -0
+
+        type = c.rec.type;
+        if ( type == DOMAIN_CONTEXT_END )
+            break;
+
+        rc = -EINVAL;
+        if ( type >= ARRAY_SIZE(fns) )
+            break;
+
+        load = fns[type].load;

While this is meant to be used by Dom0 only, I think it would be better if it nevertheless used array_access_nospec().

Quoted 8 lines
xen/common/save.c new file +8 -0
+static int load_start(struct domain *d, struct domain_ctxt_state *c)
+{
+    static struct domain_context_start s;
+    unsigned int i;
+    int rc = domain_load_ctxt_rec(c, DOMAIN_CONTEXT_START, &i, &s, sizeof(s));
+
+    if ( rc )
+        return rc;

xen/common/save.c line 299

Quoted 10 lines
xen/common/save.c new file +10 -0
+
+    if ( i )
+        return -EINVAL;
+
+    /*
+     * Make sure we are not attempting to load an image generated by a newer
+     * version of Xen.
+     */
+    if ( s.xen_major > XEN_VERSION && s.xen_minor > XEN_SUBVERSION )
+        return -EOPNOTSUPP;

Are you sure this needs to be excluded here and unilaterally? And if this is to stay, then it wants to be

    if ( s.xen_major > XEN_VERSION ||
         (s.xen_major == XEN_VERSION && s.xen_minor > XEN_SUBVERSION) )
        return -EOPNOTSUPP;
Quoted 8 lines
xen/include/xen/save.h new file +8 -0
+/*
+ * Register save and load handlers for a record type.
+ *
+ * Save handlers will be invoked in an order which copes with any inter-
+ * entry dependencies. For now this means that HEADER will come first and
+ * END will come last, all others being invoked in order of 'typecode'.
+ *
+ * Load handlers will be invoked in the order of entries present in the

xen/include/xen/save.h line 114

Quoted 10 lines
xen/include/xen/save.h new file +10 -0
+ * buffer.
+ */
+#define DOMAIN_REGISTER_CTXT_TYPE(x, s, l)                    \
+    static int __init __domain_register_##x##_ctxt_type(void) \
+    {                                                         \
+        domain_register_ctxt_type(                            \
+            DOMAIN_CONTEXT_ ## x,                             \
+            #x,                                               \
+            &(s),                                             \
+            &(l));                                            \

I don't think there's a need for each of these to consume a separate line.

Jan

#9 lore raw

Re: [PATCH v10 02/11] xen: introduce implementation of save/restore of 'domain context'

In reply to Paul Durrant
Quoted 14 lines · Paul Durrant
xen/common/save.c new file +7 -0
new file mode 100644
index 0000000000..9287b20198
@@ -0,0 +1,339 @@
 
+static int load_start(struct domain *d, struct domain_ctxt_state *c)
+{
+    static struct domain_context_start s;
+    unsigned int i;
+    int rc = domain_load_ctxt_rec(c, DOMAIN_CONTEXT_START, &i, &s, sizeof(s));
+
+    if ( rc )

xen/common/save.c line 298

Quoted 10 lines
xen/common/save.c new file +10 -0
+        return rc;
+
+    if ( i )
+        return -EINVAL;
+
+    /*
+     * Make sure we are not attempting to load an image generated by a newer
+     * version of Xen.
+     */
+    if ( s.xen_major > XEN_VERSION && s.xen_minor > XEN_SUBVERSION )

major > XEN_VERSON || (major == XEN_VERSION && minor > XEN_SUBVERSION)

~Andrew

#10 lore raw

PATCH v10 3/11 xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context

In reply to Paul Durrant

These domctls provide a mechanism to get and set 'domain context' from the toolstack. The implementation calls the domain_save_ctxt() and domain_load_ctxt() functions introduced in a previous patch.

---
Cc: Julien Grall <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v10:
- Re-base
- Add underscores and move to 64-bit image size as requested by Andrew
- Add a couple of record alignment ASSERTions
- Dropped R-b and A-b since changes are not entirely cosmetic

v4:
- Add missing zero pad checks

v3:
- Addressed comments from Julien and Jan
- Use vmalloc() rather than xmalloc_bytes()

v2:
- drop mask parameter
- const-ify some more buffers

---
 tools/flask/policy/modules/xen.if   |   4 +-
 tools/libs/ctrl/include/xenctrl.h   |   5 +
 tools/libs/ctrl/xc_domain.c         |  56 +++++++++
 xen/common/domctl.c                 | 173 ++++++++++++++++++++++++++++
 xen/include/public/domctl.h         |  39 +++++++
 xen/xsm/flask/hooks.c               |   6 +
 xen/xsm/flask/policy/access_vectors |   4 +
 7 files changed, 285 insertions(+), 2 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
index 5e2aa472b6..2e2303d684 100644
@@ -53,7 +53,7 @@ define(`create_domain_common', `
 	allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim
 			set_vnumainfo get_vnumainfo cacheflush
 			psr_cmt_op psr_alloc soft_reset
-			resource_map get_cpu_policy };
+			resource_map get_cpu_policy set_context };
 	allow $1 $2:security check_context;
 	allow $1 $2:shadow enable;
 	allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', `
 	allow $1 $2:hvm { gethvmc getparam };
 	allow $1 $2:mmu { stat pageinfo map_read };
 	allow $1 $2:domain { getaddrsize getvcpucontext pause destroy };
-	allow $1 $2:domain2 gettsc;
+	allow $1 $2:domain2 { gettsc get_context };
 	allow $1 $2:shadow { enable disable logdirty };
 ')
 
tools/libs/ctrl/include/xenctrl.h +5 -0
index 3796425e1e..754a00c67b 100644
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,
                              uint8_t *hvm_ctxt,
                              uint32_t size);
 
+int xc_domain_get_context(xc_interface *xch, uint32_t domid,
+                          void *ctxt_buf, size_t *size);
+int xc_domain_set_context(xc_interface *xch, uint32_t domid,
+                          const void *ctxt_buf, size_t size);
+
 /**
  * This function will return guest IO ABI protocol
  *
tools/libs/ctrl/xc_domain.c +56 -0

62 lines not on this page · raw message

xen/common/domctl.c +173 -0

191 lines not on this page · raw message

xen/include/public/domctl.h +39 -0
index 666aeb71bf..a3e10c03f1 100644
@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * XEN_DOMCTL_get_domain_context
+ * -----------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.
+ *                The value passed out will be the size of data written.
+ */
+struct xen_domctl_get_domain_context {
+    uint64_t size;
+    XEN_GUEST_HANDLE_64(void) buffer;
+};
+
+/* XEN_DOMCTL_set_domain_context
+ * -----------------------------
+ *
+ * buffer (IN):   The buffer from which the context data should be
+ *                copied.
+ * size (IN):     The size of the buffer from which data may be copied.
+ *                This data must include DOMAIN_SAVE_CODE_HEADER at the
+ *                start and terminate with a DOMAIN_SAVE_CODE_END record.
+ *                Any data beyond the DOMAIN_SAVE_CODE_END record will be
+ *                ignored.
+ */
+struct xen_domctl_set_domain_context {
+    uint64_t size;
+    XEN_GUEST_HANDLE_64(const_void) buffer;
+};
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1
@@ -1216,6 +1251,8 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_get_domain_context            84
+#define XEN_DOMCTL_set_domain_context            85
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -1276,6 +1313,8 @@ struct xen_domctl {
         struct xen_domctl_monitor_op        monitor_op;
         struct xen_domctl_psr_alloc         psr_alloc;
         struct xen_domctl_vuart_op          vuart_op;
+        struct xen_domctl_get_domain_context get_domain_context;
+        struct xen_domctl_set_domain_context set_domain_context;
         uint8_t                             pad[128];
     } u;
 };
xen/xsm/flask/hooks.c +6 -0
index de050cc9fe..3c6217e4ac 100644
@@ -754,6 +754,12 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_get_cpu_policy:
         return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GET_CPU_POLICY);
 
+    case XEN_DOMCTL_set_domain_context:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SET_CONTEXT);
+
+    case XEN_DOMCTL_get_domain_context:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GET_CONTEXT);
+
     default:
         return avc_unknown_permission("domctl", cmd);
     }
xen/xsm/flask/policy/access_vectors +4 -0
index 1aa0bb501c..fea0c9f143 100644
@@ -245,6 +245,10 @@ class domain2
     resource_map
 # XEN_DOMCTL_get_cpu_policy
     get_cpu_policy
+# XEN_DOMCTL_set_domain_context
+    set_context
+# XEN_DOMCTL_get_domain_context
+    get_context
 }
 
 # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
Reviewed-by on v10 3/11
#11 lore raw

Re: [PATCH v10 03/11] xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 312

Quoted 10 lines · Paul Durrant
xen/common/domctl.c +10 -0
+static int dry_run_end(void *priv, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    ASSERT(IS_ALIGNED(c->len, DOMAIN_CONTEXT_RECORD_ALIGN));
+
+    return 0;
+}
+
+static struct domain_save_ctxt_ops dry_run_ops = {

const? (same for save_ops and load_ops then)

Quoted 12 lines
xen/include/public/domctl.h +6 -0
@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * XEN_DOMCTL_get_domain_context
+ * -----------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.
+ *                The value passed out will be the size of data written.
+ */
+struct xen_domctl_get_domain_context {
+    uint64_t size;

uint64_aligned_t (also again below)?

With these adjusted

Jan

#12 lore raw

Re: [PATCH v10 03/11] xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context

Quoted 39 lines · Jan Beulich

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 312

Quoted 10 lines · Paul Durrant
xen/common/domctl.c +10 -0
+static int dry_run_end(void *priv, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    ASSERT(IS_ALIGNED(c->len, DOMAIN_CONTEXT_RECORD_ALIGN));
+
+    return 0;
+}
+
+static struct domain_save_ctxt_ops dry_run_ops = {

const? (same for save_ops and load_ops then)

Quoted 12 lines
xen/include/public/domctl.h +6 -0
@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * XEN_DOMCTL_get_domain_context
+ * -----------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.
+ *                The value passed out will be the size of data written.
+ */
+struct xen_domctl_get_domain_context {
+    uint64_t size;

uint64_aligned_t (also again below)?

Quoted 2 lines

With these adjusted

FAOD: Non-XSM hypervisor pieces only.

Jan

#13 lore raw

PATCH v10 4/11 tools/misc: add xen-domctx to present domain context

In reply to Paul Durrant

This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.

---
Cc: Andrew Cooper <[email protected]>

v10:
- Re-base
- Use err[x]()
- Keep existing A-b since modifications are trivial

v3:
- Re-worked to avoid copying onto stack
- Added optional typecode and instance arguments

v2:
- Change name from 'xen-ctx' to 'xen-domctx'

---
 .gitignore              |   1 +
 tools/misc/Makefile     |   4 +
 tools/misc/xen-domctx.c | 172 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 177 insertions(+)
 create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
index 188495783e..4d9a61124d 100644
@@ -243,6 +243,7 @@ tools/misc/xen_cpuperf
 tools/misc/xen-cpuid
 tools/misc/xen-detect
 tools/misc/xen-diag
+tools/misc/xen-domctx
 tools/misc/xen-tmem-list-parse
 tools/misc/xen-livepatch
 tools/misc/xenperf
tools/misc/Makefile +4 -0
index 7d37f297a9..fb673d0ff6 100644
@@ -32,6 +32,7 @@ INSTALL_SBIN                   += xenpm
 INSTALL_SBIN                   += xenwatchdogd
 INSTALL_SBIN                   += xen-livepatch
 INSTALL_SBIN                   += xen-diag
+INSTALL_SBIN                   += xen-domctx
 INSTALL_SBIN += $(INSTALL_SBIN-y)
 
 # Everything to be installed in a private bin/
@@ -111,6 +112,9 @@ xen-livepatch: xen-livepatch.o
 xen-diag: xen-diag.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-domctx: xen-domctx.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 xen-lowmemd: xen-lowmemd.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
 
tools/misc/xen-domctx.c new file +172 -0

172 lines not on this page · raw message

Signature · 1 line
2.20.1
#14 lore raw

PATCH v10 5/11 common/domain: add a domain context record for shared_info...

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.

---
Cc: Jan Beulich <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v10:
- Re-base
- Amend the specification now there is one
- Dropped Jan's R-b as modifications are not completely trivial

v9:
- Use macros to make the code less verbose
- Add missing check for allocation failure

v8:
- Incorporate zero-ing out of shared info fields that would be done in processing of SHARED_INFO from older stream versions

v7:
- Only restore vcpu_info and arch sub-structures for PV domains, to match processing of SHARED_INFO in xc_sr_restore_x86_pv.c
- Use additional option to domain_load_end() to ignore the record for HVM domains

v6:
- Only save compat_shared_info buffer if has_32bit_shinfo is set
- Validate flags field in load handler

v5:
- Addressed comments from Julien

v4:
- Addressed comments from Jan

v3:
- Actually dump some of the content of shared_info

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
 docs/specs/domain-context.md |  29 +++++++++
 tools/misc/xen-domctx.c      |  80 +++++++++++++++++++++++++
 xen/common/domain.c          | 113 +++++++++++++++++++++++++++++++++++
 xen/include/public/save.h    |  11 ++++
 4 files changed, 233 insertions(+)
docs/specs/domain-context.md +29 -0
index f177cf24b3..95e9f9d1ab 100644
@@ -128,6 +128,33 @@ can no longer be safely inferred.
 A record of this type terminates the image. No further data from the buffer
 should be consumed.
 
+### SHARED_INFO
+
+```
+    0       1       2       3       4       5       6       7    octet
++-------+-------+-------+-------+-------+-------+-------+-------+
+| type == 2                     | instance == 0                 |
++-------------------------------+-------------------------------+
+| length                                                        |
++-------------------------------+-------------------------------+
+| flags                         | buffer
++-------------------------------+
+...
+```
+
+\pagebreak
+The record body contains the following fields:
+
+| Field       | Description                                     |
+|-------------|-------------------------------------------------|
+| `flags`     | A bit-wise OR of the following:                 |
+|             |                                                 |
+|             | 0x00000001: The domain has 32-bit (compat)      |
+|             |             shared info                         |
+|             |                                                 |
+| `buffer`    | The shared info (`length` being architecture    |
+|             | dependent[4])                                   |
+
 * * *
 
 [1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md
@@ -135,3 +162,5 @@ should be consumed.
 [2] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/hvm/save.h
 
 [3] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/specs/libxc-migration-stream.pandoc
+
+[4] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/include/xen-foreign/reference.size
tools/misc/xen-domctx.c +80 -0

92 lines not on this page · raw message

xen/common/domain.c +113 -0

125 lines not on this page · raw message

xen/include/public/save.h +11 -0
index c4be9f570c..bccbaadd0b 100644
@@ -49,6 +49,7 @@ struct domain_context_record {
 enum {
     DOMAIN_CONTEXT_END,
     DOMAIN_CONTEXT_START,
+    DOMAIN_CONTEXT_SHARED_INFO,
     /* New types go here */
     DOMAIN_CONTEXT_NR_TYPES
 };
@@ -58,6 +59,16 @@ struct domain_context_start {
     uint32_t xen_major, xen_minor;
 };
 
+struct domain_context_shared_info {
+    uint32_t flags;
+
+#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0
+#define DOMAIN_CONTEXT_32BIT_SHARED_INFO \
+    (1U << _DOMAIN_CONTEXT_32BIT_SHARED_INFO)
+
+    uint8_t buffer[XEN_FLEX_ARRAY_DIM];
+};
+
 /* Terminating entry */
 struct domain_context_end {};
 
Signature · 1 line
2.20.1
#15 lore raw

Re: [PATCH v10 05/11] common/domain: add a domain context record for shared_info...

Quoted 7 lines · Paul Durrant
xen/common/domain.c +3 -0
@@ -1671,6 +1672,118 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+static int save_shared_info(struct domain *d, struct domain_ctxt_state *c,
+                            bool dry_run)
+{

xen/common/domain.c int continue_hypercall_on_cpu( · line 1687

Quoted 10 lines
xen/common/domain.c +10 -0
+#ifdef CONFIG_COMPAT
+    struct domain_context_shared_info s = {
+        .flags = has_32bit_shinfo(d) ? DOMAIN_CONTEXT_32BIT_SHARED_INFO : 0,
+    };
+    size_t size = has_32bit_shinfo(d) ?
+        sizeof(struct compat_shared_info) :
+        sizeof(struct shared_info);
+#else
+    struct domain_context_shared_info s = {};
+    size_t size = sizeof(struct shared_info);

All of these would imo better be expressed in terms of d->shared_info. While chances are zero that these types will change in any way, it still sets a bad precedent for people seeing this and then introducing similar disconnects elsewhere. (Same in the load handling then.)

Quoted 16 lines
xen/common/domain.c +16 -0
+static int load_shared_info(struct domain *d, struct domain_ctxt_state *c)
+{
+    struct domain_context_shared_info s = {};
+    size_t size;
+    unsigned int i;
+    int rc;
+
+    rc = domain_load_ctxt_rec_begin(c, DOMAIN_CONTEXT_SHARED_INFO, &i);
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    rc = domain_load_ctxt_rec_data(c, &s, offsetof(typeof(s), buffer));
+    if ( rc )

xen/common/domain.c int continue_hypercall_on_cpu( · line 1731

Quoted 10 lines
xen/common/domain.c +10 -0
+        return rc;
+
+    if ( s.flags & ~DOMAIN_CONTEXT_32BIT_SHARED_INFO )
+        return -EINVAL;
+
+    if ( s.flags & DOMAIN_CONTEXT_32BIT_SHARED_INFO )
+    {
+#ifdef CONFIG_COMPAT
+        d->arch.has_32bit_shinfo = true;
+        size = sizeof(struct compat_shared_info);

I realize this has been more or less this way already in prior versions, but aren't you introducing a way to have a degenerate 64-bit PV guest with 32-bit shared info (or vice versa), in that shared info bitness isn't strictly tied to guest bitness anymore? Rejecting this case may not need to live here, but it needs to be present / added somewhere.

Quoted 23 lines
xen/common/domain.c +23 -0
+#else
+        return -EINVAL;
+#endif
+    }
+    else
+        size = sizeof(struct shared_info);
+
+    if ( is_pv_domain(d) )
+    {
+        shared_info_t *shinfo = xzalloc(shared_info_t);
+
+        if ( !shinfo )
+            return -ENOMEM;
+
+        rc = domain_load_ctxt_rec_data(c, shinfo, size);
+        if ( rc )
+            goto out;
+
+        memcpy(&shared_info(d, vcpu_info), &__shared_info(d, shinfo, vcpu_info),
+               sizeof(shared_info(d, vcpu_info)));
+        memcpy(&shared_info(d, arch), &__shared_info(d, shinfo, arch),
+               sizeof(shared_info(d, arch)));
+

xen/common/domain.c int continue_hypercall_on_cpu( · line 1764

Quoted 10 lines
xen/common/domain.c +10 -0
+        memset(&shared_info(d, evtchn_pending), 0,
+               sizeof(shared_info(d, evtchn_pending)));
+        memset(&shared_info(d, evtchn_mask), 0xff,
+               sizeof(shared_info(d, evtchn_mask)));
+
+#ifdef CONFIG_X86
+        shared_info(d, arch.pfn_to_mfn_frame_list_list) = 0;
+#endif
+        for ( i = 0; i < XEN_LEGACY_MAX_VCPUS; i++ )
+            shared_info(d, vcpu_info[i].evtchn_pending_sel) = 0;

Again I realize this has been this way in earlier versions, and it was also me to ask for streamlining the code, but is this actually correct? I ask in particular in light of this comment

/*
* Compat field is never larger than native field, so cast to that as it
* is the largest memory range it is safe for the caller to modify without
* further discrimination between compat and native cases.
*/

in xen/shared.h, next to the __shared_info() #define. I can't help thinking that you'll fill only half of some of the fields in the 64-bit case.

xen/include/public/save.h struct domain_context_start { · line 65

Quoted 8 lines
xen/include/public/save.h +4 -0
@@ -58,6 +59,16 @@ struct domain_context_start {
     uint32_t xen_major, xen_minor;
 };
 
+struct domain_context_shared_info {
+    uint32_t flags;
+
+#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0

Is this separate constant actually needed for anything?

Speaking of which - wouldn't all your additions to this header better be proper name spacing citizens, by having xen_ / XEN_ prefixes?

Jan

#16 lore raw

Re: [PATCH v10 05/11] common/domain: add a domain context record for shared_info...

In reply to Paul Durrant
Quoted 5 lines · Paul Durrant
xen/include/public/save.h +0 -0
index c4be9f570c..bccbaadd0b 100644

xen/include/public/save.h struct domain_context_start { · line 67

Quoted 9 lines
xen/include/public/save.h +6 -0
@@ -58,6 +59,16 @@ struct domain_context_start {
     uint32_t xen_major, xen_minor;
 };
 
+struct domain_context_shared_info {
+    uint32_t flags;
+
+#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0
+#define DOMAIN_CONTEXT_32BIT_SHARED_INFO \
+    (1U << _DOMAIN_CONTEXT_32BIT_SHARED_INFO)

There is no need for the logarithm version of this constant.

You do however want an explicit uint32_t _rsvd; so buffer[] doesn't start at the wrong alignment for an efficient memcpy() in 64bit builds of Xen.

~Andrew

#17 lore raw

PATCH v10 6/11 x86/time: add a domain context record for tsc_info...

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

... and update xen-domctx to dump some information describing the record.

NOTE: Whilst the record is x86 specific, it is visible directly in the common
header as context record numbers should be unique across all architectures.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v10:
- Re-base
- Amend the specification now there is one
- Kept Jan's R-b as the changes are cosmetic

v8:
- Removed stray blank line

v7:
- New in v7

---
 docs/specs/domain-context.md | 32 ++++++++++++++++++++++++++++++++
 tools/misc/xen-domctx.c      | 12 ++++++++++++
 xen/arch/x86/time.c          | 30 ++++++++++++++++++++++++++++++
 xen/include/public/save.h    |  8 ++++++++
 4 files changed, 82 insertions(+)
docs/specs/domain-context.md +32 -0
index 95e9f9d1ab..8aa3466d96 100644
@@ -155,6 +155,38 @@ The record body contains the following fields:
 | `buffer`    | The shared info (`length` being architecture    |
 |             | dependent[4])                                   |
 
+### TSC_INFO
+
+```
+    0       1       2       3       4       5       6       7    octet
++-------+-------+-------+-------+-------+-------+-------+-------+
+| type == 3                     | instance == 0                 |
++-------------------------------+-------------------------------+
+| length == 20                                                  |
++-------------------------------+-------------------------------+
+| mode                          | khz                           |
++-------------------------------+-------------------------------+
+| nsec                                                          |
++-------------------------------+-------------------------------+
+| incarnation                   |
++-------------------------------+
+```
+
+The record body contains the following fields:
+
+| Field         | Description                                   |
+|---------------|-----------------------------------------------|
+| `mode`        | 0x00000000: Default (emulate if necessary)    |
+|               | 0x00000001: Always emulate                    |
+|               | 0x00000002: Never emulate                     |
+|               |                                               |
+| `khz`         | The TSC frequency in kHz                      |
+|               |                                               |
+| `nsec`        | Elapsed time in nanoseconds                   |
+|               |                                               |
+| `incarnation` | Incarnation (counter indicating how many      |
+|               | times the TSC value has been set)             |
+
 * * *
 
 [1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md
tools/misc/xen-domctx.c +12 -0
index 5ea6de50d1..1a5dfb9d5a 100644
@@ -136,6 +136,17 @@ static void dump_shared_info(void)
 #undef GET_FIELD_PTR
 }
 
+static void dump_tsc_info(void)
+{
+    struct domain_context_tsc_info *t;
+
+    GET_PTR(t);
+
+    printf("    TSC_INFO: mode: %u incarnation: %u\n"
+           "              khz %u nsec: %"PRIu64"\n",
+           t->mode, t->incarnation, t->khz, t->nsec);
+}
+
 static void dump_end(void)
 {
     struct domain_context_end *e;
@@ -225,6 +236,7 @@ int main(int argc, char **argv)
             {
             case DOMAIN_CONTEXT_START: dump_start(); break;
             case DOMAIN_CONTEXT_SHARED_INFO: dump_shared_info(); break;
+            case DOMAIN_CONTEXT_TSC_INFO: dump_tsc_info(); break;
             case DOMAIN_CONTEXT_END: dump_end(); break;
             default:
                 printf("Unknown type %u: skipping\n", rec->type);
xen/arch/x86/time.c +30 -0
index 8938c0f435..aec4bfb0f3 100644
@@ -26,6 +26,7 @@
 #include <xen/symbols.h>
 #include <xen/keyhandler.h>
 #include <xen/guest_access.h>
+#include <xen/save.h>
 #include <asm/io.h>
 #include <asm/iocap.h>
 #include <asm/msr.h>
@@ -2451,6 +2452,35 @@ int tsc_set_info(struct domain *d,
     return 0;
 }
 
+static int save_tsc_info(struct domain *d, struct domain_ctxt_state *c,
+                         bool dry_run)
+{
+    struct domain_context_tsc_info t = {};
+
+    if ( !dry_run )
+        tsc_get_info(d, &t.mode, &t.nsec, &t.khz, &t.incarnation);
+
+    return domain_save_ctxt_rec(c, DOMAIN_CONTEXT_TSC_INFO, 0, &t, sizeof(t));
+}
+
+static int load_tsc_info(struct domain *d, struct domain_ctxt_state *c)
+{
+    struct domain_context_tsc_info t;
+    unsigned int i;
+    int rc;
+
+    rc = domain_load_ctxt_rec(c, DOMAIN_CONTEXT_TSC_INFO, &i, &t, sizeof(t));
+    if ( rc )
+        return rc;
+
+    if ( i ) /* expect only a single instance */
+        return -ENXIO;
+
+    return tsc_set_info(d, t.mode, t.nsec, t.khz, t.incarnation);
+}
+
+DOMAIN_REGISTER_CTXT_TYPE(TSC_INFO, save_tsc_info, load_tsc_info);
+
 /* vtsc may incur measurable performance degradation, diagnose with this */
 static void dump_softtsc(unsigned char key)
 {
xen/include/public/save.h +8 -0
index bccbaadd0b..86881864cf 100644
@@ -50,6 +50,7 @@ enum {
     DOMAIN_CONTEXT_END,
     DOMAIN_CONTEXT_START,
     DOMAIN_CONTEXT_SHARED_INFO,
+    DOMAIN_CONTEXT_TSC_INFO,
     /* New types go here */
     DOMAIN_CONTEXT_NR_TYPES
 };
@@ -69,6 +70,13 @@ struct domain_context_shared_info {
     uint8_t buffer[XEN_FLEX_ARRAY_DIM];
 };
 
+struct domain_context_tsc_info {
+    uint32_t mode;
+    uint32_t khz;
+    uint64_t nsec;
+    uint32_t incarnation;
+};
+
 /* Terminating entry */
 struct domain_context_end {};
 
Signature · 1 line
2.20.1
#18 lore raw

Re: [PATCH v10 06/11] x86/time: add a domain context record for tsc_info...

Quoted 4 lines · Paul Durrant
docs/specs/domain-context.md +4 -0
+The record body contains the following fields:
+
+| Field         | Description                                   |
+|---------------|-----------------------------------------------|

docs/specs/domain-context.md The record body contains the following fields: · line 188

Quoted 10 lines
docs/specs/domain-context.md +10 -0
+| `mode`        | 0x00000000: Default (emulate if necessary)    |
+|               | 0x00000001: Always emulate                    |
+|               | 0x00000002: Never emulate                     |
+|               |                                               |
+| `khz`         | The TSC frequency in kHz                      |
+|               |                                               |
+| `nsec`        | Elapsed time in nanoseconds                   |
+|               |                                               |
+| `incarnation` | Incarnation (counter indicating how many      |
+|               | times the TSC value has been set)             |

It is how many set_tsc_info() (hyper)calls have been made, not how many times the guest has written to the TSC MSR.

That said, it is totally useless now that PVRDTSCP mode has gone, other than the fact that it appears in guest CPUID as an approximation of "how many times has this domain been migrated".

I.e. its a number you'll want to actively squash in your usecase.

I'm not sure whether to suggest dropping the field entirely, or not.  I highly doubt any user exists - IIRC, it was specifically for PVRDTSCP userspace to notice that the frequency may have changed, and to re-adjust its calculations.

xen/include/public/save.h enum { · line 53

Quoted 9 lines
xen/include/public/save.h +1 -0
index bccbaadd0b..86881864cf 100644
@@ -50,6 +50,7 @@ enum {
     DOMAIN_CONTEXT_END,
     DOMAIN_CONTEXT_START,
     DOMAIN_CONTEXT_SHARED_INFO,
+    DOMAIN_CONTEXT_TSC_INFO,

At a minimum, this wants an /* x86 only */ comment.  Possibly an X86 infix.

~Andrew

#19 lore raw

PATCH v10 7/11 docs/specs: add missing definitions to libxc-migration-stream

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v7:
- New in v7

---
 docs/specs/libxc-migration-stream.pandoc | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
docs/specs/libxc-migration-stream.pandoc +9 -2
index 6b0c49e97a..8aeab3b11b 100644
@@ -3,7 +3,7 @@
   Andrew Cooper <<[email protected]>>
   Wen Congyang <<[email protected]>>
   Yang Hongyang <<[email protected]>>
-% Revision 2
+% Revision 3
 
 Introduction
 ============
@@ -227,7 +227,13 @@ type         0x00000000: END
 
              0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary)
 
-             0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_
+             0x00000010: STATIC_DATA_END
+
+             0x00000011: X86_CPUID_POLICY
+
+             0x00000012: X86_MSR_POLICY
+
+             0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_
              records.
 
              0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -732,6 +738,7 @@ A typical save record for an x86 HVM guest image would look like:
 * X86_TSC_INFO
 * HVM_PARAMS
 * HVM_CONTEXT
+* END record
 
 HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect
 the validity of architectural state in the context.
Signature · 1 line
2.20.1
#20 lore raw

Re: [PATCH v10 07/11] docs/specs: add missing definitions to libxc-migration-stream

Quoted 6 lines · Paul Durrant

From: Paul Durrant <[email protected]>

The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."

Quoted 8 lines

Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'

Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")
Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")

I've committed this.  I have no idea where these hunks got lost, because I definitely did have them at some point during the mig-v3 work.

~Andrew

#21 lore raw

PATCH v10 8/11 docs / tools: specify migration v4 to include DOMAIN_CONTEXT

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informition so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.

---
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>

NOTE: Wei requested ack from Andrew

v10:
- Removed changes to xg_sr_common.c and libxc.py to make this a just documentation and header patch
- Dropped Wei's A-b in light of change

v7:
- New in v7

---
 docs/specs/libxc-migration-stream.pandoc | 62 ++++++++++++++++++------
 tools/libs/guest/xg_sr_stream_format.h   |  1 +
 2 files changed, 49 insertions(+), 14 deletions(-)
docs/specs/libxc-migration-stream.pandoc +48 -14
index 8aeab3b11b..aa6fe284f3 100644
@@ -3,7 +3,7 @@
   Andrew Cooper <<[email protected]>>
   Wen Congyang <<[email protected]>>
   Yang Hongyang <<[email protected]>>
-% Revision 3
+% Revision 4
 
 Introduction
 ============
@@ -127,7 +127,7 @@ marker      0xFFFFFFFFFFFFFFFF.
 
 id          0x58454E46 ("XENF" in ASCII).
 
-version     0x00000003.  The version of this specification.
+version     0x00000004.  The version of this specification.
 
 options     bit 0: Endianness.  0 = little-endian, 1 = big-endian.
 
@@ -209,9 +209,9 @@ type         0x00000000: END
 
              0x00000006: X86_PV_VCPU_XSAVE
 
-             0x00000007: SHARED_INFO
+             0x00000007: SHARED_INFO (deprecated)
 
-             0x00000008: X86_TSC_INFO
+             0x00000008: X86_TSC_INFO (deprecated)
 
              0x00000009: HVM_CONTEXT
 
@@ -233,7 +233,9 @@ type         0x00000000: END
 
              0x00000012: X86_MSR_POLICY
 
-             0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_
+             0x00000013: DOMAIN_CONTEXT
+
+             0x00000014 - 0x7FFFFFFF: Reserved for future _mandatory_
              records.
 
              0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS             XEN_DOMCTL_{get,set}\_vcpu_msrs
 
 \clearpage
 
-SHARED_INFO
------------
+SHARED_INFO (deprecated)
+------------------------
 
-The content of the Shared Info page.
+The content of the Shared Info page. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.
 
      0     1     2     3     4     5     6     7 octet
     +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info      Contents of the shared info page.  This record
 
 \clearpage
 
-X86_TSC_INFO
-------------
+X86_TSC_INFO (deprecated)
+-------------------------
 
 Domain TSC information, as accessed by the
-XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops.
+XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.
 
      0     1     2     3     4     5     6     7 octet
     +------------------------+------------------------+
@@ -680,6 +684,25 @@ MSR_policy       Array of xen_msr_entry_t[]'s
 
 \clearpage
 
+DOMAIN_CONTEXT
+--------------
+
+Domain context, as accessed by the
+XEN_DOMCTL_{get,set}_domain_context hypercall sub-ops.
+
+     0     1     2     3     4     5     6     7 octet
+    +-------------------------------------------------+
+    | dom_ctx                                         |
+    ...
+    +-------------------------------------------------+
+
+--------------------------------------------------------------------
+Field            Description
+-----------      ---------------------------------------------------
+dom_ctx          The Domain Context blob from Xen.
+--------------------------------------------------------------------
+
+\clearpage
 
 Layout
 ======
@@ -706,8 +729,7 @@ A typical save record for an x86 PV guest image would look like:
     * STATIC_DATA_END
 * X86_PV_P2M_FRAMES record
 * Many PAGE_DATA records
-* X86_TSC_INFO
-* SHARED_INFO record
+* DOMAIN_CONTEXT
 * VCPU context records for each online VCPU
     * X86_PV_VCPU_BASIC record
     * X86_PV_VCPU_EXTENDED record
@@ -735,7 +757,7 @@ A typical save record for an x86 HVM guest image would look like:
     * X86_{CPUID,MSR}_POLICY
     * STATIC_DATA_END
 * Many PAGE_DATA records
-* X86_TSC_INFO
+* DOMAIN_CONTEXT
 * HVM_PARAMS
 * HVM_CONTEXT
 * END record
@@ -746,6 +768,18 @@ the validity of architectural state in the context.
 Compatibility with older versions
 =================================
 
+v4 compat with v3
+-----------------
+
+A v4 stream is compatible with a v3 stream, but mandates the presence of a
+DOMAIN_CONTEXT record. This incorporates context such as the content of
+the domain's Shared Info page and the TSC information, hence the SHARED_INFO
+and TSC_INFO records are deprecated.
+It also supercedes HVM_CONTEXT and, over time, data that is currently part of
+the HVM_CONTEXT blob will move to the DOMAIN_CONTEXT blob. Xen, however, will
+continue to accept all defined HVM_CONTEXT records so a v4-compatible
+receiver can still accept an unmodified v3 stream.
+
 v3 compat with v2
 -----------------
 
tools/libs/guest/xg_sr_stream_format.h +1 -0
index 8a0da26f75..bc538bc192 100644
@@ -76,6 +76,7 @@ struct xc_sr_rhdr
 #define REC_TYPE_STATIC_DATA_END            0x00000010U
 #define REC_TYPE_X86_CPUID_POLICY           0x00000011U
 #define REC_TYPE_X86_MSR_POLICY             0x00000012U
+#define REC_TYPE_DOMAIN_CONTEXT             0x00000013U
 
 #define REC_TYPE_OPTIONAL             0x80000000U
 
Signature · 1 line
2.20.1
#22 lore raw

Re: [PATCH v10 08/11] docs / tools: specify migration v4 to include DOMAIN_CONTEXT

Quoted 14 lines · Paul Durrant
docs/specs/libxc-migration-stream.pandoc +1 -1
index 8aeab3b11b..aa6fe284f3 100644
@@ -127,7 +127,7 @@ marker      0xFFFFFFFFFFFFFFFF.
 
 id          0x58454E46 ("XENF" in ASCII).
 
-version     0x00000003.  The version of this specification.
+version     0x00000004.  The version of this specification.
 
 options     bit 0: Endianness.  0 = little-endian, 1 = big-endian.
 

docs/specs/libxc-migration-stream.pandoc type 0x00000000: END · line 212

Quoted 5 lines
docs/specs/libxc-migration-stream.pandoc +1 -1
@@ -209,9 +209,9 @@ type         0x00000000: END
 
              0x00000006: X86_PV_VCPU_XSAVE
 
-             0x00000007: SHARED_INFO
+             0x00000007: SHARED_INFO (deprecated)

"in v4" ?

docs/specs/libxc-migration-stream.pandoc X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs · line 451

Quoted 12 lines
docs/specs/libxc-migration-stream.pandoc +4 -3
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS             XEN_DOMCTL_{get,set}\_vcpu_msrs
 
 \clearpage
 
-SHARED_INFO
------------
+SHARED_INFO (deprecated)
+------------------------
 
-The content of the Shared Info page.
+The content of the Shared Info page. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.

This needs to be stricter.  A SHARED_INFO frame must not be present in a v4 stream.

Absolutely nothing good can come from having the state twice.  Moreover, ...

Quoted 6 lines
docs/specs/libxc-migration-stream.pandoc +0 -0
      0     1     2     3     4     5     6     7 octet
     +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info      Contents of the shared info page.  This record
 
 \clearpage

docs/specs/libxc-migration-stream.pandoc shared_info Contents of the shared info page. This record · line 473

Quoted 10 lines
docs/specs/libxc-migration-stream.pandoc +4 -3
 
-X86_TSC_INFO
-------------
+X86_TSC_INFO (deprecated)
+-------------------------
 
 Domain TSC information, as accessed by the
-XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops.
+XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the
+DOMAIN_CONTEXT record as of specification version 4.

... it is actively problematic for this one, as incarnation counts the number of set_tsc_info() hypercalls.

~Andrew

#23 lore raw

PATCH v10 9/11 tools/python: modify libxc.py to verify v4 stream

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

This patch adds code to verify the presence of a REC_TYPE_domain_context record in a v4 stream, as well as absence of REC_TYPE_shared_info and REC_TYPE_tsc_info records.

---
Cc: Andrew Cooper <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v10:
- New in v10

---
 tools/python/xen/migration/libxc.py | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
tools/python/xen/migration/libxc.py +20 -2
index 9881f5ced4..24fb50cbda 100644
@@ -59,6 +59,7 @@ REC_TYPE_checkpoint_dirty_pfn_list  = 0x0000000f
 REC_TYPE_static_data_end            = 0x00000010
 REC_TYPE_x86_cpuid_policy           = 0x00000011
 REC_TYPE_x86_msr_policy             = 0x00000012
+REC_TYPE_domain_context             = 0x00000013
 
 rec_type_to_str = {
     REC_TYPE_end                        : "End",
@@ -80,6 +81,7 @@ rec_type_to_str = {
     REC_TYPE_static_data_end            : "Static data end",
     REC_TYPE_x86_cpuid_policy           : "x86 CPUID policy",
     REC_TYPE_x86_msr_policy             : "x86 MSR policy",
+    REC_TYPE_domain_context             : "Domain context",
 }
 
 # page_data
@@ -156,9 +158,9 @@ class VerifyLibxc(VerifyBase):
             raise StreamError("Bad image id: Expected 0x%x, got 0x%x" %
                               (IHDR_IDENT, ident))
 
-        if not (2 <= version <= 3):
+        if not (2 <= version <= 4):
             raise StreamError(
-                "Unknown image version: Expected 2 <= ver <= 3, got %d" %
+                "Unknown image version: Expected 2 <= ver <= 4, got %d" %
                 (version, ))
 
         self.version = version
@@ -362,6 +364,9 @@ class VerifyLibxc(VerifyBase):
     def verify_record_shared_info(self, content):
         """ shared info record """
 
+        if self.version >= 4:
+            raise RecordError("Shared info record found in v4 stream")
+
         contentsz = len(content)
         if contentsz != 4096:
             raise RecordError("Length expected to be 4906 bytes, not %d" %
@@ -371,6 +376,9 @@ class VerifyLibxc(VerifyBase):
     def verify_record_tsc_info(self, content):
         """ tsc info record """
 
+        if self.version >= 4:
+            raise RecordError("TSC info record found in v4 stream")
+
         sz = calcsize(X86_TSC_INFO_FORMAT)
 
         if len(content) != sz:
@@ -476,6 +484,14 @@ class VerifyLibxc(VerifyBase):
             raise RecordError("Record length %u, expected multiple of %u" %
                               (contentsz, sz))
 
+    def verify_record_domain_context(self, content):
+        """ domain context record """
+
+        if self.version < 4:
+            raise RecordError("Domain context record found in v3 stream")
+
+        if len(content) == 0:
+            raise RecordError("Zero length domain context")
 
 record_verifiers = {
     REC_TYPE_end:
@@ -526,4 +542,6 @@ record_verifiers = {
         VerifyLibxc.verify_record_x86_cpuid_policy,
     REC_TYPE_x86_msr_policy:
         VerifyLibxc.verify_record_x86_msr_policy,
+    REC_TYPE_domain_context:
+        VerifyLibxc.verify_record_domain_context,
     }
Signature · 1 line
2.20.1
#24 lore raw

Re: [PATCH v10 09/11] tools/python: modify libxc.py to verify v4 stream

tools/python/xen/migration/libxc.py class VerifyLibxc(VerifyBase): · line 494

Quoted 12 lines · Paul Durrant
tools/python/xen/migration/libxc.py +8 -0
@@ -476,6 +484,14 @@ class VerifyLibxc(VerifyBase):
             raise RecordError("Record length %u, expected multiple of %u" %
                               (contentsz, sz))
 
+    def verify_record_domain_context(self, content):
+        """ domain context record """
+
+        if self.version < 4:
+            raise RecordError("Domain context record found in v3 stream")
+
+        if len(content) == 0:
+            raise RecordError("Zero length domain context")

This needs a recursive dissector to validate the domain context format, as it is not a private ABI within Xen.

~Andrew

#25 lore raw

PATCH v10 10/11 tools/libs/guest: add code to restore a v4 libxc stream

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

This patch adds the necessary code to accept a v4 stream, and to recognise and restore a REC_TYPE_DOMAIN_CONTEXT record.

---
Cc: Andrew Cooper <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v10:
- New in v10
- Derived from patch #8 of the v9 series
- Fix memory leak

---
 tools/libs/guest/xg_sr_common.c          |  1 +
 tools/libs/guest/xg_sr_common.h          |  3 +++
 tools/libs/guest/xg_sr_restore.c         | 24 ++++++++++++++++++++++--
 tools/libs/guest/xg_sr_restore_x86_hvm.c |  9 +++++++++
 tools/libs/guest/xg_sr_restore_x86_pv.c  |  9 +++++++++
 5 files changed, 44 insertions(+), 2 deletions(-)
tools/libs/guest/xg_sr_common.c +1 -0
index 17567ab133..f813320202 100644
@@ -39,6 +39,7 @@ static const char *const mandatory_rec_types[] =
     [REC_TYPE_STATIC_DATA_END]              = "Static data end",
     [REC_TYPE_X86_CPUID_POLICY]             = "x86 CPUID policy",
     [REC_TYPE_X86_MSR_POLICY]               = "x86 MSR policy",
+    [REC_TYPE_DOMAIN_CONTEXT]               = "Domain context",
 };
 
 const char *rec_type_to_str(uint32_t type)
tools/libs/guest/xg_sr_common.h +3 -0
index cc3ad1c394..ba9e5b0a84 100644
@@ -297,6 +297,9 @@ struct xc_sr_context
 
             /* Sender has invoked verify mode on the stream. */
             bool verify;
+
+            /* Domain context blob. */
+            struct xc_sr_blob dom_ctx;
         } restore;
     };
 
tools/libs/guest/xg_sr_restore.c +22 -2
index b57a787519..9d2bbdfaa3 100644
@@ -35,9 +35,9 @@ static int read_headers(struct xc_sr_context *ctx)
         return -1;
     }
 
-    if ( ihdr.version < 2 || ihdr.version > 3 )
+    if ( ihdr.version < 2 || ihdr.version > 4 )
     {
-        ERROR("Invalid Version: Expected 2 <= ver <= 3, Got %d",
+        ERROR("Invalid Version: Expected 2 <= ver <= 4, Got %d",
               ihdr.version);
         return -1;
     }
@@ -682,6 +682,21 @@ int handle_static_data_end(struct xc_sr_context *ctx)
     return rc;
 }
 
+/*
+ * Process a DOMAIN_CONTEXT record from the stream.
+ */
+static int handle_domain_context(struct xc_sr_context *ctx,
+                                 struct xc_sr_record *rec)
+{
+    xc_interface *xch = ctx->xch;
+    int rc = update_blob(&ctx->restore.dom_ctx, rec->data, rec->length);
+
+    if ( rc )
+        ERROR("Unable to allocate %u bytes for domain context", rec->length);
+
+    return rc;
+}
+
 static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
 {
     xc_interface *xch = ctx->xch;
@@ -709,6 +724,10 @@ static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
         rc = handle_static_data_end(ctx);
         break;
 
+    case REC_TYPE_DOMAIN_CONTEXT:
+        rc = handle_domain_context(ctx, rec);
+        break;
+
     default:
         rc = ctx->restore.ops.process_record(ctx, rec);
         break;
@@ -784,6 +803,7 @@ static void cleanup(struct xc_sr_context *ctx)
 
     free(ctx->restore.buffered_records);
     free(ctx->restore.populated_pfns);
+    free(ctx->restore.dom_ctx.ptr);
 
     if ( ctx->restore.ops.cleanup(ctx) )
         PERROR("Failed to clean up");
tools/libs/guest/xg_sr_restore_x86_hvm.c +9 -0
index d6ea6f3012..6bb164b9f0 100644
@@ -225,6 +225,15 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)
         return rc;
     }
 
+    rc = xc_domain_set_context(xch, ctx->domid,
+                               ctx->restore.dom_ctx.ptr,
+                               ctx->restore.dom_ctx.size);
+    if ( rc )
+    {
+        PERROR("Unable to restore Domain context");
+        return rc;
+    }
+
     rc = xc_dom_gnttab_seed(xch, ctx->domid, true,
                             ctx->restore.console_gfn,
                             ctx->restore.xenstore_gfn,
tools/libs/guest/xg_sr_restore_x86_pv.c +9 -0
index dc50b0f5a8..2dafad7b83 100644
@@ -1134,6 +1134,15 @@ static int x86_pv_stream_complete(struct xc_sr_context *ctx)
     if ( rc )
         return rc;
 
+    rc = xc_domain_set_context(xch, ctx->domid,
+                               ctx->restore.dom_ctx.ptr,
+                               ctx->restore.dom_ctx.size);
+    if ( rc )
+    {
+        PERROR("Unable to restore Domain context");
+        return rc;
+    }
+
     rc = xc_dom_gnttab_seed(xch, ctx->domid, false,
                             ctx->restore.console_gfn,
                             ctx->restore.xenstore_gfn,
Signature · 1 line
2.20.1
#26 lore raw

Re: [PATCH v10 10/11] tools/libs/guest: add code to restore a v4 libxc stream

Quoted 6 lines · Paul Durrant

From: Paul Durrant <[email protected]>

This patch adds the necessary code to accept a v4 stream, and to recognise and restore a REC_TYPE_DOMAIN_CONTEXT record.

Somewhere within this needs to be logic to reject the forbidden records in relevant stream versions.

Quoted 6 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +0 -0
index d6ea6f3012..6bb164b9f0 100644
@@ -225,6 +225,15 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)
         return rc;

tools/libs/guest/xg_sr_restore_x86_hvm.c static int x86_hvm_stream_complete(struct xc_sr_context *ctx) · line 235

Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +8 -0
     }
 
+    rc = xc_domain_set_context(xch, ctx->domid,
+                               ctx->restore.dom_ctx.ptr,
+                               ctx->restore.dom_ctx.size);
+    if ( rc )
+    {
+        PERROR("Unable to restore Domain context");
+        return rc;
+    }

This doesn't match where you specified the record to live in the stream, and in particular is reordered WRT HVMCONTEXT restoration.

Also, it appears to be in the middle of a block of code which needs to become if ( guest-aware ).

Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +1 -0
+
     rc = xc_dom_gnttab_seed(xch, ctx->domid, true,
                             ctx->restore.console_gfn,
                             ctx->restore.xenstore_gfn,
tools/libs/guest/xg_sr_restore_x86_pv.c +0 -0
index dc50b0f5a8..2dafad7b83 100644
@@ -1134,6 +1134,15 @@ static int x86_pv_stream_complete(struct xc_sr_context *ctx)
     if ( rc )

tools/libs/guest/xg_sr_restore_x86_pv.c static int x86_pv_stream_complete(struct xc_sr_context *ctx) · line 1144

Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_pv.c +8 -0
         return rc;
 
+    rc = xc_domain_set_context(xch, ctx->domid,
+                               ctx->restore.dom_ctx.ptr,
+                               ctx->restore.dom_ctx.size);
+    if ( rc )
+    {
+        PERROR("Unable to restore Domain context");
+        return rc;
+    }

Similar comment as HVM for the reordering.  PV guests in particular tend to be far more sensitive to the restoration order.

~Andrew

#27 lore raw

PATCH v10 11/11 tools/libs/guest: add code to save a v4 libxc stream

In reply to Paul Durrant

From: Paul Durrant <[email protected]>

This patch adds the necessary code to save a REC_TYPE_DOMAIN_CONTEXT record, and stop saving the now obsolete REC_TYPE_SHARED_INFO and REC_TYPE_TSC_INFO records for PV guests.

---
Cc: Andrew Cooper <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v10:
- New in v10
- Derived from patch #8 of the v9 series

---
 tools/libs/guest/xg_sr_common_x86.c   | 20 -----------
 tools/libs/guest/xg_sr_common_x86.h   |  6 ----
 tools/libs/guest/xg_sr_save.c         | 52 ++++++++++++++++++++++++++-
 tools/libs/guest/xg_sr_save_x86_hvm.c |  5 ---
 tools/libs/guest/xg_sr_save_x86_pv.c  | 22 ------------
 5 files changed, 51 insertions(+), 54 deletions(-)
tools/libs/guest/xg_sr_common_x86.c +0 -20
index 6f12483907..10a35b998e 100644
@@ -1,25 +1,5 @@
 #include "xg_sr_common_x86.h"
 
-int write_x86_tsc_info(struct xc_sr_context *ctx)
-{
-    xc_interface *xch = ctx->xch;
-    struct xc_sr_rec_x86_tsc_info tsc = {};
-    struct xc_sr_record rec = {
-        .type = REC_TYPE_X86_TSC_INFO,
-        .length = sizeof(tsc),
-        .data = &tsc,
-    };
-
-    if ( xc_domain_get_tsc_info(xch, ctx->domid, &tsc.mode,
-                                &tsc.nsec, &tsc.khz, &tsc.incarnation) < 0 )
-    {
-        PERROR("Unable to obtain TSC information");
-        return -1;
-    }
-
-    return write_record(ctx, &rec);
-}
-
 int handle_x86_tsc_info(struct xc_sr_context *ctx, struct xc_sr_record *rec)
 {
     xc_interface *xch = ctx->xch;
tools/libs/guest/xg_sr_common_x86.h +0 -6
index b55758c96d..e504169705 100644
@@ -3,12 +3,6 @@
 
 #include "xg_sr_common.h"
 
-/*
- * Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record
- * into the stream.
- */
-int write_x86_tsc_info(struct xc_sr_context *ctx);
-
 /*
  * Parses a X86_TSC_INFO record and applies the result to the domain.
  */
tools/libs/guest/xg_sr_save.c +51 -1

70 lines not on this page · raw message

tools/libs/guest/xg_sr_save_x86_hvm.c +0 -5
index 1634a7bc43..d44fb3fc4f 100644
@@ -193,11 +193,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)
 {
     int rc;
 
-    /* Write the TSC record. */
-    rc = write_x86_tsc_info(ctx);
-    if ( rc )
-        return rc;
-
     /* Write the HVM_CONTEXT record. */
     rc = write_hvm_context(ctx);
     if ( rc )
tools/libs/guest/xg_sr_save_x86_pv.c +0 -22
index 4964f1f7b8..3de7b19f54 100644
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)
     return rc;
 }
 
-/*
- * Writes an SHARED_INFO record into the stream.
- */
-static int write_shared_info(struct xc_sr_context *ctx)
-{
-    struct xc_sr_record rec = {
-        .type = REC_TYPE_SHARED_INFO,
-        .length = PAGE_SIZE,
-        .data = ctx->x86.pv.shinfo,
-    };
-
-    return write_record(ctx, &rec);
-}
-
 /*
  * Normalise a pagetable for the migration stream.  Performs mfn->pfn
  * conversions on the ptes.
@@ -1093,14 +1079,6 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx)
 {
     int rc;
 
-    rc = write_x86_tsc_info(ctx);
-    if ( rc )
-        return rc;
-
-    rc = write_shared_info(ctx);
-    if ( rc )
-        return rc;
-
     rc = write_all_vcpu_information(ctx);
     if ( rc )
         return rc;
Signature · 1 line
2.20.1
#28 lore raw

Re: [PATCH v10 11/11] tools/libs/guest: add code to save a v4 libxc stream

Quoted 7 lines · Paul Durrant

From: Paul Durrant <[email protected]>

This patch adds the necessary code to save a REC_TYPE_DOMAIN_CONTEXT record, and stop saving the now obsolete REC_TYPE_SHARED_INFO and REC_TYPE_TSC_INFO records for PV guests.

Looks broadly ok.

Quoted 23 lines
tools/libs/guest/xg_sr_common_x86.h +15 -0
index b55758c96d..e504169705 100644
@@ -44,6 +44,52 @@ static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type)
     return 0;
 }
 
+/*
+ * Writes a DOMAIN_CONTEXT record into the stream.
+ */
+static int write_domain_context_record(struct xc_sr_context *ctx)
+{
+    xc_interface *xch = ctx->xch;
+    struct xc_sr_record rec = {
+        .type = REC_TYPE_DOMAIN_CONTEXT,
+    };
+    size_t len = 0;
+    int rc;
+
+    rc = xc_domain_get_context(xch, ctx->domid, NULL, &len);
+    if ( rc < 0 )
+    {

tools/libs/guest/xg_sr_common_x86.h static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 71

Quoted 10 lines
tools/libs/guest/xg_sr_common_x86.h +10 -0
+        PERROR("can't get record length for dom %u\n", ctx->domid);
+        goto out;
+    }
+
+    rec.data = malloc(len);
+
+    rc = -1;
+    if ( !rec.data )
+    {
+        PERROR("can't allocate %lu bytes\n", len);

%zu, because not all versions of C have size_t the same as unsigned long.

tools/libs/guest/xg_sr_common_x86.h static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 78

Quoted 7 lines
tools/libs/guest/xg_sr_common_x86.h +7 -0
+        goto out;
+    }
+
+    rc = xc_domain_get_context(xch, ctx->domid, rec.data, &len);
+    if ( rc < 0 )
+    {
+        PERROR("can't get domain record for dom %u\n", ctx->domid);

"domain context", and above.

Quoted 11 lines
tools/libs/guest/xg_sr_save_x86_pv.c +0 -3
index 4964f1f7b8..3de7b19f54 100644
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)
     return rc;
 }
 
-/*
- * Writes an SHARED_INFO record into the stream.
- */

tools/libs/guest/xg_sr_save_x86_pv.c static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx) · line 864

Quoted 10 lines
tools/libs/guest/xg_sr_save_x86_pv.c +0 -10
-static int write_shared_info(struct xc_sr_context *ctx)
-{
-    struct xc_sr_record rec = {
-        .type = REC_TYPE_SHARED_INFO,
-        .length = PAGE_SIZE,
-        .data = ctx->x86.pv.shinfo,
-    };
-
-    return write_record(ctx, &rec);
-}

This change also wants to strip out ctx->x86.pv.shinfo, and the mapping logic.

~Andrew

#29 lore raw
In reply to Paul Durrant
Quoted 14 lines · Paul Durrant

From: Paul Durrant <[email protected]>

Paul Durrant (11):
docs / include: introduce a new framework for 'domain context' records
xen: introduce implementation of save/restore of 'domain context' xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context tools/misc: add xen-domctx to present domain context common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... docs/specs: add missing definitions to libxc-migration-stream docs / tools: specify migration v4 to include DOMAIN_CONTEXT tools/python: modify libxc.py to verify v4 stream tools/libs/guest: add code to restore a v4 libxc stream tools/libs/guest: add code to save a v4 libxc stream

Thanks - this is much better when it comes to the public API/ABI.

However, my concerns still stands.  What *else* is going in the domain context record, because you can pull the "bump the interface version and deprecated new fields" exactly once, as the libxg logic doesn't delve into domain context stream.

At the moment this does increase the downtime of the VM for no gain. What I'm trying to understand is whether this is "no gain (yet)" or whether you consider this "done" as far as cooperative migrate is concerned.

~Andrew