Skip to contentSkip to conversation

domain context infrastructure

PATCH Superseded Paul Durrant 4 participants 30 messages Apr 7, 2020 – May 13, 2020

v2 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v2 cover domain context infrastructure

From: Paul Durrant <[email protected]>

Paul Durrant (5):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context common/domain: add a domain context record for shared_info... tools/libxc: make use of domain context SHARED_INFO record...

 .gitignore                             |   1 +
 tools/flask/policy/modules/xen.if      |   4 +-
 tools/libxc/include/xenctrl.h          |   5 +
 tools/libxc/xc_domain.c                |  54 ++++
 tools/libxc/xc_sr_common.h             |   7 +-
 tools/libxc/xc_sr_common_x86.c         |  59 +++++
 tools/libxc/xc_sr_common_x86.h         |   4 +
 tools/libxc/xc_sr_common_x86_pv.c      |  53 ++++
 tools/libxc/xc_sr_common_x86_pv.h      |   3 +
 tools/libxc/xc_sr_restore_x86_pv.c     |  40 ++-
 tools/libxc/xc_sr_save_x86_pv.c        |  26 +-
 tools/libxc/xg_save_restore.h          |   1 +
 tools/misc/Makefile                    |   4 +
 tools/misc/xen-domctx.c                | 156 ++++++++++++
 xen/common/Makefile                    |   1 +
 xen/common/domain.c                    |  81 ++++++
 xen/common/domctl.c                    | 117 +++++++++
 xen/common/save.c                      | 329 +++++++++++++++++++++++++
 xen/include/public/arch-arm/hvm/save.h |   5 +
 xen/include/public/arch-x86/hvm/save.h |   5 +
 xen/include/public/domctl.h            |  44 +++-
 xen/include/public/save.h              |  92 +++++++
 xen/include/xen/save.h                 | 152 ++++++++++++
 xen/xsm/flask/hooks.c                  |   6 +
 xen/xsm/flask/policy/access_vectors    |   4 +
 25 files changed, 1201 insertions(+), 52 deletions(-)
 create mode 100644 tools/misc/xen-domctx.c
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/public/save.h
 create mode 100644 xen/include/xen/save.h

---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>

Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/5 xen/common: introduce a new framework for save/restore of 'domain' context

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework

---
 xen/common/Makefile                    |   1 +
 xen/common/save.c                      | 329 +++++++++++++++++++++++++
 xen/include/public/arch-arm/hvm/save.h |   5 +
 xen/include/public/arch-x86/hvm/save.h |   5 +
 xen/include/public/save.h              |  84 +++++++
 xen/include/xen/save.h                 | 152 ++++++++++++
 6 files changed, 576 insertions(+)
 create mode 100644 xen/common/save.c
 create mode 100644 xen/include/public/save.h
 create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
index e8cde65370..90553ba5d7 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
 obj-y += rbtree.o
 obj-y += rcupdate.o
 obj-y += rwlock.o
+obj-y += save.o
 obj-y += shutdown.o
 obj-y += softirq.o
 obj-y += sort.o
xen/common/save.c new file +329 -0

329 lines not on this page · raw message

xen/include/public/arch-arm/hvm/save.h +5 -0
index 75b8e65bcb..d5b0c15203 100644
@@ -26,6 +26,11 @@
 #ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__
 #define __XEN_PUBLIC_HVM_SAVE_ARM_H__
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif
 
 /*
xen/include/public/arch-x86/hvm/save.h +5 -0
index 773a380bc2..e61e2dbcd7 100644
@@ -648,6 +648,11 @@ struct hvm_msr {
  */
 #define HVM_SAVE_CODE_MAX 20
 
+/*
+ * Further use of HVM state is deprecated. New state records should only
+ * be added to the domain state header: public/save.h
+ */
+
 #endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */
 
 /*
xen/include/public/save.h new file +84 -0

84 lines not on this page · raw message

xen/include/xen/save.h new file +152 -0

152 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Hi Paul,

Quoted 83 lines · Paul Durrant

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework

---
  xen/common/Makefile                    |   1 +
  xen/common/save.c                      | 329 +++++++++++++++++++++++++
  xen/include/public/arch-arm/hvm/save.h |   5 +
  xen/include/public/arch-x86/hvm/save.h |   5 +
  xen/include/public/save.h              |  84 +++++++
  xen/include/xen/save.h                 | 152 ++++++++++++
  6 files changed, 576 insertions(+)

create mode 100644 xen/common/save.c
create mode 100644 xen/include/public/save.h
create mode 100644 xen/include/xen/save.h

xen/common/Makefile +1 -0
index e8cde65370..90553ba5d7 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
  obj-y += rbtree.o
  obj-y += rcupdate.o
  obj-y += rwlock.o
+obj-y += save.o
  obj-y += shutdown.o
  obj-y += softirq.o
  obj-y += sort.o
xen/common/save.c new file +19 -0
new file mode 100644
index 0000000000..6cdac3785b
@@ -0,0 +1,329 @@
+/*
+ * save.c: Save and restore PV guest state common to all domain types.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/save.h>

xen/common/save.c line 29

Quoted 10 lines
xen/common/save.c new file +10 -0
+
+union domain_copy_entry {
+    domain_write_entry write;
+    domain_read_entry read;
+};
+
+struct domain_context {
+    bool log;
+    struct domain_save_descriptor desc;
+    size_t data_len;

What is data_len?

Quoted 25 lines
xen/common/save.c new file +25 -0
+    union domain_copy_entry copy;
+    void *priv;
+};
+
+static struct {
+    const char *name;
+    bool per_vcpu;
+    domain_save_handler save;
+    domain_load_handler load;
+} handlers[DOMAIN_SAVE_CODE_MAX + 1];
+
+void __init domain_register_save_type(unsigned int tc, const char *name,
+                                      bool per_vcpu,
+                                      domain_save_handler save,
+                                      domain_load_handler load)
+{
+    BUG_ON(tc > ARRAY_SIZE(handlers));
+
+    ASSERT(!handlers[tc].save);
+    ASSERT(!handlers[tc].load);
+
+    handlers[tc].name = name;
+    handlers[tc].per_vcpu = per_vcpu;
+    handlers[tc].save = save;
+    handlers[tc].load = load;

xen/common/save.c line 64

Quoted 10 lines
xen/common/save.c new file +10 -0
+}
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len)
+{
+    int rc;
+
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv save: %s (%lu)\n", v, name,
+                 (unsigned long)len);

How about using %zu rather than a cast here?

xen/common/save.c line 67

Quoted 3 lines
xen/common/save.c new file +3 -0
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);

I can't find any answer on my question about this part. I understand the code would be buggy if this happen, but is it warrant to crash the host? Couldn't you just return an error and continue to run?

xen/common/save.c line 76

Quoted 9 lines
xen/common/save.c new file +9 -0
+
+    ASSERT(!c->data_len);
+    c->data_len = c->desc.length = len;
+
+    rc = c->copy.write(c->priv, &c->desc, sizeof(c->desc));
+    if ( rc )
+        return rc;
+
+    c->desc.length = 0;

This is confusing, why would you need to reset c->desc.length but not the rest of the fields?

Quoted 11 lines
xen/common/save.c new file +11 -0
+
+    return 0;
+}
+
+int domain_save_data(struct domain_context *c, const void *src, size_t len)
+{
+    if ( c->desc.length + len > c->data_len )
+        return -ENOSPC;
+
+    c->desc.length += len;
+

xen/common/save.c line 97

Quoted 10 lines
xen/common/save.c new file +10 -0
+    return c->copy.write(c->priv, src, len);
+}
+
+int domain_save_end(struct domain_context *c)
+{
+    /*
+     * If desc.length does not match the length specified in
+     * domain_save_begin(), there should have been more data.
+     */
+    if ( c->desc.length != c->data_len )

This suggests you know in advance the size of the record. I have seen some cases where we don't know the size in advance. A good example if when saving grants. You know the maximum of grant used by the guest but you don't know yet the number of grants used. You might need to walk the "list" twice or allocate a temporary buffer. None of them are ideal.

Another example is when saving memory, we may want to compact page informations to save space.

This problem is going to be more relevant for LiveUpdate where we need to be able to create the stream in a very short amount of time. Allocating any temporary buffer and/or walking the list twice is going to kill the performance.

I would suggest to consider a different approach where you update the record length at the end.

FWIW, this below the current approach for the LU stream (IIRC David sent a prototype recently). A record is opened using lu_stream_open_record(), you then have two way to add data:
- lu_stream_append() -> This takes a buffer and write to the stream.
- lu_stream_reserve() -> Pre-allocate space in the stream and
return a pointer to the beginning of the reserved region.
- lu_stream_end_reservation() -> Takes the actual size in parameter
and update the stream size.
- lu_stream_close_record() -> Update the header with the total
length and update the position in the stream.

xen/common/save.c line 98

Quoted 1 line
xen/common/save.c new file +1 -0
+        return -EIO;

I noticed that all the pasding check have been dropped. I still think we need implicit padding to harden the code.

Quoted 15 lines
xen/common/save.c new file +15 -0
+
+    c->data_len = 0;
+
+    return 0;
+}
+
+int domain_save(struct domain *d, domain_write_entry write, void *priv,
+                bool dry_run)
+{
+    struct domain_context c = {
+        .copy.write = write,
+        .priv = priv,
+        .log = !dry_run,
+    };
+    struct domain_save_header h = {

xen/common/save.c line 123

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .magic = DOMAIN_SAVE_MAGIC,
+        .version = DOMAIN_SAVE_VERSION,
+    };
+    struct domain_save_header e;
+    unsigned int i;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )

I can't find an answer to my question about d->is_dying. What if the guest die afterwards? What does protect us against domain_kill()?

[...]

Quoted 4 lines
xen/common/save.c new file +4 -0
+int domain_load(struct domain *d, domain_read_entry read, void *priv)
+{
+    struct domain_context c = {
+        .copy.read = read,

xen/common/save.c line 245

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .priv = priv,
+        .log = true,
+    };
+    struct domain_save_header h;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;

Same here.

Quoted 25 lines
xen/include/public/save.h new file +19 -0
new file mode 100644
index 0000000000..7e5f8752bd
@@ -0,0 +1,84 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

xen/include/public/save.h line 29

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_SAVE_H__
+#define __XEN_PUBLIC_SAVE_H__

Does this header need to be exposed outside of Xen and the tools?

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+
+#include "xen.h"
+
+/* Each entry is preceded by a descriptor */
+struct domain_save_descriptor {

xen/include/public/save.h line 44

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.
+     * Entries relating to global state should have zero in this field.
+     */
+    uint16_t vcpu_id;
+    uint32_t flags;
+    /*
+     * When restoring state this flag can be set in a descriptor to cause
+     * its content to be ignored.

Could you give examples where you would want to ignore a descriptor?

Quoted 24 lines
xen/include/public/save.h new file +24 -0
+     *
+     * NOTE: It is invalid to set this flag for HEADER or END records (see
+     *       below).
+     */
+#define _DOMAIN_SAVE_FLAG_IGNORE 0
+#define DOMAIN_SAVE_FLAG_IGNORE (1u << _DOMAIN_SAVE_FLAG_IGNORE)
+
+    /* Entry length not including this descriptor */
+    uint64_t length;
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
+ * binds these things together.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct __DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->t)
+
+/* Terminating entry */

xen/include/public/save.h line 78

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);
+
+#define DOMAIN_SAVE_MAGIC   0x53415645
+#define DOMAIN_SAVE_VERSION 0x00000001
+
+/* Initial entry */
+struct domain_save_header {
+    uint32_t magic;             /* Must be DOMAIN_SAVE_MAGIC */
+    uint32_t version;           /* Save format version */

I haven't seen any answer about xen version here. For the record:

"Let's imagine in 4.14 we introduced a bug in the saving part. This is solved in 4.15 but somehow the version is not bumped. How would you differentiate the streams saved by Xen 4.14 so you can still migrate?

If you record the version of Xen in the record automatically, then you at least have a way to differentiate the two versions."

Cheers,

Signature · 1 line
Julien Grall
#3 lore raw

RE: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 100 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 20 April 2020 18:21
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Hi Paul,

Quoted 87 lines · Paul Durrant

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework

---
  xen/common/Makefile                    |   1 +
  xen/common/save.c                      | 329 +++++++++++++++++++++++++
  xen/include/public/arch-arm/hvm/save.h |   5 +
  xen/include/public/arch-x86/hvm/save.h |   5 +
  xen/include/public/save.h              |  84 +++++++
  xen/include/xen/save.h                 | 152 ++++++++++++
  6 files changed, 576 insertions(+)

create mode 100644 xen/common/save.c
create mode 100644 xen/include/public/save.h
create mode 100644 xen/include/xen/save.h

xen/common/Makefile +1 -0
index e8cde65370..90553ba5d7 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
  obj-y += rbtree.o
  obj-y += rcupdate.o
  obj-y += rwlock.o
+obj-y += save.o
  obj-y += shutdown.o
  obj-y += softirq.o
  obj-y += sort.o
xen/common/save.c new file +23 -0
new file mode 100644
index 0000000000..6cdac3785b
@@ -0,0 +1,329 @@
+/*
+ * save.c: Save and restore PV guest state common to all domain types.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/save.h>
+
+union domain_copy_entry {
+    domain_write_entry write;
+    domain_read_entry read;
Quoted 9 lines

xen/common/save.c line 29

Quoted 6 lines
xen/common/save.c new file +6 -0
+};
+
+struct domain_context {
+    bool log;
+    struct domain_save_descriptor desc;
+    size_t data_len;

What is data_len?

It’s used for internal accounting.

Quoted 29 lines
Quoted 29 lines
xen/common/save.c new file +29 -0
+    union domain_copy_entry copy;
+    void *priv;
+};
+
+static struct {
+    const char *name;
+    bool per_vcpu;
+    domain_save_handler save;
+    domain_load_handler load;
+} handlers[DOMAIN_SAVE_CODE_MAX + 1];
+
+void __init domain_register_save_type(unsigned int tc, const char *name,
+                                      bool per_vcpu,
+                                      domain_save_handler save,
+                                      domain_load_handler load)
+{
+    BUG_ON(tc > ARRAY_SIZE(handlers));
+
+    ASSERT(!handlers[tc].save);
+    ASSERT(!handlers[tc].load);
+
+    handlers[tc].name = name;
+    handlers[tc].per_vcpu = per_vcpu;
+    handlers[tc].save = save;
+    handlers[tc].load = load;
+}
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len)
Quoted 9 lines

xen/common/save.c line 64

Quoted 6 lines
xen/common/save.c new file +6 -0
+{
+    int rc;
+
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv save: %s (%lu)\n", v, name,
+                 (unsigned long)len);

How about using %zu rather than a cast here?

Yes, that would be better.

Quoted 8 lines

xen/common/save.c line 67

Quoted 3 lines
xen/common/save.c new file +3 -0
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);

I can't find any answer on my question about this part. I understand the code would be buggy if this happen, but is it warrant to crash the host? Couldn't you just return an error and continue to run?

Since it means buggy code I could ASSERT and error out, yes.

Quoted 4 lines
Quoted 4 lines
xen/common/save.c new file +4 -0
+
+    ASSERT(!c->data_len);
+    c->data_len = c->desc.length = len;
+
Quoted 9 lines

xen/common/save.c line 76

Quoted 5 lines
xen/common/save.c new file +5 -0
+    rc = c->copy.write(c->priv, &c->desc, sizeof(c->desc));
+    if ( rc )
+        return rc;
+
+    c->desc.length = 0;

This is confusing, why would you need to reset c->desc.length but not the rest of the fields?

Because I use it to accumulate the length of the saved data and then cross check it against data_len in domain_save_end() below.

Quoted 15 lines
Quoted 15 lines
xen/common/save.c new file +15 -0
+
+    return 0;
+}
+
+int domain_save_data(struct domain_context *c, const void *src, size_t len)
+{
+    if ( c->desc.length + len > c->data_len )
+        return -ENOSPC;
+
+    c->desc.length += len;
+
+    return c->copy.write(c->priv, src, len);
+}
+
+int domain_save_end(struct domain_context *c)
Quoted 8 lines

xen/common/save.c line 97

Quoted 6 lines
xen/common/save.c new file +6 -0
+{
+    /*
+     * If desc.length does not match the length specified in
+     * domain_save_begin(), there should have been more data.
+     */
+    if ( c->desc.length != c->data_len )

This suggests you know in advance the size of the record.

That depends on what you mean by 'in advance'. I'd expect the caller of domain_save_begin() to know exactly.

Quoted 17 lines

I have seen
some cases where we don't know the size in advance. A good example if when saving grants. You know the maximum of grant used by the guest but you don't know yet the number of grants used. You might need to walk the "list" twice or allocate a temporary buffer. None of them are ideal.

Another example is when saving memory, we may want to compact page informations to save space.

This problem is going to be more relevant for LiveUpdate where we need to be able to create the stream in a very short amount of time. Allocating any temporary buffer and/or walking the list twice is going to kill the performance.

I would suggest to consider a different approach where you update the record length at the end.

Quoted 11 lines

FWIW, this below the current approach for the LU stream (IIRC David sent a prototype recently). A record is opened using lu_stream_open_record(), you then have two way to add data:
- lu_stream_append() -> This takes a buffer and write to the stream.
- lu_stream_reserve() -> Pre-allocate space in the stream and
return a pointer to the beginning of the reserved region.
- lu_stream_end_reservation() -> Takes the actual size in parameter
and update the stream size.
- lu_stream_close_record() -> Update the header with the total
length and update the position in the stream.

That sounds quite LU specific. For LM we still need to know up-front the maximal size of the buffer, and I was trying to work on the basis of never having to update previously saved data but I guess there's no actual harm in doing so, so we could avoid domain_save_begin() specifying the length.

Quoted 5 lines

xen/common/save.c line 98

Quoted 1 line
xen/common/save.c new file +1 -0
+        return -EIO;

I noticed that all the pasding check have been dropped. I still think we need implicit padding to harden the code.

I'd still view that as buggy code.

Quoted 29 lines
Quoted 15 lines
xen/common/save.c new file +15 -0
+
+    c->data_len = 0;
+
+    return 0;
+}
+
+int domain_save(struct domain *d, domain_write_entry write, void *priv,
+                bool dry_run)
+{
+    struct domain_context c = {
+        .copy.write = write,
+        .priv = priv,
+        .log = !dry_run,
+    };
+    struct domain_save_header h = {

xen/common/save.c line 123

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .magic = DOMAIN_SAVE_MAGIC,
+        .version = DOMAIN_SAVE_VERSION,
+    };
+    struct domain_save_header e;
+    unsigned int i;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )

I can't find an answer to my question about d->is_dying. What if the guest die afterwards? What does protect us against domain_kill()?

Quoted 1 line

[...]

As I said in a previous response, nothing protects against domain_kill(), this check is just supposed to avoid doing 'unnecessary' work for a domain we know is already dying. For LU though I guess we do need to save (some) state for even a dying domain, so the individual save handlers actually need to make the decision on whether they are going to do anything.

Quoted 48 lines
Quoted 5 lines
xen/common/save.c new file +4 -0
+int domain_load(struct domain *d, domain_read_entry read, void *priv)
+{
+    struct domain_context c = {
+        .copy.read = read,

xen/common/save.c line 245

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .priv = priv,
+        .log = true,
+    };
+    struct domain_save_header h;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;

Same here.

Quoted 29 lines
xen/include/public/save.h new file +23 -0
new file mode 100644
index 0000000000..7e5f8752bd
@@ -0,0 +1,84 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
Quoted 9 lines

xen/include/public/save.h line 29

Quoted 6 lines
xen/include/public/save.h new file +6 -0
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_SAVE_H__
+#define __XEN_PUBLIC_SAVE_H__

Does this header need to be exposed outside of Xen and the tools?

Probably not.

Quoted 9 lines
Quoted 9 lines
xen/include/public/save.h new file +9 -0
+
+#include "xen.h"
+
+/* Each entry is preceded by a descriptor */
+struct domain_save_descriptor {
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.
+     * Entries relating to global state should have zero in this field.
Quoted 9 lines

xen/include/public/save.h line 44

Quoted 6 lines
xen/include/public/save.h new file +6 -0
+     */
+    uint16_t vcpu_id;
+    uint32_t flags;
+    /*
+     * When restoring state this flag can be set in a descriptor to cause
+     * its content to be ignored.

Could you give examples where you would want to ignore a descriptor?

I was thinking of the case when, e.g. you want to update something in the shared_info... You save a context blob, modify the shared_info record, and then restore the context blob with all other records marked as 'ignore' since they have not been modified.

Quoted 38 lines
Quoted 24 lines
xen/include/public/save.h new file +24 -0
+     *
+     * NOTE: It is invalid to set this flag for HEADER or END records (see
+     *       below).
+     */
+#define _DOMAIN_SAVE_FLAG_IGNORE 0
+#define DOMAIN_SAVE_FLAG_IGNORE (1u << _DOMAIN_SAVE_FLAG_IGNORE)
+
+    /* Entry length not including this descriptor */
+    uint64_t length;
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
+ * binds these things together.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct __DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->t)
+
+/* Terminating entry */

xen/include/public/save.h line 78

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);
+
+#define DOMAIN_SAVE_MAGIC   0x53415645
+#define DOMAIN_SAVE_VERSION 0x00000001
+
+/* Initial entry */
+struct domain_save_header {
+    uint32_t magic;             /* Must be DOMAIN_SAVE_MAGIC */
+    uint32_t version;           /* Save format version */

I haven't seen any answer about xen version here. For the record:

Quoted 3 lines

"Let's imagine in 4.14 we introduced a bug in the saving part. This is solved in 4.15 but somehow the version is not bumped. How would you differentiate the streams saved by Xen 4.14 so you can still migrate?

I'd assume testing would at least save and restore on 4.14. If we then fixed a bug then why would we not bump the version?

Quoted 4 lines

If you record the version of Xen in the record automatically, then you at least have a way to differentiate the two versions."

OK, I guess redundant version information is not going to be harmful.

Paul

#4 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Hi Paul,

Quoted 110 lines · Paul Durrant
Quoted 100 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 20 April 2020 18:21
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Hi Paul,

Quoted 87 lines · Paul Durrant

To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.

Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.

This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.

This patch also marks the HVM-only framework as deprecated in favour of the new framework.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework

---
   xen/common/Makefile                    |   1 +
   xen/common/save.c                      | 329 +++++++++++++++++++++++++
   xen/include/public/arch-arm/hvm/save.h |   5 +
   xen/include/public/arch-x86/hvm/save.h |   5 +
   xen/include/public/save.h              |  84 +++++++
   xen/include/xen/save.h                 | 152 ++++++++++++
   6 files changed, 576 insertions(+)

create mode 100644 xen/common/save.c
create mode 100644 xen/include/public/save.h
create mode 100644 xen/include/xen/save.h

xen/common/Makefile +1 -0
index e8cde65370..90553ba5d7 100644
@@ -37,6 +37,7 @@ obj-y += radix-tree.o
   obj-y += rbtree.o
   obj-y += rcupdate.o
   obj-y += rwlock.o
+obj-y += save.o
   obj-y += shutdown.o
   obj-y += softirq.o
   obj-y += sort.o
xen/common/save.c new file +23 -0
new file mode 100644
index 0000000000..6cdac3785b
@@ -0,0 +1,329 @@
+/*
+ * save.c: Save and restore PV guest state common to all domain types.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/save.h>
+
+union domain_copy_entry {
+    domain_write_entry write;
+    domain_read_entry read;
Quoted 9 lines

xen/common/save.c line 29

Quoted 6 lines
xen/common/save.c new file +6 -0
+};
+
+struct domain_context {
+    bool log;
+    struct domain_save_descriptor desc;
+    size_t data_len;

What is data_len?

Quoted 1 line

It’s used for internal accounting.

Can you add a comment explaining it?

Quoted 46 lines
Quoted 30 lines
Quoted 30 lines
xen/common/save.c new file +29 -0
+    union domain_copy_entry copy;
+    void *priv;
+};
+
+static struct {
+    const char *name;
+    bool per_vcpu;
+    domain_save_handler save;
+    domain_load_handler load;
+} handlers[DOMAIN_SAVE_CODE_MAX + 1];
+
+void __init domain_register_save_type(unsigned int tc, const char *name,
+                                      bool per_vcpu,
+                                      domain_save_handler save,
+                                      domain_load_handler load)
+{
+    BUG_ON(tc > ARRAY_SIZE(handlers));
+
+    ASSERT(!handlers[tc].save);
+    ASSERT(!handlers[tc].load);
+
+    handlers[tc].name = name;
+    handlers[tc].per_vcpu = per_vcpu;
+    handlers[tc].save = save;
+    handlers[tc].load = load;
+}
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len)
Quoted 9 lines

xen/common/save.c line 64

Quoted 6 lines
xen/common/save.c new file +6 -0
+{
+    int rc;
+
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv save: %s (%lu)\n", v, name,
+                 (unsigned long)len);

How about using %zu rather than a cast here?

Yes, that would be better.

Quoted 4 lines
Quoted 3 lines
xen/common/save.c new file +3 -0
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
Quoted 6 lines
Quoted 4 lines

I can't find any answer on my question about this part. I understand the code would be buggy if this happen, but is it warrant to crash the host? Couldn't you just return an error and continue to run?

Since it means buggy code I could ASSERT and error out, yes.

That would be better, thanks!

Quoted 72 lines
Quoted 14 lines

xen/common/save.c line 76

Quoted 10 lines
xen/common/save.c new file +9 -0
+
+    ASSERT(!c->data_len);
+    c->data_len = c->desc.length = len;
+
+    rc = c->copy.write(c->priv, &c->desc, sizeof(c->desc));
+    if ( rc )
+        return rc;
+
+    c->desc.length = 0;

This is confusing, why would you need to reset c->desc.length but not the rest of the fields?

Because I use it to accumulate the length of the saved data and then cross check it against data_len in domain_save_end() below.

Quoted 15 lines
Quoted 15 lines
xen/common/save.c new file +15 -0
+
+    return 0;
+}
+
+int domain_save_data(struct domain_context *c, const void *src, size_t len)
+{
+    if ( c->desc.length + len > c->data_len )
+        return -ENOSPC;
+
+    c->desc.length += len;
+
+    return c->copy.write(c->priv, src, len);
+}
+
+int domain_save_end(struct domain_context *c)
Quoted 8 lines

xen/common/save.c line 97

Quoted 6 lines
xen/common/save.c new file +6 -0
+{
+    /*
+     * If desc.length does not match the length specified in
+     * domain_save_begin(), there should have been more data.
+     */
+    if ( c->desc.length != c->data_len )

This suggests you know in advance the size of the record.

That depends on what you mean by 'in advance'. I'd expect the caller of domain_save_begin() to know exactly.

Quoted 17 lines

I have seen
some cases where we don't know the size in advance. A good example if when saving grants. You know the maximum of grant used by the guest but you don't know yet the number of grants used. You might need to walk the "list" twice or allocate a temporary buffer. None of them are ideal.

Another example is when saving memory, we may want to compact page informations to save space.

This problem is going to be more relevant for LiveUpdate where we need to be able to create the stream in a very short amount of time. Allocating any temporary buffer and/or walking the list twice is going to kill the performance.

I would suggest to consider a different approach where you update the record length at the end.

Quoted 11 lines

FWIW, this below the current approach for the LU stream (IIRC David sent a prototype recently). A record is opened using lu_stream_open_record(), you then have two way to add data:
- lu_stream_append() -> This takes a buffer and write to the stream.
- lu_stream_reserve() -> Pre-allocate space in the stream and
return a pointer to the beginning of the reserved region.
- lu_stream_end_reservation() -> Takes the actual size in parameter
and update the stream size.
- lu_stream_close_record() -> Update the header with the total
length and update the position in the stream.

Quoted 1 line

That sounds quite LU specific. For LM we still need to know up-front the maximal size of the buffer,

I described the function from an LU PoV, but thsi is mostly replace lu_stream_reserve() by a function that check you have at least N bytes free in your stream.

Quoted 1 line

and I was trying to work on the basis of never having to update previously saved data but I guess there's no actual harm in doing so, so we could avoid domain_save_begin() specifying the length.

It depends what we want to achieve in term of memory space and time. If we want to avoid modifying previously saved data, then we would need to walk the elements twice or allocate a temporary buffer. They will both consume space and time.

A good example for LM is when we need to save the event channels. The maximum number of event channels is known in advance but you don't know how many of them are used. We could save all of them (even the free ones) but this is a waste of space.

To avoid the temporary buffer or walking the elements twice, our internal tree contain open-code code to rewrite the record in the event channels save function.

Quoted 38 lines
Quoted 6 lines

xen/common/save.c line 98

Quoted 2 lines
xen/common/save.c new file +1 -0
+        return -EIO;

I noticed that all the pasding check have been dropped. I still think we need implicit padding to harden the code.

I'd still view that as buggy code.

Quoted 29 lines
Quoted 15 lines
xen/common/save.c new file +15 -0
+
+    c->data_len = 0;
+
+    return 0;
+}
+
+int domain_save(struct domain *d, domain_write_entry write, void *priv,
+                bool dry_run)
+{
+    struct domain_context c = {
+        .copy.write = write,
+        .priv = priv,
+        .log = !dry_run,
+    };
+    struct domain_save_header h = {

xen/common/save.c line 123

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .magic = DOMAIN_SAVE_MAGIC,
+        .version = DOMAIN_SAVE_VERSION,
+    };
+    struct domain_save_header e;
+    unsigned int i;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )

I can't find an answer to my question about d->is_dying. What if the guest die afterwards? What does protect us against domain_kill()?

Quoted 3 lines
Quoted 1 line

[...]

As I said in a previous response, nothing protects against domain_kill(), this check is just supposed to avoid doing 'unnecessary' work for a domain we know is already dying.

Regardless LU (see below), I would argue that if a user purposefully call domain_save() on a dying domain, then he/she already accepted the cost.

Beside, this give a false idea that domain_kill() cannot be called in parallel. So I would rather drop this check unless we can make it safe.

Quoted 1 line

For LU though I guess we do need to save (some) state for even a dying domain, so the individual save handlers actually need to make the decision on whether they are going to do anything.

That's correct we would want to preserve states for dying domain.

Quoted 79 lines
Quoted 48 lines
Quoted 5 lines
xen/common/save.c new file +4 -0
+int domain_load(struct domain *d, domain_read_entry read, void *priv)
+{
+    struct domain_context c = {
+        .copy.read = read,

xen/common/save.c line 245

Quoted 10 lines
xen/common/save.c new file +10 -0
+        .priv = priv,
+        .log = true,
+    };
+    struct domain_save_header h;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;

Same here.

Quoted 29 lines
xen/include/public/save.h new file +23 -0
new file mode 100644
index 0000000000..7e5f8752bd
@@ -0,0 +1,84 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
Quoted 9 lines

xen/include/public/save.h line 29

Quoted 6 lines
xen/include/public/save.h new file +6 -0
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_SAVE_H__
+#define __XEN_PUBLIC_SAVE_H__

Does this header need to be exposed outside of Xen and the tools?

Probably not.

Quoted 9 lines
Quoted 9 lines
xen/include/public/save.h new file +9 -0
+
+#include "xen.h"
+
+/* Each entry is preceded by a descriptor */
+struct domain_save_descriptor {
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.
+     * Entries relating to global state should have zero in this field.
Quoted 9 lines

xen/include/public/save.h line 44

Quoted 6 lines
xen/include/public/save.h new file +6 -0
+     */
+    uint16_t vcpu_id;
+    uint32_t flags;
+    /*
+     * When restoring state this flag can be set in a descriptor to cause
+     * its content to be ignored.

Could you give examples where you would want to ignore a descriptor?

Quoted 1 line

I was thinking of the case when, e.g. you want to update something in the shared_info... You save a context blob, modify the shared_info record, and then restore the context blob with all other records marked as 'ignore' since they have not been modified.

How about introducing a domctl similar to XEN_DOMCTL_gethvmcontext_partial? This would only give you the shared_info record and make easier for the user to use.

Quoted 39 lines
Quoted 39 lines
Quoted 25 lines
xen/include/public/save.h new file +24 -0
+     *
+     * NOTE: It is invalid to set this flag for HEADER or END records (see
+     *       below).
+     */
+#define _DOMAIN_SAVE_FLAG_IGNORE 0
+#define DOMAIN_SAVE_FLAG_IGNORE (1u << _DOMAIN_SAVE_FLAG_IGNORE)
+
+    /* Entry length not including this descriptor */
+    uint64_t length;
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
+ * binds these things together.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct __DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->t)
+
+/* Terminating entry */

xen/include/public/save.h line 78

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);
+
+#define DOMAIN_SAVE_MAGIC   0x53415645
+#define DOMAIN_SAVE_VERSION 0x00000001
+
+/* Initial entry */
+struct domain_save_header {
+    uint32_t magic;             /* Must be DOMAIN_SAVE_MAGIC */
+    uint32_t version;           /* Save format version */

I haven't seen any answer about xen version here. For the record:

Quoted 5 lines
Quoted 3 lines

"Let's imagine in 4.14 we introduced a bug in the saving part. This is solved in 4.15 but somehow the version is not bumped. How would you differentiate the streams saved by Xen 4.14 so you can still migrate?

I'd assume testing would at least save and restore on 4.14. If we then fixed a bug then why would we not bump the version?

Do you mean we would test save/restore between 4.14 and 4.15 (and vice-versa)? If so, this may work but we need this to be part of OSSTest so we can catch this. Otherwise there are chance we don't catch regression in the stream.

Quoted 10 lines
Quoted 5 lines

If you record the version of Xen in the record automatically, then you at least have a way to differentiate the two versions."

OK, I guess redundant version information is not going to be harmful.

Paul

Signature · 1 line
Julien Grall
#5 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

In reply to Paul Durrant
Quoted 10 lines · Paul Durrant
xen/common/save.c new file +6 -0
+void __init domain_register_save_type(unsigned int tc, const char *name,
+                                      bool per_vcpu,
+                                      domain_save_handler save,
+                                      domain_load_handler load)
+{
+    BUG_ON(tc > ARRAY_SIZE(handlers));

=

xen/common/save.c new file +1 -0
+    ASSERT(!handlers[tc].save);

xen/common/save.c line 58

Quoted 10 lines
xen/common/save.c new file +10 -0
+    ASSERT(!handlers[tc].load);
+
+    handlers[tc].name = name;
+    handlers[tc].per_vcpu = per_vcpu;
+    handlers[tc].save = save;
+    handlers[tc].load = load;
+}
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len)

I find it quite odd for a function like this one to take a struct vcpu * rather than a struct domain *. See also below comment on the vcpu_id field in the public header.

Quoted 23 lines
xen/common/save.c new file +23 -0
+{
+    int rc;
+
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv save: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    ASSERT(!c->data_len);
+    c->data_len = c->desc.length = len;
+
+    rc = c->copy.write(c->priv, &c->desc, sizeof(c->desc));
+    if ( rc )
+        return rc;
+
+    c->desc.length = 0;
+
+    return 0;
+}
+
+int domain_save_data(struct domain_context *c, const void *src, size_t len)

xen/common/save.c line 91

Quoted 10 lines
xen/common/save.c new file +10 -0
+{
+    if ( c->desc.length + len > c->data_len )
+        return -ENOSPC;
+
+    c->desc.length += len;
+
+    return c->copy.write(c->priv, src, len);
+}
+
+int domain_save_end(struct domain_context *c)

I'm sure there is a reason for the split into three load/save functions (begin/data/end), but I can't figure it and the description also doesn't explain it. They're all used together only afaics, in domain_{save,load}_entry(). Or wait, there are DOMAIN_{SAVE,LOAD}_BEGIN() macros apparently allowing separate use of ..._begin(), but then it's still not clear why ..._end() need to be separate from ..._data().

xen/common/save.c line 113

Quoted 9 lines
xen/common/save.c new file +9 -0
+int domain_save(struct domain *d, domain_write_entry write, void *priv,
+                bool dry_run)
+{
+    struct domain_context c = {
+        .copy.write = write,
+        .priv = priv,
+        .log = !dry_run,
+    };
+    struct domain_save_header h = {

const? Perhaps even static?

xen/common/save.c line 124

Quoted 11 lines
xen/common/save.c new file +11 -0
+        .magic = DOMAIN_SAVE_MAGIC,
+        .version = DOMAIN_SAVE_VERSION,
+    };
+    struct domain_save_header e;
+    unsigned int i;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;

Could I talk you into using less generic an error code here, e.g. -ESRCH or -ENXIO? There look to be further uses of EINVAL that may want replacing.

Quoted 33 lines
xen/common/save.c new file +33 -0
+    domain_pause(d);
+
+    c.desc.typecode = DOMAIN_SAVE_CODE(HEADER);
+
+    rc = DOMAIN_SAVE_ENTRY(HEADER, &c, d->vcpu[0], &h, sizeof(h));
+    if ( rc )
+        goto out;
+
+    for ( i = 0; i < ARRAY_SIZE(handlers); i++ )
+    {
+        domain_save_handler save = handlers[i].save;
+
+        if ( !save )
+            continue;
+
+        memset(&c.desc, 0, sizeof(c.desc));
+        c.desc.typecode = i;
+
+        if ( handlers[i].per_vcpu )
+        {
+            struct vcpu *v;
+
+            for_each_vcpu ( d, v )
+            {
+                c.desc.vcpu_id = v->vcpu_id;
+
+                rc = save(v, &c, dry_run);
+                if ( rc )
+                    goto out;
+            }
+        }
+        else
+        {

xen/common/save.c line 168

Quoted 10 lines
xen/common/save.c new file +10 -0
+            rc = save(d->vcpu[0], &c, dry_run);
+            if ( rc )
+                goto out;
+        }
+    }
+
+    memset(&c.desc, 0, sizeof(c.desc));
+    c.desc.typecode = DOMAIN_SAVE_CODE(END);
+
+    rc = DOMAIN_SAVE_ENTRY(END, &c, d->vcpu[0], &e, 0);

By the looks of it you're passing uninitialized e here; it's just that the struct has no members. It would look less odd if you used NULL here. Otherwise please don't use literal 0, but sizeof() for the last parameter.

Quoted 4 lines
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

    if ( exact ? len != c->desc.length
               : len < c->desc.length )

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

Quoted 40 lines
xen/common/save.c new file +40 -0
+int domain_load(struct domain *d, domain_read_entry read, void *priv)
+{
+    struct domain_context c = {
+        .copy.read = read,
+        .priv = priv,
+        .log = true,
+    };
+    struct domain_save_header h;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;
+
+    rc = c.copy.read(c.priv, &c.desc, sizeof(c.desc));
+    if ( rc )
+        return rc;
+
+    if ( c.desc.typecode != DOMAIN_SAVE_CODE(HEADER) || c.desc.vcpu_id ||
+         c.desc.flags )
+        return -EINVAL;
+
+    rc = DOMAIN_LOAD_ENTRY(HEADER, &c, d->vcpu[0], &h, sizeof(h), true);
+    if ( rc )
+        return rc;
+
+    if ( h.magic != DOMAIN_SAVE_MAGIC || h.version != DOMAIN_SAVE_VERSION )
+        return -EINVAL;
+
+    domain_pause(d);
+
+    for (;;)
+    {
+        unsigned int i;
+        unsigned int flags;
+        domain_load_handler load;
+        struct vcpu *v;
+
+        rc = c.copy.read(c.priv, &c.desc, sizeof(c.desc));

xen/common/save.c line 281

Quoted 10 lines
xen/common/save.c new file +10 -0
+        if ( rc )
+            break;
+
+        rc = -EINVAL;
+
+        flags = c.desc.flags;
+        if ( flags & ~DOMAIN_SAVE_FLAG_IGNORE )
+            break;
+
+        if ( c.desc.typecode == DOMAIN_SAVE_CODE(END) ) {

Brace placement.

xen/common/save.c line 283

Quoted 2 lines
xen/common/save.c new file +2 -0
+            if ( !(flags & DOMAIN_SAVE_FLAG_IGNORE) )
+                rc = DOMAIN_LOAD_ENTRY(END, &c, d->vcpu[0], NULL, 0, true);

The public header says DOMAIN_SAVE_FLAG_IGNORE is invalid for END.

Quoted 8 lines
xen/common/save.c new file +8 -0
+            break;
+        }
+
+        i = c.desc.typecode;
+        if ( i >= ARRAY_SIZE(handlers) )
+            break;
+
+        if ( (!handlers[i].per_vcpu && c.desc.vcpu_id) ||

xen/common/save.c line 302

Quoted 10 lines
xen/common/save.c new file +10 -0
+             (c.desc.vcpu_id >= d->max_vcpus) )
+            break;
+
+        v = d->vcpu[c.desc.vcpu_id];
+
+        if ( flags & DOMAIN_SAVE_FLAG_IGNORE )
+        {
+            /* Sink the data */
+            rc = domain_load_entry(&c, c.desc.typecode, "IGNORED",
+                                   v, NULL, c.desc.length, true);

IOW the read handlers need to be able to deal with a NULL dst? Sounds a little fragile to me. Is there a reason domain_load_data() can't skip the call to read()?

Quoted 27 lines
xen/include/public/save.h new file +24 -0
@@ -0,0 +1,84 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER

xen/include/public/save.h line 34

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_SAVE_H__
+#define __XEN_PUBLIC_SAVE_H__
+
+#include "xen.h"
+
+/* Each entry is preceded by a descriptor */
+struct domain_save_descriptor {

Throughout this new public header, let's please play nice in name space terms: Prefix global scope items with xen_ / XEN_ respectively, and don't introduce violations of the standard's rules (e.g. _DOMAIN_SAVE_FLAG_IGNORE below). The latter point also goes for naming outside of the public header, of course.

xen/include/public/save.h line 37

Quoted 3 lines
xen/include/public/save.h new file +3 -0
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.

s/contain/apply/ or drop "to"?

xen/include/public/save.h line 38

Quoted 1 line
xen/include/public/save.h new file +1 -0
+     * Entries relating to global state should have zero in this field.

Is there a reason to specify zero?

xen/include/public/save.h line 40

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+     */
+    uint16_t vcpu_id;

Seeing (possibly) multi-instance records in HVM state (PIC, IO-APIC, HPET), wouldn't it be better to generalize this to ID, meaning "vCPU ID" just for per-vCPU state?

xen/include/public/save.h line 50

Quoted 10 lines
xen/include/public/save.h new file +10 -0
+    uint32_t flags;
+    /*
+     * When restoring state this flag can be set in a descriptor to cause
+     * its content to be ignored.
+     *
+     * NOTE: It is invalid to set this flag for HEADER or END records (see
+     *       below).
+     */
+#define _DOMAIN_SAVE_FLAG_IGNORE 0
+#define DOMAIN_SAVE_FLAG_IGNORE (1u << _DOMAIN_SAVE_FLAG_IGNORE)

As per your reply to Julien I think this wants further clarification on the intentions with this flag. I'm also uncertain it is a good idea to allow such partial restores - there may be dependencies between records, and hence things can easily go wrong in perhaps non-obvious ways.

xen/include/public/save.h line 53

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+    /* Entry length not including this descriptor */
+    uint64_t length;

Do you really envision descriptors with more than 4Gb of data? If so, for 32-bit purposes wouldn't this want to be uint64_aligned_t?

xen/include/public/save.h line 66

Quoted 13 lines
xen/include/public/save.h new file +13 -0
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
+ * binds these things together.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct __DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->t)

Just like the typeof() I dont think the sizeof() needs an outer pair of parentheses. I also don't see why 0 gets parenthesized.

xen/include/public/save.h line 70

Quoted 3 lines
xen/include/public/save.h new file +3 -0
+/* Terminating entry */
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);

If the header gets a __XEN__ || __XEN_TOOLS__ restriction, as suggested by Julien, using 0 here may be fine. If not, char[0] and typeof() aren't legal plain C and hence would need to be avoided.

Quoted 17 lines
xen/include/xen/save.h new file +14 -0
@@ -0,0 +1,152 @@
+/*
+ * save.h: support routines for save/restore
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *

xen/include/xen/save.h line 24

Quoted 10 lines
xen/include/xen/save.h new file +10 -0
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XEN_SAVE_H__
+#define __XEN_SAVE_H__
+
+#include <xen/sched.h>
+#include <xen/types.h>
+#include <xen/init.h>

Please sort these.

xen/include/xen/save.h line 27

Quoted 2 lines
xen/include/xen/save.h new file +2 -0
+#include <public/xen.h>
+#include <public/save.h>

The latter includes the former anyway - is the former necessary for some reason nevertheless?

xen/include/xen/save.h line 35

Quoted 7 lines
xen/include/xen/save.h new file +7 -0
+struct domain_context;
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len);
+
+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Quoted 59 lines
xen/include/xen/save.h new file +59 -0
+int domain_save_data(struct domain_context *c, const void *data, size_t len);
+int domain_save_end(struct domain_context *c);
+
+static inline int domain_save_entry(struct domain_context *c,
+                                    unsigned int tc, const char *name,
+                                    const struct vcpu *v, const void *src,
+                                    size_t len)
+{
+    int rc;
+
+    rc = domain_save_begin(c, tc, name, v, len);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, src, len);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+#define DOMAIN_SAVE_ENTRY(_x, _c, _v, _src, _len) \
+    domain_save_entry((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_src), (_len))
+
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact);
+
+#define DOMAIN_LOAD_BEGIN(_x, _c, _v, _len, _exact) \
+        domain_load_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len), \
+                          (_exact));
+
+int domain_load_data(struct domain_context *c, void *data, size_t len);
+int domain_load_end(struct domain_context *c);
+
+static inline int domain_load_entry(struct domain_context *c,
+                                    unsigned int tc, const char *name,
+                                    const struct vcpu *v, void *dst,
+                                    size_t len, bool exact)
+{
+    int rc;
+
+    rc = domain_load_begin(c, tc, name, v, len, exact);
+    if ( rc )
+        return rc;
+
+    rc = domain_load_data(c, dst, len);
+    if ( rc )
+        return rc;
+
+    return domain_load_end(c);
+}
+
+#define DOMAIN_LOAD_ENTRY(_x, _c, _v, _dst, _len, _exact) \
+    domain_load_entry((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_dst), (_len), \
+                          (_exact))
+
+/*
+ * The 'dry_run' flag indicates that the caller of domain_save() (see

xen/include/xen/save.h line 105

Quoted 10 lines
xen/include/xen/save.h new file +10 -0
+ * below) is not trying to actually acquire the data, only the size
+ * of the data. The save handler can therefore limit work to only that
+ * which is necessary to call DOMAIN_SAVE_BEGIN/ENTRY() with an accurate
+ * value for '_len'.
+ */
+typedef int (*domain_save_handler)(const struct vcpu *v,
+                                   struct domain_context *h,
+                                   bool dry_run);
+typedef int (*domain_load_handler)(struct vcpu *v,
+                                   struct domain_context *h);

Does a load handler have a need to modify struct domain_context?

Jan

#6 lore raw

RE: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 11 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02
To: Paul Durrant <[email protected]>
Cc: [email protected]; Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 9 lines
Quoted 8 lines · Paul Durrant
xen/common/save.c new file +6 -0
+void __init domain_register_save_type(unsigned int tc, const char *name,
+                                      bool per_vcpu,
+                                      domain_save_handler save,
+                                      domain_load_handler load)
+{
+    BUG_ON(tc > ARRAY_SIZE(handlers));

=

Yes.

Quoted 16 lines

xen/common/save.c line 58

Quoted 12 lines
xen/common/save.c new file +11 -0
+    ASSERT(!handlers[tc].save);
+    ASSERT(!handlers[tc].load);
+
+    handlers[tc].name = name;
+    handlers[tc].per_vcpu = per_vcpu;
+    handlers[tc].save = save;
+    handlers[tc].load = load;
+}
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len)

I find it quite odd for a function like this one to take a struct vcpu * rather than a struct domain *. See also below comment on the vcpu_id field in the public header.

I guess struct domain + vcpu_id can be used.

Quoted 35 lines
Quoted 24 lines
xen/common/save.c new file +23 -0
+{
+    int rc;
+
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv save: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    ASSERT(!c->data_len);
+    c->data_len = c->desc.length = len;
+
+    rc = c->copy.write(c->priv, &c->desc, sizeof(c->desc));
+    if ( rc )
+        return rc;
+
+    c->desc.length = 0;
+
+    return 0;
+}
+
+int domain_save_data(struct domain_context *c, const void *src, size_t len)

xen/common/save.c line 91

Quoted 10 lines
xen/common/save.c new file +10 -0
+{
+    if ( c->desc.length + len > c->data_len )
+        return -ENOSPC;
+
+    c->desc.length += len;
+
+    return c->copy.write(c->priv, src, len);
+}
+
+int domain_save_end(struct domain_context *c)
Quoted 7 lines

I'm sure there is a reason for the split into three load/save functions (begin/data/end), but I can't figure it and the description also doesn't explain it. They're all used together only afaics, in domain_{save,load}_entry(). Or wait, there are DOMAIN_{SAVE,LOAD}_BEGIN() macros apparently allowing separate use of ..._begin(), but then it's still not clear why ..._end() need to be separate from ..._data().

The split is to avoid the need to double-buffer in the save code, shared info being a case in point. If the entire save record needs to be written in one call then the shared info content would need to be copied into a newly allocated save record and then copied again into the aggregate context buffer.

Quoted 13 lines

xen/common/save.c line 113

Quoted 10 lines
xen/common/save.c new file +9 -0
+int domain_save(struct domain *d, domain_write_entry write, void *priv,
+                bool dry_run)
+{
+    struct domain_context c = {
+        .copy.write = write,
+        .priv = priv,
+        .log = !dry_run,
+    };
+    struct domain_save_header h = {

const? Perhaps even static?

Ok, static is a good idea.

Quoted 7 lines
Quoted 7 lines
xen/common/save.c new file +7 -0
+        .magic = DOMAIN_SAVE_MAGIC,
+        .version = DOMAIN_SAVE_VERSION,
+    };
+    struct domain_save_header e;
+    unsigned int i;
+    int rc;
+
Quoted 9 lines

xen/common/save.c line 124

Quoted 4 lines
xen/common/save.c new file +4 -0
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;

Could I talk you into using less generic an error code here, e.g. -ESRCH or -ENXIO? There look to be further uses of EINVAL that may want replacing.

I'm going to drop this check as it creates a problem for live update, where we actually do need to extract and restore state for dying domains.

Quoted 40 lines
Quoted 40 lines
xen/common/save.c new file +40 -0
+    domain_pause(d);
+
+    c.desc.typecode = DOMAIN_SAVE_CODE(HEADER);
+
+    rc = DOMAIN_SAVE_ENTRY(HEADER, &c, d->vcpu[0], &h, sizeof(h));
+    if ( rc )
+        goto out;
+
+    for ( i = 0; i < ARRAY_SIZE(handlers); i++ )
+    {
+        domain_save_handler save = handlers[i].save;
+
+        if ( !save )
+            continue;
+
+        memset(&c.desc, 0, sizeof(c.desc));
+        c.desc.typecode = i;
+
+        if ( handlers[i].per_vcpu )
+        {
+            struct vcpu *v;
+
+            for_each_vcpu ( d, v )
+            {
+                c.desc.vcpu_id = v->vcpu_id;
+
+                rc = save(v, &c, dry_run);
+                if ( rc )
+                    goto out;
+            }
+        }
+        else
+        {
+            rc = save(d->vcpu[0], &c, dry_run);
+            if ( rc )
+                goto out;
+        }
+    }
+
+    memset(&c.desc, 0, sizeof(c.desc));
Quoted 8 lines

xen/common/save.c line 168

Quoted 3 lines
xen/common/save.c new file +3 -0
+    c.desc.typecode = DOMAIN_SAVE_CODE(END);
+
+    rc = DOMAIN_SAVE_ENTRY(END, &c, d->vcpu[0], &e, 0);

By the looks of it you're passing uninitialized e here; it's just that the struct has no members. It would look less odd if you used NULL here. Otherwise please don't use literal 0, but sizeof() for the last parameter.

I'll init the 'e'.

Quoted 18 lines
Quoted 5 lines
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

Quoted 44 lines
Quoted 44 lines
xen/common/save.c new file +44 -0
+int domain_load(struct domain *d, domain_read_entry read, void *priv)
+{
+    struct domain_context c = {
+        .copy.read = read,
+        .priv = priv,
+        .log = true,
+    };
+    struct domain_save_header h;
+    int rc;
+
+    ASSERT(d != current->domain);
+
+    if ( d->is_dying )
+        return -EINVAL;
+
+    rc = c.copy.read(c.priv, &c.desc, sizeof(c.desc));
+    if ( rc )
+        return rc;
+
+    if ( c.desc.typecode != DOMAIN_SAVE_CODE(HEADER) || c.desc.vcpu_id ||
+         c.desc.flags )
+        return -EINVAL;
+
+    rc = DOMAIN_LOAD_ENTRY(HEADER, &c, d->vcpu[0], &h, sizeof(h), true);
+    if ( rc )
+        return rc;
+
+    if ( h.magic != DOMAIN_SAVE_MAGIC || h.version != DOMAIN_SAVE_VERSION )
+        return -EINVAL;
+
+    domain_pause(d);
+
+    for (;;)
+    {
+        unsigned int i;
+        unsigned int flags;
+        domain_load_handler load;
+        struct vcpu *v;
+
+        rc = c.copy.read(c.priv, &c.desc, sizeof(c.desc));
+        if ( rc )
+            break;
+
+        rc = -EINVAL;
Quoted 9 lines

xen/common/save.c line 281

Quoted 6 lines
xen/common/save.c new file +6 -0
+
+        flags = c.desc.flags;
+        if ( flags & ~DOMAIN_SAVE_FLAG_IGNORE )
+            break;
+
+        if ( c.desc.typecode == DOMAIN_SAVE_CODE(END) ) {

Brace placement.

Oops, yes.

Quoted 6 lines

xen/common/save.c line 283

Quoted 2 lines
xen/common/save.c new file +2 -0
+            if ( !(flags & DOMAIN_SAVE_FLAG_IGNORE) )
+                rc = DOMAIN_LOAD_ENTRY(END, &c, d->vcpu[0], NULL, 0, true);

The public header says DOMAIN_SAVE_FLAG_IGNORE is invalid for END.

Indeed, and it should be... don't know why I put that in there.

Quoted 14 lines
Quoted 14 lines
xen/common/save.c new file +14 -0
+            break;
+        }
+
+        i = c.desc.typecode;
+        if ( i >= ARRAY_SIZE(handlers) )
+            break;
+
+        if ( (!handlers[i].per_vcpu && c.desc.vcpu_id) ||
+             (c.desc.vcpu_id >= d->max_vcpus) )
+            break;
+
+        v = d->vcpu[c.desc.vcpu_id];
+
+        if ( flags & DOMAIN_SAVE_FLAG_IGNORE )
Quoted 8 lines

xen/common/save.c line 302

Quoted 4 lines
xen/common/save.c new file +4 -0
+        {
+            /* Sink the data */
+            rc = domain_load_entry(&c, c.desc.typecode, "IGNORED",
+                                   v, NULL, c.desc.length, true);

IOW the read handlers need to be able to deal with a NULL dst? Sounds a little fragile to me. Is there a reason domain_load_data() can't skip the call to read()?

Something has to move the cursor so sinking the data using a NULL dst seemed like the best way to avoid the need for a separate callback function.

Quoted 36 lines
Quoted 36 lines
xen/include/public/save.h new file +32 -0
@@ -0,0 +1,84 @@
+/*
+ * save.h
+ *
+ * Structure definitions for common PV/HVM domain state that is held by
+ * Xen and must be saved along with the domain's memory.
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_SAVE_H__
+#define __XEN_PUBLIC_SAVE_H__
+
+#include "xen.h"
+
Quoted 8 lines

xen/include/public/save.h line 34

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+/* Each entry is preceded by a descriptor */
+struct domain_save_descriptor {

Throughout this new public header, let's please play nice in name space terms: Prefix global scope items with xen_ / XEN_ respectively, and don't introduce violations of the standard's rules (e.g. _DOMAIN_SAVE_FLAG_IGNORE below). The latter point also goes for naming outside of the public header, of course.

Sorry, I just didn't pay enough attention to the cut'n'paste from the hvm save.h; I'll fix these up.

Quoted 6 lines

xen/include/public/save.h line 37

Quoted 4 lines
xen/include/public/save.h new file +3 -0
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.

s/contain/apply/ or drop "to"?

Yes, 'to' should be dropped.

Quoted 5 lines

xen/include/public/save.h line 38

Quoted 2 lines
xen/include/public/save.h new file +1 -0
+     * Entries relating to global state should have zero in this field.

Is there a reason to specify zero?

Not particularly but I thought it best to at least specify a value in all cases.

Quoted 7 lines

xen/include/public/save.h line 40

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+     */
+    uint16_t vcpu_id;

Seeing (possibly) multi-instance records in HVM state (PIC, IO-APIC, HPET), wouldn't it be better to generalize this to ID, meaning "vCPU ID" just for per-vCPU state?

True, a generic 'instance' would be needed for such records. I'll so as you suggest.

Quoted 7 lines
Quoted 7 lines
xen/include/public/save.h new file +7 -0
+    uint32_t flags;
+    /*
+     * When restoring state this flag can be set in a descriptor to cause
+     * its content to be ignored.
+     *
+     * NOTE: It is invalid to set this flag for HEADER or END records (see
+     *       below).
Quoted 9 lines

xen/include/public/save.h line 50

Quoted 3 lines
xen/include/public/save.h new file +3 -0
+     */
+#define _DOMAIN_SAVE_FLAG_IGNORE 0
+#define DOMAIN_SAVE_FLAG_IGNORE (1u << _DOMAIN_SAVE_FLAG_IGNORE)

As per your reply to Julien I think this wants further clarification on the intentions with this flag. I'm also uncertain it is a good idea to allow such partial restores - there may be dependencies between records, and hence things can easily go wrong in perhaps non-obvious ways.

OK, I'll drop this for now. It could be added later if it is needed.

Quoted 6 lines

xen/include/public/save.h line 53

Quoted 2 lines
xen/include/public/save.h new file +2 -0
+    /* Entry length not including this descriptor */
+    uint64_t length;

Do you really envision descriptors with more than 4Gb of data? If so, for 32-bit purposes wouldn't this want to be uint64_aligned_t?

I don't think we'll ever see a single record that big. I'll drop to 32 bits.

Quoted 8 lines
Quoted 8 lines
xen/include/public/save.h new file +8 -0
+};
+
+/*
+ * Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
+ * binds these things together.
+ */
+#define DECLARE_DOMAIN_SAVE_TYPE(_x, _code, _type) \
+    struct __DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; };
Quoted 9 lines

xen/include/public/save.h line 66

Quoted 5 lines
xen/include/public/save.h new file +5 -0
+
+#define DOMAIN_SAVE_CODE(_x) \
+    (sizeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->c))
+#define DOMAIN_SAVE_TYPE(_x) \
+    typeof(((struct __DOMAIN_SAVE_TYPE_##_x *)(0))->t)

Just like the typeof() I dont think the sizeof() needs an outer pair of parentheses. I also don't see why 0 gets parenthesized.

Cut'n'paste... I'll fix it.

Quoted 9 lines

xen/include/public/save.h line 70

Quoted 3 lines
xen/include/public/save.h new file +3 -0
+/* Terminating entry */
+struct domain_save_end {};
+DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end);

If the header gets a __XEN__ || __XEN_TOOLS__ restriction, as suggested by Julien, using 0 here may be fine. If not, char[0] and typeof() aren't legal plain C and hence would need to be avoided.

I'll restrict to tools.

Quoted 21 lines
Quoted 21 lines
xen/include/xen/save.h new file +18 -0
@@ -0,0 +1,152 @@
+/*
+ * save.h: support routines for save/restore
+ *
+ * Copyright Amazon.com Inc. or its affiliates.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
Quoted 9 lines

xen/include/xen/save.h line 24

Quoted 6 lines
xen/include/xen/save.h new file +6 -0
+#ifndef __XEN_SAVE_H__
+#define __XEN_SAVE_H__
+
+#include <xen/sched.h>
+#include <xen/types.h>
+#include <xen/init.h>

Please sort these.

Ok.

Quoted 6 lines

xen/include/xen/save.h line 27

Quoted 2 lines
xen/include/xen/save.h new file +2 -0
+#include <public/xen.h>
+#include <public/save.h>

The latter includes the former anyway - is the former necessary for some reason nevertheless?

No. I suspect it was at one point, but it can be dropped.

Quoted 4 lines
Quoted 4 lines
xen/include/xen/save.h new file +4 -0
+struct domain_context;
+
+int domain_save_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len);
Quoted 8 lines

xen/include/xen/save.h line 35

Quoted 3 lines
xen/include/xen/save.h new file +3 -0
+
+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Personally I think it is generally good practice to parenthesize but I can drop if you prefer.

Quoted 64 lines
Quoted 64 lines
xen/include/xen/save.h new file +63 -0
+int domain_save_data(struct domain_context *c, const void *data, size_t len);
+int domain_save_end(struct domain_context *c);
+
+static inline int domain_save_entry(struct domain_context *c,
+                                    unsigned int tc, const char *name,
+                                    const struct vcpu *v, const void *src,
+                                    size_t len)
+{
+    int rc;
+
+    rc = domain_save_begin(c, tc, name, v, len);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, src, len);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+#define DOMAIN_SAVE_ENTRY(_x, _c, _v, _src, _len) \
+    domain_save_entry((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_src), (_len))
+
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact);
+
+#define DOMAIN_LOAD_BEGIN(_x, _c, _v, _len, _exact) \
+        domain_load_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len), \
+                          (_exact));
+
+int domain_load_data(struct domain_context *c, void *data, size_t len);
+int domain_load_end(struct domain_context *c);
+
+static inline int domain_load_entry(struct domain_context *c,
+                                    unsigned int tc, const char *name,
+                                    const struct vcpu *v, void *dst,
+                                    size_t len, bool exact)
+{
+    int rc;
+
+    rc = domain_load_begin(c, tc, name, v, len, exact);
+    if ( rc )
+        return rc;
+
+    rc = domain_load_data(c, dst, len);
+    if ( rc )
+        return rc;
+
+    return domain_load_end(c);
+}
+
+#define DOMAIN_LOAD_ENTRY(_x, _c, _v, _dst, _len, _exact) \
+    domain_load_entry((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_dst), (_len), \
+                          (_exact))
+
+/*
+ * The 'dry_run' flag indicates that the caller of domain_save() (see
+ * below) is not trying to actually acquire the data, only the size
+ * of the data. The save handler can therefore limit work to only that
+ * which is necessary to call DOMAIN_SAVE_BEGIN/ENTRY() with an accurate
+ * value for '_len'.
Quoted 9 lines

xen/include/xen/save.h line 105

Quoted 6 lines
xen/include/xen/save.h new file +6 -0
+ */
+typedef int (*domain_save_handler)(const struct vcpu *v,
+                                   struct domain_context *h,
+                                   bool dry_run);
+typedef int (*domain_load_handler)(struct vcpu *v,
+                                   struct domain_context *h);

Does a load handler have a need to modify struct domain_context?

Not sure. I'll try const-ing.

Paul

Quoted 1 line

Jan

#7 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 27 lines · Paul Durrant
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02

Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 7 lines
Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

But this is load, not save - the caller supplies the data. If there's less data than can be fit, it'll be zero-extended. If there's too much data, the excess you don't know what to do with (it might be okay to tolerate it being all zero).

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
xen/common/save.c new file +7 -0
+        if ( (!handlers[i].per_vcpu && c.desc.vcpu_id) ||
+             (c.desc.vcpu_id >= d->max_vcpus) )
+            break;
+
+        v = d->vcpu[c.desc.vcpu_id];
+
+        if ( flags & DOMAIN_SAVE_FLAG_IGNORE )
Quoted 8 lines

xen/common/save.c line 302

Quoted 4 lines
xen/common/save.c new file +4 -0
+        {
+            /* Sink the data */
+            rc = domain_load_entry(&c, c.desc.typecode, "IGNORED",
+                                   v, NULL, c.desc.length, true);

IOW the read handlers need to be able to deal with a NULL dst? Sounds a little fragile to me. Is there a reason domain_load_data() can't skip the call to read()?

Quoted 3 lines

Something has to move the cursor so sinking the data using a NULL dst seemed like the best way to avoid the need for a separate callback function.

And domain_load_data() can't itself advance the cursor in such a case, with no callback involved at all?

Quoted 9 lines
Quoted 7 lines

xen/include/public/save.h line 38

Quoted 4 lines
xen/include/public/save.h new file +4 -0
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.
+     * Entries relating to global state should have zero in this field.

Is there a reason to specify zero?

Not particularly but I thought it best to at least specify a value in all cases.

A specific value is certainly a good idea; I wonder though whether an obviously invalid one (like ~0) wouldn't be better then, allowing this ID to later be assigned meaning in such cases if need be.

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+     */
+    uint16_t vcpu_id;

Seeing (possibly) multi-instance records in HVM state (PIC, IO-APIC, HPET), wouldn't it be better to generalize this to ID, meaning "vCPU ID" just for per-vCPU state?

True, a generic 'instance' would be needed for such records. I'll so as you suggest.

Which, along with my comment on domain_save_begin() taking a struct vcpu * right now, will then indeed require changing to a (struct domain *, unsigned int instance) tuple, I guess.

Quoted 10 lines
Quoted 7 lines

xen/include/xen/save.h line 35

Quoted 2 lines
xen/include/xen/save.h new file +2 -0
+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Personally I think it is generally good practice to parenthesize but I can drop if you prefer.

To be honest - it's more than just "prefer": Excess parentheses hamper readability. There shouldn't be too few, and since macros already require quite a lot of them, imo we should strive to have exactly as many as are needed.

Jan

#8 lore raw

RE: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 44 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:22
To: [email protected]
Cc: [email protected]; 'Paul Durrant' <[email protected]>; 'Andrew Cooper' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Julien Grall' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Wei Liu' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Roger Pau Monné' <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 32 lines · Paul Durrant
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02

Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

Quoted 8 lines
Quoted 2 lines

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

But this is load, not save - the caller supplies the data. If there's less data than can be fit, it'll be zero-extended. If there's too much data, the excess you don't know what to do with (it might be okay to tolerate it being all zero).

But this is a callback. The outer load function iterates over the records calling the appropriate hander for each one. Those handlers then call this function saying how much data they expect and whether they want exactly that amount, or whether they can tolerate less (i.e. zero-extend). Hence len < c->desc.length is an error, because it means the descriptor contains more data than the hander knows how to handle.

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

xen/common/save.c line 302

Quoted 11 lines
xen/common/save.c new file +11 -0
+        if ( (!handlers[i].per_vcpu && c.desc.vcpu_id) ||
+             (c.desc.vcpu_id >= d->max_vcpus) )
+            break;
+
+        v = d->vcpu[c.desc.vcpu_id];
+
+        if ( flags & DOMAIN_SAVE_FLAG_IGNORE )
+        {
+            /* Sink the data */
+            rc = domain_load_entry(&c, c.desc.typecode, "IGNORED",
+                                   v, NULL, c.desc.length, true);

IOW the read handlers need to be able to deal with a NULL dst? Sounds a little fragile to me. Is there a reason domain_load_data() can't skip the call to read()?

Quoted 7 lines
Quoted 3 lines

Something has to move the cursor so sinking the data using a NULL dst seemed like the best way to avoid the need for a separate callback function.

And domain_load_data() can't itself advance the cursor in such a case, with no callback involved at all?

How could it do that without a callback? Doing such a thing negates the utility of the ignore flag anyway since it implies the caller is going to edit the load stream in advance. Since I'm going to drop the ignore flag in v3 anyway, this is all a bit academic.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

xen/include/public/save.h line 38

Quoted 4 lines
xen/include/public/save.h new file +4 -0
+    uint16_t typecode;
+    /*
+     * Each entry will contain either to global or per-vcpu domain state.
+     * Entries relating to global state should have zero in this field.

Is there a reason to specify zero?

Quoted 7 lines
Quoted 1 line

Not particularly but I thought it best to at least specify a value in all cases.

A specific value is certainly a good idea; I wonder though whether an obviously invalid one (like ~0) wouldn't be better then, allowing this ID to later be assigned meaning in such cases if need be.

Ok, that sounds reasonable. I'll #define something for convenience.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+     */
+    uint16_t vcpu_id;

Seeing (possibly) multi-instance records in HVM state (PIC, IO-APIC, HPET), wouldn't it be better to generalize this to ID, meaning "vCPU ID" just for per-vCPU state?

Quoted 6 lines
Quoted 1 line

True, a generic 'instance' would be needed for such records. I'll so as you suggest.

Which, along with my comment on domain_save_begin() taking a struct vcpu * right now, will then indeed require changing to a (struct domain *, unsigned int instance) tuple, I guess.

Yes.

Paul

#9 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 45 lines · Paul Durrant
Quoted 36 lines

From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:22

Quoted 32 lines · Paul Durrant
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02

Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

Quoted 8 lines
Quoted 2 lines

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

But this is load, not save - the caller supplies the data. If there's less data than can be fit, it'll be zero-extended. If there's too much data, the excess you don't know what to do with (it might be okay to tolerate it being all zero).

Quoted 8 lines

But this is a callback. The outer load function iterates over the records calling the appropriate hander for each one. Those handlers then call this function saying how much data they expect and whether they want exactly that amount, or whether they can tolerate less (i.e. zero-extend). Hence
len < c->desc.length is an error, because it means the descriptor contains more data than the hander knows how to handle.

Oh, I see - "But maybe I'm getting the sense of len wrong ..." then indeed applies.

Any thoughts on tolerating the excess data being zero?

Jan

#10 lore raw

RE: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 66 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:40
To: [email protected]
Cc: [email protected]; 'Paul Durrant' <[email protected]>; 'Andrew Cooper' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Julien Grall' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Wei Liu' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Roger Pau Monné' <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 45 lines · Paul Durrant
Quoted 36 lines

From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:22

Quoted 32 lines · Paul Durrant
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02

Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

xen/common/save.c line 189

Quoted 10 lines
xen/common/save.c new file +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

Quoted 8 lines
Quoted 2 lines

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

But this is load, not save - the caller supplies the data. If there's less data than can be fit, it'll be zero-extended. If there's too much data, the excess you don't know what to do with (it might be okay to tolerate it being all zero).

Quoted 8 lines

But this is a callback. The outer load function iterates over the records calling the appropriate hander for each one. Those handlers then call this function saying how much data they expect and whether they want exactly that amount, or whether they can tolerate less (i.e. zero-extend). Hence
len < c->desc.length is an error, because it means the descriptor contains more data than the hander knows how to handle.

Quoted 5 lines

Oh, I see - "But maybe I'm getting the sense of len wrong ..." then indeed applies.

Any thoughts on tolerating the excess data being zero?

Well the point of the check here is to not tolerate excess data... Are you suggesting that it might be a reasonable idea? If so, then yes, insisting it is all zero would be an alternative.

Paul

#11 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 66 lines · Paul Durrant
Quoted 66 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:40
To: [email protected]
Cc: [email protected]; 'Paul Durrant' <[email protected]>; 'Andrew Cooper' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Julien Grall' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Wei Liu' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Roger Pau Monné' <[email protected]>
Subject: Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 53 lines · Paul Durrant
Quoted 36 lines

From: Jan Beulich <[email protected]>
Sent: 07 May 2020 08:22

Quoted 32 lines · Paul Durrant
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 29 April 2020 12:02

Quoted 4 lines · Paul Durrant
(quoted diff) +4 -0
+int domain_load_begin(struct domain_context *c, unsigned int tc,
+                      const char *name, const struct vcpu *v, size_t len,
+                      bool exact)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+    if ( c->log )
+        gdprintk(XENLOG_INFO, "%pv load: %s (%lu)\n", v, name,
+                 (unsigned long)len);
+
+    BUG_ON(tc != c->desc.typecode);
+    BUG_ON(v->vcpu_id != c->desc.vcpu_id);
+
+    if ( (exact && (len != c->desc.length)) ||
+         (len < c->desc.length) )
+        return -EINVAL;

How about

Quoted 3 lines
    if ( exact ? len != c->desc.length
               : len < c->desc.length )

Yes, that doesn't look too bad.

Quoted 4 lines

? I'm also unsure about the < - don't you mean > instead? Too little data would be compensated by zero padding, but too much data can't be dealt with. But maybe I'm getting the sense of len wrong ...

Quoted 8 lines
Quoted 2 lines

I think the < is correct. The caller needs to have at least enough space to accommodate the context record.

But this is load, not save - the caller supplies the data. If there's less data than can be fit, it'll be zero-extended. If there's too much data, the excess you don't know what to do with (it might be okay to tolerate it being all zero).

But this is a callback. The outer load function iterates over the records calling the appropriate hander for each one. Those handlers then call this function saying how much data they expect and whether they want exactly that amount, or whether they can tolerate less (i.e. zero-extend). Hence
len < c->desc.length is an error, because it means the descriptor contains more data than the hander knows how to handle.

Quoted 8 lines
Quoted 5 lines

Oh, I see - "But maybe I'm getting the sense of len wrong ..." then indeed applies.

Any thoughts on tolerating the excess data being zero?

Well the point of the check here is to not tolerate excess data... Are you suggesting that it might be a reasonable idea?

Well - it looks to be the obvious counterpart to zero-extending. I'm not going to assert though that I've thought through all possible consequences...

Jan

#12 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

In reply to Jan Beulich
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
Quoted 7 lines

xen/include/xen/save.h line 35

Quoted 2 lines
xen/include/xen/save.h new file +2 -0
+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Quoted 7 lines
Quoted 2 lines

Personally I think it is generally good practice to parenthesize but I can drop if you prefer.

To be honest - it's more than just "prefer": Excess parentheses hamper readability. There shouldn't be too few, and since macros already require quite a lot of them, imo we should strive to have exactly as many as are needed.

While I understand that too many parentheses may make the code worse, in the case of the macros, adding them for each argument is a good practice. This pretty simple to follow and avoid the mistake to forget to protect an argument correctly.

So I would let the contributor decides whether he wants to protect all the macro arguments or just as a need basis.

Cheers,

Signature · 1 line
Julien Grall
#13 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Quoted 20 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
Quoted 7 lines
Quoted 2 lines

+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Quoted 7 lines
Quoted 2 lines

Personally I think it is generally good practice to parenthesize but I can drop if you prefer.

To be honest - it's more than just "prefer": Excess parentheses hamper readability. There shouldn't be too few, and since macros already require quite a lot of them, imo we should strive to have exactly as many as are needed.

Quoted 7 lines

While I understand that too many parentheses may make the code worse, in the case of the macros, adding them for each argument is a good practice. This pretty simple to follow and avoid the mistake to forget to protect an argument correctly.

So I would let the contributor decides whether he wants to protect all the macro arguments or just as a need basis.

This is a possible alternative position to take, accepting the worse readability. But then this would please need applying in full (as far as it's possible - operands to # or ## of course can't be parenthesized):

#define DOMAIN_SAVE_BEGIN(x, c, v, len) \
        domain_save_begin((c), DOMAIN_SAVE_CODE((x)), #x, (v), (len))

which might look a little odd even to you?

As to readability - I'm sure you realize that from time to time one needs to look at preprocessor output, where parentheses used like this plus parentheses then also applied inside the invoked macro add to one another. All in all I don't really buy the "avoid the mistake to forget to protect an argument correctly" argument to outweigh the downsides. We're doing this work not on an occasional basis, but as a job. There should be a minimum level of care everyone applies.

Jan

#14 lore raw

Re: [PATCH v2 1/5] xen/common: introduce a new framework for save/restore of 'domain' context

Hi,

Quoted 34 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
Quoted 7 lines
Quoted 2 lines

+#define DOMAIN_SAVE_BEGIN(_x, _c, _v, _len) \
+        domain_save_begin((_c), DOMAIN_SAVE_CODE(_x), #_x, (_v), (_len))

In new code I'd like to ask for no leading underscores on macro parameters as well as no unnecessary parenthesization of macro parameters (e.g. when they simply get passed on to another macro or function without being part of a larger expression).

Quoted 7 lines
Quoted 2 lines

Personally I think it is generally good practice to parenthesize but I can drop if you prefer.

To be honest - it's more than just "prefer": Excess parentheses hamper readability. There shouldn't be too few, and since macros already require quite a lot of them, imo we should strive to have exactly as many as are needed.

Quoted 7 lines

While I understand that too many parentheses may make the code worse, in the case of the macros, adding them for each argument is a good practice. This pretty simple to follow and avoid the mistake to forget to protect an argument correctly.

So I would let the contributor decides whether he wants to protect all the macro arguments or just as a need basis.

This is a possible alternative position to take, accepting the worse readability. But then this would please need applying in full (as far as it's possible - operands to # or ## of course can't be parenthesized):

Quoted 4 lines
#define DOMAIN_SAVE_BEGIN(x, c, v, len) \
         domain_save_begin((c), DOMAIN_SAVE_CODE((x)), #x, (v), (len))

which might look a little odd even to you?

One pair of parenthesis looks wasteful but it doesn't bother that much.

Quoted 9 lines

As to readability - I'm sure you realize that from time to time one needs to look at preprocessor output, where parentheses used like this plus parentheses then also applied inside the invoked macro add to one another. All in all I don't really buy the "avoid the mistake to forget to protect an argument correctly" argument to outweigh the downsides. We're doing this work not on an occasional basis, but as a job. There should be a minimum level of care everyone applies.

I am sure you heard about "human error" in the past. Even if you do something all day along, you will make a mistake one day. By requesting a contributor to limit the number of parenthesis, then you increase the chance he/she will screw up one day.

You might think reviewers will catch such error, but XSA-316 proved that it is possible to miss it. I was the author of the patch and you were one the reviewer. As you can see even an experienced contributor and reviewer can make mistake... we are all human after all ;).

I don't ask to be over-proctective, but we should not lay trap to other contributors for them to fall into accidentally. In this case, Paul thinks the parentheses will help him. So I would not impose him to remove them and possibly make a mistake.

Cheers,

Signature · 1 line
Julien Grall
#15 lore raw

PATCH v2 2/5 xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

In reply to Paul Durrant

These domctls provide a mechanism to get and set domain context from the toolstack.

---
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v2:
- drop mask parameter
- const-ify some more buffers

---
 tools/flask/policy/modules/xen.if   |   4 +-
 tools/libxc/include/xenctrl.h       |   5 ++
 tools/libxc/xc_domain.c             |  54 +++++++++++++
 xen/common/domctl.c                 | 117 ++++++++++++++++++++++++++++
 xen/include/public/domctl.h         |  44 ++++++++++-
 xen/xsm/flask/hooks.c               |   6 ++
 xen/xsm/flask/policy/access_vectors |   4 +
 7 files changed, 231 insertions(+), 3 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
index 8eb2293a52..2bc9db4f64 100644
@@ -53,7 +53,7 @@ define(`create_domain_common', `
 	allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim
 			set_vnumainfo get_vnumainfo cacheflush
 			psr_cmt_op psr_alloc soft_reset
-			resource_map get_cpu_policy };
+			resource_map get_cpu_policy setcontext };
 	allow $1 $2:security check_context;
 	allow $1 $2:shadow enable;
 	allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', `
 	allow $1 $2:hvm { gethvmc getparam };
 	allow $1 $2:mmu { stat pageinfo map_read };
 	allow $1 $2:domain { getaddrsize getvcpucontext pause destroy };
-	allow $1 $2:domain2 gettsc;
+	allow $1 $2:domain2 { gettsc getcontext };
 	allow $1 $2:shadow { enable disable logdirty };
 ')
 
tools/libxc/include/xenctrl.h +5 -0
index 58fa931de1..06ca8e9a74 100644
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,
                              uint8_t *hvm_ctxt,
                              uint32_t size);
 
+int xc_domain_getcontext(xc_interface *xch, uint32_t domid,
+                         void *ctxt_buf, size_t *size);
+int xc_domain_setcontext(xc_interface *xch, uint32_t domid,
+                         const void *ctxt_buf, size_t size);
+
 /**
  * This function will return guest IO ABI protocol
  *
tools/libxc/xc_domain.c +54 -0

60 lines not on this page · raw message

xen/common/domctl.c +117 -0

135 lines not on this page · raw message

xen/include/public/domctl.h +43 -1

68 lines not on this page · raw message

xen/xsm/flask/hooks.c +6 -0
index 8af8602b46..d94d0fc125 100644
@@ -744,6 +744,12 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_get_cpu_policy:
         return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GET_CPU_POLICY);
 
+    case XEN_DOMCTL_setdomaincontext:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SETCONTEXT);
+
+    case XEN_DOMCTL_getdomaincontext:
+        return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__GETCONTEXT);
+
     default:
         return avc_unknown_permission("domctl", cmd);
     }
xen/xsm/flask/policy/access_vectors +4 -0
index c055c14c26..fccfb9de82 100644
@@ -245,6 +245,10 @@ class domain2
     resource_map
 # XEN_DOMCTL_get_cpu_policy
     get_cpu_policy
+# XEN_DOMCTL_setdomaincontext
+    setcontext
+# XEN_DOMCTL_getdomaincontext
+    getcontext
 }
 
 # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
Reviewed-by on v2 2/5
#16 lore raw

Re: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Hi Paul,

Quoted 14 lines · Paul Durrant
xen/include/public/domctl.h +1 -1
index 1ad34c35eb..8ab39acf0c 100644
@@ -38,7 +38,7 @@
  #include "hvm/save.h"
  #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000013
 
  /*
   * NB. xen_domctl.domain is an IN/OUT parameter for this operation.

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1133

Quoted 5 lines
xen/include/public/domctl.h +2 -0
@@ -1129,6 +1129,44 @@ struct xen_domctl_vuart_op {
                                   */
  };
 
+/*
+ * Get/Set domain PV context. The same struct xen_domctl_domaincontext

I think you want to update the comments to match the split.

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1135

Quoted 2 lines
xen/include/public/domctl.h +2 -0
+ * is used for both commands but with slightly different field semantics
+ * as follows:

Cheers,

Signature · 1 line
Julien Grall
#17 lore raw

RE: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Quoted 30 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 20 April 2020 18:26
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Daniel De Graaf <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Hi Paul,

Quoted 17 lines · Paul Durrant
xen/include/public/domctl.h +1 -1
index 1ad34c35eb..8ab39acf0c 100644
@@ -38,7 +38,7 @@
  #include "hvm/save.h"
  #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000013
 
  /*
   * NB. xen_domctl.domain is an IN/OUT parameter for this operation.
@@ -1129,6 +1129,44 @@ struct xen_domctl_vuart_op {
                                   */
  };
 
Quoted 4 lines

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1133

Quoted 2 lines
xen/include/public/domctl.h +2 -0
+/*
+ * Get/Set domain PV context. The same struct xen_domctl_domaincontext

I think you want to update the comments to match the split.

Oh yes.

Quoted 6 lines

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1135

Quoted 3 lines
xen/include/public/domctl.h +2 -0
+ * is used for both commands but with slightly different field semantics
+ * as follows:

Thanks,

Paul

Quoted 6 lines

Cheers,

--
Julien Grall

#18 lore raw

Re: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

In reply to Paul Durrant
Quoted 30 lines · Paul Durrant
xen/common/domctl.c +26 -0
@@ -358,6 +359,113 @@ static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo,
     return ERR_PTR(ret);
 }
 
+struct domctl_context
+{
+    void *buffer;
+    size_t len;
+    size_t cur;
+};
+
+static int accumulate_size(void *priv, const void *data, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    if ( c->len + len < c->len )
+        return -EOVERFLOW;
+
+    c->len += len;
+
+    return 0;
+}
+
+static int save_data(void *priv, const void *data, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    if ( c->len - c->cur < len )
+        return -ENOSPC;
+

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 397

Quoted 10 lines
xen/common/domctl.c +10 -0
+    memcpy(c->buffer + c->cur, data, len);
+    c->cur += len;
+
+    return 0;
+}
+
+static int getdomaincontext(struct domain *d,
+                            struct xen_domctl_getdomaincontext *gdc)
+{
+    struct domctl_context c = { };

Please can you use ZERO_BLOCK_PTR or some such for the buffer field, such that errnoeous use of the field would not end up as a (PV-controllable) NULL deref. (Yes, it's a domctl, but still.) This being common code you also want to get things right for Arm, irrespective of whether the code will be dead there for now.

Quoted 11 lines
xen/common/domctl.c +11 -0
+    int rc;
+
+    if ( d == current->domain )
+        return -EPERM;
+
+    if ( guest_handle_is_null(gdc->buffer) ) /* query for buffer size */
+    {
+        if ( gdc->size )
+            return -EINVAL;
+
+        /* dry run to acquire buffer size */

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 418

Quoted 10 lines
xen/common/domctl.c +10 -0
+        rc = domain_save(d, accumulate_size, &c, true);
+        if ( rc )
+            return rc;
+
+        gdc->size = c.len;
+        return 0;
+    }
+
+    c.len = gdc->size;
+    c.buffer = xmalloc_bytes(c.len);

What sizes are we looking at here? It may be better to use vmalloc() right from the start. If not, I'd like to advocate for using xmalloc_array() instead of xmalloc_bytes() - see the almost-XSA commit cf38b4926e2b.

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 425

Quoted 7 lines
xen/common/domctl.c +7 -0
+    if ( !c.buffer )
+        return -ENOMEM;
+
+    rc = domain_save(d, save_data, &c, false);
+
+    gdc->size = c.cur;
+    if ( !rc && copy_to_guest(gdc->buffer, c.buffer, gdc->size) )

As to my remark in patch 1 on the size field, applying to this size field too - copy_to_user{,_hvm}() don't support a 64-bit value (on y86 at least).

xen/include/public/domctl.h line 41

Quoted 8 lines
xen/include/public/domctl.h +1 -1
@@ -38,7 +38,7 @@
 #include "hvm/save.h"
 #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000013

I don't see you making any change making the interface backwards incompatible, hence no need for the bump.

Quoted 10 lines
xen/include/public/domctl.h +6 -0
@@ -1129,6 +1129,44 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * Get/Set domain PV context. The same struct xen_domctl_domaincontext
+ * is used for both commands but with slightly different field semantics
+ * as follows:
+ *
+ * XEN_DOMCTL_getdomaincontext

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1147

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should
+ *                be allocated.
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.

This leaves open whether the size also gets updated in this latter case.

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150

Quoted 3 lines
xen/include/public/domctl.h +3 -0
+ */
+struct xen_domctl_getdomaincontext {
+    uint64_t size;

If this is to remain 64-bits (with too large values suitably taken care of for all cases - see above), uint64_aligned_t please for consistency, if nothing else.

Jan

#19 lore raw

RE: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Quoted 50 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 29 April 2020 15:51
To: Paul Durrant <[email protected]>
Cc: [email protected]; Paul Durrant <[email protected]>; Daniel De Graaf <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v2 2/5] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext

Quoted 39 lines · Paul Durrant
xen/common/domctl.c +35 -0
@@ -358,6 +359,113 @@ static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo,
     return ERR_PTR(ret);
 }
 
+struct domctl_context
+{
+    void *buffer;
+    size_t len;
+    size_t cur;
+};
+
+static int accumulate_size(void *priv, const void *data, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    if ( c->len + len < c->len )
+        return -EOVERFLOW;
+
+    c->len += len;
+
+    return 0;
+}
+
+static int save_data(void *priv, const void *data, size_t len)
+{
+    struct domctl_context *c = priv;
+
+    if ( c->len - c->cur < len )
+        return -ENOSPC;
+
+    memcpy(c->buffer + c->cur, data, len);
+    c->cur += len;
+
+    return 0;
+}
+
+static int getdomaincontext(struct domain *d,
+                            struct xen_domctl_getdomaincontext *gdc)
+{
Quoted 9 lines

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 397

Quoted 1 line
xen/common/domctl.c +1 -0
+    struct domctl_context c = { };

Please can you use ZERO_BLOCK_PTR or some such for the buffer field, such that errnoeous use of the field would not end up as a (PV-controllable) NULL deref. (Yes, it's a domctl, but still.) This being common code you also want to get things right for Arm, irrespective of whether the code will be dead there for now.

Ok.

Quoted 16 lines
Quoted 16 lines
xen/common/domctl.c +16 -0
+    int rc;
+
+    if ( d == current->domain )
+        return -EPERM;
+
+    if ( guest_handle_is_null(gdc->buffer) ) /* query for buffer size */
+    {
+        if ( gdc->size )
+            return -EINVAL;
+
+        /* dry run to acquire buffer size */
+        rc = domain_save(d, accumulate_size, &c, true);
+        if ( rc )
+            return rc;
+
+        gdc->size = c.len;
Quoted 8 lines

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 418

Quoted 5 lines
xen/common/domctl.c +5 -0
+        return 0;
+    }
+
+    c.len = gdc->size;
+    c.buffer = xmalloc_bytes(c.len);

What sizes are we looking at here? It may be better to use vmalloc() right from the start.

Could be quite big so that seems reasonable.

Quoted 4 lines

If not, I'd like to advocate for using xmalloc_array() instead of xmalloc_bytes() - see the almost-XSA commit cf38b4926e2b.

Quoted 11 lines

xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 425

Quoted 7 lines
xen/common/domctl.c +7 -0
+    if ( !c.buffer )
+        return -ENOMEM;
+
+    rc = domain_save(d, save_data, &c, false);
+
+    gdc->size = c.cur;
+    if ( !rc && copy_to_guest(gdc->buffer, c.buffer, gdc->size) )

As to my remark in patch 1 on the size field, applying to this size field too - copy_to_user{,_hvm}() don't support a 64-bit value (on y86 at least).

Ok.

Quoted 7 lines
Quoted 7 lines
xen/include/public/domctl.h +0 -0
@@ -38,7 +38,7 @@
 #include "hvm/save.h"
 #include "memory.h"
 
Quoted 6 lines

xen/include/public/domctl.h line 41

Quoted 2 lines
xen/include/public/domctl.h +1 -1
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000013

I don't see you making any change making the interface backwards incompatible, hence no need for the bump.

Ok.

Quoted 15 lines
Quoted 15 lines
xen/include/public/domctl.h +11 -0
@@ -1129,6 +1129,44 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/*
+ * Get/Set domain PV context. The same struct xen_domctl_domaincontext
+ * is used for both commands but with slightly different field semantics
+ * as follows:
+ *
+ * XEN_DOMCTL_getdomaincontext
+ * ---------------------------
+ *
+ * buffer (IN):   The buffer into which the context data should be
+ *                copied, or NULL to query the buffer size that should
+ *                be allocated.
Quoted 8 lines

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1147

Quoted 5 lines
xen/include/public/domctl.h +5 -0
+ * size (IN/OUT): If 'buffer' is NULL then the value passed in must be
+ *                zero, and the value passed out will be the size of the
+ *                buffer to allocate.
+ *                If 'buffer' is non-NULL then the value passed in must
+ *                be the size of the buffer into which data may be copied.

This leaves open whether the size also gets updated in this latter case.

I'll clarify.

Quoted 8 lines

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150

Quoted 4 lines
xen/include/public/domctl.h +3 -0
+ */
+struct xen_domctl_getdomaincontext {
+    uint64_t size;

If this is to remain 64-bits (with too large values suitably taken care of for all cases - see above), uint64_aligned_t please for consistency, if nothing else.

I've changed to uint32_t.

Paul

Quoted 2 lines

Jan

#20 lore raw

PATCH v2 3/5 tools/misc: add xen-domctx to present domain context

In reply to Paul Durrant

This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Change name from 'xen-ctx' to 'xen-domctx'

---
 .gitignore              |   1 +
 tools/misc/Makefile     |   4 ++
 tools/misc/xen-domctx.c | 145 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
index 4ca679ddbc..744c0529bb 100644
@@ -208,6 +208,7 @@ tools/misc/xen_cpuperf
 tools/misc/xen-cpuid
 tools/misc/xen-detect
 tools/misc/xen-diag
+tools/misc/xen-domctx
 tools/misc/xen-tmem-list-parse
 tools/misc/xen-livepatch
 tools/misc/xenperf
tools/misc/Makefile +4 -0
index 63947bfadc..ef25524354 100644
@@ -30,6 +30,7 @@ INSTALL_SBIN                   += xenpm
 INSTALL_SBIN                   += xenwatchdogd
 INSTALL_SBIN                   += xen-livepatch
 INSTALL_SBIN                   += xen-diag
+INSTALL_SBIN                   += xen-domctx
 INSTALL_SBIN += $(INSTALL_SBIN-y)
 
 # Everything to be installed in a private bin/
@@ -108,6 +109,9 @@ xen-livepatch: xen-livepatch.o
 xen-diag: xen-diag.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-domctx: xen-domctx.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 xen-lowmemd: xen-lowmemd.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
 
tools/misc/xen-domctx.c new file +145 -0

145 lines not on this page · raw message

Signature · 1 line
2.20.1
#21 lore raw

Re: [PATCH v2 3/5] tools/misc: add xen-domctx to present domain context

tools/misc/xen-domctx.c line 80

Quoted 12 lines · Paul Durrant
tools/misc/xen-domctx.c new file +12 -0
+int main(int argc, char **argv)
+{
+    uint32_t domid;
+    unsigned int entry;
+    xc_interface *xch;
+    int rc;
+
+    if ( argc != 2 || !argv[1] || (rc = atoi(argv[1])) < 0 )
+    {
+        fprintf(stderr, "usage: %s <domid>\n", argv[0]);
+        exit(1);
+    }

Perhaps also allow dumping just a single (vCPU or other) ID?

Jan

#22 lore raw

RE: [PATCH v2 3/5] tools/misc: add xen-domctx to present domain context

Quoted 9 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 29 April 2020 16:04
To: Paul Durrant <[email protected]>
Cc: [email protected]; Paul Durrant <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v2 3/5] tools/misc: add xen-domctx to present domain context

On 07.04.2020 19:38, Paul Durrant wrote:

Quoted 15 lines

tools/misc/xen-domctx.c line 80

Quoted 12 lines
tools/misc/xen-domctx.c new file +12 -0
+int main(int argc, char **argv)
+{
+    uint32_t domid;
+    unsigned int entry;
+    xc_interface *xch;
+    int rc;
+
+    if ( argc != 2 || !argv[1] || (rc = atoi(argv[1])) < 0 )
+    {
+        fprintf(stderr, "usage: %s <domid>\n", argv[0]);
+        exit(1);
+    }

Perhaps also allow dumping just a single (vCPU or other) ID?

Yes, I can add optional type and instance arguments.

Paul

Quoted 1 line

Jan

#23 lore raw

PATCH v2 4/5 common/domain: add a domain context record for shared_info...

In reply to Paul Durrant

... and update xen-domctx to dump some information describing the record.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
 tools/misc/xen-domctx.c   | 11 ++++++
 xen/common/domain.c       | 81 +++++++++++++++++++++++++++++++++++++++
 xen/include/public/save.h | 10 ++++-
 3 files changed, 101 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +11 -0
index d663522a8b..a8d3922321 100644
@@ -59,6 +59,16 @@ static void dump_header(struct domain_save_descriptor *desc)
     off += desc->length;
 }
 
+static void dump_shared_info(struct domain_save_descriptor *desc)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) s;
+    READ(s);
+    printf("    SHARED_INFO: field_width %u buffer size: %lu\n",
+           s.field_width, desc->length - sizeof(s));
+
+    off += desc->length;
+}
+
 static void dump_end(struct domain_save_descriptor *desc)
 {
     DOMAIN_SAVE_TYPE(END) e;
@@ -125,6 +135,7 @@ int main(int argc, char **argv)
         switch (desc.typecode)
         {
         case DOMAIN_SAVE_CODE(HEADER): dump_header(&desc); break;
+        case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(&desc); break;
         case DOMAIN_SAVE_CODE(END): dump_end(&desc); return 0;
         default:
             printf("Unknown type %u: skipping\n", desc.typecode);
xen/common/domain.c +81 -0

93 lines not on this page · raw message

xen/include/public/save.h +9 -1
index 7e5f8752bd..ed994a8765 100644
@@ -79,6 +79,14 @@ struct domain_save_header {
 };
 DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint8_t field_width;
+    uint8_t pad[7];
+    uint8_t buffer[]; /* Implementation specific size */
+};
+
+DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context);
+
+#define DOMAIN_SAVE_CODE_MAX 2
 
 #endif /* __XEN_PUBLIC_SAVE_H__ */
Signature · 1 line
2.20.1
#24 lore raw

Re: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

Hi Paul,

Quoted 71 lines · Paul Durrant

... and update xen-domctx to dump some information describing the record.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
  tools/misc/xen-domctx.c   | 11 ++++++
  xen/common/domain.c       | 81 +++++++++++++++++++++++++++++++++++++++
  xen/include/public/save.h | 10 ++++-
  3 files changed, 101 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +11 -0
index d663522a8b..a8d3922321 100644
@@ -59,6 +59,16 @@ static void dump_header(struct domain_save_descriptor *desc)
      off += desc->length;
  }
 
+static void dump_shared_info(struct domain_save_descriptor *desc)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) s;
+    READ(s);
+    printf("    SHARED_INFO: field_width %u buffer size: %lu\n",
+           s.field_width, desc->length - sizeof(s));
+
+    off += desc->length;
+}
+
  static void dump_end(struct domain_save_descriptor *desc)
  {
      DOMAIN_SAVE_TYPE(END) e;
@@ -125,6 +135,7 @@ int main(int argc, char **argv)
          switch (desc.typecode)
          {
          case DOMAIN_SAVE_CODE(HEADER): dump_header(&desc); break;
+        case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(&desc); break;
          case DOMAIN_SAVE_CODE(END): dump_end(&desc); return 0;
          default:
              printf("Unknown type %u: skipping\n", desc.typecode);
xen/common/domain.c +5 -0
index 3dcd73f67c..8b72462e07 100644
@@ -33,6 +33,7 @@
  #include <xen/xenoprof.h>
  #include <xen/irq.h>
  #include <xen/argo.h>
+#include <xen/save.h>
  #include <asm/debugger.h>
  #include <asm/p2m.h>
  #include <asm/processor.h>
@@ -1646,6 +1647,86 @@ int continue_hypercall_on_cpu(
      return 0;
  }
 
+static int save_shared_info(const struct vcpu *v, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain *d = v->domain;

xen/common/domain.c int continue_hypercall_on_cpu( · line 1663

Quoted 10 lines
xen/common/domain.c +10 -0
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    int rc;
+
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, v, size);
+    if ( rc )
+        return rc;
+
+    if ( !dry_run )

NIT: I think the if is not necessary here as you don't skip that much code.

Quoted 74 lines
xen/common/domain.c +66 -0
+        ctxt.field_width =
+#ifdef CONFIG_COMPAT
+            has_32bit_shinfo(d) ? 4 :
+#endif
+            8;
+
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+static int load_shared_info(struct vcpu *v, struct domain_context *c)
+{
+    struct domain *d = v->domain;
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, v, size, true);
+    if ( rc )
+        return rc;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    for ( i = 0; i < ARRAY_SIZE(ctxt.pad); i++ )
+        if ( ctxt.pad[i] )
+            return -EINVAL;
+
+    switch ( ctxt.field_width )
+    {
+#ifdef CONFIG_COMPAT
+    case 4:
+        d->arch.has_32bit_shinfo = 1;
+        break;
+#endif
+    case 8:
+#ifdef CONFIG_COMPAT
+        d->arch.has_32bit_shinfo = 0;
+#endif
+        break;
+
+    default:
+        rc = -EINVAL;
+        break;
+    }
+
+    rc = domain_load_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_load_end(c);
+}
+
+DOMAIN_REGISTER_SAVE_RESTORE(SHARED_INFO, false, save_shared_info,
+                             load_shared_info);
+
  /*
   * Local variables:
   * mode: C
xen/include/public/save.h +0 -0
index 7e5f8752bd..ed994a8765 100644

xen/include/public/save.h struct domain_save_header { · line 85

Quoted 8 lines
xen/include/public/save.h +4 -1
@@ -79,6 +79,14 @@ struct domain_save_header {
  };
  DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint8_t field_width;
+    uint8_t pad[7];
+    uint8_t buffer[]; /* Implementation specific size */

I would recommend to use buffer[XEN_FLEX_ARRAY_DIM].

Cheers,

Signature · 1 line
Julien Grall
#25 lore raw

RE: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

Quoted 79 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 20 April 2020 18:35
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

Hi Paul,

Quoted 67 lines · Paul Durrant

... and update xen-domctx to dump some information describing the record.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
  tools/misc/xen-domctx.c   | 11 ++++++
  xen/common/domain.c       | 81 +++++++++++++++++++++++++++++++++++++++
  xen/include/public/save.h | 10 ++++-
  3 files changed, 101 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +11 -0
index d663522a8b..a8d3922321 100644
@@ -59,6 +59,16 @@ static void dump_header(struct domain_save_descriptor *desc)
      off += desc->length;
  }
 
+static void dump_shared_info(struct domain_save_descriptor *desc)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) s;
+    READ(s);
+    printf("    SHARED_INFO: field_width %u buffer size: %lu\n",
+           s.field_width, desc->length - sizeof(s));
+
+    off += desc->length;
+}
+
  static void dump_end(struct domain_save_descriptor *desc)
  {
      DOMAIN_SAVE_TYPE(END) e;
@@ -125,6 +135,7 @@ int main(int argc, char **argv)
          switch (desc.typecode)
          {
          case DOMAIN_SAVE_CODE(HEADER): dump_header(&desc); break;
+        case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(&desc); break;
          case DOMAIN_SAVE_CODE(END): dump_end(&desc); return 0;
          default:
              printf("Unknown type %u: skipping\n", desc.typecode);
xen/common/domain.c +1 -0
index 3dcd73f67c..8b72462e07 100644
@@ -33,6 +33,7 @@
  #include <xen/xenoprof.h>
  #include <xen/irq.h>
  #include <xen/argo.h>
+#include <xen/save.h>
  #include <asm/debugger.h>
  #include <asm/p2m.h>
  #include <asm/processor.h>
@@ -1646,6 +1647,86 @@ int continue_hypercall_on_cpu(
      return 0;
  }
 
Quoted 17 lines
Quoted 4 lines
xen/common/domain.c +4 -0
+static int save_shared_info(const struct vcpu *v, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain *d = v->domain;

xen/common/domain.c int continue_hypercall_on_cpu( · line 1663

Quoted 10 lines
xen/common/domain.c +10 -0
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    int rc;
+
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, v, size);
+    if ( rc )
+        return rc;
+
+    if ( !dry_run )

NIT: I think the if is not necessary here as you don't skip that much code.

I know, but it is illustrative so I'd rather keep it.

Quoted 77 lines
Quoted 77 lines
xen/common/domain.c +66 -0
+        ctxt.field_width =
+#ifdef CONFIG_COMPAT
+            has_32bit_shinfo(d) ? 4 :
+#endif
+            8;
+
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+static int load_shared_info(struct vcpu *v, struct domain_context *c)
+{
+    struct domain *d = v->domain;
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, v, size, true);
+    if ( rc )
+        return rc;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    for ( i = 0; i < ARRAY_SIZE(ctxt.pad); i++ )
+        if ( ctxt.pad[i] )
+            return -EINVAL;
+
+    switch ( ctxt.field_width )
+    {
+#ifdef CONFIG_COMPAT
+    case 4:
+        d->arch.has_32bit_shinfo = 1;
+        break;
+#endif
+    case 8:
+#ifdef CONFIG_COMPAT
+        d->arch.has_32bit_shinfo = 0;
+#endif
+        break;
+
+    default:
+        rc = -EINVAL;
+        break;
+    }
+
+    rc = domain_load_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_load_end(c);
+}
+
+DOMAIN_REGISTER_SAVE_RESTORE(SHARED_INFO, false, save_shared_info,
+                             load_shared_info);
+
  /*
   * Local variables:
   * mode: C
xen/include/public/save.h +0 -0
index 7e5f8752bd..ed994a8765 100644
@@ -79,6 +79,14 @@ struct domain_save_header {
  };
  DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
 
Quoted 8 lines

xen/include/public/save.h struct domain_save_header { · line 85

Quoted 5 lines
xen/include/public/save.h +4 -1
-#define DOMAIN_SAVE_CODE_MAX 1
+struct domain_shared_info_context {
+    uint8_t field_width;
+    uint8_t pad[7];
+    uint8_t buffer[]; /* Implementation specific size */

I would recommend to use buffer[XEN_FLEX_ARRAY_DIM].

Ok.

Paul

Quoted 4 lines

Cheers,

--
Julien Grall

#26 lore raw

Re: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

Quoted 97 lines · Paul Durrant
Quoted 79 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 20 April 2020 18:35
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

Hi Paul,

Quoted 67 lines · Paul Durrant

... and update xen-domctx to dump some information describing the record.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>

v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT

---
  tools/misc/xen-domctx.c   | 11 ++++++
  xen/common/domain.c       | 81 +++++++++++++++++++++++++++++++++++++++
  xen/include/public/save.h | 10 ++++-
  3 files changed, 101 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +11 -0
index d663522a8b..a8d3922321 100644
@@ -59,6 +59,16 @@ static void dump_header(struct domain_save_descriptor *desc)
      off += desc->length;
  }
 
+static void dump_shared_info(struct domain_save_descriptor *desc)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) s;
+    READ(s);
+    printf("    SHARED_INFO: field_width %u buffer size: %lu\n",
+           s.field_width, desc->length - sizeof(s));
+
+    off += desc->length;
+}
+
  static void dump_end(struct domain_save_descriptor *desc)
  {
      DOMAIN_SAVE_TYPE(END) e;
@@ -125,6 +135,7 @@ int main(int argc, char **argv)
          switch (desc.typecode)
          {
          case DOMAIN_SAVE_CODE(HEADER): dump_header(&desc); break;
+        case DOMAIN_SAVE_CODE(SHARED_INFO): dump_shared_info(&desc); break;
          case DOMAIN_SAVE_CODE(END): dump_end(&desc); return 0;
          default:
              printf("Unknown type %u: skipping\n", desc.typecode);
xen/common/domain.c +1 -0
index 3dcd73f67c..8b72462e07 100644
@@ -33,6 +33,7 @@
  #include <xen/xenoprof.h>
  #include <xen/irq.h>
  #include <xen/argo.h>
+#include <xen/save.h>
  #include <asm/debugger.h>
  #include <asm/p2m.h>
  #include <asm/processor.h>
@@ -1646,6 +1647,86 @@ int continue_hypercall_on_cpu(
      return 0;
  }
 
Quoted 17 lines
Quoted 4 lines
xen/common/domain.c +4 -0
+static int save_shared_info(const struct vcpu *v, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain *d = v->domain;

xen/common/domain.c int continue_hypercall_on_cpu( · line 1663

Quoted 10 lines
xen/common/domain.c +10 -0
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    int rc;
+
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, v, size);
+    if ( rc )
+        return rc;
+
+    if ( !dry_run )

NIT: I think the if is not necessary here as you don't skip that much code.

Quoted 1 line

I know, but it is illustrative so I'd rather keep it.

While I agree with the "illustrative", I'd really see this be part of the struct initializer. Plus its use here made me wonder ...

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
xen/common/domain.c +5 -0
+        ctxt.field_width =
+#ifdef CONFIG_COMPAT
+            has_32bit_shinfo(d) ? 4 :
+#endif
+            8;
Quoted 8 lines
Quoted 8 lines

xen/common/domain.c int continue_hypercall_on_cpu( · line 1676

Quoted 8 lines
xen/common/domain.c +8 -0
+
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;

... why these don't get skipped. It took me going back through earlier patches to find that there's effectively redundancy in the passed arguments in that the write callback chosen varies with whether "dry_run" is true or false.

Jan

#27 lore raw

Re: [PATCH v2 4/5] common/domain: add a domain context record for shared_info...

In reply to Paul Durrant
Quoted 5 lines · Paul Durrant
tools/misc/xen-domctx.c +0 -0
@@ -59,6 +59,16 @@ static void dump_header(struct domain_save_descriptor *desc)
     off += desc->length;
 }

tools/misc/xen-domctx.c static void dump_header(struct domain_save_descriptor *desc) · line 70

Quoted 10 lines
tools/misc/xen-domctx.c +9 -0
 
+static void dump_shared_info(struct domain_save_descriptor *desc)
+{
+    DOMAIN_SAVE_TYPE(SHARED_INFO) s;
+    READ(s);
+    printf("    SHARED_INFO: field_width %u buffer size: %lu\n",
+           s.field_width, desc->length - sizeof(s));
+
+    off += desc->length;
+}

And nothing about the actual contents of the shared info struct?

tools/misc/xen-domctx.c int continue_hypercall_on_cpu( · line 1653

Quoted 8 lines
tools/misc/xen-domctx.c +4 -0
@@ -1646,6 +1647,86 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+static int save_shared_info(const struct vcpu *v, struct domain_context *c,
+                            bool dry_run)
+{
+    struct domain *d = v->domain;

const?

Quoted 5 lines
tools/misc/xen-domctx.c +5 -0
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    int rc;
+

tools/misc/xen-domctx.c int continue_hypercall_on_cpu( · line 1668

Quoted 10 lines
tools/misc/xen-domctx.c +10 -0
+    rc = DOMAIN_SAVE_BEGIN(SHARED_INFO, c, v, size);
+    if ( rc )
+        return rc;
+
+    if ( !dry_run )
+        ctxt.field_width =
+#ifdef CONFIG_COMPAT
+            has_32bit_shinfo(d) ? 4 :
+#endif
+            8;

What are 4 and 8 to represent here? Taking Arm32 into account, the only things I could think of are sizeof(xen_ulong_t) or sizeof(guest_handle). I'd prefer if literal numbers could be avoided here, such that it would become clear what property is really meant.

Quoted 27 lines
tools/misc/xen-domctx.c +27 -0
+    rc = domain_save_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;
+
+    rc = domain_save_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_save_end(c);
+}
+
+static int load_shared_info(struct vcpu *v, struct domain_context *c)
+{
+    struct domain *d = v->domain;
+    struct domain_shared_info_context ctxt = {};
+    size_t hdr_size = offsetof(typeof(ctxt), buffer);
+    size_t size = hdr_size + PAGE_SIZE;
+    unsigned int i;
+    int rc;
+
+    rc = DOMAIN_LOAD_BEGIN(SHARED_INFO, c, v, size, true);
+    if ( rc )
+        return rc;
+
+    rc = domain_load_data(c, &ctxt, hdr_size);
+    if ( rc )
+        return rc;

tools/misc/xen-domctx.c int continue_hypercall_on_cpu( · line 1705

Quoted 10 lines
tools/misc/xen-domctx.c +10 -0
+
+    for ( i = 0; i < ARRAY_SIZE(ctxt.pad); i++ )
+        if ( ctxt.pad[i] )
+            return -EINVAL;
+
+    switch ( ctxt.field_width )
+    {
+#ifdef CONFIG_COMPAT
+    case 4:
+        d->arch.has_32bit_shinfo = 1;

true and ...

tools/misc/xen-domctx.c int continue_hypercall_on_cpu( · line 1710

Quoted 5 lines
tools/misc/xen-domctx.c +5 -0
+        break;
+#endif
+    case 8:
+#ifdef CONFIG_COMPAT
+        d->arch.has_32bit_shinfo = 0;

... false respectively, please.

Quoted 4 lines
tools/misc/xen-domctx.c +4 -0
+#endif
+        break;
+
+    default:

tools/misc/xen-domctx.c int continue_hypercall_on_cpu( · line 1724

Quoted 10 lines
tools/misc/xen-domctx.c +10 -0
+        rc = -EINVAL;
+        break;
+    }
+
+    rc = domain_load_data(c, d->shared_info, PAGE_SIZE);
+    if ( rc )
+        return rc;
+
+    return domain_load_end(c);
+}

While you check the padding fields of the header above, what about currently unused fields of the shared_info struct itself?

There's a connection between shared_info and vcpu_info - this way you may or may not restore vcpu_info - depending on guest behavior. There not being a patch in the series to deal with vcpu_info, the description here should imo at least outline the intended interaction (including e.g. ordering between the [supposed] records).

Jan

#28 lore raw

PATCH v2 5/5 tools/libxc: make use of domain context SHARED_INFO record...

In reply to Paul Durrant

... in the save/restore code.

This patch replaces direct mapping of the shared_info_frame (retrieved using XEN_DOMCTL_getdomaininfo) with save/load of the domain context SHARED_INFO record.

No modifications are made to the definition of the migration stream at this point. Subsequent patches will define a record in the libxc domain image format for passing domain context and convert the save/restore code to use that.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Re-based (now making use of DOMAIN_SAVE_FLAG_IGNORE)

---
 tools/libxc/xc_sr_common.h         |  7 +++-
 tools/libxc/xc_sr_common_x86.c     | 59 ++++++++++++++++++++++++++++++
 tools/libxc/xc_sr_common_x86.h     |  4 ++
 tools/libxc/xc_sr_common_x86_pv.c  | 53 +++++++++++++++++++++++++++
 tools/libxc/xc_sr_common_x86_pv.h  |  3 ++
 tools/libxc/xc_sr_restore_x86_pv.c | 40 ++++++++------------
 tools/libxc/xc_sr_save_x86_pv.c    | 26 ++-----------
 tools/libxc/xg_save_restore.h      |  1 +
 8 files changed, 144 insertions(+), 49 deletions(-)
tools/libxc/xc_sr_common.h +6 -1
index 5dd51ccb15..db6519cdcc 100644
@@ -287,6 +287,11 @@ struct xc_sr_context
     {
         struct /* x86 */
         {
+            struct {
+                void *buffer;
+                unsigned int len;
+            } domain_context;
+
             struct /* x86 PV guest. */
             {
                 /* 4 or 8; 32 or 64 bit domain */
@@ -314,7 +319,7 @@ struct xc_sr_context
                 /* The guest pfns containing the p2m leaves */
                 xen_pfn_t *p2m_pfns;
 
-                /* Read-only mapping of guests shared info page */
+                /* Pointer to shared_info (located in context buffer) */
                 shared_info_any_t *shinfo;
 
                 /* p2m generation count for verifying validity of local p2m. */
tools/libxc/xc_sr_common_x86.c +59 -0

65 lines not on this page · raw message

tools/libxc/xc_sr_common_x86.h +4 -0
index ebc4355bd1..501c9e52ba 100644
@@ -14,6 +14,10 @@ int write_x86_tsc_info(struct xc_sr_context *ctx);
  */
 int handle_x86_tsc_info(struct xc_sr_context *ctx, struct xc_sr_record *rec);
 
+int x86_get_context(struct xc_sr_context *ctx);
+int x86_set_context(struct xc_sr_context *ctx);
+void x86_cleanup(struct xc_sr_context *ctx);
+
 #endif
 /*
  * Local variables:
tools/libxc/xc_sr_common_x86_pv.c +53 -0

59 lines not on this page · raw message

tools/libxc/xc_sr_common_x86_pv.h +3 -0
index 2ed03309af..01442f48fb 100644
@@ -97,6 +97,9 @@ int x86_pv_domain_info(struct xc_sr_context *ctx);
  */
 int x86_pv_map_m2p(struct xc_sr_context *ctx);
 
+int x86_pv_get_shinfo(struct xc_sr_context *ctx);
+int x86_pv_set_shinfo(struct xc_sr_context *ctx);
+
 #endif
 /*
  * Local variables:
tools/libxc/xc_sr_restore_x86_pv.c +15 -25
index 904ccc462a..4dbc7f0da5 100644
@@ -864,8 +864,7 @@ static int handle_shared_info(struct xc_sr_context *ctx,
 {
     xc_interface *xch = ctx->xch;
     unsigned int i;
-    int rc = -1;
-    shared_info_any_t *guest_shinfo = NULL;
+    int rc;
     const shared_info_any_t *old_shinfo = rec->data;
 
     if ( !ctx->x86.pv.restore.seen_pv_info )
@@ -878,39 +877,30 @@ static int handle_shared_info(struct xc_sr_context *ctx,
     {
         ERROR("X86_PV_SHARED_INFO record wrong size: length %u"
               ", expected 4096", rec->length);
-        goto err;
+        return -1;
     }
 
-    guest_shinfo = xc_map_foreign_range(
-        xch, ctx->domid, PAGE_SIZE, PROT_READ | PROT_WRITE,
-        ctx->dominfo.shared_info_frame);
-    if ( !guest_shinfo )
-    {
-        PERROR("Failed to map Shared Info at mfn %#lx",
-               ctx->dominfo.shared_info_frame);
-        goto err;
-    }
+    rc = x86_pv_get_shinfo(ctx);
+    if ( rc )
+        return rc;
 
-    MEMCPY_FIELD(guest_shinfo, old_shinfo, vcpu_info, ctx->x86.pv.width);
-    MEMCPY_FIELD(guest_shinfo, old_shinfo, arch, ctx->x86.pv.width);
+    MEMCPY_FIELD(ctx->x86.pv.shinfo, old_shinfo, vcpu_info,
+                 ctx->x86.pv.width);
+    MEMCPY_FIELD(ctx->x86.pv.shinfo, old_shinfo, arch, ctx->x86.pv.width);
 
-    SET_FIELD(guest_shinfo, arch.pfn_to_mfn_frame_list_list,
+    SET_FIELD(ctx->x86.pv.shinfo, arch.pfn_to_mfn_frame_list_list,
               0, ctx->x86.pv.width);
 
-    MEMSET_ARRAY_FIELD(guest_shinfo, evtchn_pending, 0, ctx->x86.pv.width);
+    MEMSET_ARRAY_FIELD(ctx->x86.pv.shinfo, evtchn_pending, 0,
+                       ctx->x86.pv.width);
     for ( i = 0; i < XEN_LEGACY_MAX_VCPUS; i++ )
-        SET_FIELD(guest_shinfo, vcpu_info[i].evtchn_pending_sel,
+        SET_FIELD(ctx->x86.pv.shinfo, vcpu_info[i].evtchn_pending_sel,
                   0, ctx->x86.pv.width);
 
-    MEMSET_ARRAY_FIELD(guest_shinfo, evtchn_mask, 0xff, ctx->x86.pv.width);
-
-    rc = 0;
+    MEMSET_ARRAY_FIELD(ctx->x86.pv.shinfo, evtchn_mask, 0xff,
+                       ctx->x86.pv.width);
 
- err:
-    if ( guest_shinfo )
-        munmap(guest_shinfo, PAGE_SIZE);
-
-    return rc;
+    return x86_pv_set_shinfo(ctx);
 }
 
 /* restore_ops function. */
tools/libxc/xc_sr_save_x86_pv.c +3 -23
index f3ccf5bb4b..7c4fcffa92 100644
@@ -9,25 +9,6 @@ static inline bool is_canonical_address(xen_vaddr_t vaddr)
     return ((int64_t)vaddr >> 47) == ((int64_t)vaddr >> 63);
 }
 
-/*
- * Maps the guests shared info page.
- */
-static int map_shinfo(struct xc_sr_context *ctx)
-{
-    xc_interface *xch = ctx->xch;
-
-    ctx->x86.pv.shinfo = xc_map_foreign_range(
-        xch, ctx->domid, PAGE_SIZE, PROT_READ, ctx->dominfo.shared_info_frame);
-    if ( !ctx->x86.pv.shinfo )
-    {
-        PERROR("Failed to map shared info frame at mfn %#lx",
-               ctx->dominfo.shared_info_frame);
-        return -1;
-    }
-
-    return 0;
-}
-
 /*
  * Copy a list of mfns from a guest, accounting for differences between guest
  * and toolstack width.  Can fail if truncation would occur.
@@ -1041,7 +1022,7 @@ static int x86_pv_setup(struct xc_sr_context *ctx)
     if ( rc )
         return rc;
 
-    rc = map_shinfo(ctx);
+    rc = x86_pv_get_shinfo(ctx);
     if ( rc )
         return rc;
 
@@ -1112,12 +1093,11 @@ static int x86_pv_cleanup(struct xc_sr_context *ctx)
     if ( ctx->x86.pv.p2m )
         munmap(ctx->x86.pv.p2m, ctx->x86.pv.p2m_frames * PAGE_SIZE);
 
-    if ( ctx->x86.pv.shinfo )
-        munmap(ctx->x86.pv.shinfo, PAGE_SIZE);
-
     if ( ctx->x86.pv.m2p )
         munmap(ctx->x86.pv.m2p, ctx->x86.pv.nr_m2p_frames * PAGE_SIZE);
 
+    x86_cleanup(ctx);
+
     return 0;
 }
 
tools/libxc/xg_save_restore.h +1 -0
index 303081df0d..296b523963 100644
@@ -19,6 +19,7 @@
 
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
+#include <xen/save.h>
 
 /*
 ** We process save/restore/migrate in batches of pages; the below
Signature · 1 line
2.20.1
#29 lore raw

Re: [PATCH v2 5/5] tools/libxc: make use of domain context SHARED_INFO record...

Quoted 18 lines · Paul Durrant

... in the save/restore code.

This patch replaces direct mapping of the shared_info_frame (retrieved using XEN_DOMCTL_getdomaininfo) with save/load of the domain context SHARED_INFO record.

No modifications are made to the definition of the migration stream at this point. Subsequent patches will define a record in the libxc domain image format for passing domain context and convert the save/restore code to use that.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>

v2:
- Re-based (now making use of DOMAIN_SAVE_FLAG_IGNORE)

Quoted 10 lines
---
 tools/libxc/xc_sr_common.h         |  7 +++-
 tools/libxc/xc_sr_common_x86.c     | 59 ++++++++++++++++++++++++++++++
 tools/libxc/xc_sr_common_x86.h     |  4 ++
 tools/libxc/xc_sr_common_x86_pv.c  | 53 +++++++++++++++++++++++++++
 tools/libxc/xc_sr_common_x86_pv.h  |  3 ++
 tools/libxc/xc_sr_restore_x86_pv.c | 40 ++++++++------------
 tools/libxc/xc_sr_save_x86_pv.c    | 26 ++-----------
 tools/libxc/xg_save_restore.h      |  1 +
 8 files changed, 144 insertions(+), 49 deletions(-)

The underlying interface being arch-independent, shouldn't at least some of the new code go into other than xc_sr_*x86*?

Jan