domain context infrastructure
v10 cover domain context infrastructure
From: Paul Durrant <[email protected]>
Paul Durrant (11):
docs / include: introduce a new framework for 'domain context' records
xen: introduce implementation of save/restore of 'domain context' xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context tools/misc: add xen-domctx to present domain context common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... docs/specs: add missing definitions to libxc-migration-stream docs / tools: specify migration v4 to include DOMAIN_CONTEXT tools/python: modify libxc.py to verify v4 stream tools/libs/guest: add code to restore a v4 libxc stream tools/libs/guest: add code to save a v4 libxc stream
.gitignore | 1 + docs/specs/domain-context.md | 198 +++++++++++++ docs/specs/libxc-migration-stream.pandoc | 69 ++++- tools/flask/policy/modules/xen.if | 4 +- tools/libs/ctrl/include/xenctrl.h | 5 + tools/libs/ctrl/xc_domain.c | 56 ++++ tools/libs/guest/xg_sr_common.c | 1 + tools/libs/guest/xg_sr_common.h | 3 + tools/libs/guest/xg_sr_common_x86.c | 20 -- tools/libs/guest/xg_sr_common_x86.h | 6 - tools/libs/guest/xg_sr_restore.c | 24 +- tools/libs/guest/xg_sr_restore_x86_hvm.c | 9 + tools/libs/guest/xg_sr_restore_x86_pv.c | 9 + tools/libs/guest/xg_sr_save.c | 52 +++- tools/libs/guest/xg_sr_save_x86_hvm.c | 5 - tools/libs/guest/xg_sr_save_x86_pv.c | 22 -- tools/libs/guest/xg_sr_stream_format.h | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 264 ++++++++++++++++++ tools/python/xen/migration/libxc.py | 22 +- xen/arch/x86/time.c | 30 ++ xen/common/Makefile | 1 + xen/common/domain.c | 113 ++++++++ xen/common/domctl.c | 173 ++++++++++++ xen/common/save.c | 339 +++++++++++++++++++++++ xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/domctl.h | 39 +++ xen/include/public/save.h | 85 ++++++ xen/include/xen/save.h | 160 +++++++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 32 files changed, 1661 insertions(+), 74 deletions(-) create mode 100644 docs/specs/domain-context.md create mode 100644 tools/misc/xen-domctx.c create mode 100644 xen/common/save.c create mode 100644 xen/include/public/save.h create mode 100644 xen/include/xen/save.h
---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>
Signature · 1 line
2.20.1
PATCH v10 1/11 docs / include: introduce a new framework for 'domain context' records
From: Paul Durrant <[email protected]>
To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.
Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.
This patch adds the specification of a new common 'domain context' image and the supporting public header. Subsequent patches will introduce other parts of the framework, and code that will make use of it.
This patch also marks the HVM-only framework as deprecated in favour of the new framework.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v10:
- New in v10
--- docs/specs/domain-context.md | 137 +++++++++++++++++++++++++ xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/save.h | 66 ++++++++++++ 4 files changed, 213 insertions(+) create mode 100644 docs/specs/domain-context.md create mode 100644 xen/include/public/save.h
docs/specs/domain-context.md new file +137 -0
@@ -0,0 +1,137 @@ +# Domain Context (v1) + +## Background + +The design for *Non-Cooperative Migration of Guests*[1] explains that extra +information is required in the migration stream to allow a guest running PV +drivers to be migrated without its co-operation. This information includes +hypervisor state such as the set of event channels in operation and the +content of the grant table. + +There already exists in Xen a mechanism to save and restore *HVM context*[2]. +This specification describes a new framework that will replace that +mechanism and be suitable for transferring additional *PV state* records +conveying the information mentioned above. There is also on-going work to +implement *live update* of Xen where hypervisor state must be transferred in +memory from one incarnation to the next. The framework described is designed +to also be suitable for this purpose. + +## Image format + +The format will read from or written to the hypervisor in a single virtually +contiguous buffer segmented into **context records** specified in the following +sections. + +Fields within the records will always be aligned to their size. Padding and +reserved fields will always be set to zero when the context buffer is read +from the hypervisor and will be verified when written. +The endianness of numerical values will the native endianness of the +hypervisor. In the case of migration, that endianness is specified in the +*libxenctrl (libxc) Domain Image Format*[3]. + +### Record format + +All records have the following format: + +``` + 0 1 2 3 4 5 6 7 octet ++-------+-------+-------+-------+-------+-------+-------+-------+ +| type | instance | ++-------------------------------+-------------------------------+ +| length | ++---------------------------------------------------------------+ +| body +... +| | padding (0 to 7 octets) | ++-------+-------------------------------------------------------+ +``` + +\pagebreak +The fields are defined as follows: + + +| Field | Description | +|------------|--------------------------------------------------| +| `type` | A code which determines the layout and semantics | +| | of `body` | +| | | +| `instance` | The instance of the record | +| | | +| `length` | The length (in octets) of `body` | +| | | +| `body` | Zero or more octets of record data | +| | | +| `padding` | Zero to seven octets of zero-filled padding to | +| | bring the total record length up to the next | +| | 64-bit boundary | + +The `instance` field is present to distinguish multiple occurences of +a record. E.g. state that is per-vcpu may need to be described in multiple +records. + +The first record in the image is always a **START** record. The version of +the image format can be inferred from the `type` of this record. + +## Image content + +The following records are defined for the v1 image format. This set may be +extended in newer versions of the hypervisor. It is not expected that an image +saved on a newer version of Xen will need to be restored on an older version. +Therefore an image containing unrecognized record types should be rejected. + +### START + +``` + 0 1 2 3 4 5 6 7 octet ++-------+-------+-------+-------+-------+-------+-------+-------+ +| type == 1 | instance == 0 | ++-------------------------------+-------------------------------+ +| length == 8 | ++-------------------------------+-------------------------------+ +| xen_major | xen_minor | ++-------------------------------+-------------------------------+ +``` + +A type 1 **START** record implies a v1 image. If a new image format version +is needed in future then this can be indicated by a new type value for this +(first) record in the image. + +\pagebreak +The record body contains the following fields: + +| Field | Description | +|-------------|-------------------------------------------------| +| `xen_major` | The major version of Xen that created this | +| | image | +| | | +| `xen_minor` | The minor version of Xen that created this | +| | image | + +The version of Xen that created the image can be useful to the version that +is restoring the image to determine whether certain records are expected to +be present in the image. For example, a version of Xen prior to X.Y may not +generate a FOO record but Xen X.Y+ can infer its content. But Xen X.Y+1 +**must** generate the FOO record as, from that version onward, its content +can no longer be safely inferred. + +### END + +``` + 0 1 2 3 4 5 6 7 octet ++-------+-------+-------+-------+-------+-------+-------+-------+ +| type == 0 | instance == 0 | ++-------------------------------+-------------------------------+ +| length == 0 | ++---------------------------------------------------------------+ +``` + +A record of this type terminates the image. No further data from the buffer +should be consumed. + +* * * + +[1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md + +[2] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/hvm/save.h + +[3] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/specs/libxc-migration-stream.pandoc
xen/include/public/arch-arm/hvm/save.h +5 -0
@@ -26,6 +26,11 @@#ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__#define __XEN_PUBLIC_HVM_SAVE_ARM_H__ +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /*
xen/include/public/arch-x86/hvm/save.h +5 -0
@@ -648,6 +648,11 @@ struct hvm_msr {*/ #define HVM_SAVE_CODE_MAX 20 +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */ /*
xen/include/public/save.h new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records
Quoted 35 lines · Paul Durrant
xen/include/public/save.h new file +32
-0@@ -0,0 +1,66 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by Xen. +* +* Copyright Amazon.com Inc. or its affiliates. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +*/ + +#ifndef XEN_PUBLIC_SAVE_H +#define XEN_PUBLIC_SAVE_H + +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +#include "xen.h"
xen/include/public/save.h line 42
Quoted 10 lines
xen/include/public/save.h new file +10
-0+ +/* +* C structures for the Domain Context v1 format. +* See docs/specs/domain-context.md +*/ + +struct domain_context_record { +uint32_t type; +uint32_t instance; +uint64_t length;
Should this be uint64_aligned_t, such that alignof() will produce consistent values regardless of bitness of the invoking domain?
Jan
Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records
Quoted 41 lines · Jan Beulich
Quoted 40 lines · Paul Durrant
xen/include/public/save.h new file +37
-0@@ -0,0 +1,66 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by Xen. +* +* Copyright Amazon.com Inc. or its affiliates. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +*/ + +#ifndef XEN_PUBLIC_SAVE_H +#define XEN_PUBLIC_SAVE_H + +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +#include "xen.h" + +/* +* C structures for the Domain Context v1 format. +* See docs/specs/domain-context.md +*/
Quoted 8 lines
xen/include/public/save.h line 42
Quoted 5 lines
xen/include/public/save.h new file +5
-0+ +struct domain_context_record { +uint32_t type; +uint32_t instance; +uint64_t length;Should this be
uint64_aligned_t, such thatalignof()will produce consistent values regardless of bitness of the invoking domain?
Does it matter? Its just a bitstream, and can appear in the migration fd at any arbitrary alignment.
What matters is that the structure is aligned appropriately for the bitness of code operating on these fields.
Even with the tools ABI fixed to allow a 32-on-64-on-64 toolstack to function, I'm not sure that excess alignment would be appropriate. Sure
- it would be more efficient for 32bit code to align to the 8 byte boundary for the benefit of a 64bit Xen's copy_from_user(), but this alignment happens anyway because of how hypercall buffers work.
~Andrew
Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records
Quoted 54 lines · Andrew Cooper
Quoted 41 lines · Jan Beulich
Quoted 40 lines · Paul Durrant
xen/include/public/save.h new file +37
-0@@ -0,0 +1,66 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by Xen. +* +* Copyright Amazon.com Inc. or its affiliates. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +*/ + +#ifndef XEN_PUBLIC_SAVE_H +#define XEN_PUBLIC_SAVE_H + +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +#include "xen.h" + +/* +* C structures for the Domain Context v1 format. +* See docs/specs/domain-context.md +*/ Quoted 8 lines
xen/include/public/save.h line 42
Quoted 5 lines
xen/include/public/save.h new file +5
-0+ +struct domain_context_record { +uint32_t type; +uint32_t instance; +uint64_t length;Should this be
uint64_aligned_t, such thatalignof()will produce consistent values regardless of bitness of the invoking domain?Does it matter? Its just a bitstream, and can appear in the migration fd at any arbitrary alignment.
Quoted 8 lines
What matters is that the structure is aligned appropriately for the bitness of code operating on these fields.
Even with the tools ABI fixed to allow a 32-on-64-on-64 toolstack to function, I'm not sure that excess alignment would be appropriate. Sure
- it would be more efficient for 32bit code to align to the 8 byte boundary for the benefit of a 64bit Xen'scopy_from_user(), but this alignment happens anyway because of how hypercall buffers work.
It _probably_ doesn't matter (hence me having put this as a question), but a struct in the ABI doesn't mandate how it is going to be used. I don't expect this to become the (or part of an) argument to a hypercall. I also don't expect anyone needing to use alignof() on it. But by not following the common model for the tools-only parts of the public interface we'd outright exclude such uses down the road.
Jan
Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records
In reply to Paul Durrantxen/include/public/save.h line 4
Quoted 10 lines · Paul Durrant
xen/include/public/save.h new file +4
-0@@ -0,0 +1,66 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by Xen.
What exactly is, and is not in scope, for this new stream? The PV above I think refers to "paravirtual state", not PV guests.
xen/include/public/save.h line 47
Quoted 2 lines
xen/include/public/save.h new file +2
-0+#define _DOMAIN_CONTEXT_RECORD_ALIGN 3 +#define DOMAIN_CONTEXT_RECORD_ALIGN (1U << _DOMAIN_CONTEXT_RECORD_ALIGN)
Do we need the logarithm version?
xen/include/public/save.h line 54
Quoted 7 lines
xen/include/public/save.h new file +7
-0+ +enum { +DOMAIN_CONTEXT_END , +DOMAIN_CONTEXT_START , +/* New types go here */ +DOMAIN_CONTEXT_NR_TYPES +};
Does this enum ever end up in an API?
We might be ok as we're inside __XEN_TOOLS__, but enums normally cannot be used in ABI's because their size is implementation defined, and not always 4 bytes.
~Andrew
Re: [PATCH v10 01/11] docs / include: introduce a new framework for 'domain context' records
Quoted 9 lines · Andrew Cooper
xen/include/public/save.h line 54
Quoted 6 lines
xen/include/public/save.h new file +6
-0+enum { +DOMAIN_CONTEXT_END , +DOMAIN_CONTEXT_START , +/* New types go here */ +DOMAIN_CONTEXT_NR_TYPES +};Does this enum ever end up in an API?
Quoted 3 lines
We might be ok as we're inside
__XEN_TOOLS__, but enums normally cannot be used in ABI's because their size is implementation defined, and not always 4 bytes.
The only way to use this as a type (e.g. to declare a struct field) would be via typeof(), which isn't something we'd allow in the public interface (for being an extension). Hence without a tag I would think an enum is fine to have here?
Jan
PATCH v10 2/11 xen: introduce implementation of save/restore of 'domain context'
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
A previous patch introduced the specification of the 'domain context' image and the supporting public header. This patch introduces the code necessary to generate and consume such an image. The entry points to the code are domain_save_ctxt() and domain_load_ctxt(). Code to call these functions will be introduced in a subsequent patch.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
v10:
- New in v10
- Largely derived from patch #1 of the v9 series, but the code has been re-worked both functionally and cosmetically
--- xen/common/Makefile | 1 + xen/common/save.c | 339 +++++++++++++++++++++++++++++++++++++++++ xen/include/xen/save.h | 160 +++++++++++++++++++ 3 files changed, 500 insertions(+) create mode 100644 xen/common/save.c create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
@@ -37,6 +37,7 @@ obj-y += radix-tree.o obj-y += rbtree.o obj-y += rcupdate.o obj-y += rwlock.o +obj-y += save.o obj-y += shutdown.o obj-y += softirq.o obj-y += sort.o
xen/common/save.c new file +339 -0
339 lines not on this page · raw message
xen/include/xen/save.h new file +160 -0
160 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v10 02/11] xen: introduce implementation of save/restore of 'domain context'
xen/common/save.c line 49
Quoted 10 lines · Paul Durrant
xen/common/save.c new file +10
-0+void __init domain_register_ctxt_type (unsigned int type,const char *name, +domain_save_ctxt_type save, +domain_load_ctxt_type load) +{ +BUG_ON (type >=ARRAY_SIZE (fns)); + +ASSERT (!fns[type].save); +ASSERT (!fns[type].load); + + fns[type].name = name;
I expect I merely didn't spot this (perhaps just latent) issue in earlier versions: If the caller lives in code getting built into *.init.o, the string passed in will live in .init.rodata. That's a general shortcoming (if you like) of the .o -> .init.o tranformation, but I see no good alternative (or else all format strings passed to printk() and alike won't get moved either). Therefore I wonder whether it wouldn't be safer to have the struct field be e.g. char[16], assuming 15 characters will allow for meaningful names.
xen/common/save.c line 173
Quoted 7 lines
xen/common/save.c new file +7
-0+int domain_load_ctxt_rec_data (struct domain_ctxt_state *c,void *dst, +size_t len) +{ +int rc =0 ; + + c->len += len; +if (c->len > c->rec.length)
Nit: Missing blanks.
Quoted 13 lines
xen/common/save.c new file +13
-0+int domain_load_ctxt (struct domain *d,const struct domain_load_ctxt_ops *ops, +void *priv) +{ +struct domain_ctxt_state c = { .d = d, .ops.load = ops, .priv = priv, }; +domain_load_ctxt_type load; +int rc; + +ASSERT (d != current->domain); + + rc = c.ops.load->read (c.priv, &c.rec,sizeof (c.rec)); +if ( rc ) +return rc; +
xen/common/save.c line 235
Quoted 10 lines
xen/common/save.c new file +10
-0+ load = fns[DOMAIN_CONTEXT_START ].load; +BUG_ON (!load); + + rc =load (d, &c); +if ( rc ) +return rc; + +domain_pause (d); + +for (;;)
Nit: Missing blanks again.
Quoted 6 lines
xen/common/save.c new file +6
-0+ { +unsigned int type; + + rc = c.ops.load->read (c.priv, &c.rec,sizeof (c.rec)); +if ( rc ) +break ;
xen/common/save.c line 251
Quoted 10 lines
xen/common/save.c new file +10
-0+ + type = c.rec.type; +if ( type ==DOMAIN_CONTEXT_END ) +break ; + + rc = -EINVAL ; +if ( type >=ARRAY_SIZE (fns) ) +break ; + + load = fns[type].load;
While this is meant to be used by Dom0 only, I think it would be better if it nevertheless used array_access_nospec().
Quoted 8 lines
xen/common/save.c new file +8
-0+static int load_start (struct domain *d,struct domain_ctxt_state *c) +{ +static struct domain_context_start s; +unsigned int i; +int rc =domain_load_ctxt_rec (c,DOMAIN_CONTEXT_START , &i, &s,sizeof (s)); + +if ( rc ) +return rc;
xen/common/save.c line 299
Quoted 10 lines
xen/common/save.c new file +10
-0+ +if ( i ) +return -EINVAL ; + +/* +* Make sure we are not attempting to load an image generated by a newer +* version of Xen. +*/ +if ( s.xen_major >XEN_VERSION && s.xen_minor >XEN_SUBVERSION ) +return -EOPNOTSUPP ;
Are you sure this needs to be excluded here and unilaterally? And if this is to stay, then it wants to be
if ( s.xen_major > XEN_VERSION ||
(s.xen_major == XEN_VERSION && s.xen_minor > XEN_SUBVERSION) )
return -EOPNOTSUPP;Quoted 8 lines
xen/include/xen/save.h new file +8
-0+/* +* Register save and load handlers for a record type. +* +* Save handlers will be invoked in an order which copes with any inter- +* entry dependencies. For now this means that HEADER will come first and +* END will come last, all others being invoked in order of 'typecode'. +* +* Load handlers will be invoked in the order of entries present in the
xen/include/xen/save.h line 114
Quoted 10 lines
xen/include/xen/save.h new file +10
-0+* buffer. +*/ +#define DOMAIN_REGISTER_CTXT_TYPE (x, s, l) \ + static int __init __domain_register_##x##_ctxt_type(void) \ + { \ + domain_register_ctxt_type( \ + DOMAIN_CONTEXT_ ## x, \ + #x, \ + &(s), \ + &(l)); \
I don't think there's a need for each of these to consume a separate line.
Jan
Re: [PATCH v10 02/11] xen: introduce implementation of save/restore of 'domain context'
In reply to Paul DurrantQuoted 14 lines · Paul Durrant
xen/common/save.c new file +7
-0@@ -0,0 +1,339 @@ +static int load_start (struct domain *d,struct domain_ctxt_state *c) +{ +static struct domain_context_start s; +unsigned int i; +int rc =domain_load_ctxt_rec (c,DOMAIN_CONTEXT_START , &i, &s,sizeof (s)); + +if ( rc )
xen/common/save.c line 298
Quoted 10 lines
xen/common/save.c new file +10
-0+return rc; + +if ( i ) +return -EINVAL ; + +/* +* Make sure we are not attempting to load an image generated by a newer +* version of Xen. +*/ +if ( s.xen_major >XEN_VERSION && s.xen_minor >XEN_SUBVERSION )
major > XEN_VERSON || (major == XEN_VERSION && minor > XEN_SUBVERSION)
~Andrew
PATCH v10 3/11 xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context
In reply to Paul DurrantThese domctls provide a mechanism to get and set 'domain context' from the toolstack. The implementation calls the domain_save_ctxt() and domain_load_ctxt() functions introduced in a previous patch.
---
Cc: Julien Grall <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v10:
- Re-base
- Add underscores and move to 64-bit image size as requested by Andrew
- Add a couple of record alignment ASSERTions
- Dropped R-b and A-b since changes are not entirely cosmetic
v4:
- Add missing zero pad checks
v3:
- Addressed comments from Julien and Jan
- Use vmalloc() rather than xmalloc_bytes()
v2:
- drop mask parameter
- const-ify some more buffers
--- tools/flask/policy/modules/xen.if | 4 +- tools/libs/ctrl/include/xenctrl.h | 5 + tools/libs/ctrl/xc_domain.c | 56 +++++++++ xen/common/domctl.c | 173 ++++++++++++++++++++++++++++ xen/include/public/domctl.h | 39 +++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 7 files changed, 285 insertions(+), 2 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
@@ -53,7 +53,7 @@ define(`create_domain_common', ` allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim set_vnumainfo get_vnumainfo cacheflush psr_cmt_op psr_alloc soft_reset - resource_map get_cpu_policy }; + resource_map get_cpu_policy set_context }; allow $1 $2:security check_context; allow $1 $2:shadow enable; allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', ` allow $1 $2:hvm { gethvmc getparam }; allow $1 $2:mmu { stat pageinfo map_read }; allow $1 $2:domain { getaddrsize getvcpucontext pause destroy }; - allow $1 $2:domain2 gettsc; + allow $1 $2:domain2 { gettsc get_context }; allow $1 $2:shadow { enable disable logdirty }; ')
tools/libs/ctrl/include/xenctrl.h +5 -0
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,uint8_t *hvm_ctxt, uint32_t size); +int xc_domain_get_context (xc_interface *xch,uint32_t domid, +void *ctxt_buf,size_t *size); +int xc_domain_set_context (xc_interface *xch,uint32_t domid, +const void *ctxt_buf,size_t size); +/** * This function will return guest IO ABI protocol *
tools/libs/ctrl/xc_domain.c +56 -0
62 lines not on this page · raw message
xen/common/domctl.c +173 -0
191 lines not on this page · raw message
xen/include/public/domctl.h +39 -0
@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {*/ }; +/* +* XEN_DOMCTL_get_domain_context +* ----------------------------- +* +* buffer (IN): The buffer into which the context data should be +* copied, or NULL to query the buffer size that should +* be allocated. +* size (IN/OUT): If 'buffer' is NULL then the value passed in must be +* zero, and the value passed out will be the size of the +* buffer to allocate. +* If 'buffer' is non-NULL then the value passed in must +* be the size of the buffer into which data may be copied. +* The value passed out will be the size of data written. +*/ +struct xen_domctl_get_domain_context { +uint64_t size; +XEN_GUEST_HANDLE_64 (void ) buffer; +}; + +/* XEN_DOMCTL_set_domain_context +* ----------------------------- +* +* buffer (IN): The buffer from which the context data should be +* copied. +* size (IN): The size of the buffer from which data may be copied. +* This data must include DOMAIN_SAVE_CODE_HEADER at the +* start and terminate with a DOMAIN_SAVE_CODE_END record. +* Any data beyond the DOMAIN_SAVE_CODE_END record will be +* ignored. +*/ +struct xen_domctl_set_domain_context { +uint64_t size; +XEN_GUEST_HANDLE_64 (const_void) buffer; +}; +struct xen_domctl {uint32_t cmd;#define XEN_DOMCTL_createdomain 1
@@ -1216,6 +1251,8 @@ struct xen_domctl {#define XEN_DOMCTL_vuart_op 81#define XEN_DOMCTL_get_cpu_policy 82#define XEN_DOMCTL_set_cpu_policy 83 +#define XEN_DOMCTL_get_domain_context 84 +#define XEN_DOMCTL_set_domain_context 85#define XEN_DOMCTL_gdbsx_guestmemio 1000#define XEN_DOMCTL_gdbsx_pausevcpu 1001#define XEN_DOMCTL_gdbsx_unpausevcpu 1002
@@ -1276,6 +1313,8 @@ struct xen_domctl {struct xen_domctl_monitor_op monitor_op;struct xen_domctl_psr_alloc psr_alloc;struct xen_domctl_vuart_op vuart_op; +struct xen_domctl_get_domain_context get_domain_context; +struct xen_domctl_set_domain_context set_domain_context;uint8_t pad[128 ]; } u; };
xen/xsm/flask/hooks.c +6 -0
@@ -754,6 +754,12 @@ static int flask_domctl(struct domain *d, int cmd)case XEN_DOMCTL_get_cpu_policy:return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_CPU_POLICY ); +case XEN_DOMCTL_set_domain_context: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__SET_CONTEXT ); + +case XEN_DOMCTL_get_domain_context: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_CONTEXT ); +default :return avc_unknown_permission ("domctl" , cmd); }
xen/xsm/flask/policy/access_vectors +4 -0
@@ -245,6 +245,10 @@ class domain2 resource_map # XEN_DOMCTL_get_cpu_policy get_cpu_policy +# XEN_DOMCTL_set_domain_context + set_context +# XEN_DOMCTL_get_domain_context + get_context } # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
Re: [PATCH v10 03/11] xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context
xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 312
Quoted 10 lines · Paul Durrant
xen/common/domctl.c +10
-0+static int dry_run_end (void *priv,size_t len) +{ +struct domctl_context *c = priv; + +ASSERT (IS_ALIGNED (c->len,DOMAIN_CONTEXT_RECORD_ALIGN )); + +return 0 ; +} + +static struct domain_save_ctxt_ops dry_run_ops = {
const? (same for save_ops and load_ops then)
Quoted 12 lines
xen/include/public/domctl.h +6
-0@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {*/ }; +/* +* XEN_DOMCTL_get_domain_context +* ----------------------------- +* +* buffer (IN): The buffer into which the context data should be +* copied, or NULL to query the buffer size that should
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150
Quoted 10 lines
xen/include/public/domctl.h +10
-0+* be allocated. +* size (IN/OUT): If 'buffer' is NULL then the value passed in must be +* zero, and the value passed out will be the size of the +* buffer to allocate. +* If 'buffer' is non-NULL then the value passed in must +* be the size of the buffer into which data may be copied. +* The value passed out will be the size of data written. +*/ +struct xen_domctl_get_domain_context { +uint64_t size;
uint64_aligned_t (also again below)?
With these adjusted
Jan
Re: [PATCH v10 03/11] xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_context
Quoted 39 lines · Jan Beulich
xen/common/domctl.c static struct vnuma_info *vnuma_init(const struct xen_domctl_vnuma *uinfo, · line 312
Quoted 10 lines · Paul Durrant
xen/common/domctl.c +10
-0+static int dry_run_end (void *priv,size_t len) +{ +struct domctl_context *c = priv; + +ASSERT (IS_ALIGNED (c->len,DOMAIN_CONTEXT_RECORD_ALIGN )); + +return 0 ; +} + +static struct domain_save_ctxt_ops dry_run_ops = {const? (same for
save_opsandload_opsthen)Quoted 12 lines
xen/include/public/domctl.h +6
-0@@ -1132,6 +1132,41 @@ struct xen_domctl_vuart_op {*/ }; +/* +* XEN_DOMCTL_get_domain_context +* ----------------------------- +* +* buffer (IN): The buffer into which the context data should be +* copied, or NULL to query the buffer size that should xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1150
Quoted 10 lines
xen/include/public/domctl.h +10
-0+* be allocated. +* size (IN/OUT): If 'buffer' is NULL then the value passed in must be +* zero, and the value passed out will be the size of the +* buffer to allocate. +* If 'buffer' is non-NULL then the value passed in must +* be the size of the buffer into which data may be copied. +* The value passed out will be the size of data written. +*/ +struct xen_domctl_get_domain_context { +uint64_t size;
uint64_aligned_t(also again below)?
Quoted 2 lines
With these adjusted
Reviewed-by Jan Beulich <[email protected]>
FAOD: Non-XSM hypervisor pieces only.
Jan
PATCH v10 4/11 tools/misc: add xen-domctx to present domain context
In reply to Paul DurrantThis tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
---
Cc: Andrew Cooper <[email protected]>
v10:
- Re-base
- Use err[x]()
- Keep existing A-b since modifications are trivial
v3:
- Re-worked to avoid copying onto stack
- Added optional typecode and instance arguments
v2:
- Change name from 'xen-ctx' to 'xen-domctx'
--- .gitignore | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 172 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 177 insertions(+) create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
@@ -243,6 +243,7 @@ tools/misc/xen_cpuperf tools/misc/xen-cpuid tools/misc/xen-detect tools/misc/xen-diag +tools/misc/xen-domctx tools/misc/xen-tmem-list-parse tools/misc/xen-livepatch tools/misc/xenperf
tools/misc/Makefile +4 -0
@@ -32,6 +32,7 @@ INSTALL_SBIN += xenpm INSTALL_SBIN += xenwatchdogd INSTALL_SBIN += xen-livepatch INSTALL_SBIN += xen-diag +INSTALL_SBIN += xen-domctx INSTALL_SBIN += $(INSTALL_SBIN-y) # Everything to be installed in a private bin/
@@ -111,6 +112,9 @@ xen-livepatch: xen-livepatch.o xen-diag: xen-diag.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) +xen-domctx: xen-domctx.o + $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) + xen-lowmemd: xen-lowmemd.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
tools/misc/xen-domctx.c new file +172 -0
172 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v10 5/11 common/domain: add a domain context record for shared_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.
---
Cc: Jan Beulich <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v10:
- Re-base
- Amend the specification now there is one
- Dropped Jan's R-b as modifications are not completely trivial
v9:
- Use macros to make the code less verbose
- Add missing check for allocation failure
v8:
- Incorporate zero-ing out of shared info fields that would be done in processing of SHARED_INFO from older stream versions
v7:
- Only restore vcpu_info and arch sub-structures for PV domains, to match processing of SHARED_INFO in xc_sr_restore_x86_pv.c
- Use additional option to domain_load_end() to ignore the record for HVM domains
v6:
- Only save compat_shared_info buffer if has_32bit_shinfo is set
- Validate flags field in load handler
v5:
- Addressed comments from Julien
v4:
- Addressed comments from Jan
v3:
- Actually dump some of the content of shared_info
v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT
--- docs/specs/domain-context.md | 29 +++++++++ tools/misc/xen-domctx.c | 80 +++++++++++++++++++++++++ xen/common/domain.c | 113 +++++++++++++++++++++++++++++++++++ xen/include/public/save.h | 11 ++++ 4 files changed, 233 insertions(+)
docs/specs/domain-context.md +29 -0
@@ -128,6 +128,33 @@ can no longer be safely inferred. A record of this type terminates the image. No further data from the buffer should be consumed. +### SHARED_INFO + +``` + 0 1 2 3 4 5 6 7 octet ++-------+-------+-------+-------+-------+-------+-------+-------+ +| type == 2 | instance == 0 | ++-------------------------------+-------------------------------+ +| length | ++-------------------------------+-------------------------------+ +| flags | buffer ++-------------------------------+ +... +``` + +\pagebreak +The record body contains the following fields: + +| Field | Description | +|-------------|-------------------------------------------------| +| `flags` | A bit-wise OR of the following: | +| | | +| | 0x00000001: The domain has 32-bit (compat) | +| | shared info | +| | | +| `buffer` | The shared info (`length` being architecture | +| | dependent[4]) | + * * * [1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md
@@ -135,3 +162,5 @@ should be consumed. [2] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/hvm/save.h [3] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/specs/libxc-migration-stream.pandoc + +[4] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/include/xen-foreign/reference.size
tools/misc/xen-domctx.c +80 -0
92 lines not on this page · raw message
xen/common/domain.c +113 -0
125 lines not on this page · raw message
xen/include/public/save.h +11 -0
@@ -49,6 +49,7 @@ struct domain_context_record {enum {DOMAIN_CONTEXT_END ,DOMAIN_CONTEXT_START , +DOMAIN_CONTEXT_SHARED_INFO ,/* New types go here */ DOMAIN_CONTEXT_NR_TYPES };
@@ -58,6 +59,16 @@ struct domain_context_start {uint32_t xen_major, xen_minor; }; +struct domain_context_shared_info { +uint32_t flags; + +#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0 +#define DOMAIN_CONTEXT_32BIT_SHARED_INFO \ + (1U << _DOMAIN_CONTEXT_32BIT_SHARED_INFO) + +uint8_t buffer[XEN_FLEX_ARRAY_DIM ]; +}; +/* Terminating entry */ struct domain_context_end {};
Signature · 1 line
2.20.1
Re: [PATCH v10 05/11] common/domain: add a domain context record for shared_info...
Quoted 7 lines · Paul Durrant
xen/common/domain.c +3
-0@@ -1671,6 +1672,118 @@ int continue_hypercall_on_cpu(return 0 ; } +static int save_shared_info (struct domain *d,struct domain_ctxt_state *c, +bool dry_run) +{
xen/common/domain.c int continue_hypercall_on_cpu( · line 1687
Quoted 10 lines
xen/common/domain.c +10
-0+#ifdef CONFIG_COMPAT +struct domain_context_shared_info s = { + .flags =has_32bit_shinfo (d) ?DOMAIN_CONTEXT_32BIT_SHARED_INFO :0 , + }; +size_t size =has_32bit_shinfo (d) ? +sizeof (struct compat_shared_info ) : +sizeof (struct shared_info ); +#else +struct domain_context_shared_info s = {}; +size_t size =sizeof (struct shared_info );
All of these would imo better be expressed in terms of d->shared_info. While chances are zero that these types will change in any way, it still sets a bad precedent for people seeing this and then introducing similar disconnects elsewhere. (Same in the load handling then.)
Quoted 16 lines
xen/common/domain.c +16
-0+static int load_shared_info (struct domain *d,struct domain_ctxt_state *c) +{ +struct domain_context_shared_info s = {}; +size_t size; +unsigned int i; +int rc; + + rc =domain_load_ctxt_rec_begin (c,DOMAIN_CONTEXT_SHARED_INFO , &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_ctxt_rec_data (c, &s, offsetof(typeof (s ), buffer)); +if ( rc )
xen/common/domain.c int continue_hypercall_on_cpu( · line 1731
Quoted 10 lines
xen/common/domain.c +10
-0+return rc; + +if ( s.flags & ~DOMAIN_CONTEXT_32BIT_SHARED_INFO ) +return -EINVAL ; + +if ( s.flags &DOMAIN_CONTEXT_32BIT_SHARED_INFO ) + { +#ifdef CONFIG_COMPAT + d->arch.has_32bit_shinfo = true; + size =sizeof (struct compat_shared_info );
I realize this has been more or less this way already in prior versions, but aren't you introducing a way to have a degenerate 64-bit PV guest with 32-bit shared info (or vice versa), in that shared info bitness isn't strictly tied to guest bitness anymore? Rejecting this case may not need to live here, but it needs to be present / added somewhere.
Quoted 23 lines
xen/common/domain.c +23
-0+#else +return -EINVAL ; +#endif + } +else + size =sizeof (struct shared_info ); + +if (is_pv_domain (d) ) + { +shared_info_t *shinfo =xzalloc (shared_info_t); + +if ( !shinfo ) +return -ENOMEM ; + + rc =domain_load_ctxt_rec_data (c, shinfo, size); +if ( rc ) + goto out; + +memcpy (&shared_info (d, vcpu_info), &__shared_info (d, shinfo, vcpu_info), +sizeof (shared_info (d, vcpu_info))); +memcpy (&shared_info (d, arch), &__shared_info (d, shinfo, arch), +sizeof (shared_info (d, arch))); +
xen/common/domain.c int continue_hypercall_on_cpu( · line 1764
Quoted 10 lines
xen/common/domain.c +10
-0+memset (&shared_info (d, evtchn_pending),0 , +sizeof (shared_info (d, evtchn_pending))); +memset (&shared_info (d, evtchn_mask),0xff , +sizeof (shared_info (d, evtchn_mask))); + +#ifdef CONFIG_X86 +shared_info (d, arch.pfn_to_mfn_frame_list_list) =0 ; +#endif +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) +shared_info (d, vcpu_info[i].evtchn_pending_sel) =0 ;
Again I realize this has been this way in earlier versions, and it was also me to ask for streamlining the code, but is this actually correct? I ask in particular in light of this comment
/*
* Compat field is never larger than native field, so cast to that as it
* is the largest memory range it is safe for the caller to modify without
* further discrimination between compat and native cases.
*/
in xen/shared.h, next to the __shared_info() #define. I can't help thinking that you'll fill only half of some of the fields in the 64-bit case.
xen/include/public/save.h struct domain_context_start { · line 65
Quoted 8 lines
xen/include/public/save.h +4
-0@@ -58,6 +59,16 @@ struct domain_context_start {uint32_t xen_major, xen_minor; }; +struct domain_context_shared_info { +uint32_t flags; + +#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0
Is this separate constant actually needed for anything?
Speaking of which - wouldn't all your additions to this header better be proper name spacing citizens, by having xen_ / XEN_ prefixes?
Jan
Re: [PATCH v10 05/11] common/domain: add a domain context record for shared_info...
In reply to Paul DurrantQuoted 5 lines · Paul Durrant
xen/include/public/save.h +0
-0
xen/include/public/save.h struct domain_context_start { · line 67
Quoted 9 lines
xen/include/public/save.h +6
-0@@ -58,6 +59,16 @@ struct domain_context_start {uint32_t xen_major, xen_minor; }; +struct domain_context_shared_info { +uint32_t flags; + +#define _DOMAIN_CONTEXT_32BIT_SHARED_INFO 0 +#define DOMAIN_CONTEXT_32BIT_SHARED_INFO \ + (1U << _DOMAIN_CONTEXT_32BIT_SHARED_INFO)
There is no need for the logarithm version of this constant.
You do however want an explicit uint32_t _rsvd; so buffer[] doesn't start at the wrong alignment for an efficient memcpy() in 64bit builds of Xen.
~Andrew
PATCH v10 6/11 x86/time: add a domain context record for tsc_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Whilst the record is x86 specific, it is visible directly in the common
header as context record numbers should be unique across all architectures.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v10:
- Re-base
- Amend the specification now there is one
- Kept Jan's R-b as the changes are cosmetic
v8:
- Removed stray blank line
v7:
- New in v7
--- docs/specs/domain-context.md | 32 ++++++++++++++++++++++++++++++++ tools/misc/xen-domctx.c | 12 ++++++++++++ xen/arch/x86/time.c | 30 ++++++++++++++++++++++++++++++ xen/include/public/save.h | 8 ++++++++ 4 files changed, 82 insertions(+)
docs/specs/domain-context.md +32 -0
@@ -155,6 +155,38 @@ The record body contains the following fields: | `buffer` | The shared info (`length` being architecture | | | dependent[4]) | +### TSC_INFO + +``` + 0 1 2 3 4 5 6 7 octet ++-------+-------+-------+-------+-------+-------+-------+-------+ +| type == 3 | instance == 0 | ++-------------------------------+-------------------------------+ +| length == 20 | ++-------------------------------+-------------------------------+ +| mode | khz | ++-------------------------------+-------------------------------+ +| nsec | ++-------------------------------+-------------------------------+ +| incarnation | ++-------------------------------+ +``` + +The record body contains the following fields: + +| Field | Description | +|---------------|-----------------------------------------------| +| `mode` | 0x00000000: Default (emulate if necessary) | +| | 0x00000001: Always emulate | +| | 0x00000002: Never emulate | +| | | +| `khz` | The TSC frequency in kHz | +| | | +| `nsec` | Elapsed time in nanoseconds | +| | | +| `incarnation` | Incarnation (counter indicating how many | +| | times the TSC value has been set) | + * * * [1] See https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/designs/non-cooperative-migration.md
tools/misc/xen-domctx.c +12 -0
@@ -136,6 +136,17 @@ static void dump_shared_info(void)#undef GET_FIELD_PTR } +static void dump_tsc_info (void ) +{ +struct domain_context_tsc_info *t; + +GET_PTR (t); + +printf (" TSC_INFO: mode: %u incarnation: %u\n" +" khz %u nsec: %" PRIu64"\n" , + t->mode, t->incarnation, t->khz, t->nsec); +} +static void dump_end (void ) {struct domain_context_end *e;
@@ -225,6 +236,7 @@ int main(int argc, char **argv) {case DOMAIN_CONTEXT_START :dump_start ();break ;case DOMAIN_CONTEXT_SHARED_INFO :dump_shared_info ();break ; +case DOMAIN_CONTEXT_TSC_INFO :dump_tsc_info ();break ;case DOMAIN_CONTEXT_END :dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , rec->type);
xen/arch/x86/time.c +30 -0
@@ -26,6 +26,7 @@#include <xen/symbols.h> #include <xen/keyhandler.h> #include <xen/guest_access.h> +#include <xen/save.h> #include <asm/io.h> #include <asm/iocap.h> #include <asm/msr.h>
@@ -2451,6 +2452,35 @@ int tsc_set_info(struct domain *d,return 0 ; } +static int save_tsc_info (struct domain *d,struct domain_ctxt_state *c, +bool dry_run) +{ +struct domain_context_tsc_info t = {}; + +if ( !dry_run ) +tsc_get_info (d, &t.mode, &t.nsec, &t.khz, &t.incarnation); + +return domain_save_ctxt_rec (c,DOMAIN_CONTEXT_TSC_INFO ,0 , &t,sizeof (t)); +} + +static int load_tsc_info (struct domain *d,struct domain_ctxt_state *c) +{ +struct domain_context_tsc_info t; +unsigned int i; +int rc; + + rc =domain_load_ctxt_rec (c,DOMAIN_CONTEXT_TSC_INFO , &i, &t,sizeof (t)); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + +return tsc_set_info (d, t.mode, t.nsec, t.khz, t.incarnation); +} + +DOMAIN_REGISTER_CTXT_TYPE (TSC_INFO , save_tsc_info, load_tsc_info); +/* vtsc may incur measurable performance degradation, diagnose with this */ static void dump_softtsc (unsigned char key) {
xen/include/public/save.h +8 -0
@@ -50,6 +50,7 @@ enum {DOMAIN_CONTEXT_END ,DOMAIN_CONTEXT_START ,DOMAIN_CONTEXT_SHARED_INFO , +DOMAIN_CONTEXT_TSC_INFO ,/* New types go here */ DOMAIN_CONTEXT_NR_TYPES };
@@ -69,6 +70,13 @@ struct domain_context_shared_info {uint8_t buffer[XEN_FLEX_ARRAY_DIM ]; }; +struct domain_context_tsc_info { +uint32_t mode; +uint32_t khz; +uint64_t nsec; +uint32_t incarnation; +}; +/* Terminating entry */ struct domain_context_end {};
Signature · 1 line
2.20.1
Re: [PATCH v10 06/11] x86/time: add a domain context record for tsc_info...
Quoted 4 lines · Paul Durrant
docs/specs/domain-context.md +4
-0+The record body contains the following fields: + +| Field | Description | +|---------------|-----------------------------------------------|
docs/specs/domain-context.md The record body contains the following fields: · line 188
Quoted 10 lines
docs/specs/domain-context.md +10
-0+| `mode` | 0x00000000: Default (emulate if necessary) | +| | 0x00000001: Always emulate | +| | 0x00000002: Never emulate | +| | | +| `khz` | The TSC frequency in kHz | +| | | +| `nsec` | Elapsed time in nanoseconds | +| | | +| `incarnation` | Incarnation (counter indicating how many | +| | times the TSC value has been set) |
It is how many set_tsc_info() (hyper)calls have been made, not how many times the guest has written to the TSC MSR.
That said, it is totally useless now that PVRDTSCP mode has gone, other than the fact that it appears in guest CPUID as an approximation of "how many times has this domain been migrated".
I.e. its a number you'll want to actively squash in your usecase.
I'm not sure whether to suggest dropping the field entirely, or not. I highly doubt any user exists - IIRC, it was specifically for PVRDTSCP userspace to notice that the frequency may have changed, and to re-adjust its calculations.
xen/include/public/save.h enum { · line 53
Quoted 9 lines
xen/include/public/save.h +1
-0@@ -50,6 +50,7 @@ enum {DOMAIN_CONTEXT_END ,DOMAIN_CONTEXT_START ,DOMAIN_CONTEXT_SHARED_INFO , + DOMAIN_CONTEXT_TSC_INFO,
At a minimum, this wants an /* x86 only */ comment. Possibly an X86 infix.
~Andrew
PATCH v10 7/11 docs/specs: add missing definitions to libxc-migration-stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
docs/specs/libxc-migration-stream.pandoc +9 -2
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 2 +% Revision 3 Introduction ============
@@ -227,7 +227,13 @@ type 0x00000000: END 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary) - 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000010: STATIC_DATA_END + + 0x00000011: X86_CPUID_POLICY + + 0x00000012: X86_MSR_POLICY + + 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -732,6 +738,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_TSC_INFO * HVM_PARAMS * HVM_CONTEXT +* END record HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect the validity of architectural state in the context.
Signature · 1 line
2.20.1
Re: [PATCH v10 07/11] docs/specs: add missing definitions to libxc-migration-stream
Quoted 6 lines · Paul Durrant
From: Paul Durrant <[email protected]>
The
STATIC_DATA_END,X86_CPUID_POLICYandX86_MSR_POLICYrecord types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Quoted 8 lines
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
Signed-off-by Paul Durrant <[email protected]>Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")Acked-by Wei Liu <[email protected]>
I've committed this. I have no idea where these hunks got lost, because I definitely did have them at some point during the mig-v3 work.
~Andrew
PATCH v10 8/11 docs / tools: specify migration v4 to include DOMAIN_CONTEXT
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informition so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.
---
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
NOTE: Wei requested ack from Andrew
v10:
- Removed changes to xg_sr_common.c and libxc.py to make this a just documentation and header patch
- Dropped Wei's A-b in light of change
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 62 ++++++++++++++++++------ tools/libs/guest/xg_sr_stream_format.h | 1 + 2 files changed, 49 insertions(+), 14 deletions(-)
docs/specs/libxc-migration-stream.pandoc +48 -14
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 3 +% Revision 4 Introduction ============
@@ -127,7 +127,7 @@ marker 0xFFFFFFFFFFFFFFFF. id 0x58454E46 ("XENF" in ASCII). -version 0x00000003. The version of this specification. +version 0x00000004. The version of this specification. options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
@@ -209,9 +209,9 @@ type 0x00000000: END 0x00000006: X86_PV_VCPU_XSAVE - 0x00000007: SHARED_INFO + 0x00000007: SHARED_INFO (deprecated) - 0x00000008: X86_TSC_INFO + 0x00000008: X86_TSC_INFO (deprecated) 0x00000009: HVM_CONTEXT
@@ -233,7 +233,9 @@ type 0x00000000: END 0x00000012: X86_MSR_POLICY - 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000013: DOMAIN_CONTEXT + + 0x00000014 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs \clearpage -SHARED_INFO ------------ +SHARED_INFO (deprecated) +------------------------ -The content of the Shared Info page. +The content of the Shared Info page. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info Contents of the shared info page. This record \clearpage -X86_TSC_INFO ------------- +X86_TSC_INFO (deprecated) +------------------------- Domain TSC information, as accessed by the -XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. +XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +------------------------+------------------------+
@@ -680,6 +684,25 @@ MSR_policy Array of xen_msr_entry_t[]'s \clearpage +DOMAIN_CONTEXT +-------------- + +Domain context, as accessed by the +XEN_DOMCTL_{get,set}_domain_context hypercall sub-ops. + + 0 1 2 3 4 5 6 7 octet + +-------------------------------------------------+ + | dom_ctx | + ... + +-------------------------------------------------+ + +-------------------------------------------------------------------- +Field Description +----------- --------------------------------------------------- +dom_ctx The Domain Context blob from Xen. +-------------------------------------------------------------------- + +\clearpage Layout ======
@@ -706,8 +729,7 @@ A typical save record for an x86 PV guest image would look like: * STATIC_DATA_END * X86_PV_P2M_FRAMES record * Many PAGE_DATA records -* X86_TSC_INFO -* SHARED_INFO record +* DOMAIN_CONTEXT * VCPU context records for each online VCPU * X86_PV_VCPU_BASIC record * X86_PV_VCPU_EXTENDED record
@@ -735,7 +757,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_{CPUID,MSR}_POLICY * STATIC_DATA_END * Many PAGE_DATA records -* X86_TSC_INFO +* DOMAIN_CONTEXT * HVM_PARAMS * HVM_CONTEXT * END record
@@ -746,6 +768,18 @@ the validity of architectural state in the context. Compatibility with older versions ================================= +v4 compat with v3 +----------------- + +A v4 stream is compatible with a v3 stream, but mandates the presence of a +DOMAIN_CONTEXT record. This incorporates context such as the content of +the domain's Shared Info page and the TSC information, hence the SHARED_INFO +and TSC_INFO records are deprecated. +It also supercedes HVM_CONTEXT and, over time, data that is currently part of +the HVM_CONTEXT blob will move to the DOMAIN_CONTEXT blob. Xen, however, will +continue to accept all defined HVM_CONTEXT records so a v4-compatible +receiver can still accept an unmodified v3 stream. + v3 compat with v2 -----------------
tools/libs/guest/xg_sr_stream_format.h +1 -0
@@ -76,6 +76,7 @@ struct xc_sr_rhdr#define REC_TYPE_STATIC_DATA_END 0x00000010U#define REC_TYPE_X86_CPUID_POLICY 0x00000011U#define REC_TYPE_X86_MSR_POLICY 0x00000012U +#define REC_TYPE_DOMAIN_CONTEXT 0x00000013U#define REC_TYPE_OPTIONAL 0x80000000U
Signature · 1 line
2.20.1
Re: [PATCH v10 08/11] docs / tools: specify migration v4 to include DOMAIN_CONTEXT
Quoted 14 lines · Paul Durrant
docs/specs/libxc-migration-stream.pandoc +1
-1@@ -127,7 +127,7 @@ marker 0xFFFFFFFFFFFFFFFF. id 0x58454E46 ("XENF" in ASCII). -version 0x00000003. The version of this specification. +version 0x00000004. The version of this specification. options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
docs/specs/libxc-migration-stream.pandoc type 0x00000000: END · line 212
Quoted 5 lines
docs/specs/libxc-migration-stream.pandoc +1
-1@@ -209,9 +209,9 @@ type 0x00000000: END 0x00000006: X86_PV_VCPU_XSAVE - 0x00000007: SHARED_INFO + 0x00000007: SHARED_INFO (deprecated)
"in v4" ?
docs/specs/libxc-migration-stream.pandoc X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs · line 451
Quoted 12 lines
docs/specs/libxc-migration-stream.pandoc +4
-3@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs \clearpage -SHARED_INFO ------------ +SHARED_INFO (deprecated) +------------------------ -The content of the Shared Info page. +The content of the Shared Info page. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4.
This needs to be stricter. A SHARED_INFO frame must not be present in a v4 stream.
Absolutely nothing good can come from having the state twice. Moreover, ...
Quoted 6 lines
docs/specs/libxc-migration-stream.pandoc +0
-00 1 2 3 4 5 6 7 octet +-------------------------------------------------+@@ -462,11 +465,12 @@ shared_info Contents of the shared info page. This record \clearpage
docs/specs/libxc-migration-stream.pandoc shared_info Contents of the shared info page. This record · line 473
Quoted 10 lines
docs/specs/libxc-migration-stream.pandoc +4
-3-X86_TSC_INFO ------------- +X86_TSC_INFO (deprecated) +------------------------- Domain TSC information, as accessed by the -XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. +XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4.
... it is actively problematic for this one, as incarnation counts the number of set_tsc_info() hypercalls.
~Andrew
PATCH v10 9/11 tools/python: modify libxc.py to verify v4 stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
This patch adds code to verify the presence of a REC_TYPE_domain_context record in a v4 stream, as well as absence of REC_TYPE_shared_info and REC_TYPE_tsc_info records.
---
Cc: Andrew Cooper <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
v10:
- New in v10
--- tools/python/xen/migration/libxc.py | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
tools/python/xen/migration/libxc.py +20 -2
@@ -59,6 +59,7 @@ REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f REC_TYPE_static_data_end = 0x00000010 REC_TYPE_x86_cpuid_policy = 0x00000011 REC_TYPE_x86_msr_policy = 0x00000012 +REC_TYPE_domain_context = 0x00000013 rec_type_to_str = { REC_TYPE_end : "End",
@@ -80,6 +81,7 @@ rec_type_to_str = { REC_TYPE_static_data_end : "Static data end", REC_TYPE_x86_cpuid_policy : "x86 CPUID policy", REC_TYPE_x86_msr_policy : "x86 MSR policy", + REC_TYPE_domain_context : "Domain context", } # page_data
@@ -156,9 +158,9 @@ class VerifyLibxc(VerifyBase): raise StreamError("Bad image id: Expected 0x%x, got 0x%x" % (IHDR_IDENT, ident)) - if not (2 <= version <= 3): + if not (2 <= version <= 4): raise StreamError( - "Unknown image version: Expected 2 <= ver <= 3, got %d" % + "Unknown image version: Expected 2 <= ver <= 4, got %d" % (version, )) self.version = version
@@ -362,6 +364,9 @@ class VerifyLibxc(VerifyBase): def verify_record_shared_info(self, content): """ shared info record """ + if self.version >= 4: + raise RecordError("Shared info record found in v4 stream") + contentsz = len(content) if contentsz != 4096: raise RecordError("Length expected to be 4906 bytes, not %d" %
@@ -371,6 +376,9 @@ class VerifyLibxc(VerifyBase): def verify_record_tsc_info(self, content): """ tsc info record """ + if self.version >= 4: + raise RecordError("TSC info record found in v4 stream") + sz = calcsize(X86_TSC_INFO_FORMAT) if len(content) != sz:
@@ -476,6 +484,14 @@ class VerifyLibxc(VerifyBase): raise RecordError("Record length %u, expected multiple of %u" % (contentsz, sz)) + def verify_record_domain_context(self, content): + """ domain context record """ + + if self.version < 4: + raise RecordError("Domain context record found in v3 stream") + + if len(content) == 0: + raise RecordError("Zero length domain context") record_verifiers = { REC_TYPE_end:
@@ -526,4 +542,6 @@ record_verifiers = { VerifyLibxc.verify_record_x86_cpuid_policy, REC_TYPE_x86_msr_policy: VerifyLibxc.verify_record_x86_msr_policy, + REC_TYPE_domain_context: + VerifyLibxc.verify_record_domain_context, }
Signature · 1 line
2.20.1
Re: [PATCH v10 09/11] tools/python: modify libxc.py to verify v4 stream
tools/python/xen/migration/libxc.py class VerifyLibxc(VerifyBase): · line 494
Quoted 12 lines · Paul Durrant
tools/python/xen/migration/libxc.py +8
-0@@ -476,6 +484,14 @@ class VerifyLibxc(VerifyBase): raise RecordError("Record length %u, expected multiple of %u" % (contentsz, sz)) + def verify_record_domain_context(self, content): + """ domain context record """ + + if self.version < 4: + raise RecordError("Domain context record found in v3 stream") + + if len(content) == 0: + raise RecordError("Zero length domain context")
This needs a recursive dissector to validate the domain context format, as it is not a private ABI within Xen.
~Andrew
PATCH v10 10/11 tools/libs/guest: add code to restore a v4 libxc stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
This patch adds the necessary code to accept a v4 stream, and to recognise and restore a REC_TYPE_DOMAIN_CONTEXT record.
---
Cc: Andrew Cooper <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
v10:
- New in v10
- Derived from patch #8 of the v9 series
- Fix memory leak
--- tools/libs/guest/xg_sr_common.c | 1 + tools/libs/guest/xg_sr_common.h | 3 +++ tools/libs/guest/xg_sr_restore.c | 24 ++++++++++++++++++++++-- tools/libs/guest/xg_sr_restore_x86_hvm.c | 9 +++++++++ tools/libs/guest/xg_sr_restore_x86_pv.c | 9 +++++++++ 5 files changed, 44 insertions(+), 2 deletions(-)
tools/libs/guest/xg_sr_common.c +1 -0
@@ -39,6 +39,7 @@ static const char *const mandatory_rec_types[] = [REC_TYPE_STATIC_DATA_END] = "Static data end", [REC_TYPE_X86_CPUID_POLICY] = "x86 CPUID policy", [REC_TYPE_X86_MSR_POLICY] = "x86 MSR policy", + [REC_TYPE_DOMAIN_CONTEXT] = "Domain context", };const char *rec_type_to_str (uint32_t type)
tools/libs/guest/xg_sr_common.h +3 -0
@@ -297,6 +297,9 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; + +/* Domain context blob. */ +struct xc_sr_blob dom_ctx; } restore; };
tools/libs/guest/xg_sr_restore.c +22 -2
@@ -35,9 +35,9 @@ static int read_headers(struct xc_sr_context *ctx)return -1 ; } -if ( ihdr.version <2 || ihdr.version >3 ) +if ( ihdr.version <2 || ihdr.version >4 ) { -ERROR ("Invalid Version: Expected 2 <= ver <= 3, Got %d" , +ERROR ("Invalid Version: Expected 2 <= ver <= 4, Got %d" , ihdr.version);return -1 ; }
@@ -682,6 +682,21 @@ int handle_static_data_end(struct xc_sr_context *ctx)return rc; } +/* +* Process a DOMAIN_CONTEXT record from the stream. +*/ +static int handle_domain_context (struct xc_sr_context *ctx, +struct xc_sr_record *rec) +{ +xc_interface *xch = ctx->xch; +int rc =update_blob (&ctx->restore.dom_ctx, rec->data, rec->length); + +if ( rc ) +ERROR ("Unable to allocate %u bytes for domain context" , rec->length); + +return rc; +} +static int process_record (struct xc_sr_context *ctx,struct xc_sr_record *rec) {xc_interface *xch = ctx->xch;
@@ -709,6 +724,10 @@ static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec) rc =handle_static_data_end (ctx);break ; +case REC_TYPE_DOMAIN_CONTEXT : + rc =handle_domain_context (ctx, rec); +break ; +default : rc = ctx->restore.ops.process_record (ctx, rec);break ;
@@ -784,6 +803,7 @@ static void cleanup(struct xc_sr_context *ctx)free (ctx->restore.buffered_records);free (ctx->restore.populated_pfns); +free (ctx->restore.dom_ctx.ptr);if ( ctx->restore.ops.cleanup (ctx) )PERROR ("Failed to clean up" );
tools/libs/guest/xg_sr_restore_x86_hvm.c +9 -0
@@ -225,6 +225,15 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)return rc; } + rc =xc_domain_set_context (xch, ctx->domid, + ctx->restore.dom_ctx.ptr, + ctx->restore.dom_ctx.size); +if ( rc ) + { +PERROR ("Unable to restore Domain context" ); +return rc; + } + rc = xc_dom_gnttab_seed(xch, ctx->domid, true, ctx->restore.console_gfn, ctx->restore.xenstore_gfn,
tools/libs/guest/xg_sr_restore_x86_pv.c +9 -0
@@ -1134,6 +1134,15 @@ static int x86_pv_stream_complete(struct xc_sr_context *ctx)if ( rc )return rc; + rc =xc_domain_set_context (xch, ctx->domid, + ctx->restore.dom_ctx.ptr, + ctx->restore.dom_ctx.size); +if ( rc ) + { +PERROR ("Unable to restore Domain context" ); +return rc; + } + rc = xc_dom_gnttab_seed(xch, ctx->domid, false, ctx->restore.console_gfn, ctx->restore.xenstore_gfn,
Signature · 1 line
2.20.1
Re: [PATCH v10 10/11] tools/libs/guest: add code to restore a v4 libxc stream
Quoted 6 lines · Paul Durrant
From: Paul Durrant <[email protected]>
This patch adds the necessary code to accept a v4 stream, and to recognise and restore a
REC_TYPE_DOMAIN_CONTEXTrecord.Signed-off-by Paul Durrant <[email protected]>
Somewhere within this needs to be logic to reject the forbidden records in relevant stream versions.
Quoted 6 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +0
-0@@ -225,6 +225,15 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)return rc;
tools/libs/guest/xg_sr_restore_x86_hvm.c static int x86_hvm_stream_complete(struct xc_sr_context *ctx) · line 235
Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +8
-0} + rc =xc_domain_set_context (xch, ctx->domid, + ctx->restore.dom_ctx.ptr, + ctx->restore.dom_ctx.size); +if ( rc ) + { +PERROR ("Unable to restore Domain context" ); +return rc; + }
This doesn't match where you specified the record to live in the stream, and in particular is reordered WRT HVMCONTEXT restoration.
Also, it appears to be in the middle of a block of code which needs to become if ( guest-aware ).
Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_hvm.c +1
-0+ rc = xc_dom_gnttab_seed(xch, ctx->domid, true, ctx->restore.console_gfn, ctx->restore.xenstore_gfn,tools/libs/guest/xg_sr_restore_x86_pv.c +0
-0@@ -1134,6 +1134,15 @@ static int x86_pv_stream_complete(struct xc_sr_context *ctx)if ( rc )
tools/libs/guest/xg_sr_restore_x86_pv.c static int x86_pv_stream_complete(struct xc_sr_context *ctx) · line 1144
Quoted 10 lines
tools/libs/guest/xg_sr_restore_x86_pv.c +8
-0return rc; + rc =xc_domain_set_context (xch, ctx->domid, + ctx->restore.dom_ctx.ptr, + ctx->restore.dom_ctx.size); +if ( rc ) + { +PERROR ("Unable to restore Domain context" ); +return rc; + }
Similar comment as HVM for the reordering. PV guests in particular tend to be far more sensitive to the restoration order.
~Andrew
PATCH v10 11/11 tools/libs/guest: add code to save a v4 libxc stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
This patch adds the necessary code to save a REC_TYPE_DOMAIN_CONTEXT record, and stop saving the now obsolete REC_TYPE_SHARED_INFO and REC_TYPE_TSC_INFO records for PV guests.
---
Cc: Andrew Cooper <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
v10:
- New in v10
- Derived from patch #8 of the v9 series
--- tools/libs/guest/xg_sr_common_x86.c | 20 ----------- tools/libs/guest/xg_sr_common_x86.h | 6 ---- tools/libs/guest/xg_sr_save.c | 52 ++++++++++++++++++++++++++- tools/libs/guest/xg_sr_save_x86_hvm.c | 5 --- tools/libs/guest/xg_sr_save_x86_pv.c | 22 ------------ 5 files changed, 51 insertions(+), 54 deletions(-)
tools/libs/guest/xg_sr_common_x86.c +0 -20
@@ -1,25 +1,5 @@#include "xg_sr_common_x86.h" -int write_x86_tsc_info (struct xc_sr_context *ctx) -{ -xc_interface *xch = ctx->xch; -struct xc_sr_rec_x86_tsc_info tsc = {}; -struct xc_sr_record rec = { - .type =REC_TYPE_X86_TSC_INFO , - .length =sizeof (tsc), - .data = &tsc, - }; - -if (xc_domain_get_tsc_info (xch, ctx->domid, &tsc.mode, - &tsc.nsec, &tsc.khz, &tsc.incarnation) <0 ) - { -PERROR ("Unable to obtain TSC information" ); -return -1 ; - } - -return write_record (ctx, &rec); -} -int handle_x86_tsc_info (struct xc_sr_context *ctx,struct xc_sr_record *rec) {xc_interface *xch = ctx->xch;
tools/libs/guest/xg_sr_common_x86.h +0 -6
@@ -3,12 +3,6 @@#include "xg_sr_common.h" -/* -* Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record -* into the stream. -*/ -int write_x86_tsc_info (struct xc_sr_context *ctx); -/* * Parses a X86_TSC_INFO record and applies the result to the domain. */
tools/libs/guest/xg_sr_save.c +51 -1
70 lines not on this page · raw message
tools/libs/guest/xg_sr_save_x86_hvm.c +0 -5
@@ -193,11 +193,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; -/* Write the TSC record. */ - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; -/* Write the HVM_CONTEXT record. */ rc =write_hvm_context (ctx);if ( rc )
tools/libs/guest/xg_sr_save_x86_pv.c +0 -22
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)return rc; } -/* -* Writes an SHARED_INFO record into the stream. -*/ -static int write_shared_info (struct xc_sr_context *ctx) -{ -struct xc_sr_record rec = { - .type =REC_TYPE_SHARED_INFO , - .length =PAGE_SIZE , - .data = ctx->x86.pv.shinfo, - }; - -return write_record (ctx, &rec); -} -/* * Normalise a pagetable for the migration stream. Performs mfn->pfn * conversions on the ptes.
@@ -1093,14 +1079,6 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; - - rc =write_shared_info (ctx); -if ( rc ) -return rc; - rc =write_all_vcpu_information (ctx);if ( rc )return rc;
Signature · 1 line
2.20.1
Re: [PATCH v10 11/11] tools/libs/guest: add code to save a v4 libxc stream
Quoted 7 lines · Paul Durrant
From: Paul Durrant <[email protected]>
This patch adds the necessary code to save a
REC_TYPE_DOMAIN_CONTEXTrecord, and stop saving the now obsoleteREC_TYPE_SHARED_INFOandREC_TYPE_TSC_INFOrecords for PV guests.Signed-off-by Paul Durrant <[email protected]>
Looks broadly ok.
Quoted 23 lines
tools/libs/guest/xg_sr_common_x86.h +15
-0@@ -44,6 +44,52 @@ static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type)return 0 ; } +/* +* Writes a DOMAIN_CONTEXT record into the stream. +*/ +static int write_domain_context_record (struct xc_sr_context *ctx) +{ +xc_interface *xch = ctx->xch; +struct xc_sr_record rec = { + .type =REC_TYPE_DOMAIN_CONTEXT , + }; +size_t len =0 ; +int rc; + + rc =xc_domain_get_context (xch, ctx->domid,NULL , &len); +if ( rc <0 ) + {
tools/libs/guest/xg_sr_common_x86.h static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 71
Quoted 10 lines
tools/libs/guest/xg_sr_common_x86.h +10
-0+PERROR ("can't get record length for dom %u\n" , ctx->domid); + goto out; + } + + rec.data =malloc (len); + + rc =-1 ; +if ( !rec.data ) + { +PERROR ("can't allocate %lu bytes\n" , len);
%zu, because not all versions of C have size_t the same as unsigned long.
tools/libs/guest/xg_sr_common_x86.h static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 78
Quoted 7 lines
tools/libs/guest/xg_sr_common_x86.h +7
-0+ goto out; + } + + rc =xc_domain_get_context (xch, ctx->domid, rec.data, &len); +if ( rc <0 ) + { +PERROR ("can't get domain record for dom %u\n" , ctx->domid);
"domain context", and above.
Quoted 11 lines
tools/libs/guest/xg_sr_save_x86_pv.c +0
-3@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)return rc; } -/* -* Writes an SHARED_INFO record into the stream. -*/
tools/libs/guest/xg_sr_save_x86_pv.c static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx) · line 864
Quoted 10 lines
tools/libs/guest/xg_sr_save_x86_pv.c +0
-10-static int write_shared_info (struct xc_sr_context *ctx) -{ -struct xc_sr_record rec = { - .type =REC_TYPE_SHARED_INFO , - .length =PAGE_SIZE , - .data = ctx->x86.pv.shinfo, - }; - -return write_record (ctx, &rec); -}
This change also wants to strip out ctx->x86.pv.shinfo, and the mapping logic.
~Andrew
Quoted 14 lines · Paul Durrant
From: Paul Durrant <[email protected]>
Paul Durrant (11):
docs / include: introduce a new framework for 'domain context' records
xen: introduce implementation of save/restore of 'domain context' xen/common/domctl: introduce XEN_DOMCTL_get/set_domain_contexttools/misc: add xen-domctx to present domain context common/domain: add a domain context record forshared_info... x86/time: add a domain context record fortsc_info... docs/specs: add missing definitions to libxc-migration-stream docs / tools: specify migration v4 to includeDOMAIN_CONTEXTtools/python: modifylibxc.pyto verify v4 streamtools/libs/guest: add code to restore a v4 libxc streamtools/libs/guest: add code to save a v4 libxc stream
Thanks - this is much better when it comes to the public API/ABI.
However, my concerns still stands. What *else* is going in the domain context record, because you can pull the "bump the interface version and deprecated new fields" exactly once, as the libxg logic doesn't delve into domain context stream.
At the moment this does increase the downtime of the VM for no gain. What I'm trying to understand is whether this is "no gain (yet)" or whether you consider this "done" as far as cooperative migrate is concerned.
~Andrew