domain context infrastructure
v9 cover domain context infrastructure
From: Paul Durrant <[email protected]>
Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to include DOMAIN_CONTEXT common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
.gitignore | 1 + docs/specs/libxc-migration-stream.pandoc | 69 ++++- tools/flask/policy/modules/xen.if | 4 +- tools/libs/ctrl/include/xenctrl.h | 5 + tools/libs/ctrl/xc_domain.c | 56 ++++ tools/libs/guest/xg_sr_common.c | 1 + tools/libs/guest/xg_sr_common.h | 3 + tools/libs/guest/xg_sr_common_x86.c | 20 -- tools/libs/guest/xg_sr_common_x86.h | 6 - tools/libs/guest/xg_sr_restore.c | 45 +++- tools/libs/guest/xg_sr_save.c | 52 +++- tools/libs/guest/xg_sr_save_x86_hvm.c | 5 - tools/libs/guest/xg_sr_save_x86_pv.c | 22 -- tools/libs/guest/xg_sr_stream_format.h | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 289 +++++++++++++++++++++ tools/python/xen/migration/libxc.py | 2 + xen/arch/x86/time.c | 34 ++- xen/common/Makefile | 1 + xen/common/domain.c | 105 ++++++++ xen/common/domctl.c | 173 +++++++++++++ xen/common/save.c | 315 +++++++++++++++++++++++ xen/include/asm-x86/time.h | 5 +- xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/domctl.h | 41 +++ xen/include/public/save.h | 111 ++++++++ xen/include/xen/save.h | 170 ++++++++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 30 files changed, 1485 insertions(+), 75 deletions(-) create mode 100644 tools/misc/xen-domctx.c create mode 100644 xen/common/save.c create mode 100644 xen/include/public/save.h create mode 100644 xen/include/xen/save.h
---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>
Signature · 1 line
2.20.1
PATCH v9 1/8 xen/common: introduce a new framework for save/restore of 'domain' context
To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.
Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.
This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.
This patch also marks the HVM-only framework as deprecated in favour of the new framework.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v7:
- Add an option to domain_load_end() to ignore unconsumed data, which will be needed by a subsequent patch
- Kept acks since the modification is very small
v4:
- Addressed further comments from Jan
v3:
- Addressed comments from Julien and Jan
- Save handlers no longer need to state entry length up-front
- Save handlers expected to deal with multiple instances internally
- Entries are now auto-padded to 8 byte boundary
v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework
--- xen/common/Makefile | 1 + xen/common/save.c | 315 +++++++++++++++++++++++++ xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/save.h | 89 +++++++ xen/include/xen/save.h | 170 +++++++++++++ 6 files changed, 585 insertions(+) create mode 100644 xen/common/save.c create mode 100644 xen/include/public/save.h create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
@@ -37,6 +37,7 @@ obj-y += radix-tree.o obj-y += rbtree.o obj-y += rcupdate.o obj-y += rwlock.o +obj-y += save.o obj-y += shutdown.o obj-y += softirq.o obj-y += sort.o
xen/common/save.c new file +315 -0
315 lines not on this page · raw message
xen/include/public/arch-arm/hvm/save.h +5 -0
@@ -26,6 +26,11 @@#ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__#define __XEN_PUBLIC_HVM_SAVE_ARM_H__ +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /*
xen/include/public/arch-x86/hvm/save.h +5 -0
@@ -648,6 +648,11 @@ struct hvm_msr {*/ #define HVM_SAVE_CODE_MAX 20 +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */ /*
xen/include/public/save.h new file +89 -0
89 lines not on this page · raw message
xen/include/xen/save.h new file +170 -0
170 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context
xen/common/save.c line 2
Quoted 8 lines · Paul Durrant
xen/common/save.c new file +2
-0@@ -0,0 +1,315 @@ +/* +* save.c: Save and restore PV guest state common to all domain types.
This description will be stale by the time your work is complete.
xen/common/save.c line 85
Quoted 11 lines
xen/common/save.c new file +11
-0+int domain_save_data (struct domain_context *c,const void *src,size_t len) +{ +int rc = c->ops.save->append (c->priv, src, len); + +if ( !rc ) + c->len += len; + +return rc; +} + +#define DOMAIN_SAVE_ALIGN 8
This is part of the stream ABI.
xen/common/save.c line 90
Quoted 5 lines
xen/common/save.c new file +5
-0+ +int domain_save_end (struct domain_context *c) +{ +struct domain *d = c->domain; +size_t len =ROUNDUP (c->len,DOMAIN_SAVE_ALIGN ) - c->len;/* padding */
DOMAIN_SAVE_ALIGN - (c->len & (DOMAIN_SAVE_ALIGN - 1))
isn't vulnerable to overflow.
Quoted 6 lines
xen/common/save.c new file +6
-0+int rc; + +if ( len ) + { +static const uint8_t pad[DOMAIN_SAVE_ALIGN ] = {}; +
xen/common/save.c line 106
Quoted 10 lines
xen/common/save.c new file +10
-0+ rc =domain_save_data (c, pad, len); + +if ( rc ) +return rc; + } +ASSERT (IS_ALIGNED (c->len,DOMAIN_SAVE_ALIGN )); + +if ( c->name ) +gdprintk (XENLOG_INFO ,"%pd save: %s[%u] +%zu (-%zu)\n" , d, c->name, + c->desc.instance, c->len, len);
IMO, this is unhelpful to print out. It also appears to be the only use of the c->name field.
It also creates obscure and hard to follow logic based on dry_run.
Quoted 56 lines
xen/include/public/save.h new file +50
-0@@ -0,0 +1,89 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by +* Xen and must be saved along with the domain's memory. +* +* Copyright Amazon.com Inc. or its affiliates. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +*/ + +#ifndef XEN_PUBLIC_SAVE_H +#define XEN_PUBLIC_SAVE_H + +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +#include "xen.h" + +/* Entry data is preceded by a descriptor */ +struct domain_save_descriptor { +uint16_t typecode; + +/* +* Instance number of the entry (since there may be multiple of some +* types of entries). +*/ +uint16_t instance; + +/* Entry length not including this descriptor */ +uint32_t length; +}; + +/* +* Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE
xen/include/public/save.h line 60
Quoted 10 lines
xen/include/public/save.h new file +10
-0+* binds these things together, although it is not intended that the +* resulting type is ever instantiated. +*/ +#define DECLARE_DOMAIN_SAVE_TYPE (_x, _code, _type) \ + struct DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; }; + +#define DOMAIN_SAVE_CODE (_x) \ + (sizeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->c)) +#define DOMAIN_SAVE_TYPE (_x) \ + typeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->t)
I realise this is going to make me very unpopular, but NACK.
This is straight up obfuscation with no redeeming properties. I know you've copied it from the exist HVMCONTEXT infrastructure, but it is obnoxious to use there (particularly in the domain builder) and not an example wanting copying.
Furthermore, the code will be simpler and easier to follow without it.
Secondly, and more importantly, I do not see anything in docs/specs/ describing the binary format of this stream, and I'm going to insist that one appears, ahead of this patch in the series.
In doing so, you're hopefully going to discover the bug with the older HVMCONTEXT stream which makes the version field fairly pointless (more below).
It should describe how to forward compatibly extend the stream, and under what circumstances the version number can/should change. It also needs to describe the alignment and extending rules which ...
xen/include/public/save.h line 68
Quoted 8 lines
xen/include/public/save.h new file +8
-0+ +/* +* All entries will be zero-padded to the next 64-bit boundary when saved, +* so there is no need to include trailing pad fields in structure +* definitions. +* When loading, entries will be zero-extended if the load handler reads +* beyond the length specified in the descriptor. +*/
... shouldn't be this.
The current zero extending property was an emergency hack to fix an ABI breakage which had gone unnoticed for a couple of releases. The work to implement it created several very hard to debug breakages in Xen.
A properly designed stream shouldn't need auto-extending behaviour, and the legibility of the code is improved by not having it.
It is a trick which can stay up your sleeve for an emergency, in the hope you'll never have to use it.
Quoted 5 lines
xen/include/public/save.h new file +5
-0+ +/* Terminating entry */ +struct domain_save_end {}; +DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end); +
xen/include/public/save.h line 83
Quoted 10 lines
xen/include/public/save.h new file +10
-0+#define DOMAIN_SAVE_MAGIC 0x53415645 +#define DOMAIN_SAVE_VERSION 0x00000001 + +/* Initial entry */ +struct domain_save_header { +uint32_t magic;/* Must be DOMAIN_SAVE_MAGIC */ +uint16_t xen_major, xen_minor;/* Xen version */ +uint32_t version;/* Save format version */ +}; +DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);
The layout problem with the stream is the fact that this header doesn't come first.
In the eventual future where uint16_t won't be sufficient for instance, and uint32_t might not be sufficient for len, the version number is going to have to be bumped, in order to change the descriptor layout.
Overall, this patch needs to be a minimum of two. First a written document which is the authoritative stream ABI, and the second which is this implementation. The header describing the stream format should not be substantively different from xg_sr_stream_format.h
~Andrew
P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.
Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context
On Fri, Oct 02, 2020 at 10:20:18PM +0100, Andrew Cooper wrote:
[...]
Quoted 3 lines
P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.
For Rust the header is going to be generated by a tool called bindgen. It doesn't like nested macros, so I would be all for a simpler C header file if we can help it.
Wei.
RE: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context
In reply to Andrew CooperQuoted 13 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:20
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Julien Grall <[email protected]>; Jan Beulich <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' contextOn 24/09/2020 14:10, Paul Durrant wrote:
Quoted 11 lines
xen/common/save.c line 2
Quoted 8 lines
xen/common/save.c new file +2
-0@@ -0,0 +1,315 @@ +/* +* save.c: Save and restore PV guest state common to all domain types. This description will be stale by the time your work is complete.
True now, I'll just drop the 'PV'
Quoted 5 lines
Quoted 5 lines
xen/common/save.c new file +5
-0+int domain_save_data (struct domain_context *c,const void *src,size_t len) +{ +int rc = c->ops.save->append (c->priv, src, len); + +if ( !rc )
Quoted 9 lines
xen/common/save.c line 85
Quoted 6 lines
xen/common/save.c new file +6
-0+ c->len += len; + +return rc; +} + +#define DOMAIN_SAVE_ALIGN 8This is part of the stream ABI.
And what's actually the problem with defining it here?
Quoted 10 lines
xen/common/save.c line 90
Quoted 5 lines
xen/common/save.c new file +5
-0+ +int domain_save_end (struct domain_context *c) +{ +struct domain *d = c->domain; +size_t len =ROUNDUP (c->len,DOMAIN_SAVE_ALIGN ) - c->len;/* padding */
DOMAIN_SAVE_ALIGN- (c->len& (DOMAIN_SAVE_ALIGN- 1))isn't vulnerable to overflow.
...and significantly uglier code. What's actually wrong with what I wrote?
Quoted 20 lines
Quoted 6 lines
xen/common/save.c new file +6
-0+int rc; + +if ( len ) + { +static const uint8_t pad[DOMAIN_SAVE_ALIGN ] = {}; +xen/common/save.c line 106
Quoted 10 lines
xen/common/save.c new file +10
-0+ rc =domain_save_data (c, pad, len); + +if ( rc ) +return rc; + } +ASSERT (IS_ALIGNED (c->len,DOMAIN_SAVE_ALIGN )); + +if ( c->name ) +gdprintk (XENLOG_INFO ,"%pd save: %s[%u] +%zu (-%zu)\n" , d, c->name, + c->desc.instance, c->len, len);IMO, this is unhelpful to print out. It also appears to be the only use of the
c->namefield.
Quoted 2 lines
It also creates obscure and hard to follow logic based on
dry_run.
I'll drop it to debug. I personally find it helpful and would prefer to keep it.
Quoted 69 lines
Quoted 56 lines
xen/include/public/save.h new file +50
-0@@ -0,0 +1,89 @@ +/* +* save.h +* +* Structure definitions for common PV/HVM domain state that is held by +* Xen and must be saved along with the domain's memory. +* +* Copyright Amazon.com Inc. or its affiliates. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +*/ + +#ifndef XEN_PUBLIC_SAVE_H +#define XEN_PUBLIC_SAVE_H + +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +#include "xen.h" + +/* Entry data is preceded by a descriptor */ +struct domain_save_descriptor { +uint16_t typecode; + +/* +* Instance number of the entry (since there may be multiple of some +* types of entries). +*/ +uint16_t instance; + +/* Entry length not including this descriptor */ +uint32_t length; +}; + +/* +* Each entry has a type associated with it. DECLARE_DOMAIN_SAVE_TYPE xen/include/public/save.h line 60
Quoted 10 lines
xen/include/public/save.h new file +10
-0+* binds these things together, although it is not intended that the +* resulting type is ever instantiated. +*/ +#define DECLARE_DOMAIN_SAVE_TYPE (_x, _code, _type) \ + struct DOMAIN_SAVE_TYPE_##_x { char c[_code]; _type t; }; + +#define DOMAIN_SAVE_CODE (_x) \ + (sizeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->c)) +#define DOMAIN_SAVE_TYPE (_x) \ + typeof(((struct DOMAIN_SAVE_TYPE_##_x *)0)->t)I realise this is going to make me very unpopular, but NACK.
Quoted 7 lines
This is straight up obfuscation with no redeeming properties. I know you've copied it from the exist HVMCONTEXT infrastructure, but it is obnoxious to use there (particularly in the domain builder) and not an example wanting copying.
Furthermore, the code will be simpler and easier to follow without it.
OK, I can drop it if you so vehemently object.
Quoted 4 lines
Secondly, and more importantly, I do not see anything in docs/specs/ describing the binary format of this stream, and I'm going to insist that one appears, ahead of this patch in the series.
I can certainly put something there if you wish.
Quoted 8 lines
In doing so, you're hopefully going to discover the bug with the older HVMCONTEXT stream which makes the version field fairly pointless (more below).
It should describe how to forward compatibly extend the stream, and under what circumstances the version number can/should change. It also needs to describe the alignment and extending rules which ...
Quoted 11 lines
xen/include/public/save.h line 68
Quoted 8 lines
xen/include/public/save.h new file +8
-0+ +/* +* All entries will be zero-padded to the next 64-bit boundary when saved, +* so there is no need to include trailing pad fields in structure +* definitions. +* When loading, entries will be zero-extended if the load handler reads +* beyond the length specified in the descriptor. +*/ ... shouldn't be this.
Auto-padding was explicitly requested by Julien and extending (with zeroes or otherwise) is the necessary corollary (since the save handlers are not explicitly padding to the alignment boundary).
Quoted 10 lines
The current zero extending property was an emergency hack to fix an ABI breakage which had gone unnoticed for a couple of releases. The work to implement it created several very hard to debug breakages in Xen.
A properly designed stream shouldn't need auto-extending behaviour, and the legibility of the code is improved by not having it.
It is a trick which can stay up your sleeve for an emergency, in the hope you'll never have to use it.
The zero-extending here is different; it does not form part of the record. It is merely there to make sure the alignment constraint is met.
Quoted 10 lines
Quoted 10 lines
xen/include/public/save.h new file +10
-0+ +/* Terminating entry */ +struct domain_save_end {}; +DECLARE_DOMAIN_SAVE_TYPE(END, 0, struct domain_save_end); + +#define DOMAIN_SAVE_MAGIC 0x53415645 +#define DOMAIN_SAVE_VERSION 0x00000001 + +/* Initial entry */ +struct domain_save_header {
Quoted 9 lines
xen/include/public/save.h line 83
Quoted 5 lines
xen/include/public/save.h new file +5
-0+uint32_t magic;/* Must be DOMAIN_SAVE_MAGIC */ +uint16_t xen_major, xen_minor;/* Xen version */ +uint32_t version;/* Save format version */ +}; +DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header);The layout problem with the stream is the fact that this header doesn't come first.
? It most certainly does some first as is evident from the load and save functions. But I will add a document that states it, as requested.
Quoted 10 lines
In the eventual future where
uint16_twon't be sufficient for instance, anduint32_tmight not be sufficient for len, the version number is going to have to be bumped, in order to change the descriptor layout.Overall, this patch needs to be a minimum of two. First a written document which is the authoritative stream ABI, and the second which is this implementation. The header describing the stream format should not be substantively different from
xg_sr_stream_format.h
Ok.
Quoted 5 lines
~Andrew
P.S. Another good reason for having extremely simple header files is for the poor sole trying to write a Go/Rust/other binding for this in some likely not-to-distant future.
Fine. I'm happy to drop the macro/type magic if no-one feels it is necessary.
Paul
Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context
Quoted 11 lines · Paul Durrant
Quoted 11 lines
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:20xen/common/save.c line 90
Quoted 4 lines · Paul Durrant
xen/common/save.c new file +4
-0+int domain_save_end (struct domain_context *c) +{ +struct domain *d = c->domain; +size_t len =ROUNDUP (c->len,DOMAIN_SAVE_ALIGN ) - c->len;/* padding */
DOMAIN_SAVE_ALIGN- (c->len& (DOMAIN_SAVE_ALIGN- 1))
Quoted 4 lines
Quoted 2 lines
isn't vulnerable to overflow.
...and significantly uglier code. What's actually wrong with what I wrote?
I don't think there's anything "wrong" or "vulnerable" here, but I still can see Andrew's point. The "vulnerable" aspect applies only in the (highly hypothetical I think) cases of either sizeof(size_t) < sizeof(int) or size_t being a signed type, afaict. But since it's easy (and imo not "significantly uglier") to write code that is free of any wrapping or overflowing behavior, I think it is sensible to actually write it that way.
Jan
Re: [PATCH v9 1/8] xen/common: introduce a new framework for save/restore of 'domain' context
In reply to Paul Durrantxen/include/xen/save.h line 99
Quoted 10 lines · Paul Durrant
xen/include/xen/save.h new file +10
-0+/* +* The 'dry_run' flag indicates that the caller of domain_save() (see below) +* is not trying to actually acquire the data, only the size of the data. +* The save handler can therefore limit work to only that which is necessary +* to call domain_save_data() the correct number of times with accurate values +* for 'len'. +*/ +typedef int (*domain_save_handler )(const struct domain *d, +struct domain_context *c, +bool dry_run);
Sorry - missed this the first time around. This cannot take a const domain.
Doing so prevents putting (amongst other things), event channel details into the stream, because you won't be able to take the domain's event lock, and having the domain paused isn't good enough protection.
Removing this const will reduce the churn in subsequent patches somewhat.
~Andrew
PATCH v9 2/8 xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext
In reply to Paul DurrantThese domctls provide a mechanism to get and set domain context from the toolstack.
---
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v4:
- Add missing zero pad checks
v3:
- Addressed comments from Julien and Jan
- Use vmalloc() rather than xmalloc_bytes()
v2:
- drop mask parameter
- const-ify some more buffers
--- tools/flask/policy/modules/xen.if | 4 +- tools/libs/ctrl/include/xenctrl.h | 5 + tools/libs/ctrl/xc_domain.c | 56 +++++++++ xen/common/domctl.c | 173 ++++++++++++++++++++++++++++ xen/include/public/domctl.h | 41 +++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 7 files changed, 287 insertions(+), 2 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
@@ -53,7 +53,7 @@ define(`create_domain_common', ` allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim set_vnumainfo get_vnumainfo cacheflush psr_cmt_op psr_alloc soft_reset - resource_map get_cpu_policy }; + resource_map get_cpu_policy setcontext }; allow $1 $2:security check_context; allow $1 $2:shadow enable; allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', ` allow $1 $2:hvm { gethvmc getparam }; allow $1 $2:mmu { stat pageinfo map_read }; allow $1 $2:domain { getaddrsize getvcpucontext pause destroy }; - allow $1 $2:domain2 gettsc; + allow $1 $2:domain2 { gettsc getcontext }; allow $1 $2:shadow { enable disable logdirty }; ')
tools/libs/ctrl/include/xenctrl.h +5 -0
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,uint8_t *hvm_ctxt, uint32_t size); +int xc_domain_getcontext (xc_interface *xch,uint32_t domid, +void *ctxt_buf,size_t *size); +int xc_domain_setcontext (xc_interface *xch,uint32_t domid, +const void *ctxt_buf,size_t size); +/** * This function will return guest IO ABI protocol *
tools/libs/ctrl/xc_domain.c +56 -0
62 lines not on this page · raw message
xen/common/domctl.c +173 -0
191 lines not on this page · raw message
xen/include/public/domctl.h +41 -0
59 lines not on this page · raw message
xen/xsm/flask/hooks.c +6 -0
@@ -744,6 +744,12 @@ static int flask_domctl(struct domain *d, int cmd)case XEN_DOMCTL_get_cpu_policy:return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_CPU_POLICY ); +case XEN_DOMCTL_setdomaincontext: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__SETCONTEXT ); + +case XEN_DOMCTL_getdomaincontext: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GETCONTEXT ); +default :return avc_unknown_permission ("domctl" , cmd); }
xen/xsm/flask/policy/access_vectors +4 -0
@@ -245,6 +245,10 @@ class domain2 resource_map # XEN_DOMCTL_get_cpu_policy get_cpu_policy +# XEN_DOMCTL_setdomaincontext + setcontext +# XEN_DOMCTL_getdomaincontext + getcontext } # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
Re: [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext
Quoted 5 lines · Paul Durrant
These domctls provide a mechanism to get and set domain context from the toolstack.
Signed-off-by Paul Durrant <[email protected]>Reviewed-by Julien Grall <[email protected]>
Re: [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext
In reply to Paul DurrantQuoted 14 lines · Paul Durrant
xen/include/public/domctl.h +6
-0@@ -1130,6 +1130,43 @@ struct xen_domctl_vuart_op {*/ }; +/* +* XEN_DOMCTL_getdomaincontext +* --------------------------- +* +* buffer (IN): The buffer into which the context data should be +* copied, or NULL to query the buffer size that should
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1148
Quoted 10 lines
xen/include/public/domctl.h +10
-0+* be allocated. +* size (IN/OUT): If 'buffer' is NULL then the value passed in must be +* zero, and the value passed out will be the size of the +* buffer to allocate. +* If 'buffer' is non-NULL then the value passed in must +* be the size of the buffer into which data may be copied. +* The value passed out will be the size of data written. +*/ +struct xen_domctl_getdomaincontext { +uint32_t size;
This series is full of mismatched 32/64bit sizes, with several truncation bugs in the previous patch.
Just use a 64bit size here. Life is too short to go searching for all the other truncation bug when this stream tips over 4G, and its not like there is a shortage of space in this structure.
Quoted 25 lines
xen/include/public/domctl.h +21
-0+uint32_t pad; +XEN_GUEST_HANDLE_64 (void ) buffer; +}; + +/* XEN_DOMCTL_setdomaincontext +* --------------------------- +* +* buffer (IN): The buffer from which the context data should be +* copied. +* size (IN): The size of the buffer from which data may be copied. +* This data must include DOMAIN_SAVE_CODE_HEADER at the +* start and terminate with a DOMAIN_SAVE_CODE_END record. +* Any data beyond the DOMAIN_SAVE_CODE_END record will be +* ignored. +*/ +struct xen_domctl_setdomaincontext { +uint32_t size; +uint32_t pad; +XEN_GUEST_HANDLE_64 (const_void) buffer; +}; +struct xen_domctl {uint32_t cmd;#define XEN_DOMCTL_createdomain 1
xen/include/public/domctl.h struct xen_domctl { · line 1255
Quoted 5 lines
xen/include/public/domctl.h +2
-0@@ -1214,6 +1251,8 @@ struct xen_domctl {#define XEN_DOMCTL_vuart_op 81#define XEN_DOMCTL_get_cpu_policy 82#define XEN_DOMCTL_set_cpu_policy 83 +#define XEN_DOMCTL_getdomaincontext 84 +#define XEN_DOMCTL_setdomaincontext 85
So, we've currently got:
#define XEN_DOMCTL_setvcpucontext 12 #define XEN_DOMCTL_getvcpucontext 13 #define XEN_DOMCTL_gethvmcontext 33 #define XEN_DOMCTL_sethvmcontext 34 #define XEN_DOMCTL_set_ext_vcpucontext 42 #define XEN_DOMCTL_get_ext_vcpucontext 43 #define XEN_DOMCTL_gethvmcontext_partial 55 #define XEN_DOMCTL_setvcpuextstate 62 #define XEN_DOMCTL_getvcpuextstate 63
which are doing alarmingly related things for vcpus. (As an amusing exercise to the reader, figure out which are PV specific and which are HVM specific. Hint: they're not disjoint sets.)
I know breaking with tradition is sacrilege, but at the very minimum, can we get some underscores in that name so you can at least read the words which make it up more easily.
~Andrew
RE: [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext
Quoted 44 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 22:58
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; Julien Grall <[email protected]>; Daniel De Graaf <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 2/8] xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontextCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 14 lines · Paul Durrant
xen/include/public/domctl.h +6
-0@@ -1130,6 +1130,43 @@ struct xen_domctl_vuart_op {*/ }; +/* +* XEN_DOMCTL_getdomaincontext +* --------------------------- +* +* buffer (IN): The buffer into which the context data should be +* copied, or NULL to query the buffer size that should xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1148
Quoted 10 lines
xen/include/public/domctl.h +10
-0+* be allocated. +* size (IN/OUT): If 'buffer' is NULL then the value passed in must be +* zero, and the value passed out will be the size of the +* buffer to allocate. +* If 'buffer' is non-NULL then the value passed in must +* be the size of the buffer into which data may be copied. +* The value passed out will be the size of data written. +*/ +struct xen_domctl_getdomaincontext { +uint32_t size;This series is full of mismatched 32/64bit sizes, with several truncation bugs in the previous patch.
Quoted 4 lines
Just use a 64bit size here. Life is too short to go searching for all the other truncation bug when this stream tips over 4G, and its not like there is a shortage of space in this structure.
Ok.
Quoted 43 lines
Quoted 25 lines
xen/include/public/domctl.h +21
-0+uint32_t pad; +XEN_GUEST_HANDLE_64 (void ) buffer; +}; + +/* XEN_DOMCTL_setdomaincontext +* --------------------------- +* +* buffer (IN): The buffer from which the context data should be +* copied. +* size (IN): The size of the buffer from which data may be copied. +* This data must include DOMAIN_SAVE_CODE_HEADER at the +* start and terminate with a DOMAIN_SAVE_CODE_END record. +* Any data beyond the DOMAIN_SAVE_CODE_END record will be +* ignored. +*/ +struct xen_domctl_setdomaincontext { +uint32_t size; +uint32_t pad; +XEN_GUEST_HANDLE_64 (const_void) buffer; +}; +struct xen_domctl {uint32_t cmd;#define XEN_DOMCTL_createdomain 1xen/include/public/domctl.h struct xen_domctl { · line 1255
Quoted 5 lines
xen/include/public/domctl.h +2
-0@@ -1214,6 +1251,8 @@ struct xen_domctl {#define XEN_DOMCTL_vuart_op 81#define XEN_DOMCTL_get_cpu_policy 82#define XEN_DOMCTL_set_cpu_policy 83 +#define XEN_DOMCTL_getdomaincontext 84 +#define XEN_DOMCTL_setdomaincontext 85So, we've currently got:
#define XEN_DOMCTL_setvcpucontext 12 #define XEN_DOMCTL_getvcpucontext 13 #define XEN_DOMCTL_gethvmcontext 33 #define XEN_DOMCTL_sethvmcontext 34 #define XEN_DOMCTL_set_ext_vcpucontext 42 #define XEN_DOMCTL_get_ext_vcpucontext 43 #define XEN_DOMCTL_gethvmcontext_partial 55 #define XEN_DOMCTL_setvcpuextstate 62 #define XEN_DOMCTL_getvcpuextstate 63
Quoted 4 lines
which are doing alarmingly related things for vcpus. (As an amusing exercise to the reader, figure out which are PV specific and which are HVM specific. Hint: they're not disjoint sets.)
Yes, hence the desire to come up with something common.
Quoted 5 lines
I know breaking with tradition is sacrilege, but at the very minimum, can we get some underscores in that name so you can at least read the words which make it up more easily.
Sure.
Paul
Quoted 1 line
~Andrew
PATCH v9 3/8 tools/misc: add xen-domctx to present domain context
In reply to Paul DurrantThis tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
NOTE: Ian requested ack from Andrew
v3:
- Re-worked to avoid copying onto stack
- Added optional typecode and instance arguments
v2:
- Change name from 'xen-ctx' to 'xen-domctx'
--- .gitignore | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 200 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 205 insertions(+) create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
@@ -233,6 +233,7 @@ tools/misc/xen_cpuperf tools/misc/xen-cpuid tools/misc/xen-detect tools/misc/xen-diag +tools/misc/xen-domctx tools/misc/xen-tmem-list-parse tools/misc/xen-livepatch tools/misc/xenperf
tools/misc/Makefile +4 -0
@@ -32,6 +32,7 @@ INSTALL_SBIN += xenpm INSTALL_SBIN += xenwatchdogd INSTALL_SBIN += xen-livepatch INSTALL_SBIN += xen-diag +INSTALL_SBIN += xen-domctx INSTALL_SBIN += $(INSTALL_SBIN-y) # Everything to be installed in a private bin/
@@ -111,6 +112,9 @@ xen-livepatch: xen-livepatch.o xen-diag: xen-diag.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) +xen-domctx: xen-domctx.o + $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) + xen-lowmemd: xen-lowmemd.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
tools/misc/xen-domctx.c new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context
Quoted 6 lines · Paul Durrant
This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
Signed-off-by Paul Durrant <[email protected]>Acked-by Ian Jackson <[email protected]>
Re: [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context
In reply to Paul DurrantQuoted 11 lines · Paul Durrant
This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
Signed-off-by Paul Durrant <[email protected]>Acked-by Ian Jackson <[email protected]>---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>NOTE: Ian requested ack from Andrew
I mean - its a fairly throwaway piece of misc userspace, so ack.
However, it is largely superseded by the changes you need to make to verify-stream-v2 so you might want to bear that in mind.
Also, I wonder if it is wise in general that we're throwing so many misc debugging tools into sbin.
Quoted 10 lines
tools/misc/xen-domctx.c new file +10
-0+#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xen/domctl.h> +#include <xen/save.h>
tools/misc/xen-domctx.c line 46
Quoted 10 lines
tools/misc/xen-domctx.c new file +10
-0+ +static void *buf =NULL ; +static size_t len, off; + +#define GET_PTR (_x) \ + do { \ + if ( len - off < sizeof(*(_x)) ) \ + { \ + fprintf(stderr, \ + "error: need another %lu bytes, only %lu available\n", \
%zu is the correct formatter for size_t.
tools/misc/xen-domctx.c line 48
Quoted 2 lines
tools/misc/xen-domctx.c new file +2
-0+ sizeof(*(_x)), len - off); \ +exit (1 ); \
Your error handling will be far more simple by using err() instead of opencoding it.
~Andrew
RE: [PATCH v9 3/8] tools/misc: add xen-domctx to present domain context
Quoted 26 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 23:40
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 3/8]tools/misc: add xen-domctx to present domain contextCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 11 lines · Paul Durrant
This tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
Signed-off-by Paul Durrant <[email protected]>Acked-by Ian Jackson <[email protected]>---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>NOTE: Ian requested ack from Andrew
Quoted 8 lines
I mean - its a fairly throwaway piece of misc userspace, so ack.
However, it is largely superseded by the changes you need to make to verify-stream-v2 so you might want to bear that in mind.
Also, I wonder if it is wise in general that we're throwing so many misc debugging tools into sbin.
The intention is to eventually replace hvm context so we'll need a parallel tool to replace xen-hvmctx, unless we want to retire it.
Quoted 14 lines
Quoted 14 lines
tools/misc/xen-domctx.c new file +14
-0+#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xen/domctl.h> +#include <xen/save.h> + +static void *buf =NULL ; +static size_t len, off; +
Quoted 9 lines
tools/misc/xen-domctx.c line 46
Quoted 6 lines
tools/misc/xen-domctx.c new file +6
-0+#define GET_PTR (_x) \ + do { \ + if ( len - off < sizeof(*(_x)) ) \ + { \ + fprintf(stderr, \ + "error: need another %lu bytes, only %lu available\n", \%zu is the correct formatter for
size_t.
True, missed that.
Quoted 6 lines
tools/misc/xen-domctx.c line 48
Quoted 2 lines
tools/misc/xen-domctx.c new file +2
-0+ sizeof(*(_x)), len - off); \ +exit (1 ); \Your error handling will be far more simple by using
err()instead of opencoding it.
Ok.
Paul
Quoted 1 line
~Andrew
PATCH v9 4/8 docs/specs: add missing definitions to libxc-migration-stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
docs/specs/libxc-migration-stream.pandoc +9 -2
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 2 +% Revision 3 Introduction ============
@@ -227,7 +227,13 @@ type 0x00000000: END 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary) - 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000010: STATIC_DATA_END + + 0x00000011: X86_CPUID_POLICY + + 0x00000012: X86_MSR_POLICY + + 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -732,6 +738,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_TSC_INFO * HVM_PARAMS * HVM_CONTEXT +* END record HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect the validity of architectural state in the context.
Signature · 1 line
2.20.1
Re: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream
Quoted 6 lines · Paul Durrant
From: Paul Durrant <[email protected]>
The
STATIC_DATA_END,X86_CPUID_POLICYandX86_MSR_POLICYrecord types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Quoted 7 lines
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
Signed-off-by Paul Durrant <[email protected]>Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")
Re: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream
In reply to Paul DurrantQuoted 6 lines · Paul Durrant
From: Paul Durrant <[email protected]>
The
STATIC_DATA_END,X86_CPUID_POLICYandX86_MSR_POLICYrecord types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Quoted 7 lines
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
Signed-off-by Paul Durrant <[email protected]>Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")
Oops sorry. I swear I had these at one point. I can only presume it got swallowed by a rebase at some point.
~Andrew
RE: [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-stream
Quoted 25 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 02 October 2020 23:42
To: Paul Durrant <[email protected]>; [email protected]
Cc: Durrant, Paul <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 4/8] docs/specs: add missing definitions to libxc-migration-streamCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 10 lines · Paul Durrant
From: Paul Durrant <[email protected]>
The
STATIC_DATA_END,X86_CPUID_POLICYandX86_MSR_POLICYrecord types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
Quoted 7 lines
Quoted 3 lines
Signed-off-by Paul Durrant <[email protected]>Fixes 6f71b5b1506 ("docs/migration Specify migration v3 and STATIC_DATA_END")Fixes ddd273d8863 ("docs/migration: Specify X86_{CPUID,MSR}_POLICY records")Oops sorry. I swear I had these at one point. I can only presume it got swallowed by a rebase at some point.
Can I take that as an R-b?
Paul
Quoted 1 line
~Andrew
PATCH v9 5/8 docs / tools: specific migration v4 to include DOMAIN_CONTEXT
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 62 ++++++++++++++++++------ tools/libs/guest/xg_sr_common.c | 1 + tools/libs/guest/xg_sr_stream_format.h | 1 + tools/python/xen/migration/libxc.py | 2 + 4 files changed, 52 insertions(+), 14 deletions(-)
docs/specs/libxc-migration-stream.pandoc +48 -14
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 3 +% Revision 4 Introduction ============
@@ -127,7 +127,7 @@ marker 0xFFFFFFFFFFFFFFFF. id 0x58454E46 ("XENF" in ASCII). -version 0x00000003. The version of this specification. +version 0x00000004. The version of this specification. options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
@@ -209,9 +209,9 @@ type 0x00000000: END 0x00000006: X86_PV_VCPU_XSAVE - 0x00000007: SHARED_INFO + 0x00000007: SHARED_INFO (deprecated) - 0x00000008: X86_TSC_INFO + 0x00000008: X86_TSC_INFO (deprecated) 0x00000009: HVM_CONTEXT
@@ -233,7 +233,9 @@ type 0x00000000: END 0x00000012: X86_MSR_POLICY - 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000013: DOMAIN_CONTEXT + + 0x00000014 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs \clearpage -SHARED_INFO ------------ +SHARED_INFO (deprecated) +------------------------ -The content of the Shared Info page. +The content of the Shared Info page. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info Contents of the shared info page. This record \clearpage -X86_TSC_INFO ------------- +X86_TSC_INFO (deprecated) +------------------------- Domain TSC information, as accessed by the -XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. +XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +------------------------+------------------------+
@@ -680,6 +684,25 @@ MSR_policy Array of xen_msr_entry_t[]'s \clearpage +DOMAIN_CONTEXT +-------------- + +Domain context, as accessed by the +XEN_DOMCTL_{get,set}domaincontext hypercall sub-ops. + + 0 1 2 3 4 5 6 7 octet + +-------------------------------------------------+ + | dom_ctx | + ... + +-------------------------------------------------+ + +-------------------------------------------------------------------- +Field Description +----------- --------------------------------------------------- +dom_ctx The Domain Context blob from Xen. +-------------------------------------------------------------------- + +\clearpage Layout ======
@@ -706,8 +729,7 @@ A typical save record for an x86 PV guest image would look like: * STATIC_DATA_END * X86_PV_P2M_FRAMES record * Many PAGE_DATA records -* X86_TSC_INFO -* SHARED_INFO record +* DOMAIN_CONTEXT * VCPU context records for each online VCPU * X86_PV_VCPU_BASIC record * X86_PV_VCPU_EXTENDED record
@@ -735,7 +757,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_{CPUID,MSR}_POLICY * STATIC_DATA_END * Many PAGE_DATA records -* X86_TSC_INFO +* DOMAIN_CONTEXT * HVM_PARAMS * HVM_CONTEXT * END record
@@ -746,6 +768,18 @@ the validity of architectural state in the context. Compatibility with older versions ================================= +v4 compat with v3 +----------------- + +A v4 stream is compatible with a v3 stream, but mandates the presence of a +DOMAIN_CONTEXT record. This incorporates context such as the content of +the domain's Shared Info page and the TSC information, hence the SHARED_INFO +and TSC_INFO records are deprecated. +It also supercedes HVM_CONTEXT and, over time, data that is currently part of +the HVM_CONTEXT blob will move to the DOMAIN_CONTEXT blob. Xen, however, will +continue to accept all defined HVM_CONTEXT records so a v4-compatible +receiver can still accept an unmodified v3 stream. + v3 compat with v2 -----------------
tools/libs/guest/xg_sr_common.c +1 -0
@@ -39,6 +39,7 @@ static const char *const mandatory_rec_types[] = [REC_TYPE_STATIC_DATA_END] = "Static data end", [REC_TYPE_X86_CPUID_POLICY] = "x86 CPUID policy", [REC_TYPE_X86_MSR_POLICY] = "x86 MSR policy", + [REC_TYPE_DOMAIN_CONTEXT] = "Domain context", };const char *rec_type_to_str (uint32_t type)
tools/libs/guest/xg_sr_stream_format.h +1 -0
@@ -76,6 +76,7 @@ struct xc_sr_rhdr#define REC_TYPE_STATIC_DATA_END 0x00000010U#define REC_TYPE_X86_CPUID_POLICY 0x00000011U#define REC_TYPE_X86_MSR_POLICY 0x00000012U +#define REC_TYPE_DOMAIN_CONTEXT 0x00000013U#define REC_TYPE_OPTIONAL 0x80000000U
tools/python/xen/migration/libxc.py +2 -0
@@ -59,6 +59,7 @@ REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f REC_TYPE_static_data_end = 0x00000010 REC_TYPE_x86_cpuid_policy = 0x00000011 REC_TYPE_x86_msr_policy = 0x00000012 +REC_TYPE_domain_context = 0x00000013 rec_type_to_str = { REC_TYPE_end : "End",
@@ -80,6 +81,7 @@ rec_type_to_str = { REC_TYPE_static_data_end : "Static data end", REC_TYPE_x86_cpuid_policy : "x86 CPUID policy", REC_TYPE_x86_msr_policy : "x86 MSR policy", + REC_TYPE_domain_context : "Domain context", } # page_data
Signature · 1 line
2.20.1
Re: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT
Quoted 14 lines · Paul Durrant
From: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new
DOMAIN_CONTEXTrecord in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so theSHARED_INFOandTSC_INFOrecords are deprecated. It is intended that, in future, this record will contain state currently present in theHVM_CONTEXTrecord. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify anHVM_CONTEXTrecord andXEN_DOMCTL_sethvmcontextwill continue to accept all content of that record that may be present in a version 3 stream.
Quoted 1 line
Signed-off-by Paul Durrant <[email protected]>
I would like Andrew to give an explicit ack on this patch.
Re: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT
In reply to Paul DurrantQuoted 14 lines · Paul Durrant
From: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new
DOMAIN_CONTEXTrecord in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so theSHARED_INFOandTSC_INFOrecords are deprecated. It is intended that, in future, this record will contain state currently present in theHVM_CONTEXTrecord. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify anHVM_CONTEXTrecord andXEN_DOMCTL_sethvmcontextwill continue to accept all content of that record that may be present in a version 3 stream.
Quoted 1 line
Signed-off-by Paul Durrant <[email protected]>
What's the plan for the remainder of the work? We don't want to burn multiple version numbers for each bit of incremental work.
One option might be to specify the full extent of the work, and use an environment variable to alter the behaviour of the sending side, while it is still work-in-progress.
~Andrew
RE: [PATCH v9 5/8] docs / tools: specific migration v4 to include DOMAIN_CONTEXT
Quoted 25 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 05 October 2020 11:09
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Marek Marczykowski-Górecki <[email protected]>
Subject: Re: [PATCH v9 5/8] docs / tools: specific migration v4 to includeDOMAIN_CONTEXTQuoted 14 lines · Paul Durrant
From: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new
DOMAIN_CONTEXTrecord in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so theSHARED_INFOandTSC_INFOrecords are deprecated. It is intended that, in future, this record will contain state currently present in theHVM_CONTEXTrecord. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify anHVM_CONTEXTrecord andXEN_DOMCTL_sethvmcontextwill continue to accept all content of that record that may be present in a version 3 stream.
Quoted 9 lines
Quoted 1 line
Signed-off-by Paul Durrant <[email protected]>What's the plan for the remainder of the work? We don't want to burn multiple version numbers for each bit of incremental work.
One option might be to specify the full extent of the work, and use an environment variable to alter the behaviour of the sending side, while it is still work-in-progress.
The other missing part for transparent migration is xenstore content, but I'd expect that to sit at the next level up so I'm not anticipating any more churn at this level.
Paul
PATCH v9 6/8 common/domain: add a domain context record for shared_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v9:
- Use macros to make the code less verbose
- Add missing check for allocation failure
v8:
- Incorporate zero-ing out of shared info fields that would be done in processing of SHARED_INFO from older stream versions
v7:
- Only restore vcpu_info and arch sub-structures for PV domains, to match processing of SHARED_INFO in xc_sr_restore_x86_pv.c
- Use additional option to domain_load_end() to ignore the record for HVM domains
v6:
- Only save compat_shared_info buffer if has_32bit_shinfo is set
- Validate flags field in load handler
v5:
- Addressed comments from Julien
v4:
- Addressed comments from Jan
v3:
- Actually dump some of the content of shared_info
v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT
--- tools/misc/xen-domctx.c | 78 ++++++++++++++++++++++++++++ xen/common/domain.c | 105 ++++++++++++++++++++++++++++++++++++++ xen/include/public/save.h | 13 ++++- 3 files changed, 195 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +78 -0
@@ -31,6 +31,7 @@#include <errno.h> #include <xenctrl.h> +#include <xen-tools/libs.h> #include <xen/xen.h> #include <xen/domctl.h> #include <xen/save.h>
@@ -61,6 +62,82 @@ static void dump_header(void) } +static void print_binary (const char *prefix,const void *val,size_t size, +const char *suffix) +{ +printf ("%s" , prefix); + +while ( size-- ) + { +uint8_t octet = *(const uint8_t *)val++; +unsigned int i; + +for ( i =0 ; i <8 ; i++ ) + { +printf ("%u" , octet &1 ); + octet >>=1 ; + } + } + +printf ("%s" , suffix); +} + +static void dump_shared_info (void ) +{ +DOMAIN_SAVE_TYPE (SHARED_INFO) *s; +bool has_32bit_shinfo; +shared_info_any_t *info; +unsigned int i, n; + +GET_PTR (s); + has_32bit_shinfo = s->flags &DOMAIN_SAVE_32BIT_SHINFO ; + +printf (" SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n" , + has_32bit_shinfo ?"true" :"false" , s->buffer_size); + + info = (shared_info_any_t *)s->buffer; + +#define GET_FIELD_PTR (_f) \ + (has_32bit_shinfo ? \ + (const void *)&(info->x32._f) : \ + (const void *)&(info->x64._f)) +#define GET_FIELD_SIZE (_f) \ + (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f)) +#define GET_FIELD (_f) \ + (has_32bit_shinfo ? info->x32._f : info->x64._f) + + n = has_32bit_shinfo ? +ARRAY_SIZE (info->x32.evtchn_pending) : +ARRAY_SIZE (info->x64.evtchn_pending); + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_pending: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_pending[0 ]), +GET_FIELD_SIZE (evtchn_pending[0 ]),"\n" ); + } + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_mask: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_mask[0 ]), +GET_FIELD_SIZE (evtchn_mask[0 ]),"\n" ); + } + +printf (" wc: version: %u sec: %u nsec: %u\n" , +GET_FIELD (wc_version),GET_FIELD (wc_sec),GET_FIELD (wc_nsec)); + +#undef GET_FIELD +#undef GET_FIELD_SIZE +#undef GET_FIELD_PTR +} +static void dump_end (void ) {DOMAIN_SAVE_TYPE (END) *e;
@@ -173,6 +250,7 @@ int main(int argc, char **argv)switch (desc->typecode) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ; +case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);
xen/common/domain.c +105 -0
117 lines not on this page · raw message
xen/include/public/save.h +12 -1
@@ -82,7 +82,18 @@ struct domain_save_header { }; DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header); -#define DOMAIN_SAVE_CODE_MAX 1 +struct domain_shared_info_context { +uint32_t flags; + +#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001 + +uint32_t buffer_size; +uint8_t buffer[XEN_FLEX_ARRAY_DIM ];/* Implementation specific size */ +}; + +DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context); + +#define DOMAIN_SAVE_CODE_MAX 2#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
Signature · 1 line
2.20.1
Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...
Quoted 11 lines · Paul Durrant
From: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-onlySHARED_INFOrecord done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.Signed-off-by Paul Durrant <[email protected]>
Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...
In reply to Paul DurrantQuoted 11 lines · Paul Durrant
From: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-onlySHARED_INFOrecord done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.Signed-off-by Paul Durrant <[email protected]>
Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...
In reply to Paul DurrantQuoted 73 lines · Paul Durrant
tools/misc/xen-domctx.c +58
-0@@ -31,6 +31,7 @@#include <errno.h> #include <xenctrl.h> +#include <xen-tools/libs.h> #include <xen/xen.h> #include <xen/domctl.h> #include <xen/save.h> @@ -61,6 +62,82 @@ static void dump_header(void) } +static void print_binary (const char *prefix,const void *val,size_t size, +const char *suffix) +{ +printf ("%s" , prefix); + +while ( size-- ) + { +uint8_t octet = *(const uint8_t *)val++; +unsigned int i; + +for ( i =0 ; i <8 ; i++ ) + { +printf ("%u" , octet &1 ); + octet >>=1 ; + } + } + +printf ("%s" , suffix); +} + +static void dump_shared_info (void ) +{ +DOMAIN_SAVE_TYPE (SHARED_INFO) *s; +bool has_32bit_shinfo; +shared_info_any_t *info; +unsigned int i, n; + +GET_PTR (s); + has_32bit_shinfo = s->flags &DOMAIN_SAVE_32BIT_SHINFO ; + +printf (" SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n" , + has_32bit_shinfo ?"true" :"false" , s->buffer_size); + + info = (shared_info_any_t *)s->buffer; + +#define GET_FIELD_PTR (_f) \ + (has_32bit_shinfo ? \ + (const void *)&(info->x32._f) : \ + (const void *)&(info->x64._f)) +#define GET_FIELD_SIZE (_f) \ + (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f)) +#define GET_FIELD (_f) \ + (has_32bit_shinfo ? info->x32._f : info->x64._f) + + n = has_32bit_shinfo ? +ARRAY_SIZE (info->x32.evtchn_pending) : +ARRAY_SIZE (info->x64.evtchn_pending); + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_pending: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_pending[0 ]), +GET_FIELD_SIZE (evtchn_pending[0 ]),"\n" ); + }
tools/misc/xen-domctx.c static void dump_header(void) · line 131
Quoted 10 lines
tools/misc/xen-domctx.c +10
-0+ +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_mask: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_mask[0 ]), +GET_FIELD_SIZE (evtchn_mask[0 ]),"\n" ); + }
What about domains using FIFO? This is meaningless for them.
tools/misc/xen-domctx.c static void dump_header(void) · line 134
Quoted 3 lines
tools/misc/xen-domctx.c +3
-0+ +printf (" wc: version: %u sec: %u nsec: %u\n" , +GET_FIELD (wc_version),GET_FIELD (wc_sec),GET_FIELD (wc_nsec));
wc_sec_hi is also a rather critical field in this calculation.
Quoted 80 lines
tools/misc/xen-domctx.c +7
-0+ +#undef GET_FIELD +#undef GET_FIELD_SIZE +#undef GET_FIELD_PTR +} +static void dump_end (void ) {DOMAIN_SAVE_TYPE (END) *e;@@ -173,6 +250,7 @@ int main(int argc, char **argv)switch (desc->typecode) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ; +case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);xen/common/domain.c +48
-0@@ -33,6 +33,7 @@#include <xen/xenoprof.h> #include <xen/irq.h> #include <xen/argo.h> +#include <xen/save.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> @@ -1657,6 +1658,110 @@ int continue_hypercall_on_cpu(return 0 ; } +static int save_shared_info (const struct domain *d,struct domain_context *c, +bool dry_run) +{ +struct domain_shared_info_context ctxt = { +#ifdef CONFIG_COMPAT + .flags =has_32bit_shinfo (d) ?DOMAIN_SAVE_32BIT_SHINFO :0 , + .buffer_size =has_32bit_shinfo (d) ? +sizeof (struct compat_shared_info ) : +sizeof (struct shared_info ), +#else + .buffer_size =sizeof (struct shared_info ), +#endif + }; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +int rc; + + rc =DOMAIN_SAVE_BEGIN (SHARED_INFO , c,0 ); +if ( rc ) +return rc; + + rc =domain_save_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + + rc =domain_save_data (c, d->shared_info, ctxt.buffer_size); +if ( rc ) +return rc; + +return domain_save_end (c); +} + +static int load_shared_info (struct domain *d,struct domain_context *c) +{ +struct domain_shared_info_context ctxt; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_BEGIN (SHARED_INFO , c, &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_data (c, &ctxt, hdr_size); +if ( rc )
xen/common/domain.c int continue_hypercall_on_cpu( · line 1717
Quoted 10 lines
xen/common/domain.c +10
-0+return rc; + +if ( ctxt.buffer_size >sizeof (shared_info_t) || + (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO ) ) +return -EINVAL ; + +if ( ctxt.flags &DOMAIN_SAVE_32BIT_SHINFO ) + { +#ifdef CONFIG_COMPAT +has_32bit_shinfo (d) = true;
d->arch.has_32bit_shinfo
Quoted 5 lines
xen/common/domain.c +5
-0+#else +return -EINVAL ; +#endif + } +
xen/common/domain.c int continue_hypercall_on_cpu( · line 1732
Quoted 10 lines
xen/common/domain.c +10
-0+if (is_pv_domain (d) ) + { +shared_info_t *shinfo =xmalloc (shared_info_t); + +if ( !shinfo ) +return -ENOMEM ; + + rc =domain_load_data (c, shinfo,sizeof (*shinfo)); +if ( rc ) + goto out;
There's no need for a memory allocation, or to double buffer this data. You can memcpy() straight out of the context record.
Quoted 4 lines
xen/common/domain.c +4
-0+ +memcpy (&shared_info (d, vcpu_info), &__shared_info (d, shinfo, vcpu_info), +sizeof (shared_info (d, vcpu_info))); + memcpy(&shared_info(d, arch), &__shared_info(d, shinfo, arch),
xen/common/domain.c int continue_hypercall_on_cpu( · line 1746
Quoted 10 lines
xen/common/domain.c +10
-0+ sizeof(shared_info(d, arch))); + +memset (&shared_info (d, evtchn_pending),0 , +sizeof (shared_info (d, evtchn_pending))); +memset (&shared_info (d, evtchn_mask),0xff , +sizeof (shared_info (d, evtchn_mask))); + +shared_info (d, arch.pfn_to_mfn_frame_list_list) =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) +shared_info (d, vcpu_info[i].evtchn_pending_sel) =0 ;
What is the plan for transparent migrate here? While this is ok for regular migrate, its definitely not for transparent.
Quoted 26 lines
xen/common/domain.c +18
-0+ + rc =domain_load_end (c, false); + + out: +xfree (shinfo); + } +else +/* +* No modifications to shared_info are required for restoring non-PV +* domains. +*/ + rc =domain_load_end (c, true); + +return rc; +} + +DOMAIN_REGISTER_SAVE_LOAD (SHARED_INFO , save_shared_info, load_shared_info); +/* * Local variables: * mode: C xen/include/public/save.h +0
-0
xen/include/public/save.h struct domain_save_header { · line 90
Quoted 10 lines
xen/include/public/save.h +6
-1@@ -82,7 +82,18 @@ struct domain_save_header { }; DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header); -#define DOMAIN_SAVE_CODE_MAX 1 +struct domain_shared_info_context { +uint32_t flags; + +#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001 + +uint32_t buffer_size;
This struct is already wrapped with a header including a size which encompasses buffer.
Multiple overlapping size fields is an easy way to memory corruption, because it causes ambiguity as to which one is right.
~Andrew
RE: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...
Quoted 87 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 05 October 2020 11:40
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v9 6/8] common/domain: add a domain context record forshared_info...Quoted 77 lines · Paul Durrant
tools/misc/xen-domctx.c +62
-0@@ -31,6 +31,7 @@#include <errno.h> #include <xenctrl.h> +#include <xen-tools/libs.h> #include <xen/xen.h> #include <xen/domctl.h> #include <xen/save.h> @@ -61,6 +62,82 @@ static void dump_header(void) } +static void print_binary (const char *prefix,const void *val,size_t size, +const char *suffix) +{ +printf ("%s" , prefix); + +while ( size-- ) + { +uint8_t octet = *(const uint8_t *)val++; +unsigned int i; + +for ( i =0 ; i <8 ; i++ ) + { +printf ("%u" , octet &1 ); + octet >>=1 ; + } + } + +printf ("%s" , suffix); +} + +static void dump_shared_info (void ) +{ +DOMAIN_SAVE_TYPE (SHARED_INFO) *s; +bool has_32bit_shinfo; +shared_info_any_t *info; +unsigned int i, n; + +GET_PTR (s); + has_32bit_shinfo = s->flags &DOMAIN_SAVE_32BIT_SHINFO ; + +printf (" SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n" , + has_32bit_shinfo ?"true" :"false" , s->buffer_size); + + info = (shared_info_any_t *)s->buffer; + +#define GET_FIELD_PTR (_f) \ + (has_32bit_shinfo ? \ + (const void *)&(info->x32._f) : \ + (const void *)&(info->x64._f)) +#define GET_FIELD_SIZE (_f) \ + (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f)) +#define GET_FIELD (_f) \ + (has_32bit_shinfo ? info->x32._f : info->x64._f) + + n = has_32bit_shinfo ? +ARRAY_SIZE (info->x32.evtchn_pending) : +ARRAY_SIZE (info->x64.evtchn_pending); + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_pending: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_pending[0 ]), +GET_FIELD_SIZE (evtchn_pending[0 ]),"\n" ); + } + +for ( i =0 ; i < n; i++ ) + { + const char *prefix = !i ?
Quoted 9 lines
tools/misc/xen-domctx.c static void dump_header(void) · line 131
Quoted 6 lines
tools/misc/xen-domctx.c +6
-0+ " evtchn_mask: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_mask[0 ]), +GET_FIELD_SIZE (evtchn_mask[0 ]),"\n" ); + }What about domains using FIFO? This is meaningless for them.
Indeed, but this is essentially a debug tool so I'd rather it just dumped everything that might be useful.
Quoted 6 lines
tools/misc/xen-domctx.c static void dump_header(void) · line 134
Quoted 3 lines
tools/misc/xen-domctx.c +3
-0+ +printf (" wc: version: %u sec: %u nsec: %u\n" , +GET_FIELD (wc_version),GET_FIELD (wc_sec),GET_FIELD (wc_nsec));
wc_sec_hiis also a rather critical field in this calculation.
Ok.
Quoted 84 lines
Quoted 84 lines
tools/misc/xen-domctx.c +7
-0+ +#undef GET_FIELD +#undef GET_FIELD_SIZE +#undef GET_FIELD_PTR +} +static void dump_end (void ) {DOMAIN_SAVE_TYPE (END) *e;@@ -173,6 +250,7 @@ int main(int argc, char **argv)switch (desc->typecode) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ; +case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);xen/common/domain.c +52
-0@@ -33,6 +33,7 @@#include <xen/xenoprof.h> #include <xen/irq.h> #include <xen/argo.h> +#include <xen/save.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> @@ -1657,6 +1658,110 @@ int continue_hypercall_on_cpu(return 0 ; } +static int save_shared_info (const struct domain *d,struct domain_context *c, +bool dry_run) +{ +struct domain_shared_info_context ctxt = { +#ifdef CONFIG_COMPAT + .flags =has_32bit_shinfo (d) ?DOMAIN_SAVE_32BIT_SHINFO :0 , + .buffer_size =has_32bit_shinfo (d) ? +sizeof (struct compat_shared_info ) : +sizeof (struct shared_info ), +#else + .buffer_size =sizeof (struct shared_info ), +#endif + }; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +int rc; + + rc =DOMAIN_SAVE_BEGIN (SHARED_INFO , c,0 ); +if ( rc ) +return rc; + + rc =domain_save_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + + rc =domain_save_data (c, d->shared_info, ctxt.buffer_size); +if ( rc ) +return rc; + +return domain_save_end (c); +} + +static int load_shared_info (struct domain *d,struct domain_context *c) +{ +struct domain_shared_info_context ctxt; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_BEGIN (SHARED_INFO , c, &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + +if ( ctxt.buffer_size >sizeof (shared_info_t) || + (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO ) )
Quoted 9 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1717
Quoted 6 lines
xen/common/domain.c +6
-0+return -EINVAL ; + +if ( ctxt.flags &DOMAIN_SAVE_32BIT_SHINFO ) + { +#ifdef CONFIG_COMPAT +has_32bit_shinfo (d) = true;
d->arch.has_32bit_shinfo
If you'd prefer, ok.
Quoted 10 lines
Quoted 10 lines
xen/common/domain.c +10
-0+#else +return -EINVAL ; +#endif + } + +if (is_pv_domain (d) ) + { +shared_info_t *shinfo =xmalloc (shared_info_t); + +if ( !shinfo )
Quoted 9 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1732
Quoted 5 lines
xen/common/domain.c +5
-0+return -ENOMEM ; + + rc =domain_load_data (c, shinfo,sizeof (*shinfo)); +if ( rc ) + goto out;There's no need for a memory allocation, or to double buffer this data. You can
memcpy()straight out of the context record.
That would mean re-working the way that domain_load_data() works. I'd really rather not.
Quoted 9 lines
Quoted 9 lines
xen/common/domain.c +9
-0+ +memcpy (&shared_info (d, vcpu_info), &__shared_info (d, shinfo, vcpu_info), +sizeof (shared_info (d, vcpu_info))); +memcpy (&shared_info (d, arch), &__shared_info (d, shinfo, arch), +sizeof (shared_info (d, arch))); + +memset (&shared_info (d, evtchn_pending),0 , +sizeof (shared_info (d, evtchn_pending))); + memset(&shared_info(d, evtchn_mask), 0xff,
Quoted 9 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1746
Quoted 5 lines
xen/common/domain.c +5
-0+ sizeof(shared_info(d, evtchn_mask))); + +shared_info (d, arch.pfn_to_mfn_frame_list_list) =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) +shared_info (d, vcpu_info[i].evtchn_pending_sel) =0 ;What is the plan for transparent migrate here? While this is ok for regular migrate, its definitely not for transparent.
Quite true, as evidenced that this is inside 'if ( is_pv_domain(d) )'. It is not yet clear how much of the shared info we need for transparent migrate. It may be nothing.
Quoted 40 lines
Quoted 26 lines
xen/common/domain.c +18
-0+ + rc =domain_load_end (c, false); + + out: +xfree (shinfo); + } +else +/* +* No modifications to shared_info are required for restoring non-PV +* domains. +*/ + rc =domain_load_end (c, true); + +return rc; +} + +DOMAIN_REGISTER_SAVE_LOAD (SHARED_INFO , save_shared_info, load_shared_info); +/* * Local variables: * mode: C xen/include/public/save.h +0
-0xen/include/public/save.h struct domain_save_header { · line 90
Quoted 10 lines
xen/include/public/save.h +6
-1@@ -82,7 +82,18 @@ struct domain_save_header { }; DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header); -#define DOMAIN_SAVE_CODE_MAX 1 +struct domain_shared_info_context { +uint32_t flags; + +#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001 + +uint32_t buffer_size;This struct is already wrapped with a header including a size which encompasses buffer.
Quoted 3 lines
Multiple overlapping size fields is an easy way to memory corruption, because it causes ambiguity as to which one is right.
The record size currently includes padding. I'm re-working that in v10 and so this size can be dropped.
Paul
Re: [PATCH v9 6/8] common/domain: add a domain context record for shared_info...
In reply to Andrew CooperQuoted 21 lines · Andrew Cooper
Quoted 20 lines · Paul Durrant
xen/common/domain.c +20
-0+static int load_shared_info (struct domain *d,struct domain_context *c) +{ +struct domain_shared_info_context ctxt; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_BEGIN (SHARED_INFO , c, &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + +if ( ctxt.buffer_size >sizeof (shared_info_t) || + (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO ) )
Quoted 8 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1717
Quoted 6 lines
xen/common/domain.c +6
-0+return -EINVAL ; + +if ( ctxt.flags &DOMAIN_SAVE_32BIT_SHINFO ) + { +#ifdef CONFIG_COMPAT +has_32bit_shinfo (d) = true;
d->arch.has_32bit_shinfo
But this is common code, i.e. using d->arch directly is a layering violation. I know your dislike of lvalues disguised by function- like macros, but what do you do?
Jan
PATCH v9 7/8 x86/time: add a domain context record for tsc_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Whilst the record definition is x86 specific, it is visible directly
in the common header as context record numbers should be unique across all architectures.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v8:
- Removed stray blank line
v7:
- New in v7
--- tools/misc/xen-domctx.c | 11 +++++++++++ xen/arch/x86/time.c | 34 +++++++++++++++++++++++++++++++++- xen/include/asm-x86/time.h | 5 +++-- xen/include/public/save.h | 13 ++++++++++++- 4 files changed, 59 insertions(+), 4 deletions(-)
tools/misc/xen-domctx.c +11 -0
@@ -59,7 +59,17 @@ static void dump_header(void)printf (" HEADER: magic %#x, version %u\n" , h->magic, h->version); +} + +static void dump_tsc_info (void ) +{ +DOMAIN_SAVE_TYPE (TSC_INFO) *t; + +GET_PTR (t); +printf (" TSC_INFO: mode: %u incarnation: %u\n" +" khz %u elapsed_nsec: %" PRIu64"\n" , + t->mode, t->incarnation, t->khz, t->elapsed_nsec); } static void print_binary(const char *prefix, const void *val, size_t size,
@@ -251,6 +261,7 @@ int main(int argc, char **argv) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ;case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ; +case DOMAIN_SAVE_CODE (TSC_INFO ):dump_tsc_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);
xen/arch/x86/time.c +33 -1
@@ -26,6 +26,7 @@#include <xen/symbols.h> #include <xen/keyhandler.h> #include <xen/guest_access.h> +#include <xen/save.h> #include <asm/io.h> #include <asm/iocap.h> #include <asm/msr.h>
@@ -2334,7 +2335,7 @@ int host_tsc_is_safe(void)* called to collect tsc-related data only for save file or live * migrate; called after last rdtsc is done on this incarnation */ -void tsc_get_info (struct domain *d,uint32_t *tsc_mode, +void tsc_get_info (const struct domain *d,uint32_t *tsc_mode,uint64_t *elapsed_nsec,uint32_t *gtsc_khz,uint32_t *incarnation) {
@@ -2451,6 +2452,37 @@ int tsc_set_info(struct domain *d,return 0 ; } +static int save_tsc_info (const struct domain *d,struct domain_context *c, +bool dry_run) +{ +struct domain_tsc_info_context ctxt; + +if ( !dry_run ) +tsc_get_info (d, &ctxt.mode, &ctxt.elapsed_nsec, &ctxt.khz, + &ctxt.incarnation); + +return DOMAIN_SAVE_ENTRY (TSC_INFO , c,0 , &ctxt,sizeof (ctxt)); +} + +static int load_tsc_info (struct domain *d,struct domain_context *c) +{ +struct domain_tsc_info_context ctxt; +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_ENTRY (TSC_INFO , c, &i, &ctxt,sizeof (ctxt)); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + +return tsc_set_info (d, ctxt.mode, ctxt.elapsed_nsec, ctxt.khz, + ctxt.incarnation); +} + +DOMAIN_REGISTER_SAVE_LOAD (TSC_INFO , save_tsc_info, load_tsc_info); +/* vtsc may incur measurable performance degradation, diagnose with this */ static void dump_softtsc (unsigned char key) {
xen/include/asm-x86/time.h +3 -2
@@ -59,8 +59,9 @@ u64 gtsc_to_gtime(struct domain *d, u64 tsc);int tsc_set_info (struct domain *d,uint32_t tsc_mode,uint64_t elapsed_nsec,uint32_t gtsc_khz,uint32_t incarnation); -void tsc_get_info (struct domain *d,uint32_t *tsc_mode,uint64_t *elapsed_nsec, -uint32_t *gtsc_khz,uint32_t *incarnation); +void tsc_get_info (const struct domain *d,uint32_t *tsc_mode, +uint64_t *elapsed_nsec,uint32_t *gtsc_khz, +uint32_t *incarnation);void force_update_vcpu_system_time (struct vcpu *v);
xen/include/public/save.h +12 -1
@@ -93,7 +93,18 @@ struct domain_shared_info_context { DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context); -#define DOMAIN_SAVE_CODE_MAX 2 +#if defined(__i386__) || defined(__x86_64__) +struct domain_tsc_info_context { +uint32_t mode; +uint32_t incarnation; +uint64_t elapsed_nsec; +uint32_t khz; +}; + +DECLARE_DOMAIN_SAVE_TYPE(TSC_INFO, 3, struct domain_tsc_info_context); +#endif + +#define DOMAIN_SAVE_CODE_MAX 3#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
Signature · 1 line
2.20.1
Re: [PATCH v9 7/8] x86/time: add a domain context record for tsc_info...
Quoted 10 lines · Paul Durrant
From: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Whilst the record definition is x86 specific, it is visible directly
in the common header as context record numbers should be unique across all architectures.Signed-off-by Paul Durrant <[email protected]>Reviewed-by Jan Beulich <[email protected]>
PATCH v9 8/8 tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and bump the version.
This patch implements version 4 of the migration stream by adding the code necessary to save and restore DOMAIN_CONTEXT records, and removing the code to save the SHARED_INFO and TSC_INFO records (as these are deprecated in version 4).
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
v7:
- New in v7
--- tools/libs/guest/xg_sr_common.h | 3 ++ tools/libs/guest/xg_sr_common_x86.c | 20 ----------- tools/libs/guest/xg_sr_common_x86.h | 6 ---- tools/libs/guest/xg_sr_restore.c | 45 +++++++++++++++++++++-- tools/libs/guest/xg_sr_save.c | 52 ++++++++++++++++++++++++++- tools/libs/guest/xg_sr_save_x86_hvm.c | 5 --- tools/libs/guest/xg_sr_save_x86_pv.c | 22 ------------ 7 files changed, 97 insertions(+), 56 deletions(-)
tools/libs/guest/xg_sr_common.h +3 -0
@@ -298,6 +298,9 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; + +/* Domain context blob. */ +struct xc_sr_blob context; } restore; };
tools/libs/guest/xg_sr_common_x86.c +0 -20
@@ -1,25 +1,5 @@#include "xg_sr_common_x86.h" -int write_x86_tsc_info (struct xc_sr_context *ctx) -{ -xc_interface *xch = ctx->xch; -struct xc_sr_rec_x86_tsc_info tsc = {}; -struct xc_sr_record rec = { - .type =REC_TYPE_X86_TSC_INFO , - .length =sizeof (tsc), - .data = &tsc, - }; - -if (xc_domain_get_tsc_info (xch, ctx->domid, &tsc.mode, - &tsc.nsec, &tsc.khz, &tsc.incarnation) <0 ) - { -PERROR ("Unable to obtain TSC information" ); -return -1 ; - } - -return write_record (ctx, &rec); -} -int handle_x86_tsc_info (struct xc_sr_context *ctx,struct xc_sr_record *rec) {xc_interface *xch = ctx->xch;
tools/libs/guest/xg_sr_common_x86.h +0 -6
@@ -3,12 +3,6 @@#include "xg_sr_common.h" -/* -* Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record -* into the stream. -*/ -int write_x86_tsc_info (struct xc_sr_context *ctx); -/* * Parses a X86_TSC_INFO record and applies the result to the domain. */
tools/libs/guest/xg_sr_restore.c +43 -2
82 lines not on this page · raw message
tools/libs/guest/xg_sr_save.c +51 -1
70 lines not on this page · raw message
tools/libs/guest/xg_sr_save_x86_hvm.c +0 -5
@@ -193,11 +193,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; -/* Write the TSC record. */ - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; -/* Write the HVM_CONTEXT record. */ rc =write_hvm_context (ctx);if ( rc )
tools/libs/guest/xg_sr_save_x86_pv.c +0 -22
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)return rc; } -/* -* Writes an SHARED_INFO record into the stream. -*/ -static int write_shared_info (struct xc_sr_context *ctx) -{ -struct xc_sr_record rec = { - .type =REC_TYPE_SHARED_INFO , - .length =PAGE_SIZE , - .data = ctx->x86.pv.shinfo, - }; - -return write_record (ctx, &rec); -} -/* * Normalise a pagetable for the migration stream. Performs mfn->pfn * conversions on the ptes.
@@ -1093,14 +1079,6 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; - - rc =write_shared_info (ctx); -if ( rc ) -return rc; - rc =write_all_vcpu_information (ctx);if ( rc )return rc;
Signature · 1 line
2.20.1
Re: [PATCH v9 8/8] tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
Quoted 10 lines · Paul Durrant
From: Paul Durrant <[email protected]>
... and bump the version.
This patch implements version 4 of the migration stream by adding the code necessary to save and restore
DOMAIN_CONTEXTrecords, and removing the code to save theSHARED_INFOandTSC_INFOrecords (as these are deprecated in version 4).Signed-off-by Paul Durrant <[email protected]>
Again, I think Andrew should have a look at this as well.
Re: [PATCH v9 8/8] tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
In reply to Paul DurrantQuoted 10 lines · Paul Durrant
From: Paul Durrant <[email protected]>
... and bump the version.
This patch implements version 4 of the migration stream by adding the code necessary to save and restore
DOMAIN_CONTEXTrecords, and removing the code to save theSHARED_INFOandTSC_INFOrecords (as these are deprecated in version 4).Signed-off-by Paul Durrant <[email protected]>
This really needs to be at least 3 patches.
First to adjust tools/python/scripts/verify-stream-v2 to understand the new changes in the stream.
My testing tends to include running the script over the result of xl save, and using a script in place of libxl-save-helper which tee's the stream through the script, which lets you test in-line migrate. (I wonder if it would be better to add a pass-through mode to the script and give libxl a way of running it in the same way as it currently runs covert-legacy-stream.)
Next, a patch updating the receive side only to understand the new changes in the stream. In particular, this makes it far easier to confirm that backwards compatibility is maintained.
Finally, a patch updating the sending side, if applicable. (I've got an alternative suggestion to avoid burning a load of major version numbers, but will follow up on a different patch with that).
Quoted 21 lines
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>v7:
- New in v7--- tools/libs/guest/xg_sr_common.h | 3 ++ tools/libs/guest/xg_sr_common_x86.c | 20 ----------- tools/libs/guest/xg_sr_common_x86.h | 6 ---- tools/libs/guest/xg_sr_restore.c | 45 +++++++++++++++++++++-- tools/libs/guest/xg_sr_save.c | 52 ++++++++++++++++++++++++++- tools/libs/guest/xg_sr_save_x86_hvm.c | 5 --- tools/libs/guest/xg_sr_save_x86_pv.c | 22 ------------ 7 files changed, 97 insertions(+), 56 deletions(-)tools/libs/guest/xg_sr_common.h +0
-0
tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 303
Quoted 6 lines
tools/libs/guest/xg_sr_common.h +3
-0@@ -298,6 +298,9 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; + +/* Domain context blob. */ +struct xc_sr_blob context;
We already have
ctx->x86.hvm.restore.context
and are now gaining
ctx->restore.context
This is concerning close to being ambiguous. How about dom_context ?
Also, you leak the memory allocation. Free it in xg_sr_restore.c:cleanup().
Quoted 11 lines
tools/libs/guest/xg_sr_restore.c +3
-0@@ -529,6 +529,20 @@ static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx)return rc; } +static int stream_complete (struct xc_sr_context *ctx) +{ +xc_interface *xch = ctx->xch;
tools/libs/guest/xg_sr_restore.c static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx) · line 544
Quoted 10 lines
tools/libs/guest/xg_sr_restore.c +10
-0+int rc; + + rc =xc_domain_setcontext (xch, ctx->domid, + ctx->restore.context.ptr, + ctx->restore.context.size); +if ( rc <0 ) +PERROR ("Unable to restore domain context" ); + +return rc; +}
Please put this in the PV and HVM stream_complete() hooks.
This is somewhat of a layering violation and enforcing an order which might not be appropriate in all cases.
~Andrew
Quoted 20 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Paul Durrant
Sent: Thursday, September 24, 2020 9:10 AM
To: [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH v9 0/8] domain context infrastructureFrom: Paul Durrant <[email protected]>
Quoted 10 lines
Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontexttools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to includeDOMAIN_CONTEXTcommon/domain: add a domain context record forshared_info... x86/time: add a domain context record fortsc_info...tools/libxc: addDOMAIN_CONTEXTrecords to the migration stream...
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the series.
Thanks,
Tamas
Quoted 44 lines
-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 24 September 2020 20:36
To: Paul Durrant <[email protected]>; [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [PATCH v9 0/8] domain context infrastructureQuoted 18 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Paul Durrant
Sent: Thursday, September 24, 2020 9:10 AM
To: [email protected]
Cc: Paul Durrant <[email protected]>; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH v9 0/8] domain context infrastructureFrom: Paul Durrant <[email protected]>
Quoted 10 lines
Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontexttools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to includeDOMAIN_CONTEXTcommon/domain: add a domain context record forshared_info... x86/time: add a domain context record fortsc_info...tools/libxc: addDOMAIN_CONTEXTrecords to the migration stream...
Quoted 5 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to the series.
Hi Tamas,
Done. See https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/heads/domain-save14
Cheers,
Paul
Quoted 9 lines
Quoted 4 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to theseries.
Quoted 1 line
Hi Tamas,
Quoted 7 lines
Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14Cheers,
Paul
Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function copy_pv_domaincontext for now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.
Thanks,
Tamas
Quoted 34 lines
-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructureCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 9 lines
Quoted 4 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to theseries.
Quoted 1 line
Hi Tamas,
Quoted 7 lines
Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14Cheers,
Paul
Quoted 8 lines
Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at
https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the functioncopy_pv_domaincontextfor now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.
Hi Tamas,
The code structure appears to be ok... just some cosmetic tweaks:
- I think you should call the function simply 'copy_domaincontext' as the idea is that all state (including what is now in hvm context) will be consolidated
- The prevailing style in domctl.c AFAICS is that assignments are mostly not done inside if statements. Personally I think this is a good thing.
Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a v10... but I'm currently hoping that won't be necessary.
Cheers,
Paul
Quoted 3 lines
Thanks,
Tamas
Quoted 44 lines · Paul
Quoted 35 lines
-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructureCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 9 lines
Quoted 4 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to theseries.
Quoted 1 line
Hi Tamas,
Quoted 7 lines
Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14Cheers,
Paul
Quoted 8 lines
Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that would be called during VM forking. Please take a look at
https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdiff;h=1843ca7302e415317fdb9a63b3a4
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the functioncopy_pv_domaincontextfor now as that seemed like the most appropriate description for it. Please let me know if this looks good to you. I'm still testing it but if everything checks out it would be nice to just append this patch to your series.
Quoted 5 lines
Hi Tamas,
The code structure appears to be ok... just some cosmetic tweaks:
- I think you should call the function simply '
copy_domaincontext' as the idea is that all state (including what is now in hvm context) will be consolidated
Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.
Quoted 1 line
- The prevailing style in
domctl.cAFAICS is that assignments are mostly not done inside if statements. Personally I think this is a good thing.
I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.
Quoted 2 lines
Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a v10... but I'm currently hoping that won't be necessary.
I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.
Tamas
Quoted 88 lines
-----Original Message-----
From: Tamas K Lengyel <[email protected]>
Sent: 29 September 2020 13:06
To: Durrant, Paul <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected]; [email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructureCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 31 lines · Paul
Quoted 31 lines
-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructureCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 13 lines
Quoted 4 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to theseries.
Quoted 1 line
Hi Tamas,
Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortlog;h=refs/h
eads/domain-save14Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Cheers,
Paul
Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that wouldbe
Quoted 2 lines
Quoted 2 lines
called during VM forking. Please take a look at
Quoted 1 line
Quoted 1 line
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function
copy_pv_domaincontextQuoted 1 line
Quoted 1 line
for now as that seemed like the most appropriate description for it. Please let me know if this
looks
Quoted 1 line
Quoted 1 line
good to you. I'm still testing it but if everything checks out it would be nice to just append
this
Quoted 7 lines
Quoted 1 line
patch to your series.
Hi Tamas,
The code structure appears to be ok... just some cosmetic tweaks:
- I think you should call the function simply '
copy_domaincontext' as the idea is that all state(including what is now in hvm context) will be consolidated
Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.
Quoted 1 line
- The prevailing style in
domctl.cAFAICS is that assignments are mostly not done inside ifstatements. Personally I think this is a good thing.
I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Once you have something ready to go then I'd be happy to tag it onto my series if I need to do a
v10... but I'm currently hoping that won't be necessary.
I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.
It may be a while before I have everything moved over so you may still want to go ahead with this patch if the delay is likely to block things. Also, without this I assume any records I port over from HVM context (and hence remove the save code) are going to cause breakage for VM forking?
Paul
Quoted 1 line
Tamas
Quoted 128 lines
-----Original Message-----
From: Durrant, Paul <[email protected]>
Sent: Tuesday, September 29, 2020 8:14 AM
To: Tamas K Lengyel <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected]; xen- [email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu <[email protected]>
Subject: RE: [PATCH v9 0/8] domain context infrastructureQuoted 5 lines
-----Original Message-----
From: Tamas K Lengyel <[email protected]>
Sent: 29 September 2020 13:06
To: Durrant, Paul <[email protected]>
Cc: Lengyel, Tamas <[email protected]>; [email protected];Quoted 8 lines
[email protected]; Andrew Cooper <[email protected]>; Daniel De Graaf <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Marek Marczykowski-Górecki <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Wei Liu
Quoted 4 lines
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructure
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and
know the content is safe.
Quoted 4 lines
On Tue, Sep 29, 2020 at 7:54 AM Durrant, Paul <[email protected]>
Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Lengyel, Tamas <[email protected]>
Sent: 28 September 2020 15:17
To: [email protected]; [email protected]
Cc: Durrant, Paul <[email protected]>; 'Andrew Cooper' <[email protected]>; 'Daniel De Graaf'<[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson'
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
<[email protected]>; 'Jan Beulich' <[email protected]>; 'Julien Grall' <[email protected]>; 'Marek Marczykowski-Górecki'
<[email protected]>; 'Roger Pau Monné'
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; 'Stefano Stabellini' <[email protected]>;
'Volodymyr Babchuk'
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; 'Wei Liu' <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v9 0/8] domain context infrastructureCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender
and know the content is safe.
Quoted 68 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Hi Paul,
Could you push a git branch somewhere for this series? I would like to see this being integrated with VM forking and if its not too much effort just create the patch for that so that it could be appended to theseries.
Quoted 1 line
Hi Tamas,
Done. See
https://xenbits.xen.org/gitweb/?p=people/pauldu/xen.git;a=shortl
og;h=refs/h
eads/domain-save14Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Cheers,
Paul
Hi Paul,
I added a small patch that would save & load the PV context from one domain to another that wouldbe
Quoted 2 lines
Quoted 2 lines
called during VM forking. Please take a look at
https://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=commitdif
f;h=1843ca7302e415317fdb9a63b3a4Quoted 2 lines
Quoted 2 lines
d29a385dc766;hp=8149296fdf80c73727e61cea6fe3251aecf8b333. I called the function
copy_pv_domaincontextQuoted 2 lines
Quoted 2 lines
for now as that seemed like the most appropriate description for it. Please let me know if this
looks
Quoted 2 lines
Quoted 2 lines
good to you. I'm still testing it but if everything checks out it would be nice to just append
this
Quoted 8 lines
Quoted 1 line
patch to your series.
Hi Tamas,
The code structure appears to be ok... just some cosmetic tweaks:
- I think you should call the function simply '
copy_domaincontext' as the idea is that all state(including what is now in hvm context) will be consolidated
Sure, I wasn't entirely clear about whether this will be limited to PV context or if it will eventually add the hvm stuff too. Right now I still would have to do that separately.
Quoted 2 lines
- The prevailing style in
domctl.cAFAICS is that assignments are mostly not done inside ifstatements. Personally I think this is a good thing.
I think it cuts down on function sizes when all that is being done after an assigment is a NULL-check. No need for a separate line for it but I also don't care that much. So if it's more important to whoever maintains this to keep the style consistent in this regard I can change it.
Quoted 3 lines
Once you have something ready to go then I'd be happy to tag it
onto my series if I need to do av10... but I'm currently hoping that won't be necessary.
Quoted 8 lines
Quoted 3 lines
I think I'll wait until HVM context is included in the framework as well so that we can just switch over everything at once.
It may be a while before I have everything moved over so you may still want to go ahead with this patch if the delay is likely to block things. Also, without this I assume any records I port over from HVM context (and hence remove the save code) are going to cause breakage for VM forking?
If you do remove the existing hvm_save/hvm_load functions then yes, that would pretty much break VM forking right away. We use those in hvm_copy_context_and_params. So if you do make significant changes in there I would ask that you at least compile test with CONFIG_MEM_SHARING enabled. That would be the point where we would want to start using this new copy_domaincontext function. Right now just for the PV stuff it's not critical as we do copy those internally ourself, it would just be nicer to use this plumbing you add that will be shared with domain save/restore/migrate.
Tamas