domain context infrastructure
v8 cover domain context infrastructure
From: Paul Durrant <[email protected]>
Paul Durrant (8):
xen/common: introduce a new framework for save/restore of 'domain'
context
xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext tools/misc: add xen-domctx to present domain context docs/specs: add missing definitions to libxc-migration-stream docs / tools: specific migration v4 to include DOMAIN_CONTEXT common/domain: add a domain context record for shared_info... x86/time: add a domain context record for tsc_info... tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
.gitignore | 1 + docs/specs/libxc-migration-stream.pandoc | 69 ++++- tools/flask/policy/modules/xen.if | 4 +- tools/libxc/include/xenctrl.h | 5 + tools/libxc/xc_domain.c | 56 ++++ tools/libxc/xc_sr_common.c | 1 + tools/libxc/xc_sr_common.h | 3 + tools/libxc/xc_sr_common_x86.c | 20 -- tools/libxc/xc_sr_common_x86.h | 6 - tools/libxc/xc_sr_restore.c | 45 +++- tools/libxc/xc_sr_save.c | 52 +++- tools/libxc/xc_sr_save_x86_hvm.c | 5 - tools/libxc/xc_sr_save_x86_pv.c | 22 -- tools/libxc/xc_sr_stream_format.h | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 289 +++++++++++++++++++++ tools/python/xen/migration/libxc.py | 2 + xen/arch/x86/time.c | 34 ++- xen/common/Makefile | 1 + xen/common/domain.c | 144 +++++++++++ xen/common/domctl.c | 173 +++++++++++++ xen/common/save.c | 315 +++++++++++++++++++++++ xen/include/asm-x86/time.h | 5 +- xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/domctl.h | 41 +++ xen/include/public/save.h | 111 ++++++++ xen/include/xen/save.h | 170 ++++++++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 30 files changed, 1524 insertions(+), 75 deletions(-) create mode 100644 tools/misc/xen-domctx.c create mode 100644 xen/common/save.c create mode 100644 xen/include/public/save.h create mode 100644 xen/include/xen/save.h
---
Cc: Andrew Cooper <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>
Signature · 1 line
2.20.1
PATCH v8 1/8 xen/common: introduce a new framework for save/restore of 'domain' context
To allow enlightened HVM guests (i.e. those that have PV drivers) to be migrated without their co-operation it will be necessary to transfer 'PV' state such as event channel state, grant entry state, etc.
Currently there is a framework (entered via the hvm_save/load() functions) that allows a domain's 'HVM' (architectural) state to be transferred but 'PV' state is also common with pure PV guests and so this framework is not really suitable.
This patch adds the new public header and low level implementation of a new common framework, entered via the domain_save/load() functions. Subsequent patches will introduce other parts of the framework, and code that will make use of it within the current version of the libxc migration stream.
This patch also marks the HVM-only framework as deprecated in favour of the new framework.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v7:
- Add an option to domain_load_end() to ignore unconsumed data, which will be needed by a subsequent patch
- Kept acks since the modification is very small
v4:
- Addressed further comments from Jan
v3:
- Addressed comments from Julien and Jan
- Save handlers no longer need to state entry length up-front
- Save handlers expected to deal with multiple instances internally
- Entries are now auto-padded to 8 byte boundary
v2:
- Allow multi-stage save/load to avoid the need to double-buffer
- Get rid of the masks and add an 'ignore' flag instead
- Create copy function union to preserve const save buffer
- Deprecate HVM-only framework
--- xen/common/Makefile | 1 + xen/common/save.c | 315 +++++++++++++++++++++++++ xen/include/public/arch-arm/hvm/save.h | 5 + xen/include/public/arch-x86/hvm/save.h | 5 + xen/include/public/save.h | 89 +++++++ xen/include/xen/save.h | 170 +++++++++++++ 6 files changed, 585 insertions(+) create mode 100644 xen/common/save.c create mode 100644 xen/include/public/save.h create mode 100644 xen/include/xen/save.h
xen/common/Makefile +1 -0
@@ -37,6 +37,7 @@ obj-y += radix-tree.o obj-y += rbtree.o obj-y += rcupdate.o obj-y += rwlock.o +obj-y += save.o obj-y += shutdown.o obj-y += softirq.o obj-y += sort.o
xen/common/save.c new file +315 -0
315 lines not on this page · raw message
xen/include/public/arch-arm/hvm/save.h +5 -0
@@ -26,6 +26,11 @@#ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__#define __XEN_PUBLIC_HVM_SAVE_ARM_H__ +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /*
xen/include/public/arch-x86/hvm/save.h +5 -0
@@ -648,6 +648,11 @@ struct hvm_msr {*/ #define HVM_SAVE_CODE_MAX 20 +/* +* Further use of HVM state is deprecated. New state records should only +* be added to the domain state header: public/save.h +*/ +#endif /* __XEN_PUBLIC_HVM_SAVE_X86_H__ */ /*
xen/include/public/save.h new file +89 -0
89 lines not on this page · raw message
xen/include/xen/save.h new file +170 -0
170 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v8 2/8 xen/common/domctl: introduce XEN_DOMCTL_get/setdomaincontext
In reply to Paul DurrantThese domctls provide a mechanism to get and set domain context from the toolstack.
---
Cc: Daniel De Graaf <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v4:
- Add missing zero pad checks
v3:
- Addressed comments from Julien and Jan
- Use vmalloc() rather than xmalloc_bytes()
v2:
- drop mask parameter
- const-ify some more buffers
--- tools/flask/policy/modules/xen.if | 4 +- tools/libxc/include/xenctrl.h | 5 + tools/libxc/xc_domain.c | 56 +++++++++ xen/common/domctl.c | 173 ++++++++++++++++++++++++++++ xen/include/public/domctl.h | 41 +++++++ xen/xsm/flask/hooks.c | 6 + xen/xsm/flask/policy/access_vectors | 4 + 7 files changed, 287 insertions(+), 2 deletions(-)
tools/flask/policy/modules/xen.if +2 -2
@@ -53,7 +53,7 @@ define(`create_domain_common', ` allow $1 $2:domain2 { set_cpu_policy settsc setscheduler setclaim set_vnumainfo get_vnumainfo cacheflush psr_cmt_op psr_alloc soft_reset - resource_map get_cpu_policy }; + resource_map get_cpu_policy setcontext }; allow $1 $2:security check_context; allow $1 $2:shadow enable; allow $1 $2:mmu { map_read map_write adjust memorymap physmap pinpage mmuext_op updatemp };
@@ -97,7 +97,7 @@ define(`migrate_domain_out', ` allow $1 $2:hvm { gethvmc getparam }; allow $1 $2:mmu { stat pageinfo map_read }; allow $1 $2:domain { getaddrsize getvcpucontext pause destroy }; - allow $1 $2:domain2 gettsc; + allow $1 $2:domain2 { gettsc getcontext }; allow $1 $2:shadow { enable disable logdirty }; ')
tools/libxc/include/xenctrl.h +5 -0
@@ -867,6 +867,11 @@ int xc_domain_hvm_setcontext(xc_interface *xch,uint8_t *hvm_ctxt, uint32_t size); +int xc_domain_getcontext (xc_interface *xch,uint32_t domid, +void *ctxt_buf,size_t *size); +int xc_domain_setcontext (xc_interface *xch,uint32_t domid, +const void *ctxt_buf,size_t size); +/** * This function will return guest IO ABI protocol *
tools/libxc/xc_domain.c +56 -0
62 lines not on this page · raw message
xen/common/domctl.c +173 -0
191 lines not on this page · raw message
xen/include/public/domctl.h +41 -0
59 lines not on this page · raw message
xen/xsm/flask/hooks.c +6 -0
@@ -745,6 +745,12 @@ static int flask_domctl(struct domain *d, int cmd)case XEN_DOMCTL_get_cpu_policy:return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_CPU_POLICY ); +case XEN_DOMCTL_setdomaincontext: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__SETCONTEXT ); + +case XEN_DOMCTL_getdomaincontext: +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GETCONTEXT ); +default :return avc_unknown_permission ("domctl" , cmd); }
xen/xsm/flask/policy/access_vectors +4 -0
@@ -245,6 +245,10 @@ class domain2 resource_map # XEN_DOMCTL_get_cpu_policy get_cpu_policy +# XEN_DOMCTL_setdomaincontext + setcontext +# XEN_DOMCTL_getdomaincontext + getcontext } # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.20.1
PATCH v8 3/8 tools/misc: add xen-domctx to present domain context
In reply to Paul DurrantThis tool is analogous to 'xen-hvmctx' which presents HVM context. Subsequent patches will add 'dump' functions when new records are introduced.
---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
NOTE: Ian requested ack from Andrew
v3:
- Re-worked to avoid copying onto stack
- Added optional typecode and instance arguments
v2:
- Change name from 'xen-ctx' to 'xen-domctx'
--- .gitignore | 1 + tools/misc/Makefile | 4 + tools/misc/xen-domctx.c | 200 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 205 insertions(+) create mode 100644 tools/misc/xen-domctx.c
.gitignore +1 -0
@@ -212,6 +212,7 @@ tools/misc/xen_cpuperf tools/misc/xen-cpuid tools/misc/xen-detect tools/misc/xen-diag +tools/misc/xen-domctx tools/misc/xen-tmem-list-parse tools/misc/xen-livepatch tools/misc/xenperf
tools/misc/Makefile +4 -0
@@ -31,6 +31,7 @@ INSTALL_SBIN += xenpm INSTALL_SBIN += xenwatchdogd INSTALL_SBIN += xen-livepatch INSTALL_SBIN += xen-diag +INSTALL_SBIN += xen-domctx INSTALL_SBIN += $(INSTALL_SBIN-y) # Everything to be installed in a private bin/
@@ -114,6 +115,9 @@ xen-livepatch: xen-livepatch.o xen-diag: xen-diag.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) +xen-domctx: xen-domctx.o + $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) + xen-lowmemd: xen-lowmemd.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
tools/misc/xen-domctx.c new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v8 4/8 docs/specs: add missing definitions to libxc-migration-stream
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
The STATIC_DATA_END, X86_CPUID_POLICY and X86_MSR_POLICY record types have sections explaining what they are but their values are not defined. Indeed their values are defined as "Reserved for future mandatory records."
Also, the spec revision is adjusted to match the migration stream version and an END record is added to the description of a 'typical save record for and x86 HVM guest.'
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
docs/specs/libxc-migration-stream.pandoc +9 -2
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 2 +% Revision 3 Introduction ============
@@ -227,7 +227,13 @@ type 0x00000000: END 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary) - 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000010: STATIC_DATA_END + + 0x00000011: X86_CPUID_POLICY + + 0x00000012: X86_MSR_POLICY + + 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -732,6 +738,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_TSC_INFO * HVM_PARAMS * HVM_CONTEXT +* END record HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect the validity of architectural state in the context.
Signature · 1 line
2.20.1
PATCH v8 5/8 docs / tools: specific migration v4 to include DOMAIN_CONTEXT
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
A new 'domain context' framework was recently introduced to facilitate transfer of state for both PV and HVM guests. Hence this patch mandates the presence of a new DOMAIN_CONTEXT record in version 4 of the libxc migration stream.
This record will incorprate the content of the domain's 'shared_info' page and the TSC informaiton so the SHARED_INFO and TSC_INFO records are deprecated. It is intended that, in future, this record will contain state currently present in the HVM_CONTEXT record. However, for compatibility with earlier migration streams, the version 4 stream format continues to specify an HVM_CONTEXT record and XEN_DOMCTL_sethvmcontext will continue to accept all content of that record that may be present in a version 3 stream.
---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Marek Marczykowski-Górecki" <[email protected]>
v7:
- New in v7
--- docs/specs/libxc-migration-stream.pandoc | 62 ++++++++++++++++++------ tools/libxc/xc_sr_common.c | 1 + tools/libxc/xc_sr_stream_format.h | 1 + tools/python/xen/migration/libxc.py | 2 + 4 files changed, 52 insertions(+), 14 deletions(-)
docs/specs/libxc-migration-stream.pandoc +48 -14
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 3 +% Revision 4 Introduction ============
@@ -127,7 +127,7 @@ marker 0xFFFFFFFFFFFFFFFF. id 0x58454E46 ("XENF" in ASCII). -version 0x00000003. The version of this specification. +version 0x00000004. The version of this specification. options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
@@ -209,9 +209,9 @@ type 0x00000000: END 0x00000006: X86_PV_VCPU_XSAVE - 0x00000007: SHARED_INFO + 0x00000007: SHARED_INFO (deprecated) - 0x00000008: X86_TSC_INFO + 0x00000008: X86_TSC_INFO (deprecated) 0x00000009: HVM_CONTEXT
@@ -233,7 +233,9 @@ type 0x00000000: END 0x00000012: X86_MSR_POLICY - 0x00000013 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000013: DOMAIN_CONTEXT + + 0x00000014 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -442,10 +444,11 @@ X86_PV_VCPU_MSRS XEN_DOMCTL_{get,set}\_vcpu_msrs \clearpage -SHARED_INFO ------------ +SHARED_INFO (deprecated) +------------------------ -The content of the Shared Info page. +The content of the Shared Info page. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +-------------------------------------------------+
@@ -462,11 +465,12 @@ shared_info Contents of the shared info page. This record \clearpage -X86_TSC_INFO ------------- +X86_TSC_INFO (deprecated) +------------------------- Domain TSC information, as accessed by the -XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. +XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops. This is incorporated into the +DOMAIN_CONTEXT record as of specification version 4. 0 1 2 3 4 5 6 7 octet +------------------------+------------------------+
@@ -680,6 +684,25 @@ MSR_policy Array of xen_msr_entry_t[]'s \clearpage +DOMAIN_CONTEXT +-------------- + +Domain context, as accessed by the +XEN_DOMCTL_{get,set}domaincontext hypercall sub-ops. + + 0 1 2 3 4 5 6 7 octet + +-------------------------------------------------+ + | dom_ctx | + ... + +-------------------------------------------------+ + +-------------------------------------------------------------------- +Field Description +----------- --------------------------------------------------- +dom_ctx The Domain Context blob from Xen. +-------------------------------------------------------------------- + +\clearpage Layout ======
@@ -706,8 +729,7 @@ A typical save record for an x86 PV guest image would look like: * STATIC_DATA_END * X86_PV_P2M_FRAMES record * Many PAGE_DATA records -* X86_TSC_INFO -* SHARED_INFO record +* DOMAIN_CONTEXT * VCPU context records for each online VCPU * X86_PV_VCPU_BASIC record * X86_PV_VCPU_EXTENDED record
@@ -735,7 +757,7 @@ A typical save record for an x86 HVM guest image would look like: * X86_{CPUID,MSR}_POLICY * STATIC_DATA_END * Many PAGE_DATA records -* X86_TSC_INFO +* DOMAIN_CONTEXT * HVM_PARAMS * HVM_CONTEXT * END record
@@ -746,6 +768,18 @@ the validity of architectural state in the context. Compatibility with older versions ================================= +v4 compat with v3 +----------------- + +A v4 stream is compatible with a v3 stream, but mandates the presence of a +DOMAIN_CONTEXT record. This incorporates context such as the content of +the domain's Shared Info page and the TSC information, hence the SHARED_INFO +and TSC_INFO records are deprecated. +It also supercedes HVM_CONTEXT and, over time, data that is currently part of +the HVM_CONTEXT blob will move to the DOMAIN_CONTEXT blob. Xen, however, will +continue to accept all defined HVM_CONTEXT records so a v4-compatible +receiver can still accept an unmodified v3 stream. + v3 compat with v2 -----------------
tools/libxc/xc_sr_common.c +1 -0
@@ -39,6 +39,7 @@ static const char *const mandatory_rec_types[] = [REC_TYPE_STATIC_DATA_END] = "Static data end", [REC_TYPE_X86_CPUID_POLICY] = "x86 CPUID policy", [REC_TYPE_X86_MSR_POLICY] = "x86 MSR policy", + [REC_TYPE_DOMAIN_CONTEXT] = "Domain context", };const char *rec_type_to_str (uint32_t type)
tools/libxc/xc_sr_stream_format.h +1 -0
@@ -76,6 +76,7 @@ struct xc_sr_rhdr#define REC_TYPE_STATIC_DATA_END 0x00000010U#define REC_TYPE_X86_CPUID_POLICY 0x00000011U#define REC_TYPE_X86_MSR_POLICY 0x00000012U +#define REC_TYPE_DOMAIN_CONTEXT 0x00000013U#define REC_TYPE_OPTIONAL 0x80000000U
tools/python/xen/migration/libxc.py +2 -0
@@ -59,6 +59,7 @@ REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f REC_TYPE_static_data_end = 0x00000010 REC_TYPE_x86_cpuid_policy = 0x00000011 REC_TYPE_x86_msr_policy = 0x00000012 +REC_TYPE_domain_context = 0x00000013 rec_type_to_str = { REC_TYPE_end : "End",
@@ -80,6 +81,7 @@ rec_type_to_str = { REC_TYPE_static_data_end : "Static data end", REC_TYPE_x86_cpuid_policy : "x86 CPUID policy", REC_TYPE_x86_msr_policy : "x86 MSR policy", + REC_TYPE_domain_context : "Domain context", } # page_data
Signature · 1 line
2.20.1
PATCH v8 6/8 common/domain: add a domain context record for shared_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Processing of the content during restore is currently limited to
PV domains, and matches processing of the PV-only SHARED_INFO record done by libxc. All content is, however, saved such that restore processing can be modified in future without requiring a new record format.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
v8:
- Incorporate zero-ing out of shared info fields that would be done in processing of SHARED_INFO from older stream versions
v7:
- Only restore vcpu_info and arch sub-structures for PV domains, to match processing of SHARED_INFO in xc_sr_restore_x86_pv.c
- Use additional option to domain_load_end() to ignore the record for HVM domains
v6:
- Only save compat_shared_info buffer if has_32bit_shinfo is set
- Validate flags field in load handler
v5:
- Addressed comments from Julien
v4:
- Addressed comments from Jan
v3:
- Actually dump some of the content of shared_info
v2:
- Drop the header change to define a 'Xen' page size and instead use a variable length struct now that the framework makes this is feasible
- Guard use of 'has_32bit_shinfo' in common code with CONFIG_COMPAT
--- tools/misc/xen-domctx.c | 78 +++++++++++++++++++++ xen/common/domain.c | 144 ++++++++++++++++++++++++++++++++++++++ xen/include/public/save.h | 13 +++- 3 files changed, 234 insertions(+), 1 deletion(-)
tools/misc/xen-domctx.c +78 -0
@@ -31,6 +31,7 @@#include <errno.h> #include <xenctrl.h> +#include <xen-tools/libs.h> #include <xen/xen.h> #include <xen/domctl.h> #include <xen/save.h>
@@ -61,6 +62,82 @@ static void dump_header(void) } +static void print_binary (const char *prefix,const void *val,size_t size, +const char *suffix) +{ +printf ("%s" , prefix); + +while ( size-- ) + { +uint8_t octet = *(const uint8_t *)val++; +unsigned int i; + +for ( i =0 ; i <8 ; i++ ) + { +printf ("%u" , octet &1 ); + octet >>=1 ; + } + } + +printf ("%s" , suffix); +} + +static void dump_shared_info (void ) +{ +DOMAIN_SAVE_TYPE (SHARED_INFO) *s; +bool has_32bit_shinfo; +shared_info_any_t *info; +unsigned int i, n; + +GET_PTR (s); + has_32bit_shinfo = s->flags &DOMAIN_SAVE_32BIT_SHINFO ; + +printf (" SHARED_INFO: has_32bit_shinfo: %s buffer_size: %u\n" , + has_32bit_shinfo ?"true" :"false" , s->buffer_size); + + info = (shared_info_any_t *)s->buffer; + +#define GET_FIELD_PTR (_f) \ + (has_32bit_shinfo ? \ + (const void *)&(info->x32._f) : \ + (const void *)&(info->x64._f)) +#define GET_FIELD_SIZE (_f) \ + (has_32bit_shinfo ? sizeof(info->x32._f) : sizeof(info->x64._f)) +#define GET_FIELD (_f) \ + (has_32bit_shinfo ? info->x32._f : info->x64._f) + + n = has_32bit_shinfo ? +ARRAY_SIZE (info->x32.evtchn_pending) : +ARRAY_SIZE (info->x64.evtchn_pending); + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_pending: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_pending[0 ]), +GET_FIELD_SIZE (evtchn_pending[0 ]),"\n" ); + } + +for ( i =0 ; i < n; i++ ) + { +const char *prefix = !i ? +" evtchn_mask: " : +" " ; + +print_binary (prefix,GET_FIELD_PTR (evtchn_mask[0 ]), +GET_FIELD_SIZE (evtchn_mask[0 ]),"\n" ); + } + +printf (" wc: version: %u sec: %u nsec: %u\n" , +GET_FIELD (wc_version),GET_FIELD (wc_sec),GET_FIELD (wc_nsec)); + +#undef GET_FIELD +#undef GET_FIELD_SIZE +#undef GET_FIELD_PTR +} +static void dump_end (void ) {DOMAIN_SAVE_TYPE (END) *e;
@@ -173,6 +250,7 @@ int main(int argc, char **argv)switch (desc->typecode) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ; +case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);
xen/common/domain.c +144 -0
156 lines not on this page · raw message
xen/include/public/save.h +12 -1
@@ -82,7 +82,18 @@ struct domain_save_header { }; DECLARE_DOMAIN_SAVE_TYPE(HEADER, 1, struct domain_save_header); -#define DOMAIN_SAVE_CODE_MAX 1 +struct domain_shared_info_context { +uint32_t flags; + +#define DOMAIN_SAVE_32BIT_SHINFO 0x00000001 + +uint32_t buffer_size; +uint8_t buffer[XEN_FLEX_ARRAY_DIM ];/* Implementation specific size */ +}; + +DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context); + +#define DOMAIN_SAVE_CODE_MAX 2#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
Signature · 1 line
2.20.1
Re: [PATCH v8 6/8] common/domain: add a domain context record for shared_info...
Quoted 26 lines · Paul Durrant
xen/common/domain.c +26
-0+static int load_shared_info (struct domain *d,struct domain_context *c) +{ +struct domain_shared_info_context ctxt; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_BEGIN (SHARED_INFO , c, &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + +if ( ctxt.buffer_size >sizeof (shared_info_t) || + (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO ) ) +return -EINVAL ; + +if ( ctxt.flags &DOMAIN_SAVE_32BIT_SHINFO ) + { +#ifdef CONFIG_COMPAT +has_32bit_shinfo (d) = true;
xen/common/domain.c int continue_hypercall_on_cpu( · line 1729
Quoted 10 lines
xen/common/domain.c +10
-0+#else +return -EINVAL ; +#endif + } + +if (is_pv_domain (d) ) + { +shared_info_t *shinfo =xmalloc (shared_info_t); + + rc =domain_load_data (c, shinfo,sizeof (*shinfo));
You need to check the allocation's success first. But of course the question is why you don't read directly into d->shared_info. The domain is paused at this point, isn't it?
Quoted 53 lines
xen/common/domain.c +53
-0+if ( rc ) + { +xfree (shinfo); +return rc; + } + +#ifdef CONFIG_COMPAT +if (has_32bit_shinfo (d) ) + { +memcpy (&d->shared_info->compat.vcpu_info, + &shinfo->compat.vcpu_info, +sizeof (d->shared_info->compat.vcpu_info)); +memcpy (&d->shared_info->compat.arch, + &shinfo->compat.arch, +sizeof (d->shared_info->compat.vcpu_info)); +memset (&d->shared_info->compat.evtchn_pending, +0 , +sizeof (d->shared_info->compat.evtchn_pending)); +memset (&d->shared_info->compat.evtchn_mask, +0xff , +sizeof (d->shared_info->compat.evtchn_mask)); + + d->shared_info->compat.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info->compat.vcpu_info[i].evtchn_pending_sel =0 ; + } +else + { +memcpy (&d->shared_info->native.vcpu_info, + &shinfo->native.vcpu_info, +sizeof (d->shared_info->native.vcpu_info)); +memcpy (&d->shared_info->native.arch, + &shinfo->native.arch, +sizeof (d->shared_info->native.arch)); +memset (&d->shared_info->native.evtchn_pending, +0 , +sizeof (d->shared_info->compat.evtchn_pending)); +memset (&d->shared_info->native.evtchn_mask, +0xff , +sizeof (d->shared_info->native.evtchn_mask)); + + d->shared_info->native.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info->native.vcpu_info[i].evtchn_pending_sel =0 ; + } +#else +memcpy (&d->shared_info->vcpu_info, + &shinfo->vcpu_info, +sizeof (d->shared_info->vcpu_info)); +memcpy (&d->shared_info->arch, + &shinfo->arch, +sizeof (d->shared_info->shared)); + memset(&d->shared_info->evtchn_pending,
xen/common/domain.c int continue_hypercall_on_cpu( · line 1792
Quoted 10 lines
xen/common/domain.c +10
-0+0 , + sizeof(d->shared_info->evtchn_pending)); +memset (&d->shared_info->evtchn_mask, +0xff , +sizeof (d->shared_info->evtchn_mask)); + + d->shared_info.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info.vcpu_info[i].evtchn_pending_sel =0 ; +#endif
A lot of redundancy; maybe it gets better if indeed you stop reading into an intermediate buffer.
xen/common/domain.c int continue_hypercall_on_cpu( · line 1799
Quoted 6 lines
xen/common/domain.c +6
-0+xfree (shinfo); + + rc =domain_load_end (c, false); + } +else + rc =domain_load_end (c, true);
Perhaps at least a brief comment here wouldn't hurt regarding the needs (or lack thereof) for HVM / Arm?
Jan
RE: [PATCH v8 6/8] common/domain: add a domain context record for shared_info...
Quoted 46 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 16 September 2020 15:43
To: Paul Durrant <[email protected]>
Cc: [email protected]; Durrant, Paul <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [EXTERNAL] [PATCH v8 6/8] common/domain: add a domain context record forshared_info...CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 30 lines · Paul Durrant
xen/common/domain.c +30
-0+static int load_shared_info (struct domain *d,struct domain_context *c) +{ +struct domain_shared_info_context ctxt; +size_t hdr_size = offsetof(typeof (ctxt ), buffer); +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_BEGIN (SHARED_INFO , c, &i); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + + rc =domain_load_data (c, &ctxt, hdr_size); +if ( rc ) +return rc; + +if ( ctxt.buffer_size >sizeof (shared_info_t) || + (ctxt.flags & ~DOMAIN_SAVE_32BIT_SHINFO ) ) +return -EINVAL ; + +if ( ctxt.flags &DOMAIN_SAVE_32BIT_SHINFO ) + { +#ifdef CONFIG_COMPAT +has_32bit_shinfo (d) = true; +#else +return -EINVAL ; +#endif + }
Quoted 8 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1729
Quoted 6 lines
xen/common/domain.c +6
-0+ +if (is_pv_domain (d) ) + { +shared_info_t *shinfo =xmalloc (shared_info_t); + + rc =domain_load_data (c, shinfo,sizeof (*shinfo));You need to check the allocation's success first.
Oops, yes.
Quoted 4 lines
But of course the
question is why you don't read directly intod->shared_info. The domain is paused at this point, isn't it?
It is, but domain_create() may have initialized something outside of the areas we want to touch; I'm trying to follow what the existing restore code in toolstack currently does.
Quoted 58 lines
Quoted 58 lines
xen/common/domain.c +58
-0+if ( rc ) + { +xfree (shinfo); +return rc; + } + +#ifdef CONFIG_COMPAT +if (has_32bit_shinfo (d) ) + { +memcpy (&d->shared_info->compat.vcpu_info, + &shinfo->compat.vcpu_info, +sizeof (d->shared_info->compat.vcpu_info)); +memcpy (&d->shared_info->compat.arch, + &shinfo->compat.arch, +sizeof (d->shared_info->compat.vcpu_info)); +memset (&d->shared_info->compat.evtchn_pending, +0 , +sizeof (d->shared_info->compat.evtchn_pending)); +memset (&d->shared_info->compat.evtchn_mask, +0xff , +sizeof (d->shared_info->compat.evtchn_mask)); + + d->shared_info->compat.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info->compat.vcpu_info[i].evtchn_pending_sel =0 ; + } +else + { +memcpy (&d->shared_info->native.vcpu_info, + &shinfo->native.vcpu_info, +sizeof (d->shared_info->native.vcpu_info)); +memcpy (&d->shared_info->native.arch, + &shinfo->native.arch, +sizeof (d->shared_info->native.arch)); +memset (&d->shared_info->native.evtchn_pending, +0 , +sizeof (d->shared_info->compat.evtchn_pending)); +memset (&d->shared_info->native.evtchn_mask, +0xff , +sizeof (d->shared_info->native.evtchn_mask)); + + d->shared_info->native.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info->native.vcpu_info[i].evtchn_pending_sel =0 ; + } +#else +memcpy (&d->shared_info->vcpu_info, + &shinfo->vcpu_info, +sizeof (d->shared_info->vcpu_info)); +memcpy (&d->shared_info->arch, + &shinfo->arch, +sizeof (d->shared_info->shared)); +memset (&d->shared_info->evtchn_pending, +0 , +sizeof (d->shared_info->evtchn_pending)); +memset (&d->shared_info->evtchn_mask, +0xff , +sizeof (d->shared_info->evtchn_mask));
Quoted 8 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1792
Quoted 5 lines
xen/common/domain.c +5
-0+ + d->shared_info.arch.pfn_to_mfn_frame_list_list =0 ; +for ( i =0 ; i <XEN_LEGACY_MAX_VCPUS ; i++ ) + d->shared_info.vcpu_info[i].evtchn_pending_sel =0 ; +#endif A lot of redundancy; maybe it gets better if indeed you stop reading into an intermediate buffer.
As I said above, I think it is better if I continue to use an intermediate buffer but I'll see if I can add some macro magic to make this less verbose.
Quoted 11 lines
xen/common/domain.c int continue_hypercall_on_cpu( · line 1799
Quoted 7 lines
xen/common/domain.c +6
-0+xfree (shinfo); + + rc =domain_load_end (c, false); + } +else + rc =domain_load_end (c, true);Perhaps at least a brief comment here wouldn't hurt regarding the needs (or lack thereof) for HVM / Arm?
Ok.
Paul
Quoted 1 line
Jan
PATCH v8 7/8 x86/time: add a domain context record for tsc_info...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and update xen-domctx to dump some information describing the record.
NOTE: Whilst the record definition is x86 specific, it is visible directly
in the common header as context record numbers should be unique across all architectures.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v8:
- Removed stray blank line
v7:
- New in v7
--- tools/misc/xen-domctx.c | 11 +++++++++++ xen/arch/x86/time.c | 34 +++++++++++++++++++++++++++++++++- xen/include/asm-x86/time.h | 5 +++-- xen/include/public/save.h | 13 ++++++++++++- 4 files changed, 59 insertions(+), 4 deletions(-)
tools/misc/xen-domctx.c +11 -0
@@ -59,7 +59,17 @@ static void dump_header(void)printf (" HEADER: magic %#x, version %u\n" , h->magic, h->version); +} + +static void dump_tsc_info (void ) +{ +DOMAIN_SAVE_TYPE (TSC_INFO) *t; + +GET_PTR (t); +printf (" TSC_INFO: mode: %u incarnation: %u\n" +" khz %u elapsed_nsec: %" PRIu64"\n" , + t->mode, t->incarnation, t->khz, t->elapsed_nsec); } static void print_binary(const char *prefix, const void *val, size_t size,
@@ -251,6 +261,7 @@ int main(int argc, char **argv) {case DOMAIN_SAVE_CODE (HEADER ):dump_header ();break ;case DOMAIN_SAVE_CODE (SHARED_INFO ):dump_shared_info ();break ; +case DOMAIN_SAVE_CODE (TSC_INFO ):dump_tsc_info ();break ;case DOMAIN_SAVE_CODE (END ):dump_end ();break ;default :printf ("Unknown type %u: skipping\n" , desc->typecode);
xen/arch/x86/time.c +33 -1
@@ -26,6 +26,7 @@#include <xen/symbols.h> #include <xen/keyhandler.h> #include <xen/guest_access.h> +#include <xen/save.h> #include <asm/io.h> #include <asm/iocap.h> #include <asm/msr.h>
@@ -2334,7 +2335,7 @@ int host_tsc_is_safe(void)* called to collect tsc-related data only for save file or live * migrate; called after last rdtsc is done on this incarnation */ -void tsc_get_info (struct domain *d,uint32_t *tsc_mode, +void tsc_get_info (const struct domain *d,uint32_t *tsc_mode,uint64_t *elapsed_nsec,uint32_t *gtsc_khz,uint32_t *incarnation) {
@@ -2453,6 +2454,37 @@ int tsc_set_info(struct domain *d,return 0 ; } +static int save_tsc_info (const struct domain *d,struct domain_context *c, +bool dry_run) +{ +struct domain_tsc_info_context ctxt; + +if ( !dry_run ) +tsc_get_info (d, &ctxt.mode, &ctxt.elapsed_nsec, &ctxt.khz, + &ctxt.incarnation); + +return DOMAIN_SAVE_ENTRY (TSC_INFO , c,0 , &ctxt,sizeof (ctxt)); +} + +static int load_tsc_info (struct domain *d,struct domain_context *c) +{ +struct domain_tsc_info_context ctxt; +unsigned int i; +int rc; + + rc =DOMAIN_LOAD_ENTRY (TSC_INFO , c, &i, &ctxt,sizeof (ctxt)); +if ( rc ) +return rc; + +if ( i )/* expect only a single instance */ +return -ENXIO ; + +return tsc_set_info (d, ctxt.mode, ctxt.elapsed_nsec, ctxt.khz, + ctxt.incarnation); +} + +DOMAIN_REGISTER_SAVE_LOAD (TSC_INFO , save_tsc_info, load_tsc_info); +/* vtsc may incur measurable performance degradation, diagnose with this */ static void dump_softtsc (unsigned char key) {
xen/include/asm-x86/time.h +3 -2
@@ -59,8 +59,9 @@ u64 gtsc_to_gtime(struct domain *d, u64 tsc);int tsc_set_info (struct domain *d,uint32_t tsc_mode,uint64_t elapsed_nsec,uint32_t gtsc_khz,uint32_t incarnation); -void tsc_get_info (struct domain *d,uint32_t *tsc_mode,uint64_t *elapsed_nsec, -uint32_t *gtsc_khz,uint32_t *incarnation); +void tsc_get_info (const struct domain *d,uint32_t *tsc_mode, +uint64_t *elapsed_nsec,uint32_t *gtsc_khz, +uint32_t *incarnation);void force_update_vcpu_system_time (struct vcpu *v);
xen/include/public/save.h +12 -1
@@ -93,7 +93,18 @@ struct domain_shared_info_context { DECLARE_DOMAIN_SAVE_TYPE(SHARED_INFO, 2, struct domain_shared_info_context); -#define DOMAIN_SAVE_CODE_MAX 2 +#if defined(__i386__) || defined(__x86_64__) +struct domain_tsc_info_context { +uint32_t mode; +uint32_t incarnation; +uint64_t elapsed_nsec; +uint32_t khz; +}; + +DECLARE_DOMAIN_SAVE_TYPE(TSC_INFO, 3, struct domain_tsc_info_context); +#endif + +#define DOMAIN_SAVE_CODE_MAX 3#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
Signature · 1 line
2.20.1
PATCH v8 8/8 tools/libxc: add DOMAIN_CONTEXT records to the migration stream...
In reply to Paul DurrantFrom: Paul Durrant <[email protected]>
... and bump the version.
This patch implements version 4 of the migration stream by adding the code necessary to save and restore DOMAIN_CONTEXT records, and removing the code to save the SHARED_INFO and TSC_INFO records (as these are deprecated in version 4).
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
v7:
- New in v7
--- tools/libxc/xc_sr_common.h | 3 ++ tools/libxc/xc_sr_common_x86.c | 20 ------------ tools/libxc/xc_sr_common_x86.h | 6 ---- tools/libxc/xc_sr_restore.c | 45 +++++++++++++++++++++++++-- tools/libxc/xc_sr_save.c | 52 +++++++++++++++++++++++++++++++- tools/libxc/xc_sr_save_x86_hvm.c | 5 --- tools/libxc/xc_sr_save_x86_pv.c | 22 -------------- 7 files changed, 97 insertions(+), 56 deletions(-)
tools/libxc/xc_sr_common.h +3 -0
@@ -298,6 +298,9 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; + +/* Domain context blob. */ +struct xc_sr_blob context; } restore; };
tools/libxc/xc_sr_common_x86.c +0 -20
@@ -1,25 +1,5 @@#include "xc_sr_common_x86.h" -int write_x86_tsc_info (struct xc_sr_context *ctx) -{ -xc_interface *xch = ctx->xch; -struct xc_sr_rec_x86_tsc_info tsc = {}; -struct xc_sr_record rec = { - .type =REC_TYPE_X86_TSC_INFO , - .length =sizeof (tsc), - .data = &tsc, - }; - -if (xc_domain_get_tsc_info (xch, ctx->domid, &tsc.mode, - &tsc.nsec, &tsc.khz, &tsc.incarnation) <0 ) - { -PERROR ("Unable to obtain TSC information" ); -return -1 ; - } - -return write_record (ctx, &rec); -} -int handle_x86_tsc_info (struct xc_sr_context *ctx,struct xc_sr_record *rec) {xc_interface *xch = ctx->xch;
tools/libxc/xc_sr_common_x86.h +0 -6
@@ -3,12 +3,6 @@#include "xc_sr_common.h" -/* -* Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record -* into the stream. -*/ -int write_x86_tsc_info (struct xc_sr_context *ctx); -/* * Parses a X86_TSC_INFO record and applies the result to the domain. */
tools/libxc/xc_sr_restore.c +43 -2
82 lines not on this page · raw message
tools/libxc/xc_sr_save.c +51 -1
70 lines not on this page · raw message
tools/libxc/xc_sr_save_x86_hvm.c +0 -5
@@ -193,11 +193,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; -/* Write the TSC record. */ - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; -/* Write the HVM_CONTEXT record. */ rc =write_hvm_context (ctx);if ( rc )
tools/libxc/xc_sr_save_x86_pv.c +0 -22
@@ -849,20 +849,6 @@ static int write_x86_pv_p2m_frames(struct xc_sr_context *ctx)return rc; } -/* -* Writes an SHARED_INFO record into the stream. -*/ -static int write_shared_info (struct xc_sr_context *ctx) -{ -struct xc_sr_record rec = { - .type =REC_TYPE_SHARED_INFO , - .length =PAGE_SIZE , - .data = ctx->x86.pv.shinfo, - }; - -return write_record (ctx, &rec); -} -/* * Normalise a pagetable for the migration stream. Performs mfn->pfn * conversions on the ptes.
@@ -1093,14 +1079,6 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; - rc =write_x86_tsc_info (ctx); -if ( rc ) -return rc; - - rc =write_shared_info (ctx); -if ( rc ) -return rc; - rc =write_all_vcpu_information (ctx);if ( rc )return rc;
Signature · 1 line
2.20.1