leftover from 2020
v20210601 cover leftover from 2020
Various unreviewed changes, rebase to 57f68dfd2d.
Olaf Hering (38):
tools: add API to work with sevaral bits at once
xl: fix description of migrate --debug
tools: create libxensaverestore
tools: add readv_exact to libxenctrl
tools: add xc_is_known_page_type to libxenctrl
tools: use xc_is_known_page_type
tools: unify type checking for data pfns in migration stream
tools: show migration transfer rate in send_dirty_pages tools/guest: prepare to allocate arrays once tools/guest: save: move batch_pfns
tools/guest: save: move mfns array
tools/guest: save: move types array
tools/guest: save: move errors array
tools/guest: save: move iov array
tools/guest: save: move rec_pfns array
tools/guest: save: move guest_data array
tools/guest: save: move local_pages array
tools/guest: restore: move pfns array
tools/guest: restore: move types array
tools/guest: restore: move mfns array
tools/guest: restore: move map_errs array
tools/guest: restore: move mfns array in populate_pfns tools/guest: restore: move pfns array in populate_pfns tools/guest: restore: split record processing tools/guest: restore: split handle_page_data tools/guest: restore: write data directly into guest
tools: recognize LIBXL_API_VERSION for 4.16
tools: adjust libxl_domain_suspend to receive a struct props
tools: change struct precopy_stats to precopy_stats_t
tools: add callback to libxl for precopy_policy and precopy_stats_t
tools: add --max_iters to libxl_domain_suspend
tools: add --min_remaining to libxl_domain_suspend
tools: add --abort_if_busy to libxl_domain_suspend
tools: add API for expandable bitmaps
tools: use xg_sr_bitmap for populated_pfns
tools: use superpages during restore of HVM guest
tools: remove migration stream verify code hotplug/Linux: fix starting of xenstored with restarting systemd
.gitignore | 2 +
docs/man/xl.1.pod.in | 24 +-
tools/hotplug/Linux/init.d/xencommons.in | 2 +-
tools/hotplug/Linux/launch-xenstore.in | 40 +-
.../Linux/systemd/xenstored.service.in | 2 +-
tools/include/libxl.h | 32 +-
tools/include/xenguest.h | 186 -----
tools/include/xensaverestore.h | 207 ++++++
tools/libs/Makefile | 1 +
tools/libs/ctrl/xc_bitops.h | 25 +
tools/libs/ctrl/xc_private.c | 55 +-
tools/libs/ctrl/xc_private.h | 34 +
tools/libs/guest/Makefile | 11 -
tools/libs/guest/xg_dom_x86.c | 5 -
tools/libs/guest/xg_offline_page.c | 1 -
tools/libs/guest/xg_private.h | 5 +
tools/libs/guest/xg_sr_restore_x86_hvm.c | 274 --------
tools/libs/light/Makefile | 4 +-
tools/libs/light/libxl_dom_save.c | 24 +
tools/libs/light/libxl_domain.c | 10 +-
tools/libs/light/libxl_internal.h | 7 +
tools/libs/light/libxl_save_helper.c | 1 +
tools/libs/light/libxl_save_msgs_gen.pl | 5 +-
tools/libs/light/libxl_stream_write.c | 9 +-
tools/libs/light/libxl_types.idl | 1 +
tools/libs/saverestore/Makefile | 38 ++
.../xg_sr_common.c => saverestore/common.c} | 77 ++-
.../xg_sr_common.h => saverestore/common.h} | 224 +++++-
.../common_x86.c} | 2 +-
.../common_x86.h} | 2 +-
.../common_x86_pv.c} | 2 +-
.../common_x86_pv.h} | 2 +-
.../nomigrate.c} | 0
.../xg_sr_restore.c => saverestore/restore.c} | 598 ++++++++--------
tools/libs/saverestore/restore_x86_hvm.c | 644 ++++++++++++++++++
.../restore_x86_pv.c} | 70 +-
.../xg_sr_save.c => saverestore/save.c} | 209 +++---
.../save_restore.h} | 2 -
.../save_x86_hvm.c} | 7 +-
.../save_x86_pv.c} | 33 +-
.../stream_format.h} | 0
tools/libs/uselibs.mk | 4 +-
tools/ocaml/libs/xl/xenlight_stubs.c | 3 +-
tools/xl/xl_cmdtable.c | 26 +-
tools/xl/xl_migrate.c | 54 +-
tools/xl/xl_saverestore.c | 3 +-
46 files changed, 1970 insertions(+), 997 deletions(-)
create mode 100644 tools/include/xensaverestore.h
delete mode 100644 tools/libs/guest/xg_sr_restore_x86_hvm.c
create mode 100644 tools/libs/saverestore/Makefile
rename tools/libs/{guest/xg_sr_common.c => saverestore/common.c} (71%)
rename tools/libs/{guest/xg_sr_common.h => saverestore/common.h} (70%)
rename tools/libs/{guest/xg_sr_common_x86.c => saverestore/common_x86.c} (99%)
rename tools/libs/{guest/xg_sr_common_x86.h => saverestore/common_x86.h} (98%)
rename tools/libs/{guest/xg_sr_common_x86_pv.c => saverestore/common_x86_pv.c} (99%)
rename tools/libs/{guest/xg_sr_common_x86_pv.h => saverestore/common_x86_pv.h} (98%)
rename tools/libs/{guest/xg_nomigrate.c => saverestore/nomigrate.c} (100%)
rename tools/libs/{guest/xg_sr_restore.c => saverestore/restore.c} (67%)
create mode 100644 tools/libs/saverestore/restore_x86_hvm.c
rename tools/libs/{guest/xg_sr_restore_x86_pv.c => saverestore/restore_x86_pv.c} (94%)
rename tools/libs/{guest/xg_sr_save.c => saverestore/save.c} (85%)
rename tools/libs/{guest/xg_save_restore.h => saverestore/save_restore.h} (98%)
rename tools/libs/{guest/xg_sr_save_x86_hvm.c => saverestore/save_x86_hvm.c} (96%)
rename tools/libs/{guest/xg_sr_save_x86_pv.c => saverestore/save_x86_pv.c} (97%)
rename tools/libs/{guest/xg_sr_stream_format.h => saverestore/stream_format.h} (100%)PATCH v20210601 1/38 tools: add API to work with sevaral bits at once
Introduce new API to test if a fixed number of bits is clear or set, and clear or set them all at once.
The caller has to make sure the input bitnumber is a multiply of BITS_PER_LONG.
This API avoids the loop over each bit in a known range just to see if all of them are either clear or set.
--- tools/libs/ctrl/xc_bitops.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
tools/libs/ctrl/xc_bitops.h +25 -0
@@ -77,4 +77,29 @@ static inline void bitmap_or(void *_dst, const void *_other, dst[i] |= other[i]; } +static inline int test_bit_long_set (unsigned long nr_base,const void *_addr) +{ +const unsigned long *addr = _addr; +unsigned long val = addr[nr_base /BITS_PER_LONG ]; +return val == ~0 ; +} + +static inline int test_bit_long_clear (unsigned long nr_base,const void *_addr) +{ +const unsigned long *addr = _addr; +unsigned long val = addr[nr_base /BITS_PER_LONG ]; +return val ==0 ; +} + +static inline void clear_bit_long (unsigned long nr_base,void *_addr) +{ +unsigned long *addr = _addr; + addr[nr_base /BITS_PER_LONG ] =0 ; +} + +static inline void set_bit_long (unsigned long nr_base,void *_addr) +{ +unsigned long *addr = _addr; + addr[nr_base /BITS_PER_LONG ] = ~0 ; +}#endif /* XC_BITOPS_H */
Re: [PATCH v20210601 01/38] tools: add API to work with sevaral bits at once
Quoted 2 lines · Olaf Hering
Introduce new API to test if a fixed number of bits is clear or set, and clear or set them all at once.
More precise: the new functions check whether BITS_PER_LONG bits are set or clear.
Quoted 1 line
The caller has to make sure the input bitnumber is a multiply of
BITS_PER_LONG.
s/multiply/multiple/
Quoted 14 lines
This API avoids the loop over each bit in a known range just to see if all of them are either clear or set.
Signed-off-by Olaf Hering <[email protected]>--- tools/libs/ctrl/xc_bitops.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)tools/libs/ctrl/xc_bitops.h +0
-0
tools/libs/ctrl/xc_bitops.h static inline void bitmap_or(void *_dst, const void *_other, · line 80
Quoted 4 lines
tools/libs/ctrl/xc_bitops.h +1
-0@@ -77,4 +77,29 @@ static inline void bitmap_or(void *_dst, const void *_other, dst[i] |= other[i]; } +static inline int test_bit_long_set (unsigned long nr_base,const void *_addr)
Make return type bool (here and below)?
tools/libs/ctrl/xc_bitops.h static inline void bitmap_or(void *_dst, const void *_other, · line 83
Quoted 3 lines
tools/libs/ctrl/xc_bitops.h +3
-0+{ +const unsigned long *addr = _addr; +unsigned long val = addr[nr_base /BITS_PER_LONG ];
Add a blank line here (same below).
Quoted 12 lines
tools/libs/ctrl/xc_bitops.h +11
-0+ return val == ~0; > +} + +static inline int test_bit_long_clear (unsigned long nr_base,const void *_addr) +{ +const unsigned long *addr = _addr; +unsigned long val = addr[nr_base /BITS_PER_LONG ]; +return val ==0 ; +} + +static inline void clear_bit_long (unsigned long nr_base,void *_addr) +{
tools/libs/ctrl/xc_bitops.h static inline void bitmap_or(void *_dst, const void *_other, · line 105
Quoted 10 lines
tools/libs/ctrl/xc_bitops.h +9
-0+unsigned long *addr = _addr; + addr[nr_base / BITS_PER_LONG] = 0; +} + +static inline void set_bit_long (unsigned long nr_base,void *_addr) +{ +unsigned long *addr = _addr; + addr[nr_base /BITS_PER_LONG ] = ~0 ; +}#endif /* XC_BITOPS_H */
Juergen
PATCH v20210601 2/38 xl: fix description of migrate --debug
In reply to Olaf Heringxl migrate --debug used to track every pfn in every batch of pages. But these times are gone. Adjust the help text to tell what --debug is supposed to do today.
--- docs/man/xl.1.pod.in | 4 +++- tools/libs/guest/xg_sr_save.c | 2 +- tools/xl/xl_cmdtable.c | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)
docs/man/xl.1.pod.in +3 -1
@@ -481,7 +481,9 @@ domain. =item B<--debug> -Display huge (!) amount of debug information during the migration process. +Verify transferred domU page data. All memory will be transferred one more +time to the destination host while the domU is paused, and compared with +the result of the inital transfer while the domU was still running. =item B<-p>
tools/libs/guest/xg_sr_save.c +1 -1
@@ -752,7 +752,7 @@ static int send_domain_memory_live(struct xc_sr_context *ctx)if ( rc ) goto out; -if ( ctx->save.debug && ctx->stream_type !=XC_STREAM_PLAIN ) +if ( ctx->save.debug ) { rc =verify_frames (ctx);if ( rc )
tools/xl/xl_cmdtable.c +1 -1
@@ -172,7 +172,7 @@ const struct cmd_spec cmd_table[] = {" migrate-receive [-d -e]\n" "-e Do not wait in the background (on <host>) for the death\n" " of the domain.\n" -"--debug Print huge (!) amount of debug during the migration process.\n" +"--debug Verify transferred domU page data.\n" "-p Do not unpause domain after migrating it.\n" "-D Preserve the domain id" },
Re: [PATCH v20210601 02/38] xl: fix description of migrate --debug
Quoted 16 lines · Olaf Hering
xl migrate --debug used to track every pfn in every batch of pages. But these times are gone. Adjust the help text to tell what --debug is supposed to do today.
Signed-off-by Olaf Hering <[email protected]>--- docs/man/xl.1.pod.in | 4 +++- tools/libs/guest/xg_sr_save.c | 2 +- tools/xl/xl_cmdtable.c | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)docs/man/xl.1.pod.in +0
-0
docs/man/xl.1.pod.in domain. · line 486
Quoted 7 lines
docs/man/xl.1.pod.in +3
-1@@ -481,7 +481,9 @@ domain. =item B<--debug> -Display huge (!) amount of debug information during the migration process. +Verify transferred domU page data. All memory will be transferred one more +time to the destination host while the domU is paused, and compared with +the result of the inital transfer while the domU was still running.
Shouldn't you adapt (or remove?) this paragraph with patch 37?
tools/libs/guest/xg_sr_save.c static int send_domain_memory_live(struct xc_sr_context *ctx) · line 755
Quoted 13 lines
docs/man/xl.1.pod.in +0
-0=item B<-p>tools/libs/guest/xg_sr_save.c +1
-1@@ -752,7 +752,7 @@ static int send_domain_memory_live(struct xc_sr_context *ctx)if ( rc ) goto out; -if ( ctx->save.debug && ctx->stream_type !=XC_STREAM_PLAIN ) +if ( ctx->save.debug )
This is no documentation change IMO. You should either mention this modification in the commit message, or put it into a separate patch.
Juergen
Re: [PATCH v20210601 02/38] xl: fix description of migrate --debug
Am Wed, 2 Jun 2021 08:09:00 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 4 lines
tools/libs/guest/xg_sr_save.c static int send_domain_memory_live(struct xc_sr_context *ctx) · line 755
Quoted 2 lines
tools/libs/guest/xg_sr_save.c +1
-1-if ( ctx->save.debug && ctx->stream_type !=XC_STREAM_PLAIN ) +if ( ctx->save.debug )This is no documentation change IMO. You should either mention this modification in the commit message, or put it into a separate patch.
I think the conclusion was that this branch is dead code because only the stream_type==XC_STREAM_PLAIN code branch sets debug.
So far there is a decision to be made about this code branch:
- document it
- remove it
- fix it
The latter might be impossible due to lack of APIs to query the usage of a given page.
But, perhaps remus and/or colo does not suffer from apparent corruption. One day I should create a remus and a colo setup to exercise these code paths.
Olaf
Re: [PATCH v20210601 02/38] xl: fix description of migrate --debug
Quoted 8 lines · Olaf Hering
Am Wed, 2 Jun 2021 08:09:00 +0200
schrieb Juergen Gross <[email protected]>:Quoted 4 lines
tools/libs/guest/xg_sr_save.c static int send_domain_memory_live(struct xc_sr_context *ctx) · line 755
Quoted 2 lines
tools/libs/guest/xg_sr_save.c +1
-1-if ( ctx->save.debug && ctx->stream_type !=XC_STREAM_PLAIN ) +if ( ctx->save.debug )This is no documentation change IMO. You should either mention this modification in the commit message, or put it into a separate patch.
Quoted 8 lines
I think the conclusion was that this branch is dead code because only the stream_type==XC_STREAM_PLAIN code branch sets debug.
So far there is a decision to be made about this code branch:
- document it
- remove it
- fix it
I don't mind either way. I just pointed out an inconsistency between patch title and content. When searching for a patch causing a regression I usually skip documentation patches.
Juergen
PATCH v20210602 2/38 xl: fix description of migrate --debug
In reply to Olaf Heringxl migrate --debug used to track every pfn in every batch of pages. But these times are gone. The code in xc_domain_save is the consumer of this knob, but it considers it only for the remus and colo case.
Adjust the help text to tell what --debug does today: Nothing.
v02:
- the option has no effect anymore
--- docs/man/xl.1.pod.in | 2 +- tools/xl/xl_cmdtable.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
docs/man/xl.1.pod.in +1 -1
@@ -481,7 +481,7 @@ domain. =item B<--debug> -Display huge (!) amount of debug information during the migration process. +This option has no effect. It is preserved for compatibility reasons. =item B<-p>
tools/xl/xl_cmdtable.c +1 -1
@@ -172,7 +172,7 @@ const struct cmd_spec cmd_table[] = {" migrate-receive [-d -e]\n" "-e Do not wait in the background (on <host>) for the death\n" " of the domain.\n" -"--debug Print huge (!) amount of debug during the migration process.\n" +"--debug Ignored.\n" "-p Do not unpause domain after migrating it.\n" "-D Preserve the domain id" },
Re: [PATCH v20210602 02/38] xl: fix description of migrate --debug
Quoted 10 lines · Olaf Hering
xl migrate --debug used to track every pfn in every batch of pages. But these times are gone. The code in
xc_domain_saveis the consumer of this knob, but it considers it only for the remus and colo case.Adjust the help text to tell what --debug does today: Nothing.
Signed-off-by Olaf Hering <[email protected]>v02:
- the option has no effect anymore
Juergen
PATCH v20210601 3/38 tools: create libxensaverestore
In reply to Olaf HeringMove all save/restore related code from libxenguest.so into a separate library libxensaverestore.so. The only consumer is libxl-save-helper. There is no need to have the moved code mapped all the time in binaries where libxenguest.so is used.
According to size(1) the change is:
text data bss dec hex filename
187183 4304 48 191535 2ec2f guest/libxenguest.so.4.15.0
124106 3376 48 127530 1f22a guest/libxenguest.so.4.15.0 67841 1872 8 69721 11059 saverestore/libxensaverestore.so.4.15.0
While touching the files anyway, take the opportunity to drop the reduntant xg_sr_ filename prefix.
v4:
- drop xg_ prefix from filenames (jgross)
- drop sr_ prefix from filenames (jbeulich)
v3:
- repost in time for 4.16
v2:
- copy also license header
- move xg_nomigrate.c
- add size(1) output to commit msg
- remove change from libxl_create.c
---
.gitignore | 2 +
tools/include/xenguest.h | 186 ----------------
tools/include/xensaverestore.h | 208 ++++++++++++++++++
tools/libs/Makefile | 1 +
tools/libs/guest/Makefile | 11 -
tools/libs/guest/xg_offline_page.c | 1 -
tools/libs/light/Makefile | 4 +-
tools/libs/light/libxl_internal.h | 1 +
tools/libs/light/libxl_save_helper.c | 1 +
tools/libs/light/libxl_save_msgs_gen.pl | 2 +-
tools/libs/saverestore/Makefile | 38 ++++
.../xg_sr_common.c => saverestore/common.c} | 2 +-
.../xg_sr_common.h => saverestore/common.h} | 16 +-
.../common_x86.c} | 2 +-
.../common_x86.h} | 2 +-
.../common_x86_pv.c} | 2 +-
.../common_x86_pv.h} | 2 +-
.../nomigrate.c} | 0
.../xg_sr_restore.c => saverestore/restore.c} | 2 +-
.../restore_x86_hvm.c} | 2 +-
.../restore_x86_pv.c} | 2 +-
.../xg_sr_save.c => saverestore/save.c} | 2 +-
.../save_restore.h} | 2 -
.../save_x86_hvm.c} | 2 +-
.../save_x86_pv.c} | 2 +-
.../stream_format.h} | 0
tools/libs/uselibs.mk | 4 +-
27 files changed, 282 insertions(+), 217 deletions(-)
create mode 100644 tools/include/xensaverestore.h
create mode 100644 tools/libs/saverestore/Makefile
rename tools/libs/{guest/xg_sr_common.c => saverestore/common.c} (99%)
rename tools/libs/{guest/xg_sr_common.h => saverestore/common.h} (98%)
rename tools/libs/{guest/xg_sr_common_x86.c => saverestore/common_x86.c} (99%)
rename tools/libs/{guest/xg_sr_common_x86.h => saverestore/common_x86.h} (98%)
rename tools/libs/{guest/xg_sr_common_x86_pv.c => saverestore/common_x86_pv.c} (99%)
rename tools/libs/{guest/xg_sr_common_x86_pv.h => saverestore/common_x86_pv.h} (98%)
rename tools/libs/{guest/xg_nomigrate.c => saverestore/nomigrate.c} (100%)
rename tools/libs/{guest/xg_sr_restore.c => saverestore/restore.c} (99%)
rename tools/libs/{guest/xg_sr_restore_x86_hvm.c => saverestore/restore_x86_hvm.c} (99%)
rename tools/libs/{guest/xg_sr_restore_x86_pv.c => saverestore/restore_x86_pv.c} (99%)
rename tools/libs/{guest/xg_sr_save.c => saverestore/save.c} (99%)
rename tools/libs/{guest/xg_save_restore.h => saverestore/save_restore.h} (98%)
rename tools/libs/{guest/xg_sr_save_x86_hvm.c => saverestore/save_x86_hvm.c} (99%)
rename tools/libs/{guest/xg_sr_save_x86_pv.c => saverestore/save_x86_pv.c} (99%)
rename tools/libs/{guest/xg_sr_stream_format.h => saverestore/stream_format.h} (100%).gitignore +2 -0
@@ -147,6 +147,8 @@ tools/libs/light/test_timedereg tools/libs/light/test_fdderegrace tools/libs/light/tmp.* tools/libs/light/xenlight.pc +tools/libs/saverestore/libxensaverestore.map +tools/libs/saverestore/xensaverestore.pc tools/libs/stat/_paths.h tools/libs/stat/headers.chk tools/libs/stat/libxenstat.map
tools/include/xenguest.h +0 -186
198 lines not on this page · raw message
tools/include/xensaverestore.h new file +208 -0
208 lines not on this page · raw message
tools/libs/Makefile +1 -0
@@ -12,6 +12,7 @@ SUBDIRS-y += devicemodel SUBDIRS-y += ctrl SUBDIRS-y += guest SUBDIRS-y += hypfs +SUBDIRS-y += saverestore SUBDIRS-y += store SUBDIRS-y += stat SUBDIRS-$(CONFIG_Linux) += vchan
tools/libs/guest/Makefile +0 -11
@@ -10,18 +10,7 @@ SRCS-y += xg_private.c SRCS-y += xg_domain.c SRCS-y += xg_suspend.c ifeq ($(CONFIG_MIGRATE),y) -SRCS-y += xg_sr_common.c -SRCS-$(CONFIG_X86) += xg_sr_common_x86.c -SRCS-$(CONFIG_X86) += xg_sr_common_x86_pv.c -SRCS-$(CONFIG_X86) += xg_sr_restore_x86_pv.c -SRCS-$(CONFIG_X86) += xg_sr_restore_x86_hvm.c -SRCS-$(CONFIG_X86) += xg_sr_save_x86_pv.c -SRCS-$(CONFIG_X86) += xg_sr_save_x86_hvm.c -SRCS-y += xg_sr_restore.c -SRCS-y += xg_sr_save.c SRCS-y += xg_offline_page.c -else -SRCS-y += xg_nomigrate.c endif CFLAGS += -I$(XEN_libxenctrl)
tools/libs/guest/xg_offline_page.c +0 -1
@@ -29,7 +29,6 @@#include "xc_private.h" #include "xg_private.h" -#include "xg_save_restore.h" struct pte_backup_entry {
tools/libs/light/Makefile +2 -2
@@ -179,7 +179,7 @@ $(ACPI_OBJS) $(ACPI_PIC_OBJS): CFLAGS += -I. -DLIBACPI_STDUTILS=\"$(CURDIR)/libx $(TEST_PROG_OBJS) _libxl.api-for-check: CFLAGS += $(CFLAGS_libxentoollog) $(CFLAGS_libxentoolcore) libxl_dom.o libxl_dom.opic: CFLAGS += -I$(XEN_ROOT)/tools # include libacpi/x86.h libxl_x86_acpi.o libxl_x86_acpi.opic: CFLAGS += -I$(XEN_ROOT)/tools -$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenevtchn) $(CFLAGS_libxenguest) +$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenevtchn) $(CFLAGS_libxensaverestore) testidl.o: CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight) testidl.c: libxl_types.idl gentest.py $(XEN_INCLUDE)/libxl.h $(AUTOINCS)
@@ -241,7 +241,7 @@ test_%: test_%.o test_common.o libxenlight_test.so $(CC) $(LDFLAGS) -o $@ $^ $(filter-out %libxenlight.so, $(LDLIBS_libxenlight)) $(LDLIBS_libxentoollog) $(LDLIBS_libxentoolcore) -lyajl $(APPEND_LDFLAGS) libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so - $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxentoollog) $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxentoolcore) $(APPEND_LDFLAGS) + $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxentoollog) $(LDLIBS_libxenctrl) $(LDLIBS_libxensaverestore) $(LDLIBS_libxentoolcore) $(APPEND_LDFLAGS) testidl: testidl.o libxenlight.so $(CC) $(LDFLAGS) -o $@ testidl.o $(LDLIBS_libxenlight) $(LDLIBS_libxentoollog) $(LDLIBS_libxentoolcore) $(APPEND_LDFLAGS)
tools/libs/light/libxl_internal.h +1 -0
@@ -56,6 +56,7 @@#define XC_WANT_COMPAT_MAP_FOREIGN_API #include <xenctrl.h> #include <xenguest.h> +#include <xensaverestore.h> #include <xenhypfs.h> #include <xen-tools/libs.h>
tools/libs/light/libxl_save_helper.c +1 -0
@@ -48,6 +48,7 @@#include "xenctrl.h" #include "xenguest.h" +#include "xensaverestore.h" #include "_libxl_save_msgs_helper.h" /*----- logger -----*/
tools/libs/light/libxl_save_msgs_gen.pl +1 -1
@@ -72,7 +72,7 @@ END_BOTH END_CALLOUT #include <xenctrl.h> -#include <xenguest.h> +#include <xensaverestore.h> #include "_libxl_save_msgs_${ah}.h" END_HELPER
tools/libs/saverestore/Makefile new file +38 -0
@@ -0,0 +1,38 @@ +XEN_ROOT = $(CURDIR)/../../.. +include $(XEN_ROOT)/tools/Rules.mk + +ifeq ($(CONFIG_MIGRATE),y) +SRCS-y += common.c +SRCS-$(CONFIG_X86) += common_x86.c +SRCS-$(CONFIG_X86) += common_x86_pv.c +SRCS-$(CONFIG_X86) += restore_x86_pv.c +SRCS-$(CONFIG_X86) += restore_x86_hvm.c +SRCS-$(CONFIG_X86) += save_x86_pv.c +SRCS-$(CONFIG_X86) += save_x86_hvm.c +SRCS-y += restore.c +SRCS-y += save.c +else +SRCS-y += nomigrate.c +endif + +CFLAGS += -I$(XEN_libxenctrl) +CFLAGS += -I$(XEN_libxenguest) + +-include $(XEN_TARGET_ARCH)/Makefile + +CFLAGS += -Werror -Wmissing-prototypes +CFLAGS += -I. -I./include $(CFLAGS_xeninclude) +CFLAGS += -D__XEN_TOOLS__ +CFLAGS += -include $(XEN_ROOT)/tools/config.h +# Needed for asprintf() +CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE + +LIBHEADER := xensaverestore.h + +NO_HEADERS_CHK := y + +include $(XEN_ROOT)/tools/libs/libs.mk + +.PHONY: cleanlocal +cleanlocal: + rm -f libxensaverestore.map
tools/libs/saverestore/common.c renamed from tools/libs/guest/xg_sr_common.c +1 -1
@@ -1,6 +1,6 @@#include <assert.h> -#include "xg_sr_common.h" +#include "common.h" #include <xen-tools/libs.h>
tools/libs/saverestore/common.h renamed from tools/libs/guest/xg_sr_common.h +14 -2
@@ -1,13 +1,25 @@#ifndef __COMMON__H#define __COMMON__H +#include <unistd.h> +#include <errno.h> #include <stdbool.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/mman.h> +#include <sys/types.h> +#include <sys/stat.h> + +#include "xc_private.h" +#include "xenguest.h" +#include "xensaverestore.h" #include "xg_private.h" -#include "xg_save_restore.h" +#include "save_restore.h" #include "xc_bitops.h" -#include "xg_sr_stream_format.h" +#include "stream_format.h" /* String representation of Domain Header types. */ const char *dhdr_type_to_str (uint32_t type);
tools/libs/saverestore/common_x86.c renamed from tools/libs/guest/xg_sr_common_x86.c +1 -1
@@ -1,4 +1,4 @@ -#include "xg_sr_common_x86.h" +#include "common_x86.h" int write_x86_tsc_info (struct xc_sr_context *ctx) {
tools/libs/saverestore/common_x86.h renamed from tools/libs/guest/xg_sr_common_x86.h +1 -1
@@ -1,7 +1,7 @@#ifndef __COMMON_X86__H#define __COMMON_X86__H -#include "xg_sr_common.h" +#include "common.h" /* * Obtains a domains TSC information from Xen and writes a X86_TSC_INFO record
tools/libs/saverestore/common_x86_pv.c renamed from tools/libs/guest/xg_sr_common_x86_pv.c +1 -1
@@ -1,6 +1,6 @@#include <assert.h> -#include "xg_sr_common_x86_pv.h" +#include "common_x86_pv.h" xen_pfn_t mfn_to_pfn (struct xc_sr_context *ctx,xen_pfn_t mfn) {
tools/libs/saverestore/common_x86_pv.h renamed from tools/libs/guest/xg_sr_common_x86_pv.h +1 -1
@@ -1,7 +1,7 @@#ifndef __COMMON_X86_PV_H#define __COMMON_X86_PV_H -#include "xg_sr_common_x86.h" +#include "common_x86.h" /* Virtual address ranges reserved for hypervisor. */ #define HYPERVISOR_VIRT_START_X86_64 0xFFFF800000000000ULL
tools/libs/saverestore/nomigrate.c renamed from tools/libs/guest/xg_nomigrate.c +0 -0
tools/libs/saverestore/restore.c renamed from tools/libs/guest/xg_sr_restore.c +1 -1
@@ -2,7 +2,7 @@#include <assert.h> -#include "xg_sr_common.h" +#include "common.h" /* * Read and validate the Image and Domain headers.
tools/libs/saverestore/restore_x86_hvm.c renamed from tools/libs/guest/xg_sr_restore_x86_hvm.c +1 -1
@@ -1,7 +1,7 @@#include <assert.h> #include <arpa/inet.h> -#include "xg_sr_common_x86.h" +#include "common_x86.h" /* * Process an HVM_CONTEXT record from the stream.
tools/libs/saverestore/restore_x86_pv.c renamed from tools/libs/guest/xg_sr_restore_x86_pv.c +1 -1
@@ -1,6 +1,6 @@#include <assert.h> -#include "xg_sr_common_x86_pv.h" +#include "common_x86_pv.h" static xen_pfn_t pfn_to_mfn (const struct xc_sr_context *ctx,xen_pfn_t pfn) {
tools/libs/saverestore/save.c renamed from tools/libs/guest/xg_sr_save.c +1 -1
@@ -1,7 +1,7 @@#include <assert.h> #include <arpa/inet.h> -#include "xg_sr_common.h" +#include "common.h" /* * Writes an Image header and Domain header into the stream.
tools/libs/saverestore/save_restore.h renamed from tools/libs/guest/xg_save_restore.h +0 -2
@@ -15,8 +15,6 @@* License along with this library; If not, see <http://www.gnu.org/licenses/>. */ -#include "xc_private.h" -#include <xen/foreign/x86_32.h> #include <xen/foreign/x86_64.h>
tools/libs/saverestore/save_x86_hvm.c renamed from tools/libs/guest/xg_sr_save_x86_hvm.c +1 -1
@@ -1,6 +1,6 @@#include <assert.h> -#include "xg_sr_common_x86.h" +#include "common_x86.h" #include <xen/hvm/params.h>
tools/libs/saverestore/save_x86_pv.c renamed from tools/libs/guest/xg_sr_save_x86_pv.c +1 -1
@@ -1,7 +1,7 @@#include <assert.h> #include <limits.h> -#include "xg_sr_common_x86_pv.h" +#include "common_x86_pv.h" /* Check a 64 bit virtual address for being canonical. */ static inline bool is_canonical_address (xen_vaddr_t vaddr)
tools/libs/saverestore/stream_format.h renamed from tools/libs/guest/xg_sr_stream_format.h +0 -0
tools/libs/uselibs.mk +3 -1
@@ -20,6 +20,8 @@ LIBS_LIBS += ctrl USELIBS_ctrl := toollog call evtchn gnttab foreignmemory devicemodel LIBS_LIBS += guest USELIBS_guest := evtchn ctrl +LIBS_LIBS += saverestore +USELIBS_saverestore := guest ctrl LIBS_LIBS += store USELIBS_store := toolcore LIBS_LIBS += vchan
@@ -27,7 +29,7 @@ USELIBS_vchan := toollog store gnttab evtchn LIBS_LIBS += stat USELIBS_stat := ctrl store LIBS_LIBS += light -USELIBS_light := toollog evtchn toolcore ctrl store hypfs guest +USELIBS_light := toollog evtchn toolcore ctrl store hypfs guest saverestore LIBS_LIBS += util USELIBS_util := light FILENAME_util := xlutil
PATCH v20210601 4/38 tools: add readv_exact to libxenctrl
In reply to Olaf HeringRead a batch of iovec's.
In the common case of short reads, finish individual iov's with read_exact.
--- tools/libs/ctrl/xc_private.c | 55 +++++++++++++++++++++++++++++++++++- tools/libs/ctrl/xc_private.h | 1 + 2 files changed, 55 insertions(+), 1 deletion(-)
tools/libs/ctrl/xc_private.c +54 -1
@@ -659,8 +659,23 @@ int write_exact(int fd, const void *data, size_t size)#if defined(__MINIOS__)/* -* MiniOS's libc doesn't know about writev(). Implement it as multiple write()s. +* MiniOS's libc doesn't know about readv/writev(). +* Implement it as multiple read/write()s. */ +int readv_exact (int fd,const struct iovec *iov,int iovcnt) +{ +int rc, i; + +for ( i =0 ; i < iovcnt; ++i ) + { + rc =read_exact (fd, iov[i].iov_base, iov[i].iov_len); +if ( rc ) +return rc; + } + +return 0 ; +} +int writev_exact (int fd,const struct iovec *iov,int iovcnt) {int rc, i;
@@ -675,6 +690,44 @@ int writev_exact(int fd, const struct iovec *iov, int iovcnt)return 0 ; }#else +int readv_exact (int fd,const struct iovec *iov,int iovcnt) +{ +int rc =0 , idx =0 ; +ssize_t len; + +while ( idx < iovcnt ) + { + len =readv (fd, &iov[idx],min (iovcnt - idx,IOV_MAX )); +if ( len ==-1 && errno ==EINTR ) +continue ; +if ( len <=0 ) + { + rc =-1 ; + goto out; + } +while ( len >0 && idx < iovcnt ) + { +if ( len >= iov[idx].iov_len ) + { + len -= iov[idx].iov_len; + } +else + { +void *p = iov[idx].iov_base + len; +size_t l = iov[idx].iov_len - len; + + rc =read_exact (fd, p, l); +if ( rc ) + goto out; + len =0 ; + } + idx++; + } + } +out: +return rc; +} +int writev_exact (int fd,const struct iovec *iov,int iovcnt) {struct iovec *local_iov =NULL ;
tools/libs/ctrl/xc_private.h +1 -0
@@ -441,6 +441,7 @@ int xc_flush_mmu_updates(xc_interface *xch, struct xc_mmu *mmu);/* Return 0 on success; -1 on error setting errno. */ int read_exact (int fd,void *data,size_t size);/* EOF => -1, errno=0 */ +int readv_exact (int fd,const struct iovec *iov,int iovcnt);int write_exact (int fd,const void *data,size_t size);int writev_exact (int fd,const struct iovec *iov,int iovcnt);
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
Quoted 15 lines · Olaf Hering
Read a batch of iovec's.
In the common case of short reads, finish individual iov's with
read_exact.Signed-off-by Olaf Hering <[email protected]>--- tools/libs/ctrl/xc_private.c | 55 +++++++++++++++++++++++++++++++++++- tools/libs/ctrl/xc_private.h | 1 + 2 files changed, 55 insertions(+), 1 deletion(-)tools/libs/ctrl/xc_private.c +0
-0
tools/libs/ctrl/xc_private.c int write_exact(int fd, const void *data, size_t size) · line 662
Quoted 4 lines
tools/libs/ctrl/xc_private.c +0
-1@@ -659,8 +659,23 @@ int write_exact(int fd, const void *data, size_t size)#if defined(__MINIOS__)/* -* MiniOS's libc doesn't know about writev(). Implement it as multiple
write()s.
Quoted 27 lines
tools/libs/ctrl/xc_private.c +19
-0+* MiniOS's libc doesn't know about readv/writev(). +* Implement it as multiple read/write()s. */ +int readv_exact (int fd,const struct iovec *iov,int iovcnt) +{ +int rc, i; + +for ( i =0 ; i < iovcnt; ++i ) + { + rc =read_exact (fd, iov[i].iov_base, iov[i].iov_len); +if ( rc ) +return rc; + } + +return 0 ; +} +int writev_exact (int fd,const struct iovec *iov,int iovcnt) {int rc, i;@@ -675,6 +690,44 @@ int writev_exact(int fd, const struct iovec *iov, int iovcnt) return 0; }#else +int readv_exact (int fd,const struct iovec *iov,int iovcnt) +{ +int rc =0 , idx =0 ;
tools/libs/ctrl/xc_private.c int writev_exact(int fd, const struct iovec *iov, int iovcnt) · line 705
Quoted 10 lines
tools/libs/ctrl/xc_private.c +10
-0+ssize_t len; + +while ( idx < iovcnt ) + { + len =readv (fd, &iov[idx],min (iovcnt - idx,IOV_MAX )); +if ( len ==-1 && errno ==EINTR ) +continue ; +if ( len <=0 ) + { + rc =-1 ;
Is EOF really an error?
Quoted 7 lines
tools/libs/ctrl/xc_private.c +7
-0+ goto out; + } +while ( len >0 && idx < iovcnt ) + { +if ( len >= iov[idx].iov_len ) + { + len -= iov[idx].iov_len;
tools/libs/ctrl/xc_private.c int writev_exact(int fd, const struct iovec *iov, int iovcnt) · line 722
Quoted 10 lines
tools/libs/ctrl/xc_private.c +10
-0+ } +else + { +void *p = iov[idx].iov_base + len; +size_t l = iov[idx].iov_len - len; + + rc =read_exact (fd, p, l); +if ( rc ) + goto out; + len =0 ;
This will stop the loop, even if idx hasn't reached iovcnt.
tools/libs/ctrl/xc_private.c int writev_exact(int fd, const struct iovec *iov, int iovcnt) · line 730
Quoted 8 lines
tools/libs/ctrl/xc_private.c +8
-0+ } + idx++; + } + } +out: +return rc; +} +
Juergen
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
Am Wed, 2 Jun 2021 08:30:08 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 7 lines · Olaf Hering
tools/libs/ctrl/xc_private.c int writev_exact(int fd, const struct iovec *iov, int iovcnt) · line 705
Quoted 5 lines
tools/libs/ctrl/xc_private.c +4
-0+int readv_exact (int fd,const struct iovec *iov,int iovcnt) + if ( len <= 0 ) + { + rc =-1 ;Is EOF really an error?
I think yes, that's what "exact" implies IMO.
Quoted 1 line
This will stop the loop, even if idx hasn't reached iovcnt.
Yes, it will trigger yet another readv().
The "while" might be a leftover from a variant which used repeated read_exact to finish the function. It should become a "if", and the "len = 0" can be removed.
Olaf
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
Am Wed, 2 Jun 2021 12:57:10 +0200
schrieb Olaf Hering <[email protected]>:
Quoted 5 lines
Quoted 1 line
This will stop the loop, even if idx hasn't reached iovcnt.
Yes, it will trigger yet another
readv().The "while" might be a leftover from a variant which used repeated
read_exactto finish the function. It should become a "if", and the "len = 0" can be removed.
I think the code is correct.
The while() loop checks if the last iov had a short read, and finishes that iov with read_exact.
I will add a comment, if that helps.
Olaf
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
In reply to Olaf HeringQuoted 4 lines · Olaf Hering
Am Wed, 2 Jun 2021 08:30:08 +0200
schrieb Juergen Gross <[email protected]>:Quoted 1 line · Olaf Hering
Quoted 8 lines
Quoted 6 lines
tools/libs/ctrl/xc_private.c int writev_exact(int fd, const struct iovec *iov, int iovcnt) · line 705
Quoted 5 lines
tools/libs/ctrl/xc_private.c +4
-0+int readv_exact (int fd,const struct iovec *iov,int iovcnt) + if ( len <= 0 ) + { + rc =-1 ;Is EOF really an error?
I think yes, that's what "exact" implies IMO.
Shouldn't you check for zero length iovec elements as in the writev_exact() case then?
Quoted 3 lines
Quoted 1 line
This will stop the loop, even if idx hasn't reached iovcnt.
Yes, it will trigger yet another
readv().
Ah, right.
Juergen
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
Am Wed, 2 Jun 2021 13:41:02 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 2 lines
Shouldn't you check for zero length iovec elements as in the
writev_exact()case then?
I will double check if this is a hard requirement.
Olaf
Re: [PATCH v20210601 04/38] tools: add readv_exact to libxenctrl
In reply to Juergen GrossAm Wed, 2 Jun 2021 13:41:02 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 2 lines
Shouldn't you check for zero length iovec elements as in the
writev_exact()case then?
It is not clear to me what the purpose of skipping zero-length iov's at the beginning of the iov array in writev_exact is.
I assume the syscall (or libc) will just skip them anyway. The code would not handle empty iov's in the middle of the array. Perhaps writev_exact should be done in the same way as my readv_exact, that would also remove the need for malloc. I think in the past I had a patch to do just that. Maybe just locally, maybe I sent it to this list.
In case of readv_exact, only the trailing incomplete iov will be filled with read_exact. I think the length parameter for read_exact can not become zero, "len" would already be zero. Zero-length iov's will be skipped by the loop AFAICS.
Olaf
PATCH v20210601 5/38 tools: add xc_is_known_page_type to libxenctrl
In reply to Olaf HeringUsers of xc_get_pfn_type_batch may want to sanity check the data returned by Xen. Add a simple helper for this purpose.
--- tools/libs/ctrl/xc_private.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
tools/libs/ctrl/xc_private.h +33 -0
@@ -421,6 +421,39 @@ void *xc_map_foreign_ranges(xc_interface *xch, uint32_t dom,int xc_get_pfn_type_batch (xc_interface *xch,uint32_t dom,unsigned int num,xen_pfn_t *); +/* Sanitiy check for types returned by Xen */ +static inline bool xc_is_known_page_type (xen_pfn_t type) +{ +bool ret; + +switch (type) + { +case XEN_DOMCTL_PFINFO_NOTAB : + +case XEN_DOMCTL_PFINFO_L1TAB : +case XEN_DOMCTL_PFINFO_L1TAB |XEN_DOMCTL_PFINFO_LPINTAB : + +case XEN_DOMCTL_PFINFO_L2TAB : +case XEN_DOMCTL_PFINFO_L2TAB |XEN_DOMCTL_PFINFO_LPINTAB : + +case XEN_DOMCTL_PFINFO_L3TAB : +case XEN_DOMCTL_PFINFO_L3TAB |XEN_DOMCTL_PFINFO_LPINTAB : + +case XEN_DOMCTL_PFINFO_L4TAB : +case XEN_DOMCTL_PFINFO_L4TAB |XEN_DOMCTL_PFINFO_LPINTAB : + +case XEN_DOMCTL_PFINFO_XTAB : +case XEN_DOMCTL_PFINFO_XALLOC : +case XEN_DOMCTL_PFINFO_BROKEN : + ret = true; +break ; +default : + ret = false; +break ; + } +return ret; +} +void bitmap_64_to_byte (uint8_t *bp,const uint64_t *lp,int nbits);void bitmap_byte_to_64 (uint64_t *lp,const uint8_t *bp,int nbits);
Re: [PATCH v20210601 05/38] tools: add xc_is_known_page_type to libxenctrl
Quoted 13 lines · Olaf Hering
Users of
xc_get_pfn_type_batchmay want to sanity check the data returned by Xen. Add a simple helper for this purpose.Signed-off-by Olaf Hering <[email protected]>--- tools/libs/ctrl/xc_private.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)tools/libs/ctrl/xc_private.h +0
-0
tools/libs/ctrl/xc_private.h void *xc_map_foreign_ranges(xc_interface *xch, uint32_t dom, · line 429
Quoted 9 lines
tools/libs/ctrl/xc_private.h +6
-0@@ -421,6 +421,39 @@ void *xc_map_foreign_ranges(xc_interface *xch, uint32_t dom,int xc_get_pfn_type_batch (xc_interface *xch,uint32_t dom,unsigned int num,xen_pfn_t *); +/* Sanitiy check for types returned by Xen */ +static inline bool xc_is_known_page_type (xen_pfn_t type) +{ +bool ret; + + switch (type)
I think you should not imply the planned use case here. It would be better to use "switch (type & XEN_DOMCTL_PFINFO_LTAB_MASK)".
I'm on the edge regarding putting the new function into xc_private.h. In the end your use case is _not_ to call the new function from libxenctrl.
Juergen
Re: [PATCH v20210601 05/38] tools: add xc_is_known_page_type to libxenctrl
Am Wed, 2 Jun 2021 08:51:45 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 6 lines
I think you should not imply the planned use case here. It would be better to use "switch (type &
XEN_DOMCTL_PFINFO_LTAB_MASK)".I'm on the edge regarding putting the new function into
xc_private.h. In the end your use case is _not_ to call the new function from libxenctrl.
I'm not sure what that means.
One or the other has to be rebased to the new state.
Olaf
Re: [PATCH v20210601 05/38] tools: add xc_is_known_page_type to libxenctrl
Quoted 6 lines · Olaf Hering
Am Wed, 2 Jun 2021 08:51:45 +0200
schrieb Juergen Gross <[email protected]>:Quoted 3 lines
I think you should not imply the planned use case here. It would be better to use "switch (type &
XEN_DOMCTL_PFINFO_LTAB_MASK)".
Quoted 6 lines
Quoted 3 lines
I'm on the edge regarding putting the new function into
xc_private.h. In the end your use case is _not_ to call the new function from libxenctrl.I'm not sure what that means.
The name xc_private.h indicates that this header file is supposed to be used only inside of libxenctrl. I know that this isn't true today, but I don't think new misuses should be added to this file.
Quoted 1 line
One or the other has to be rebased to the new state.
You can add the same functions to some libxensaverestore prvate header instead.
Juergen
PATCH v20210601 6/38 tools: use xc_is_known_page_type
In reply to Olaf HeringVerify pfn type on sending side, also verify incoming batch of pfns.
--- tools/libs/saverestore/restore.c | 3 +-- tools/libs/saverestore/save.c | 6 ++++++ 2 files changed, 7 insertions(+), 2 deletions(-)
tools/libs/saverestore/restore.c +1 -2
@@ -406,8 +406,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) } type = (pages->pfn[i] &PAGE_DATA_TYPE_MASK ) >>32 ; -if ( ((type >>XEN_DOMCTL_PFINFO_LTAB_SHIFT ) >=5 ) && - ((type >>XEN_DOMCTL_PFINFO_LTAB_SHIFT ) <=8 ) ) +if (xc_is_known_page_type (type) == false ) {ERROR ("Invalid type %#" PRIx32" for pfn %#" PRIpfn" (index %u)" , type, pfn, i);
tools/libs/saverestore/save.c +6 -0
@@ -147,6 +147,12 @@ static int write_batch(struct xc_sr_context *ctx)for ( i =0 ; i < nr_pfns; ++i ) { +if (xc_is_known_page_type (types[i]) == false ) + { +ERROR ("Wrong type %#" PRIpfn" for pfn %#" PRIpfn, types[i], mfns[i]); + goto err; + } +switch ( types[i] ) {case XEN_DOMCTL_PFINFO_BROKEN :
Re: [PATCH v20210601 06/38] tools: use xc_is_known_page_type
Quoted 3 lines · Olaf Hering
Verify pfn type on sending side, also verify incoming batch of pfns.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 7/38 tools: unify type checking for data pfns in migration stream
In reply to Olaf HeringIntroduce a helper which decides if a given pfn type has data for the migration stream.
No change in behavior intended.
--- tools/libs/saverestore/common.h | 17 ++++++++++++++++ tools/libs/saverestore/restore.c | 34 +++++--------------------------- tools/libs/saverestore/save.c | 14 ++----------- 3 files changed, 24 insertions(+), 41 deletions(-)
tools/libs/saverestore/common.h +17 -0
@@ -467,6 +467,23 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,/* Handle a STATIC_DATA_END record. */ int handle_static_data_end (struct xc_sr_context *ctx); +static inline bool page_type_has_stream_data (uint32_t type) +{ +bool ret; + +switch (type) + { +case XEN_DOMCTL_PFINFO_XTAB : +case XEN_DOMCTL_PFINFO_XALLOC : +case XEN_DOMCTL_PFINFO_BROKEN : + ret = false; +break ; +default : + ret = true; +break ; + } +return ret; +}#endif /* * Local variables:
tools/libs/saverestore/restore.c +5 -29
@@ -152,9 +152,8 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,for ( i =0 ; i < count; ++i ) { -if ( (!types || (types && - (types[i] !=XEN_DOMCTL_PFINFO_XTAB && - types[i] !=XEN_DOMCTL_PFINFO_BROKEN ))) && +if ( (!types || + (types &&page_type_has_stream_data (types[i]) == true)) && !pfn_is_populated (ctx, original_pfns[i]) ) { rc =pfn_set_populated (ctx, original_pfns[i]);
@@ -233,25 +232,8 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, { ctx->restore.ops.set_page_type (ctx, pfns[i], types[i]); -switch ( types[i] ) - { -case XEN_DOMCTL_PFINFO_NOTAB : - -case XEN_DOMCTL_PFINFO_L1TAB : -case XEN_DOMCTL_PFINFO_L1TAB |XEN_DOMCTL_PFINFO_LPINTAB : - -case XEN_DOMCTL_PFINFO_L2TAB : -case XEN_DOMCTL_PFINFO_L2TAB |XEN_DOMCTL_PFINFO_LPINTAB : - -case XEN_DOMCTL_PFINFO_L3TAB : -case XEN_DOMCTL_PFINFO_L3TAB |XEN_DOMCTL_PFINFO_LPINTAB : - -case XEN_DOMCTL_PFINFO_L4TAB : -case XEN_DOMCTL_PFINFO_L4TAB |XEN_DOMCTL_PFINFO_LPINTAB : - +if (page_type_has_stream_data (types[i]) == true ) mfns[nr_pages++] = ctx->restore.ops.pfn_to_gfn (ctx, pfns[i]); -break ; - } }/* Nothing to do? */
@@ -271,14 +253,8 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,for ( i =0 , j =0 ; i < count; ++i ) { -switch ( types[i] ) - { -case XEN_DOMCTL_PFINFO_XTAB : -case XEN_DOMCTL_PFINFO_BROKEN : -case XEN_DOMCTL_PFINFO_XALLOC : -/* No page data to deal with. */ +if (page_type_has_stream_data (types[i]) == false )continue ; - }if ( map_errs[j] ) {
@@ -413,7 +389,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) goto err; } -if ( type <XEN_DOMCTL_PFINFO_BROKEN ) +if (page_type_has_stream_data (type) == true )/* NOTAB and all L1 through L4 tables (including pinned) should * have a page worth of data in the record. */ pages_of_data++;
tools/libs/saverestore/save.c +2 -12
@@ -153,13 +153,8 @@ static int write_batch(struct xc_sr_context *ctx) goto err; } -switch ( types[i] ) - { -case XEN_DOMCTL_PFINFO_BROKEN : -case XEN_DOMCTL_PFINFO_XALLOC : -case XEN_DOMCTL_PFINFO_XTAB : +if (page_type_has_stream_data (types[i]) == false )continue ; - } mfns[nr_pages++] = mfns[i]; }
@@ -177,13 +172,8 @@ static int write_batch(struct xc_sr_context *ctx)for ( i =0 , p =0 ; i < nr_pfns; ++i ) { -switch ( types[i] ) - { -case XEN_DOMCTL_PFINFO_BROKEN : -case XEN_DOMCTL_PFINFO_XALLOC : -case XEN_DOMCTL_PFINFO_XTAB : +if (page_type_has_stream_data (types[i]) == false )continue ; - }if ( errors[p] ) {
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
Quoted 45 lines · Olaf Hering
Introduce a helper which decides if a given pfn type has data for the migration stream.
No change in behavior intended.
Signed-off-by Olaf Hering <[email protected]>--- tools/libs/saverestore/common.h | 17 ++++++++++++++++ tools/libs/saverestore/restore.c | 34 +++++--------------------------- tools/libs/saverestore/save.c | 14 ++----------- 3 files changed, 24 insertions(+), 41 deletions(-)tools/libs/saverestore/common.h +17
-0@@ -467,6 +467,23 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,/* Handle a STATIC_DATA_END record. */ int handle_static_data_end (struct xc_sr_context *ctx); +static inline bool page_type_has_stream_data (uint32_t type) +{ +bool ret; + +switch (type) + { +case XEN_DOMCTL_PFINFO_XTAB : +case XEN_DOMCTL_PFINFO_XALLOC : +case XEN_DOMCTL_PFINFO_BROKEN : + ret = false; +break ; +default : + ret = true; +break ; + } +return ret; +}#endif /* * Local variables: tools/libs/saverestore/restore.c +0
-0
tools/libs/saverestore/restore.c int populate_pfns(struct xc_sr_context *ctx, unsigned int count, · line 156
Quoted 8 lines
tools/libs/saverestore/restore.c +2
-3@@ -152,9 +152,8 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,for ( i =0 ; i < count; ++i ) { - if ( (!types || (types && - (types[i] != XEN_DOMCTL_PFINFO_XTAB && - types[i] != XEN_DOMCTL_PFINFO_BROKEN))) && + if ( (!types || + (types && page_type_has_stream_data(types[i]) == true)) &&
What about XEN_DOMCTL_PFINFO_XALLOC? Is this case impossible here, or are you changing behavior?
Juergen
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
Am Wed, 2 Jun 2021 08:59:13 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 2 lines
What about
XEN_DOMCTL_PFINFO_XALLOC? Is this case impossible here, or are you changing behavior?
I think XEN_DOMCTL_PFINFO_XALLOC is a type that does not exist in practice.
So, no change in behavior.
Olaf
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
Quoted 7 lines · Olaf Hering
Am Wed, 2 Jun 2021 08:59:13 +0200
schrieb Juergen Gross <[email protected]>:Quoted 2 lines
What about
XEN_DOMCTL_PFINFO_XALLOC? Is this case impossible here, or are you changing behavior?I think
XEN_DOMCTL_PFINFO_XALLOCis a type that does not exist in practice.
Oh, indeed. It was used ages ago by migration code (I could find it being used e.g. in Xen 4.2, but not in 4.7). BTW, the use case was to not shatter superpages on migration...
Quoted 1 line
So, no change in behavior.
Maybe XEN_DOMCTL_PFINFO_XALLOC should no longer be supported, i.e. a stream containing it should be rejected? This might be a worthwhile modification of patch 5.
Juergen
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
Am Wed, 2 Jun 2021 14:03:40 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 3 lines
Maybe
XEN_DOMCTL_PFINFO_XALLOCshould no longer be supported, i.e. a stream containing it should be rejected? This might be a worthwhile modification of patch 5.
If there is a desire to actively break incoming domUs from pre-4.6 hosts, this should probably be done in a separate change.
I have not checked if such migration works today with unmodified xen.git#staging.
Olaf
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
Quoted 10 lines · Olaf Hering
Am Wed, 2 Jun 2021 14:03:40 +0200
schrieb Juergen Gross <[email protected]>:Quoted 3 lines
Maybe
XEN_DOMCTL_PFINFO_XALLOCshould no longer be supported, i.e. a stream containing it should be rejected? This might be a worthwhile modification of patch 5.If there is a desire to actively break incoming domUs from pre-4.6 hosts, this should probably be done in a separate change.
I have not checked if such migration works today with unmodified xen.git#staging.
At least with your patch this might change now, as you are modifying the behavior in case of XEN_DOMCTL_PFINFO_XALLOC.
Officially pre-4.6 hosts are not supported regarding LM. So your patch should be fine in this regard. OTH I'd rather have a clear failure than a maybe weird behavior in case a stream is containing a record with XEN_DOMCTL_PFINFO_XALLOC.
Juergen
Re: [PATCH v20210601 07/38] tools: unify type checking for data pfns in migration stream
In reply to Juergen GrossAm Wed, 2 Jun 2021 08:59:13 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 12 lines
tools/libs/saverestore/restore.c int populate_pfns(struct xc_sr_context *ctx, unsigned int count, · line 156
Quoted 9 lines
tools/libs/saverestore/restore.c +2
-3@@ -152,9 +152,8 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count,for ( i =0 ; i < count; ++i ) { - if ( (!types || (types && - (types[i] != XEN_DOMCTL_PFINFO_XTAB && - types[i] != XEN_DOMCTL_PFINFO_BROKEN))) && + if ( (!types || + (types && page_type_has_stream_data(types[i]) == true)) &&What about
XEN_DOMCTL_PFINFO_XALLOC? Is this case impossible here, or are you changing behavior?
I guess this needs to be handled somehow, a large enough HVM domU will have XEN_DOMCTL_PFINFO_XALLOC in the stream. Not sure why this was thrown away with the v2 format.
Olaf
PATCH v20210601 8/38 tools: show migration transfer rate in send_dirty_pages
In reply to Olaf HeringShow how fast domU pages are transferred in each iteration.
The relevant data is how fast the pfns travel, not so much how much protocol overhead exists. So the reported MiB/sec is just for pfns.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 47 +++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)
tools/libs/saverestore/common.h +2 -0
@@ -250,6 +250,8 @@ struct xc_sr_contextbool debug;unsigned long p2m_size; +size_t pages_sent; +size_t overhead_sent;struct precopy_stats stats;
tools/libs/saverestore/save.c +47 -0
79 lines not on this page · raw message
Re: [PATCH v20210601 08/38] tools: show migration transfer rate in send_dirty_pages
Quoted 36 lines · Olaf Hering
Show how fast domU pages are transferred in each iteration.
The relevant data is how fast the pfns travel, not so much how much protocol overhead exists. So the reported MiB/sec is just for pfns.
Signed-off-by Olaf Hering <[email protected]>--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 47 +++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)tools/libs/saverestore/common.h +2
-0@@ -250,6 +250,8 @@ struct xc_sr_contextbool debug;unsigned long p2m_size; +size_t pages_sent; +size_t overhead_sent;struct precopy_stats stats;tools/libs/saverestore/save.c +1
-0@@ -1,5 +1,6 @@#include <assert.h> #include <arpa/inet.h> +#include <time.h> #include "common.h"
tools/libs/saverestore/save.c static int write_batch(struct xc_sr_context *ctx) · line 243
Quoted 5 lines
tools/libs/saverestore/save.c +2
-0@@ -238,6 +239,8 @@ static int write_batch(struct xc_sr_context *ctx) iov[3 ].iov_len = nr_pfns *sizeof (*rec_pfns); iovcnt =4 ; + ctx->save.pages_sent += nr_pages; + ctx->save.overhead_sent += sizeof(rec) + sizeof(hdr) + nr_pfns *
sizeof(*rec_pfns);
tools/libs/saverestore/save.c static int suspend_domain(struct xc_sr_context *ctx) · line 367
Quoted 12 lines
tools/libs/saverestore/save.c +5
-0if ( nr_pages ) {@@ -357,6 +360,43 @@ static int suspend_domain(struct xc_sr_context *ctx)return 0 ; } +static void show_transfer_rate (struct xc_sr_context *ctx,struct timespec *start) +{ +xc_interface *xch = ctx->xch; +struct timespec end = {}, diff = {}; +size_t ms, MiB_sec = ctx->save.pages_sent *PAGE_SIZE ;
I'd rather not initialize MiB_sec here ...
tools/libs/saverestore/save.c static int suspend_domain(struct xc_sr_context *ctx) · line 369
Quoted 2 lines
tools/libs/saverestore/save.c +2
-0+ +if (!MiB_sec)
... and test for ctx->save.pages_sent to be non-zero here.
Quoted 9 lines
tools/libs/saverestore/save.c +9
-0+return ; + +if (clock_gettime (CLOCK_MONOTONIC , &end) ) +PERROR ("clock_gettime" ); + +if ( (end.tv_nsec - start->tv_nsec) <0 ) + { + diff.tv_sec = end.tv_sec - start->tv_sec -1 ; + diff.tv_nsec = end.tv_nsec - start->tv_nsec + (1000U *1000U *1000U );
tools/libs/saverestore/save.c static int suspend_domain(struct xc_sr_context *ctx) · line 388
Quoted 10 lines
tools/libs/saverestore/save.c +10
-0+ } +else + { + diff.tv_sec = end.tv_sec - start->tv_sec; + diff.tv_nsec = end.tv_nsec - start->tv_nsec; + } + + ms = (diff.tv_nsec / (1000U *1000U )); +if (!ms) + ms =1 ;
I'd move this ...
tools/libs/saverestore/save.c static int suspend_domain(struct xc_sr_context *ctx) · line 389
Quoted 1 line
tools/libs/saverestore/save.c +1
-0+ ms += (diff.tv_sec *1000U );
... below this.
tools/libs/saverestore/save.c static int suspend_domain(struct xc_sr_context *ctx) · line 393
Quoted 4 lines
tools/libs/saverestore/save.c +4
-0+ + MiB_sec *=1000U ; + MiB_sec /= ms; + MiB_sec /=1024U *1024U ;
Avoid MiB_sec holding bytes per second for some time and use:
MiB_sec = ((ctx->save.pages_sent * PAGE_SIZE * 1000) / ms) /
(1024U * 1024U);
Juergen
Re: [PATCH v20210601 08/38] tools: show migration transfer rate in send_dirty_pages
Am Wed, 2 Jun 2021 09:10:44 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 2 lines
MiB_sec= ((ctx->save.pages_sent*PAGE_SIZE* 1000) / ms) /(1024U * 1024U);
I'm not sure: how does this improve the patch?
Olaf
Re: [PATCH v20210601 08/38] tools: show migration transfer rate in send_dirty_pages
Quoted 7 lines · Olaf Hering
Am Wed, 2 Jun 2021 09:10:44 +0200
schrieb Juergen Gross <[email protected]>:Quoted 2 lines
MiB_sec= ((ctx->save.pages_sent*PAGE_SIZE* 1000) / ms) /(1024U * 1024U);I'm not sure: how does this improve the patch?
The scattered calculation makes it much harder to verify it (at least for me).
And initializing a variable named "MiB_sec" with a value being clearly bytes doesn't help.
Juergen
PATCH v20210601 9/38 tools/guest: prepare to allocate arrays once
In reply to Olaf HeringThe hotpath 'send_dirty_pages' is supposed to do just one thing: sending. The other end 'handle_page_data' is supposed to do just receiving.
But instead both do other costly work like memory allocations and data moving. Do the allocations once, the array sizes are a compiletime constant. Avoid unneeded copying of data by receiving data directly into mapped guest memory.
This patch is just prepartion, subsequent changes will populate the arrays.
Once all changes are applied, migration of a busy HVM domU changes like that:
Without this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3 xen_testing):
2020-10-29 10:23:10.711+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 55.324905335 sec, 203 MiB/sec: Internal error 2020-10-29 10:23:35.115+0000: xc: show_transfer_rate: 16829632 bytes + 2097552 pages in 24.401179720 sec, 335 MiB/sec: Internal error 2020-10-29 10:23:59.436+0000: xc: show_transfer_rate: 16829032 bytes + 2097478 pages in 24.319025928 sec, 336 MiB/sec: Internal error 2020-10-29 10:24:23.844+0000: xc: show_transfer_rate: 16829024 bytes + 2097477 pages in 24.406992500 sec, 335 MiB/sec: Internal error 2020-10-29 10:24:48.292+0000: xc: show_transfer_rate: 16828912 bytes + 2097463 pages in 24.446489027 sec, 335 MiB/sec: Internal error 2020-10-29 10:25:01.816+0000: xc: show_transfer_rate: 16836080 bytes + 2098356 pages in 13.447091818 sec, 609 MiB/sec: Internal error
With this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3 xen_unstable):
2020-10-28 21:26:05.074+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 52.564054368 sec, 213 MiB/sec: Internal error 2020-10-28 21:26:23.527+0000: xc: show_transfer_rate: 16830040 bytes + 2097603 pages in 18.450592015 sec, 444 MiB/sec: Internal error 2020-10-28 21:26:41.926+0000: xc: show_transfer_rate: 16830944 bytes + 2097717 pages in 18.397862306 sec, 445 MiB/sec: Internal error 2020-10-28 21:27:00.339+0000: xc: show_transfer_rate: 16829176 bytes + 2097498 pages in 18.411973339 sec, 445 MiB/sec: Internal error 2020-10-28 21:27:18.643+0000: xc: show_transfer_rate: 16828592 bytes + 2097425 pages in 18.303326695 sec, 447 MiB/sec: Internal error 2020-10-28 21:27:26.289+0000: xc: show_transfer_rate: 16835952 bytes + 2098342 pages in 7.579846749 sec, 1081 MiB/sec: Internal error
--- tools/libs/saverestore/common.h | 8 ++++++++ tools/libs/saverestore/restore.c | 8 ++++++++ tools/libs/saverestore/save.c | 4 +++- 3 files changed, 19 insertions(+), 1 deletion(-)
tools/libs/saverestore/common.h +8 -0
@@ -223,6 +223,12 @@ static inline int update_blob(struct xc_sr_blob *blob,return 0 ; } +struct xc_sr_save_arrays { +}; + +struct xc_sr_restore_arrays { +}; +struct xc_sr_context {xc_interface *xch;
@@ -260,6 +266,7 @@ struct xc_sr_contextunsigned long *deferred_pages;unsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf; +struct xc_sr_save_arrays *m; } save; struct /* Restore data. */
@@ -311,6 +318,7 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; +struct xc_sr_restore_arrays *m; } restore; };
tools/libs/saverestore/restore.c +8 -0
@@ -739,6 +739,13 @@ static int setup(struct xc_sr_context *ctx) } ctx->restore.allocated_rec_num =DEFAULT_BUF_RECORDS ; + ctx->restore.m =malloc (sizeof (*ctx->restore.m)); +if ( !ctx->restore.m ) { +ERROR ("Unable to allocate memory for arrays" ); + rc =-1 ; + goto err; + } + err:return rc; }
@@ -757,6 +764,7 @@ static void cleanup(struct xc_sr_context *ctx)xc_hypercall_buffer_free_pages ( xch, dirty_bitmap,NRPAGES (bitmap_size (ctx->restore.p2m_size))); +free (ctx->restore.m);free (ctx->restore.buffered_records);free (ctx->restore.populated_pfns);
tools/libs/saverestore/save.c +3 -1
@@ -853,8 +853,9 @@ static int setup(struct xc_sr_context *ctx) ctx->save.batch_pfns =malloc (MAX_BATCH_SIZE *sizeof (*ctx->save.batch_pfns)); ctx->save.deferred_pages =bitmap_alloc (ctx->save.p2m_size); + ctx->save.m =malloc (sizeof (*ctx->save.m)); -if ( !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages ) +if ( !ctx->save.m || !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages ) {ERROR ("Unable to allocate memory for dirty bitmaps, batch pfns and" " deferred pages" );
@@ -886,6 +887,7 @@ static void cleanup(struct xc_sr_context *ctx) NRPAGES(bitmap_size(ctx->save.p2m_size)));free (ctx->save.deferred_pages);free (ctx->save.batch_pfns); +free (ctx->save.m); }/*
Re: [PATCH v20210601 09/38] tools/guest: prepare to allocate arrays once
Quoted 10 lines · Olaf Hering
The hotpath '
send_dirty_pages' is supposed to do just one thing: sending. The other end 'handle_page_data' is supposed to do just receiving.But instead both do other costly work like memory allocations and data moving. Do the allocations once, the array sizes are a compiletime constant. Avoid unneeded copying of data by receiving data directly into mapped guest memory.
This patch is just prepartion, subsequent changes will populate the arrays.
Once all changes are applied, migration of a busy HVM domU changes like
that:
Quoted 29 lines
Without this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3
xen_testing):2020-10-29 10:23:10.711+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 55.324905335 sec, 203 MiB/sec: Internal error 2020-10-29 10:23:35.115+0000: xc: show_transfer_rate: 16829632 bytes + 2097552 pages in 24.401179720 sec, 335 MiB/sec: Internal error 2020-10-29 10:23:59.436+0000: xc: show_transfer_rate: 16829032 bytes + 2097478 pages in 24.319025928 sec, 336 MiB/sec: Internal error 2020-10-29 10:24:23.844+0000: xc: show_transfer_rate: 16829024 bytes + 2097477 pages in 24.406992500 sec, 335 MiB/sec: Internal error 2020-10-29 10:24:48.292+0000: xc: show_transfer_rate: 16828912 bytes + 2097463 pages in 24.446489027 sec, 335 MiB/sec: Internal error 2020-10-29 10:25:01.816+0000: xc: show_transfer_rate: 16836080 bytes + 2098356 pages in 13.447091818 sec, 609 MiB/sec: Internal errorWith this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3
xen_unstable):2020-10-28 21:26:05.074+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 52.564054368 sec, 213 MiB/sec: Internal error 2020-10-28 21:26:23.527+0000: xc: show_transfer_rate: 16830040 bytes + 2097603 pages in 18.450592015 sec, 444 MiB/sec: Internal error 2020-10-28 21:26:41.926+0000: xc: show_transfer_rate: 16830944 bytes + 2097717 pages in 18.397862306 sec, 445 MiB/sec: Internal error 2020-10-28 21:27:00.339+0000: xc: show_transfer_rate: 16829176 bytes + 2097498 pages in 18.411973339 sec, 445 MiB/sec: Internal error 2020-10-28 21:27:18.643+0000: xc: show_transfer_rate: 16828592 bytes + 2097425 pages in 18.303326695 sec, 447 MiB/sec: Internal error 2020-10-28 21:27:26.289+0000: xc: show_transfer_rate: 16835952 bytes + 2098342 pages in 7.579846749 sec, 1081 MiB/sec: Internal errorSigned-off-by Olaf Hering <[email protected]>--- tools/libs/saverestore/common.h | 8 ++++++++ tools/libs/saverestore/restore.c | 8 ++++++++ tools/libs/saverestore/save.c | 4 +++- 3 files changed, 19 insertions(+), 1 deletion(-)tools/libs/saverestore/common.h +0
-0
tools/libs/saverestore/common.h static inline int update_blob(struct xc_sr_blob *blob, · line 230
Quoted 8 lines
tools/libs/saverestore/common.h +5
-0@@ -223,6 +223,12 @@ static inline int update_blob(struct xc_sr_blob *blob,return 0 ; } +struct xc_sr_save_arrays { +}; + +struct xc_sr_restore_arrays { +};
Can you please add the mfns/pfns arrays to above types, as ...
Quoted 25 lines
tools/libs/saverestore/common.h +3
-0+struct xc_sr_context {xc_interface *xch;@@ -260,6 +266,7 @@ struct xc_sr_contextunsigned long *deferred_pages;unsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf; +struct xc_sr_save_arrays *m; } save; struct /* Restore data. */@@ -311,6 +318,7 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; +struct xc_sr_restore_arrays *m; } restore; };tools/libs/saverestore/restore.c +0
-0
tools/libs/saverestore/restore.c static int setup(struct xc_sr_context *ctx) · line 743
Quoted 5 lines
tools/libs/saverestore/restore.c +2
-0@@ -739,6 +739,13 @@ static int setup(struct xc_sr_context *ctx) } ctx->restore.allocated_rec_num =DEFAULT_BUF_RECORDS ; + ctx->restore.m =malloc (sizeof (*ctx->restore.m)); +if ( !ctx->restore.m ) {
... this case might trigger without the full series applied, due to allocating zero bytes (same for the save side below).
Juergen
Re: [PATCH v20210601 09/38] tools/guest: prepare to allocate arrays once
Am Wed, 2 Jun 2021 09:29:08 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 5 lines
tools/libs/saverestore/restore.c static int setup(struct xc_sr_context *ctx) · line 743
Quoted 2 lines
tools/libs/saverestore/restore.c +2
-0+ ctx->restore.m =malloc (sizeof (*ctx->restore.m)); +if ( !ctx->restore.m ) {... this case might trigger without the full series applied, due to allocating zero bytes (same for the save side below).
Such bisection point with a libc that returns NULL would be just bad luck.
See git-bisect(1) "Avoiding testing a commit" how to deal with it, in the unlikely case it actually triggers.
Olaf
Re: [PATCH v20210601 09/38] tools/guest: prepare to allocate arrays once
Quoted 10 lines · Olaf Hering
Am Wed, 2 Jun 2021 09:29:08 +0200
schrieb Juergen Gross <[email protected]>:Quoted 5 lines
tools/libs/saverestore/restore.c static int setup(struct xc_sr_context *ctx) · line 743
Quoted 2 lines
tools/libs/saverestore/restore.c +2
-0+ ctx->restore.m =malloc (sizeof (*ctx->restore.m)); +if ( !ctx->restore.m ) {... this case might trigger without the full series applied, due to allocating zero bytes (same for the save side below).
Such bisection point with a libc that returns NULL would be just bad luck.
Sure, but sending a patch which is known to break bisecting is bad behavior. You could even add a dummy element (with a comment indicating its purpose) which could be removed when the first "real" structure element is being added.
Quoted 1 line
See git-bisect(1) "Avoiding testing a commit" how to deal with it, in the unlikely case it actually triggers.
It can be avoided, yes, but you need to search for the reason the failure occurred first. And this debugging effort should be avoided if possible.
Juergen
PATCH v20210601 10/38 tools/guest: save: move batch_pfns
In reply to Olaf HeringThe batch_pfns array is already allocated in advance. Move it into the preallocated area.
--- tools/libs/saverestore/common.h | 2 +- tools/libs/saverestore/save.c | 25 +++++++++++-------------- 2 files changed, 12 insertions(+), 15 deletions(-)
tools/libs/saverestore/common.h +1 -1
@@ -224,6 +224,7 @@ static inline int update_blob(struct xc_sr_blob *blob, }struct xc_sr_save_arrays { +xen_pfn_t batch_pfns[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
@@ -261,7 +262,6 @@ struct xc_sr_contextstruct precopy_stats stats; -xen_pfn_t *batch_pfns;unsigned int nr_batch_pfns;unsigned long *deferred_pages;unsigned long nr_deferred_pages;
tools/libs/saverestore/save.c +11 -14
@@ -77,7 +77,7 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)/* * Writes a batch of memory as a PAGE_DATA record into the stream. The batch -* is constructed in ctx->save.batch_pfns. +* is constructed in ctx->save.m->batch_pfns. * * This function: * - gets the types for each pfn in the batch.
@@ -128,12 +128,12 @@ static int write_batch(struct xc_sr_context *ctx)for ( i =0 ; i < nr_pfns; ++i ) { types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn (ctx, - ctx->save.batch_pfns[i]); + ctx->save.m->batch_pfns[i]);/* Likely a ballooned page. */ if ( mfns[i] ==INVALID_MFN ) { -set_bit (ctx->save.batch_pfns[i], ctx->save.deferred_pages); +set_bit (ctx->save.m->batch_pfns[i], ctx->save.deferred_pages); ++ctx->save.nr_deferred_pages; } }
@@ -179,7 +179,7 @@ static int write_batch(struct xc_sr_context *ctx)if ( errors[p] ) {ERROR ("Mapping of pfn %#" PRIpfn" (mfn %#" PRIpfn") failed %d" , - ctx->save.batch_pfns[i], mfns[p], errors[p]); + ctx->save.m->batch_pfns[i], mfns[p], errors[p]); goto err; }
@@ -193,7 +193,7 @@ static int write_batch(struct xc_sr_context *ctx) {if ( rc ==-1 && errno ==EAGAIN ) { -set_bit (ctx->save.batch_pfns[i], ctx->save.deferred_pages); +set_bit (ctx->save.m->batch_pfns[i], ctx->save.deferred_pages); ++ctx->save.nr_deferred_pages; types[i] =XEN_DOMCTL_PFINFO_XTAB ; --nr_pages;
@@ -224,7 +224,7 @@ static int write_batch(struct xc_sr_context *ctx) rec.length += nr_pages *PAGE_SIZE ;for ( i =0 ; i < nr_pfns; ++i ) - rec_pfns[i] = ((uint64_t )(types[i]) <<32 ) | ctx->save.batch_pfns[i]; + rec_pfns[i] = ((uint64_t )(types[i]) <<32 ) | ctx->save.m->batch_pfns[i]; iov[0 ].iov_base = &rec.type; iov[0 ].iov_len =sizeof (rec.type);
@@ -296,9 +296,9 @@ static int flush_batch(struct xc_sr_context *ctx)if ( !rc ) { -VALGRIND_MAKE_MEM_UNDEFINED (ctx->save.batch_pfns, +VALGRIND_MAKE_MEM_UNDEFINED (ctx->save.m->batch_pfns,MAX_BATCH_SIZE * -sizeof (*ctx->save.batch_pfns)); +sizeof (*ctx->save.m->batch_pfns)); }return rc;
@@ -315,7 +315,7 @@ static int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn) rc =flush_batch (ctx);if ( rc ==0 ) - ctx->save.batch_pfns[ctx->save.nr_batch_pfns++] = pfn; + ctx->save.m->batch_pfns[ctx->save.nr_batch_pfns++] = pfn;return rc; }
@@ -850,14 +850,12 @@ static int setup(struct xc_sr_context *ctx) dirty_bitmap =xc_hypercall_buffer_alloc_pages ( xch, dirty_bitmap,NRPAGES (bitmap_size (ctx->save.p2m_size))); - ctx->save.batch_pfns =malloc (MAX_BATCH_SIZE * -sizeof (*ctx->save.batch_pfns)); ctx->save.deferred_pages =bitmap_alloc (ctx->save.p2m_size); ctx->save.m =malloc (sizeof (*ctx->save.m)); -if ( !ctx->save.m || !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages ) +if ( !ctx->save.m || !dirty_bitmap || !ctx->save.deferred_pages ) { -ERROR ("Unable to allocate memory for dirty bitmaps, batch pfns and" +ERROR ("Unable to allocate memory for dirty bitmaps and" " deferred pages" ); rc =-1 ; errno =ENOMEM ;
@@ -886,7 +884,6 @@ static void cleanup(struct xc_sr_context *ctx)xc_hypercall_buffer_free_pages (xch, dirty_bitmap,NRPAGES (bitmap_size (ctx->save.p2m_size)));free (ctx->save.deferred_pages); -free (ctx->save.batch_pfns);free (ctx->save.m); }
Re: [PATCH v20210601 10/38] tools/guest: save: move batch_pfns
Quoted 4 lines · Olaf Hering
The
batch_pfnsarray is already allocated in advance. Move it into the preallocated area.Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 11/38 tools/guest: save: move mfns array
In reply to Olaf HeringRemove allocation from hotpath, move mfns array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++----- 2 files changed, 4 insertions(+), 5 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -225,6 +225,8 @@ static inline int update_blob(struct xc_sr_blob *blob,struct xc_sr_save_arrays {xen_pfn_t batch_pfns[MAX_BATCH_SIZE ]; +/* write_batch: Mfns of the batch pfns. */ +xen_pfn_t mfns[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +2 -5
@@ -88,7 +88,7 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)static int write_batch (struct xc_sr_context *ctx) {xc_interface *xch = ctx->xch; -xen_pfn_t *mfns =NULL , *types =NULL ; +xen_pfn_t *mfns = ctx->save.m->mfns, *types =NULL ;void *guest_mapping =NULL ;void **guest_data =NULL ;void **local_pages =NULL ;
@@ -105,8 +105,6 @@ static int write_batch(struct xc_sr_context *ctx)assert (nr_pfns !=0 ); -/* Mfns of the batch pfns. */ - mfns =malloc (nr_pfns *sizeof (*mfns));/* Types of the batch pfns. */ types =malloc (nr_pfns *sizeof (*types));/* Errors from attempting to map the gfns. */
@@ -118,7 +116,7 @@ static int write_batch(struct xc_sr_context *ctx)/* iovec[] for writev(). */ iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); -if ( !mfns || !types || !errors || !guest_data || !local_pages || !iov ) +if ( !types || !errors || !guest_data || !local_pages || !iov ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -277,7 +275,6 @@ static int write_batch(struct xc_sr_context *ctx)free (guest_data);free (errors);free (types); -free (mfns);return rc; }
Re: [PATCH v20210601 11/38] tools/guest: save: move mfns array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move mfns array into preallocated space.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 12/38 tools/guest: save: move types array
In reply to Olaf HeringRemove allocation from hotpath, move types array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++----- 2 files changed, 4 insertions(+), 5 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -227,6 +227,8 @@ struct xc_sr_save_arrays {xen_pfn_t batch_pfns[MAX_BATCH_SIZE ];/* write_batch: Mfns of the batch pfns. */ xen_pfn_t mfns[MAX_BATCH_SIZE ]; +/* write_batch: Types of the batch pfns. */ +xen_pfn_t types[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +2 -5
@@ -88,7 +88,7 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)static int write_batch (struct xc_sr_context *ctx) {xc_interface *xch = ctx->xch; -xen_pfn_t *mfns = ctx->save.m->mfns, *types =NULL ; +xen_pfn_t *mfns = ctx->save.m->mfns, *types = ctx->save.m->types;void *guest_mapping =NULL ;void **guest_data =NULL ;void **local_pages =NULL ;
@@ -105,8 +105,6 @@ static int write_batch(struct xc_sr_context *ctx)assert (nr_pfns !=0 ); -/* Types of the batch pfns. */ - types =malloc (nr_pfns *sizeof (*types));/* Errors from attempting to map the gfns. */ errors =malloc (nr_pfns *sizeof (*errors));/* Pointers to page data to send. Mapped gfns or local allocations. */
@@ -116,7 +114,7 @@ static int write_batch(struct xc_sr_context *ctx)/* iovec[] for writev(). */ iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); -if ( !types || !errors || !guest_data || !local_pages || !iov ) +if ( !errors || !guest_data || !local_pages || !iov ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -274,7 +272,6 @@ static int write_batch(struct xc_sr_context *ctx)free (local_pages);free (guest_data);free (errors); -free (types);return rc; }
Re: [PATCH v20210601 12/38] tools/guest: save: move types array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move types array into preallocated space.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 13/38 tools/guest: save: move errors array
In reply to Olaf HeringRemove allocation from hotpath, move errors array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++----- 2 files changed, 4 insertions(+), 5 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -229,6 +229,8 @@ struct xc_sr_save_arrays {xen_pfn_t mfns[MAX_BATCH_SIZE ];/* write_batch: Types of the batch pfns. */ xen_pfn_t types[MAX_BATCH_SIZE ]; +/* write_batch: Errors from attempting to map the gfns. */ +int errors[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +2 -5
@@ -92,7 +92,7 @@ static int write_batch(struct xc_sr_context *ctx)void *guest_mapping =NULL ;void **guest_data =NULL ;void **local_pages =NULL ; -int *errors =NULL , rc =-1 ; +int *errors = ctx->save.m->errors, rc =-1 ;unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;unsigned int nr_pfns = ctx->save.nr_batch_pfns;void *page, *orig_page;
@@ -105,8 +105,6 @@ static int write_batch(struct xc_sr_context *ctx)assert (nr_pfns !=0 ); -/* Errors from attempting to map the gfns. */ - errors =malloc (nr_pfns *sizeof (*errors));/* Pointers to page data to send. Mapped gfns or local allocations. */ guest_data =calloc (nr_pfns,sizeof (*guest_data));/* Pointers to locally allocated pages. Need freeing. */
@@ -114,7 +112,7 @@ static int write_batch(struct xc_sr_context *ctx)/* iovec[] for writev(). */ iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); -if ( !errors || !guest_data || !local_pages || !iov ) +if ( !guest_data || !local_pages || !iov ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -271,7 +269,6 @@ static int write_batch(struct xc_sr_context *ctx)free (iov);free (local_pages);free (guest_data); -free (errors);return rc; }
Re: [PATCH v20210601 13/38] tools/guest: save: move errors array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move errors array into preallocated space.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 14/38 tools/guest: save: move iov array
In reply to Olaf HeringRemove allocation from hotpath, move iov array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++----- 2 files changed, 4 insertions(+), 5 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -231,6 +231,8 @@ struct xc_sr_save_arrays {xen_pfn_t types[MAX_BATCH_SIZE ];/* write_batch: Errors from attempting to map the gfns. */ int errors[MAX_BATCH_SIZE ]; +/* write_batch: iovec[] for writev(). */ +struct iovec iov[MAX_BATCH_SIZE +4 ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +2 -5
@@ -97,7 +97,7 @@ static int write_batch(struct xc_sr_context *ctx)unsigned int nr_pfns = ctx->save.nr_batch_pfns;void *page, *orig_page;uint64_t *rec_pfns =NULL ; -struct iovec *iov =NULL ;int iovcnt =0 ; +struct iovec *iov = ctx->save.m->iov;int iovcnt =0 ;struct xc_sr_rec_page_data_header hdr = {0 };struct xc_sr_record rec = { .type =REC_TYPE_PAGE_DATA ,
@@ -109,10 +109,8 @@ static int write_batch(struct xc_sr_context *ctx) guest_data =calloc (nr_pfns,sizeof (*guest_data));/* Pointers to locally allocated pages. Need freeing. */ local_pages =calloc (nr_pfns,sizeof (*local_pages)); -/* iovec[] for writev(). */ - iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); -if ( !guest_data || !local_pages || !iov ) +if ( !guest_data || !local_pages ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -266,7 +264,6 @@ static int write_batch(struct xc_sr_context *ctx)xenforeignmemory_unmap (xch->fmem, guest_mapping, nr_pages_mapped);for ( i =0 ; local_pages && i < nr_pfns; ++i )free (local_pages[i]); -free (iov);free (local_pages);free (guest_data);
Re: [PATCH v20210601 14/38] tools/guest: save: move iov array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move iov array into preallocated space.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 15/38 tools/guest: save: move rec_pfns array
In reply to Olaf HeringRemove allocation from hotpath, move rec_pfns array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 11 +---------- 2 files changed, 3 insertions(+), 10 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -233,6 +233,8 @@ struct xc_sr_save_arrays {int errors[MAX_BATCH_SIZE ];/* write_batch: iovec[] for writev(). */ struct iovec iov[MAX_BATCH_SIZE +4 ]; +/* write_batch */ +uint64_t rec_pfns[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +1 -10
@@ -96,7 +96,7 @@ static int write_batch(struct xc_sr_context *ctx)unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;unsigned int nr_pfns = ctx->save.nr_batch_pfns;void *page, *orig_page; -uint64_t *rec_pfns =NULL ; +uint64_t *rec_pfns = ctx->save.m->rec_pfns;struct iovec *iov = ctx->save.m->iov;int iovcnt =0 ;struct xc_sr_rec_page_data_header hdr = {0 };struct xc_sr_record rec = {
@@ -201,14 +201,6 @@ static int write_batch(struct xc_sr_context *ctx) } } - rec_pfns =malloc (nr_pfns *sizeof (*rec_pfns)); -if ( !rec_pfns ) - { -ERROR ("Unable to allocate %zu bytes of memory for page data pfn list" , - nr_pfns *sizeof (*rec_pfns)); - goto err; - } - hdr.count = nr_pfns; rec.length =sizeof (hdr);
@@ -259,7 +251,6 @@ static int write_batch(struct xc_sr_context *ctx) rc = ctx->save.nr_batch_pfns =0 ; err: -free (rec_pfns);if ( guest_mapping )xenforeignmemory_unmap (xch->fmem, guest_mapping, nr_pages_mapped);for ( i =0 ; local_pages && i < nr_pfns; ++i )
Re: [PATCH v20210601 15/38] tools/guest: save: move rec_pfns array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move
rec_pfnsarray into preallocated space.Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 16/38 tools/guest: save: move guest_data array
In reply to Olaf HeringRemove allocation from hotpath, move guest_data array into preallocated space.
Because this was allocated with calloc:
Adjust the loop to clear unused entries as needed.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -235,6 +235,8 @@ struct xc_sr_save_arrays {struct iovec iov[MAX_BATCH_SIZE +4 ];/* write_batch */ uint64_t rec_pfns[MAX_BATCH_SIZE ]; +/* write_batch: Pointers to page data to send. Mapped gfns or local allocations. */ +void *guest_data[MAX_BATCH_SIZE ]; };struct xc_sr_restore_arrays {
tools/libs/saverestore/save.c +6 -5
@@ -90,7 +90,7 @@ static int write_batch(struct xc_sr_context *ctx)xc_interface *xch = ctx->xch;xen_pfn_t *mfns = ctx->save.m->mfns, *types = ctx->save.m->types;void *guest_mapping =NULL ; -void **guest_data =NULL ; +void **guest_data = ctx->save.m->guest_data;void **local_pages =NULL ;int *errors = ctx->save.m->errors, rc =-1 ;unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;
@@ -105,12 +105,10 @@ static int write_batch(struct xc_sr_context *ctx)assert (nr_pfns !=0 ); -/* Pointers to page data to send. Mapped gfns or local allocations. */ - guest_data =calloc (nr_pfns,sizeof (*guest_data));/* Pointers to locally allocated pages. Need freeing. */ local_pages =calloc (nr_pfns,sizeof (*local_pages)); -if ( !guest_data || !local_pages ) +if ( !local_pages ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -166,7 +164,10 @@ static int write_batch(struct xc_sr_context *ctx)for ( i =0 , p =0 ; i < nr_pfns; ++i ) {if (page_type_has_stream_data (types[i]) == false ) + { + guest_data[i] =NULL ;continue ; + }if ( errors[p] ) {
@@ -183,6 +184,7 @@ static int write_batch(struct xc_sr_context *ctx)if ( rc ) { + guest_data[i] =NULL ;if ( rc ==-1 && errno ==EAGAIN ) {set_bit (ctx->save.m->batch_pfns[i], ctx->save.deferred_pages);
@@ -256,7 +258,6 @@ static int write_batch(struct xc_sr_context *ctx)for ( i =0 ; local_pages && i < nr_pfns; ++i )free (local_pages[i]);free (local_pages); -free (guest_data);return rc; }
Re: [PATCH v20210601 16/38] tools/guest: save: move guest_data array
Quoted 1 line · Olaf Hering
Remove allocation from hotpath, move
guest_dataarray into preallocated
space.
Quoted 5 lines
Because this was allocated with calloc:
Adjust the loop to clear unused entries as needed.Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 17/38 tools/guest: save: move local_pages array
In reply to Olaf HeringRemove allocation from hotpath, move local_pages array into preallocated space.
Adjust the code to use the src page as is in case of HVM.
In case of PV the page may need to be normalised, use an private memory area for this purpose.
--- tools/libs/saverestore/common.h | 22 ++++++++++--------- tools/libs/saverestore/save.c | 25 +++------------------ tools/libs/saverestore/save_x86_hvm.c | 5 +++-- tools/libs/saverestore/save_x86_pv.c | 31 ++++++++++++++++++--------- 4 files changed, 39 insertions(+), 44 deletions(-)
tools/libs/saverestore/common.h +12 -10
@@ -45,16 +45,12 @@ struct xc_sr_save_ops* Optionally transform the contents of a page from being specific to the * sending environment, to being generic for the stream. * -* The page of data at the end of 'page' may be a read-only mapping of a -* running guest; it must not be modified. If no transformation is -* required, the callee should leave '*pages' untouched. +* The page of data '*src' may be a read-only mapping of a running guest; +* it must not be modified. If no transformation is required, the callee +* should leave '*src' untouched, and return it via '**ptr'. * -* If a transformation is required, the callee should allocate themselves -* a local page using malloc() and return it via '*page'. -* -* The caller shall free() '*page' in all cases. In the case that the -* callee encounters an error, it should *NOT* free() the memory it -* allocated for '*page'. +* If a transformation is required, the callee should provide the +* transformed page in a private buffer and return it via '**ptr'. * * It is valid to fail with EAGAIN if the transformation is not able to be * completed at this point. The page shall be retried later.
@@ -62,7 +58,7 @@ struct xc_sr_save_ops* @returns 0 for success, -1 for failure, with errno appropriately set. */ int (*normalise_page)(struct xc_sr_context *ctx,xen_pfn_t type, -void **page); +void *src,unsigned int idx,void **ptr);/** * Set up local environment to save a domain. (Typically querying
@@ -383,6 +379,12 @@ struct xc_sr_contextunion { +struct + { +/* Used by write_batch for modified pages. */ +void *normalised_pages; + } save; +struct {/* State machine for the order of received records. */
tools/libs/saverestore/save.c +3 -22
@@ -91,11 +91,10 @@ static int write_batch(struct xc_sr_context *ctx)xen_pfn_t *mfns = ctx->save.m->mfns, *types = ctx->save.m->types;void *guest_mapping =NULL ;void **guest_data = ctx->save.m->guest_data; -void **local_pages =NULL ;int *errors = ctx->save.m->errors, rc =-1 ;unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;unsigned int nr_pfns = ctx->save.nr_batch_pfns; -void *page, *orig_page; +void *src;uint64_t *rec_pfns = ctx->save.m->rec_pfns;struct iovec *iov = ctx->save.m->iov;int iovcnt =0 ;struct xc_sr_rec_page_data_header hdr = {0 };
@@ -105,16 +104,6 @@ static int write_batch(struct xc_sr_context *ctx)assert (nr_pfns !=0 ); -/* Pointers to locally allocated pages. Need freeing. */ - local_pages =calloc (nr_pfns,sizeof (*local_pages)); - -if ( !local_pages ) - { -ERROR ("Unable to allocate arrays for a batch of %u pages" , - nr_pfns); - goto err; - } -for ( i =0 ; i < nr_pfns; ++i ) { types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx,
@@ -176,11 +165,8 @@ static int write_batch(struct xc_sr_context *ctx) goto err; } - orig_page = page = guest_mapping + (p *PAGE_SIZE ); - rc = ctx->save.ops.normalise_page (ctx, types[i], &page); - -if ( orig_page != page ) - local_pages[i] = page; + src = guest_mapping + (p *PAGE_SIZE ); + rc = ctx->save.ops.normalise_page (ctx, types[i], src, i, &guest_data[i]);if ( rc ) {
@@ -195,8 +181,6 @@ static int write_batch(struct xc_sr_context *ctx)else goto err; } -else - guest_data[i] = page; rc =-1 ; ++p;
@@ -255,9 +239,6 @@ static int write_batch(struct xc_sr_context *ctx) err:if ( guest_mapping )xenforeignmemory_unmap (xch->fmem, guest_mapping, nr_pages_mapped); -for ( i =0 ; local_pages && i < nr_pfns; ++i ) -free (local_pages[i]); -free (local_pages);return rc; }
tools/libs/saverestore/save_x86_hvm.c +3 -2
@@ -129,9 +129,10 @@ static xen_pfn_t x86_hvm_pfn_to_gfn(const struct xc_sr_context *ctx,return pfn; } -static int x86_hvm_normalise_page (struct xc_sr_context *ctx, -xen_pfn_t type,void **page) +static int x86_hvm_normalise_page (struct xc_sr_context *ctx,xen_pfn_t type, +void *src,unsigned int idx,void **ptr) { + *ptr = src;return 0 ; }
tools/libs/saverestore/save_x86_pv.c +21 -10
@@ -999,29 +999,31 @@ static xen_pfn_t x86_pv_pfn_to_gfn(const struct xc_sr_context *ctx,* save_ops function. Performs pagetable normalisation on appropriate pages. */ static int x86_pv_normalise_page (struct xc_sr_context *ctx,xen_pfn_t type, -void **page) +void *src,unsigned int idx,void **ptr) {xc_interface *xch = ctx->xch; -void *local_page;int rc; +void *dst; type &=XEN_DOMCTL_PFINFO_LTABTYPE_MASK ;if ( type <XEN_DOMCTL_PFINFO_L1TAB || type >XEN_DOMCTL_PFINFO_L4TAB ) + { + *ptr = src;return 0 ; + } - local_page =malloc (PAGE_SIZE ); -if ( !local_page ) +if ( idx >=MAX_BATCH_SIZE ) { -ERROR ("Unable to allocate scratch page" ); - rc =-1 ; - goto out; +ERROR ("idx %u out of range" , idx); + errno =ERANGE ; +return -1 ; } - rc =normalise_pagetable (ctx, *page, local_page, type); - *page = local_page; + dst = ctx->x86.pv.save.normalised_pages + idx *PAGE_SIZE ; + rc =normalise_pagetable (ctx, src, dst, type); + *ptr = dst; - out:return rc; }
@@ -1031,8 +1033,16 @@ static int x86_pv_normalise_page(struct xc_sr_context *ctx, xen_pfn_t type,*/ static int x86_pv_setup (struct xc_sr_context *ctx) { +xc_interface *xch = ctx->xch;int rc; + ctx->x86.pv.save.normalised_pages =malloc (MAX_BATCH_SIZE *PAGE_SIZE ); +if ( !ctx->x86.pv.save.normalised_pages ) + { +PERROR ("Failed to allocate normalised_pages" ); +return -1 ; + } + rc =x86_pv_domain_info (ctx);if ( rc )return rc;
@@ -1118,6 +1128,7 @@ static int x86_pv_check_vm_state(struct xc_sr_context *ctx)static int x86_pv_cleanup (struct xc_sr_context *ctx) { +free (ctx->x86.pv.save.normalised_pages);free (ctx->x86.pv.p2m_pfns);if ( ctx->x86.pv.p2m )
Re: [PATCH v20210601 17/38] tools/guest: save: move local_pages array
Quoted 7 lines · Olaf Hering
Remove allocation from hotpath, move
local_pagesarray into preallocated space.Adjust the code to use the src page as is in case of HVM.
In case of PV the page may need to be normalised, use an private memory area for this purpose.Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 18/38 tools/guest: restore: move pfns array
In reply to Olaf HeringRemove allocation from hotpath, move pfns array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/restore.c | 6 ++---- 2 files changed, 4 insertions(+), 4 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -236,6 +236,8 @@ struct xc_sr_save_arrays { };struct xc_sr_restore_arrays { +/* handle_page_data */ +xen_pfn_t pfns[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +2 -4
@@ -315,7 +315,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)unsigned int i, pages_of_data =0 ;int rc =-1 ; -xen_pfn_t *pfns =NULL , pfn; +xen_pfn_t *pfns = ctx->restore.m->pfns, pfn;uint32_t *types =NULL , type;/*
@@ -363,9 +363,8 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) goto err; } - pfns =malloc (pages->count *sizeof (*pfns)); types =malloc (pages->count *sizeof (*types)); -if ( !pfns || !types ) +if ( !types ) {ERROR ("Unable to allocate enough memory for %u pfns" , pages->count);
@@ -412,7 +411,6 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) &pages->pfn[pages->count]); err:free (types); -free (pfns);return rc; }
Re: [PATCH v20210601 18/38] tools/guest: restore: move pfns array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move pfns array into preallocated space.
Signed-off-by Olaf Hering <[email protected]>
Juergen
PATCH v20210601 19/38 tools/guest: restore: move types array
In reply to Olaf HeringRemove allocation from hotpath, move types array into preallocated space.
--- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 12 +----------- 2 files changed, 2 insertions(+), 11 deletions(-)
tools/libs/saverestore/common.h +1 -0
@@ -238,6 +238,7 @@ struct xc_sr_save_arrays {struct xc_sr_restore_arrays {/* handle_page_data */ xen_pfn_t pfns[MAX_BATCH_SIZE ]; +uint32_t types[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +1 -11
@@ -316,7 +316,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)int rc =-1 ;xen_pfn_t *pfns = ctx->restore.m->pfns, pfn; -uint32_t *types =NULL , type; +uint32_t *types = ctx->restore.m->types, type;/* * v2 compatibility only exists for x86 streams. This is a bit of a
@@ -363,14 +363,6 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) goto err; } - types =malloc (pages->count *sizeof (*types)); -if ( !types ) - { -ERROR ("Unable to allocate enough memory for %u pfns" , - pages->count); - goto err; - } -for ( i =0 ; i < pages->count; ++i ) { pfn = pages->pfn[i] &PAGE_DATA_PFN_MASK ;
@@ -410,8 +402,6 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) rc =process_page_data (ctx, pages->count, pfns, types, &pages->pfn[pages->count]); err: -free (types); -return rc; }
Re: [PATCH v20210601 19/38] tools/guest: restore: move types array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move types array into preallocated space.
Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 20/38 tools/guest: restore: move mfns array
In reply to Olaf HeringRemove allocation from hotpath, move mfns array into preallocated space.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/restore.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -239,6 +239,8 @@ struct xc_sr_restore_arrays {/* handle_page_data */ xen_pfn_t pfns[MAX_BATCH_SIZE ];uint32_t types[MAX_BATCH_SIZE ]; +/* process_page_data */ +xen_pfn_t mfns[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +2 -3
@@ -205,7 +205,7 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, xen_pfn_t *pfns, uint32_t *types, void *page_data) {xc_interface *xch = ctx->xch; -xen_pfn_t *mfns =malloc (count *sizeof (*mfns)); +xen_pfn_t *mfns = ctx->restore.m->mfns;int *map_errs =malloc (count *sizeof (*map_errs));int rc;void *mapping =NULL , *guest_page =NULL ;
@@ -213,7 +213,7 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, j,/* j indexes the subset of pfns we decide to map. */ nr_pages =0 ; -if ( !mfns || !map_errs ) +if ( !map_errs ) { rc =-1 ; ERROR("Failed to allocate %zu bytes to process page data",
@@ -299,7 +299,6 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,xenforeignmemory_unmap (xch->fmem, mapping, nr_pages);free (map_errs); -free (mfns);return rc; }
Re: [PATCH v20210601 20/38] tools/guest: restore: move mfns array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move mfns array into preallocated space.
Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 21/38 tools/guest: restore: move map_errs array
In reply to Olaf HeringRemove allocation from hotpath, move map_errs array into preallocated space.
--- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 12 +----------- 2 files changed, 2 insertions(+), 11 deletions(-)
tools/libs/saverestore/common.h +1 -0
@@ -241,6 +241,7 @@ struct xc_sr_restore_arrays {uint32_t types[MAX_BATCH_SIZE ];/* process_page_data */ xen_pfn_t mfns[MAX_BATCH_SIZE ]; +int map_errs[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +1 -11
@@ -206,21 +206,13 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, {xc_interface *xch = ctx->xch;xen_pfn_t *mfns = ctx->restore.m->mfns; -int *map_errs =malloc (count *sizeof (*map_errs)); +int *map_errs = ctx->restore.m->map_errs;int rc;void *mapping =NULL , *guest_page =NULL ;unsigned int i,/* i indexes the pfns from the record. */ j,/* j indexes the subset of pfns we decide to map. */ nr_pages =0 ; -if ( !map_errs ) - { - rc =-1 ; -ERROR ("Failed to allocate %zu bytes to process page data" , - count * (sizeof (*mfns) +sizeof (*map_errs))); - goto err; - } - rc =populate_pfns (ctx, count, pfns, types);if ( rc ) {
@@ -298,8 +290,6 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,if ( mapping )xenforeignmemory_unmap (xch->fmem, mapping, nr_pages); -free (map_errs); -return rc; }
Re: [PATCH v20210601 21/38] tools/guest: restore: move map_errs array
Quoted 3 lines · Olaf Hering
Remove allocation from hotpath, move
map_errsarray into preallocated space.Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 22/38 tools/guest: restore: move mfns array in populate_pfns
In reply to Olaf HeringRemove allocation from hotpath, move populate_pfns mfns array into preallocated space. Use some prefix to avoid conflict with an array used in handle_page_data.
--- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/restore.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-)
tools/libs/saverestore/common.h +2 -0
@@ -242,6 +242,8 @@ struct xc_sr_restore_arrays {/* process_page_data */ xen_pfn_t mfns[MAX_BATCH_SIZE ];int map_errs[MAX_BATCH_SIZE ]; +/* populate_pfns */ +xen_pfn_t pp_mfns[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +2 -3
@@ -138,12 +138,12 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count, const xen_pfn_t *original_pfns, const uint32_t *types) { xc_interface *xch = ctx->xch; -xen_pfn_t *mfns =malloc (count *sizeof (*mfns)), +xen_pfn_t *mfns = ctx->restore.m->pp_mfns, *pfns =malloc (count *sizeof (*pfns));unsigned int i, nr_pfns =0 ;int rc =-1 ; -if ( !mfns || !pfns ) +if ( !pfns ) {ERROR ("Failed to allocate %zu bytes for populating the physmap" ,2 * count *sizeof (*mfns));
@@ -191,7 +191,6 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count, err:free (pfns); -free (mfns);return rc; }
Re: [PATCH v20210601 22/38] tools/guest: restore: move mfns array in populate_pfns
Quoted 4 lines · Olaf Hering
Remove allocation from hotpath, move
populate_pfnsmfns array into preallocated space. Use some prefix to avoid conflict with an array used inhandle_page_data.Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 23/38 tools/guest: restore: move pfns array in populate_pfns
In reply to Olaf HeringRemove allocation from hotpath, move populate_pfns' pfns array into preallocated space. Use some prefix to avoid conflict with an array used in handle_page_data.
--- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 11 +---------- 2 files changed, 2 insertions(+), 10 deletions(-)
tools/libs/saverestore/common.h +1 -0
@@ -244,6 +244,7 @@ struct xc_sr_restore_arrays {int map_errs[MAX_BATCH_SIZE ];/* populate_pfns */ xen_pfn_t pp_mfns[MAX_BATCH_SIZE ]; +xen_pfn_t pp_pfns[MAX_BATCH_SIZE ]; };struct xc_sr_context
tools/libs/saverestore/restore.c +1 -10
@@ -139,17 +139,10 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count, {xc_interface *xch = ctx->xch;xen_pfn_t *mfns = ctx->restore.m->pp_mfns, - *pfns =malloc (count *sizeof (*pfns)); + *pfns = ctx->restore.m->pp_pfns;unsigned int i, nr_pfns =0 ;int rc =-1 ; -if ( !pfns ) - { -ERROR ("Failed to allocate %zu bytes for populating the physmap" , -2 * count *sizeof (*mfns)); - goto err; - } -for ( i =0 ; i < count; ++i ) { if ( (!types ||
@@ -190,8 +183,6 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned int count, rc =0 ; err: -free (pfns); -return rc; }
Re: [PATCH v20210601 23/38] tools/guest: restore: move pfns array in populate_pfns
Quoted 4 lines · Olaf Hering
Remove allocation from hotpath, move
populate_pfns' pfns array into preallocated space. Use some prefix to avoid conflict with an array used inhandle_page_data.Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 24/38 tools/guest: restore: split record processing
In reply to Olaf Heringhandle_page_data must be able to read directly into mapped guest memory. This will avoid unneccesary memcpy calls for data which can be consumed verbatim.
Rearrange the code to allow decisions based on the incoming record.
This change is preparation for future changes in handle_page_data, no change in behavior is intended.
--- tools/libs/saverestore/common.c | 33 ++++++++++++--------- tools/libs/saverestore/common.h | 4 ++- tools/libs/saverestore/restore.c | 49 ++++++++++++++++++++++---------- tools/libs/saverestore/save.c | 7 ++++- 4 files changed, 63 insertions(+), 30 deletions(-)
tools/libs/saverestore/common.c +20 -13
@@ -91,26 +91,33 @@ int write_split_record(struct xc_sr_context *ctx, struct xc_sr_record *rec,return -1 ; } -int read_record (struct xc_sr_context *ctx,int fd,struct xc_sr_record *rec) +int read_record_header (struct xc_sr_context *ctx,int fd,struct xc_sr_rhdr *rhdr) {xc_interface *xch = ctx->xch; -struct xc_sr_rhdr rhdr; -size_t datasz; -if (read_exact (fd, &rhdr,sizeof (rhdr)) ) +if (read_exact (fd, rhdr,sizeof (*rhdr)) ) {PERROR ("Failed to read Record Header from stream" );return -1 ; } -if ( rhdr.length >REC_LENGTH_MAX ) +if ( rhdr->length >REC_LENGTH_MAX ) { -ERROR ("Record (0x%08x, %s) length %#x exceeds max (%#x)" , rhdr.type, -rec_type_to_str (rhdr.type), rhdr.length,REC_LENGTH_MAX ); +ERROR ("Record (0x%08x, %s) length %#x exceeds max (%#x)" , rhdr->type, +rec_type_to_str (rhdr->type), rhdr->length,REC_LENGTH_MAX );return -1 ; } - datasz =ROUNDUP (rhdr.length,REC_ALIGN_ORDER ); +return 0 ; +} + +int read_record_data (struct xc_sr_context *ctx,int fd,struct xc_sr_rhdr *rhdr, +struct xc_sr_record *rec) +{ +xc_interface *xch = ctx->xch; +size_t datasz; + + datasz =ROUNDUP (rhdr->length,REC_ALIGN_ORDER );if ( datasz ) {
@@ -119,7 +126,7 @@ int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)if ( !rec->data ) {ERROR ("Unable to allocate %zu bytes for record data (0x%08x, %s)" , - datasz, rhdr.type,rec_type_to_str (rhdr.type)); + datasz, rhdr->type,rec_type_to_str (rhdr->type));return -1 ; }
@@ -128,18 +135,18 @@ int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)free (rec->data); rec->data =NULL ;PERROR ("Failed to read %zu bytes of data for record (0x%08x, %s)" , - datasz, rhdr.type,rec_type_to_str (rhdr.type)); + datasz, rhdr->type,rec_type_to_str (rhdr->type));return -1 ; } }else rec->data =NULL ; - rec->type = rhdr.type; - rec->length = rhdr.length; + rec->type = rhdr->type; + rec->length = rhdr->length;return 0 ; -}; +}static void __attribute__((unused))build_assertions (void ) {
tools/libs/saverestore/common.h +3 -1
@@ -487,7 +487,9 @@ static inline int write_record(struct xc_sr_context *ctx,* * On failure, the contents of the record structure are undefined. */ -int read_record (struct xc_sr_context *ctx,int fd,struct xc_sr_record *rec); +int read_record_header (struct xc_sr_context *ctx,int fd,struct xc_sr_rhdr *rhdr); +int read_record_data (struct xc_sr_context *ctx,int fd,struct xc_sr_rhdr *rhdr, +struct xc_sr_record *rec);/* * This would ideally be private in restore.c, but is needed by
tools/libs/saverestore/restore.c +34 -15
125 lines not on this page · raw message
tools/libs/saverestore/save.c +6 -1
@@ -590,6 +590,7 @@ static int send_memory_live(struct xc_sr_context *ctx)static int colo_merge_secondary_dirty_bitmap (struct xc_sr_context *ctx) {xc_interface *xch = ctx->xch; +struct xc_sr_rhdr rhdr;struct xc_sr_record rec;uint64_t *pfns =NULL ;uint64_t pfn;
@@ -598,7 +599,11 @@ static int colo_merge_secondary_dirty_bitmap(struct xc_sr_context *ctx) DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, &ctx->save.dirty_bitmap_hbuf); - rc =read_record (ctx, ctx->save.recv_fd, &rec); + rc =read_record_header (ctx, ctx->save.recv_fd, &rhdr); +if ( rc ) + goto err; + + rc =read_record_data (ctx, ctx->save.recv_fd, &rhdr, &rec);if ( rc ) goto err;
Re: [PATCH v20210601 24/38] tools/guest: restore: split record processing
Quoted 2 lines · Olaf Hering
handle_page_datamust be able to read directly into mapped guest memory. This will avoid unneccesary memcpy calls for data which can be consumed
verbatim.
Quoted 7 lines
Rearrange the code to allow decisions based on the incoming record.
This change is preparation for future changes in
handle_page_data, no change in behavior is intended.Signed-off-by Olaf Hering<[email protected]>
Juergen
PATCH v20210601 25/38 tools/guest: restore: split handle_page_data
In reply to Olaf Heringhandle_page_data must be able to read directly into mapped guest memory. This will avoid unneccesary memcpy calls for data that can be consumed verbatim.
Split the various steps of record processing:
- move processing to handle_buffered_page_data
- adjust xenforeignmemory_map to set errno in case of failure
- adjust verify mode to set errno in case of failure
This change is preparation for future changes in handle_page_data, no change in behavior is intended.
--- tools/libs/saverestore/common.h | 9 + tools/libs/saverestore/restore.c | 343 ++++++++++++++++++++----------- 2 files changed, 231 insertions(+), 121 deletions(-)
tools/libs/saverestore/common.h +9 -0
@@ -242,9 +242,14 @@ struct xc_sr_restore_arrays {/* process_page_data */ xen_pfn_t mfns[MAX_BATCH_SIZE ];int map_errs[MAX_BATCH_SIZE ]; +void *guest_data[MAX_BATCH_SIZE ]; +/* populate_pfns */ xen_pfn_t pp_mfns[MAX_BATCH_SIZE ];xen_pfn_t pp_pfns[MAX_BATCH_SIZE ]; + +/* Must be the last member */ +struct xc_sr_rec_page_data_header pages; };struct xc_sr_context
@@ -335,7 +340,11 @@ struct xc_sr_context/* Sender has invoked verify mode on the stream. */ bool verify; +void *verify_buf; +struct xc_sr_restore_arrays *m; +void *guest_mapping; +uint32_t nr_mapped_pages; } restore; };
tools/libs/saverestore/restore.c +222 -121
412 lines not on this page · raw message
PATCH v20210601 26/38 tools/guest: restore: write data directly into guest
In reply to Olaf HeringRead incoming migration stream directly into the guest memory. This avoids the memory allocation and copying, and the resulting performance penalty.
--- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 132 ++++++++++++++++++++++++++++++- 2 files changed, 129 insertions(+), 4 deletions(-)
tools/libs/saverestore/common.h +1 -0
@@ -243,6 +243,7 @@ struct xc_sr_restore_arrays {xen_pfn_t mfns[MAX_BATCH_SIZE ];int map_errs[MAX_BATCH_SIZE ];void *guest_data[MAX_BATCH_SIZE ]; +struct iovec iov[MAX_BATCH_SIZE ];/* populate_pfns */ xen_pfn_t pp_mfns[MAX_BATCH_SIZE ];
tools/libs/saverestore/restore.c +128 -4
145 lines not on this page · raw message
PATCH v20210601 27/38 tools: recognize LIBXL_API_VERSION for 4.16
In reply to Olaf HeringThis is required by upcoming API changes.
--- tools/include/libxl.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/include/libxl.h +2 -1
@@ -729,7 +729,8 @@ typedef struct libxl__ctx libxl_ctx;#if LIBXL_API_VERSION !=0x040200 &&LIBXL_API_VERSION !=0x040300 && \LIBXL_API_VERSION !=0x040400 &&LIBXL_API_VERSION !=0x040500 && \LIBXL_API_VERSION !=0x040700 &&LIBXL_API_VERSION !=0x040800 && \ -LIBXL_API_VERSION !=0x041300 &&LIBXL_API_VERSION !=0x041400 +LIBXL_API_VERSION !=0x041300 &&LIBXL_API_VERSION !=0x041400 && \ +LIBXL_API_VERSION !=0x041600 #error Unknown LIBXL_API_VERSION#endif #endif
PATCH v20210601 28/38 tools: adjust libxl_domain_suspend to receive a struct props
In reply to Olaf HeringUpcoming changes will pass more knobs down to xc_domain_save. Adjust the libxl_domain_suspend API to allow easy adding of additional knobs.
No change in behavior intented.
--- tools/include/libxl.h | 26 +++++++++++++++++++++++--- tools/libs/light/libxl_domain.c | 7 ++++--- tools/ocaml/libs/xl/xenlight_stubs.c | 3 ++- tools/xl/xl_migrate.c | 9 ++++++--- tools/xl/xl_saverestore.c | 3 ++- 5 files changed, 37 insertions(+), 11 deletions(-)
tools/include/libxl.h +23 -3
@@ -1706,12 +1706,32 @@ static inline int libxl_retrieve_domain_configuration_0x041200( libxl_retrieve_domain_configuration_0x041200#endif +/* +* LIBXL_HAVE_DOMAIN_SUSPEND_PROPS indicates that the +* libxl_domain_suspend_props() function takes a props struct. +*/ +#define LIBXL_HAVE_DOMAIN_SUSPEND_PROPS 1 + +typedef struct { +uint32_t flags;/* LIBXL_SUSPEND_* */ +}libxl_domain_suspend_props ; +#define LIBXL_SUSPEND_DEBUG 1 +#define LIBXL_SUSPEND_LIVE 2 + int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, - int flags, /* LIBXL_SUSPEND_* */ + libxl_domain_suspend_props *props, const libxl_asyncop_how *ao_how) LIBXL_EXTERNAL_CALLERS_ONLY; -#define LIBXL_SUSPEND_DEBUG 1 -#define LIBXL_SUSPEND_LIVE 2 +#if defined(LIBXL_API_VERSION ) &&LIBXL_API_VERSION <0x041600 +static inline int libxl_domain_suspend_0x041500 (libxl_ctx *ctx,uint32_t domid, +int fd,int flags,/* LIBXL_SUSPEND_* */ +const libxl_asyncop_how *ao_how) +{ +libxl_domain_suspend_props props = { .flags = flags, }; +return libxl_domain_suspend (ctx, domid, fd, &props, ao_how); +} +#define libxl_domain_suspend libxl_domain_suspend_0x041500 +#endif /* * Only suspend domain, do not save its state to file, do not destroy it.
tools/libs/light/libxl_domain.c +4 -3
@@ -505,7 +505,8 @@ static void domain_suspend_cb(libxl__egc *egc, } -int libxl_domain_suspend (libxl_ctx *ctx,uint32_t domid,int fd,int flags, +int libxl_domain_suspend (libxl_ctx *ctx,uint32_t domid,int fd, +libxl_domain_suspend_props *props,const libxl_asyncop_how *ao_how) {AO_CREATE (ctx, domid, ao_how);
@@ -526,8 +527,8 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags, dss->domid = domid; dss->fd = fd; dss->type = type; - dss->live = flags &LIBXL_SUSPEND_LIVE ; - dss->debug = flags &LIBXL_SUSPEND_DEBUG ; + dss->live = props->flags &LIBXL_SUSPEND_LIVE ; + dss->debug = props->flags &LIBXL_SUSPEND_DEBUG ; dss->checkpointed_stream =LIBXL_CHECKPOINTED_STREAM_NONE ; rc = libxl__fd_flags_modify_save(gc, dss->fd,
tools/ocaml/libs/xl/xenlight_stubs.c +2 -1
@@ -614,10 +614,11 @@ value stub_libxl_domain_suspend(value ctx, value domid, value fd, value async, vint ret;uint32_t c_domid =Int_val (domid);int c_fd =Int_val (fd); +libxl_domain_suspend_props props = {};libxl_asyncop_how *ao_how =aohow_val (async);caml_enter_blocking_section (); - ret =libxl_domain_suspend (CTX , c_domid, c_fd,0 , ao_how); + ret =libxl_domain_suspend (CTX , c_domid, c_fd, &props, ao_how);caml_leave_blocking_section ();free (ao_how);
tools/xl/xl_migrate.c +6 -3
@@ -186,7 +186,10 @@ static void migrate_domain(uint32_t domid, int preserve_domid,char *away_domname;char rc_buf;uint8_t *config_data; -int config_len, flags =LIBXL_SUSPEND_LIVE ; +int config_len; +libxl_domain_suspend_props props = { + .flags =LIBXL_SUSPEND_LIVE , + };save_domain_core_begin (domid, preserve_domid, override_config_file, &config_data, &config_len);
@@ -205,8 +208,8 @@ static void migrate_domain(uint32_t domid, int preserve_domid,xtl_stdiostream_adjust_flags (logger,XTL_STDIOSTREAM_HIDE_PROGRESS ,0 );if (debug) - flags |=LIBXL_SUSPEND_DEBUG ; - rc =libxl_domain_suspend (ctx, domid, send_fd, flags,NULL ); + props.flags |=LIBXL_SUSPEND_DEBUG ; + rc =libxl_domain_suspend (ctx, domid, send_fd, &props,NULL );if (rc) {fprintf (stderr,"migration sender: libxl_domain_suspend failed" " (rc=%d)\n" , rc);
tools/xl/xl_saverestore.c +2 -1
@@ -130,6 +130,7 @@ static int save_domain(uint32_t domid, int preserve_domid,int fd;uint8_t *config_data;int config_len; +libxl_domain_suspend_props props = {};save_domain_core_begin (domid, preserve_domid, override_config_file, &config_data, &config_len);
@@ -146,7 +147,7 @@ static int save_domain(uint32_t domid, int preserve_domid,save_domain_core_writeconfig (fd, filename, config_data, config_len); -int rc =libxl_domain_suspend (ctx, domid, fd,0 ,NULL ); +int rc =libxl_domain_suspend (ctx, domid, fd, &props,NULL );close (fd);if (rc <0 ) {
PATCH v20210601 29/38 tools: change struct precopy_stats to precopy_stats_t
In reply to Olaf HeringThis will help libxl_save_msgs_gen.pl to copy the struct as a region of memory.
No change in behavior intented.
--- tools/include/xensaverestore.h | 7 +++---- tools/libs/saverestore/common.h | 2 +- tools/libs/saverestore/save.c | 6 +++--- 3 files changed, 7 insertions(+), 8 deletions(-)
tools/include/xensaverestore.h +3 -4
@@ -23,18 +23,17 @@#define XCFLAGS_DEBUG (1 << 1)/* For save's precopy_policy(). */ -struct precopy_stats -{ +typedef struct {unsigned int iteration;unsigned long total_written;long dirty_count;/* -1 if unknown */ -}; +}precopy_stats_t ;/* * A precopy_policy callback may not be running in the same address * space as libxc an so precopy_stats is passed by value. */ -typedef int (*precopy_policy_t )(struct precopy_stats ,void *); +typedef int (*precopy_policy_t )(precopy_stats_t ,void *);/* callbacks provided by xc_domain_save */ struct save_callbacks {
tools/libs/saverestore/common.h +1 -1
@@ -283,7 +283,7 @@ struct xc_sr_contextsize_t pages_sent;size_t overhead_sent; -struct precopy_stats stats; +precopy_stats_t stats;unsigned int nr_batch_pfns;unsigned long *deferred_pages;
tools/libs/saverestore/save.c +3 -3
@@ -489,7 +489,7 @@ static int update_progress_string(struct xc_sr_context *ctx, char **str)#define SPP_MAX_ITERATIONS 5#define SPP_TARGET_DIRTY_COUNT 50 -static int simple_precopy_policy (struct precopy_stats stats,void *user) +static int simple_precopy_policy (precopy_stats_t stats,void *user) { return ((stats.dirty_count >= 0 && stats.dirty_count < SPP_TARGET_DIRTY_COUNT) ||
@@ -516,13 +516,13 @@ static int send_memory_live(struct xc_sr_context *ctx)precopy_policy_t precopy_policy = ctx->save.callbacks->precopy_policy;void *data = ctx->save.callbacks->data; -struct precopy_stats *policy_stats; +precopy_stats_t *policy_stats; rc =update_progress_string (ctx, &progress_str);if ( rc ) goto out; - ctx->save.stats = (struct precopy_stats ){ + ctx->save.stats = (precopy_stats_t ){ .dirty_count = ctx->save.p2m_size, }; policy_stats = &ctx->save.stats;
PATCH v20210601 30/38 tools: add callback to libxl for precopy_policy and precopy_stats_t
In reply to Olaf HeringThis duplicates simple_precopy_policy. To recap its purpose:
- do up to 5 iterations of copying dirty domU memory to target, including the initial copying of all domU memory, excluding the final copying while the domU is suspended
- do fewer iterations in case the domU dirtied less than 50 pages
Take the opportunity to also move xen_pfn_t into qw().
--- tools/libs/light/libxl_dom_save.c | 19 +++++++++++++++++++ tools/libs/light/libxl_internal.h | 2 ++ tools/libs/light/libxl_save_msgs_gen.pl | 3 ++- 3 files changed, 23 insertions(+), 1 deletion(-)
tools/libs/light/libxl_dom_save.c +19 -0
@@ -373,6 +373,24 @@ int libxl__save_emulator_xenstore_data(libxl__domain_save_state *dss,return rc; } +static int libxl__domain_save_precopy_policy (precopy_stats_t stats,void *user) +{ +libxl__save_helper_state *shs = user; +libxl__domain_save_state *dss = shs->caller_state; +STATE_AO_GC (dss->ao); + +LOGD (DEBUG , shs->domid,"iteration %u dirty_count %ld total_written %lu" , + stats.iteration, stats.dirty_count, stats.total_written); +if (stats.dirty_count >=0 && stats.dirty_count <LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT ) + goto stop_copy; +if (stats.iteration >=LIBXL_XGS_POLICY_MAX_ITERATIONS ) + goto stop_copy; +return XGS_POLICY_CONTINUE_PRECOPY ; + +stop_copy: +return XGS_POLICY_STOP_AND_COPY ; +} +/*----- main code for saving, in order of execution -----*/ void libxl__domain_save (libxl__egc *egc,libxl__domain_save_state *dss)
@@ -430,6 +448,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss) callbacks->suspend = libxl__domain_suspend_callback; callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty; + callbacks->precopy_policy = libxl__domain_save_precopy_policy; dss->sws.ao = dss->ao; dss->sws.dss = dss;
tools/libs/light/libxl_internal.h +2 -0
@@ -125,6 +125,8 @@#define DOMID_XS_PATH "domid"#define PVSHIM_BASENAME "xen-shim"#define PVSHIM_CMDLINE "pv-shim console=xen,pv" +#define LIBXL_XGS_POLICY_MAX_ITERATIONS 5 +#define LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT 50/* Size macros. */ #define __AC (X ,Y ) (X##Y)
tools/libs/light/libxl_save_msgs_gen.pl +2 -1
@@ -23,6 +23,7 @@ our @msgs = ( STRING doing_what), 'unsigned long', 'done', 'unsigned long', 'total'] ], + [ 'scxW', "precopy_policy", ['precopy_stats_t', 'stats'] ], [ 'srcxA', "suspend", [] ], [ 'srcxA', "postcopy", [] ], [ 'srcxA', "checkpoint", [] ],
@@ -142,7 +143,7 @@ static void bytes_put(unsigned char *const buf, int *len, END -foreach my $simpletype (qw(int uint16_t uint32_t unsigned), 'unsigned long', 'xen_pfn_t') { +foreach my $simpletype (qw(int uint16_t uint32_t unsigned precopy_stats_t xen_pfn_t), 'unsigned long') { my $typeid = typeid($simpletype); $out_body{'callout'} .= <<END; static int ${typeid}_get(const unsigned char **msg,
PATCH v20210601 31/38 tools: add --max_iters to libxl_domain_suspend
In reply to Olaf HeringMigrating a large, and potentially busy, domU will take more time than neccessary due to excessive number of copying iterations.
Allow to host admin to control the number of iterations which copy cumulated domU dirty pages to the target host.
The default remains 5, which means one initial iteration to copy the entire domU memory, and up to 4 additional iterations to copy dirty memory from the still running domU. After the given number of iterations the domU is suspended, remaining dirty memory is copied and the domU is finally moved to the target host.
This patch adjusts xl(1) and the libxl API.
External users check LIBXL_HAVE_DOMAIN_SUSPEND_PROPS for the availibility of the new .max_iters property.
--- docs/man/xl.1.pod.in | 4 ++++ tools/include/libxl.h | 1 + tools/libs/light/libxl_dom_save.c | 2 +- tools/libs/light/libxl_domain.c | 1 + tools/libs/light/libxl_internal.h | 1 + tools/xl/xl_cmdtable.c | 3 ++- tools/xl/xl_migrate.c | 10 +++++++++- 7 files changed, 19 insertions(+), 3 deletions(-)
docs/man/xl.1.pod.in +4 -0
@@ -496,6 +496,10 @@ such that it will be identical on the destination host, unless that configuration is overridden using the B<-C> option. Note that it is not possible to use this option for a 'localhost' migration. +=item B<--max_iters> I<iterations> + +Number of copy iterations before final suspend+move (default: 5) + =back =item B<remus> [I<OPTIONS>] I<domain-id> I<host>
tools/include/libxl.h +1 -0
@@ -1714,6 +1714,7 @@ static inline int libxl_retrieve_domain_configuration_0x041200(typedef struct {uint32_t flags;/* LIBXL_SUSPEND_* */ +uint32_t max_iters; }libxl_domain_suspend_props ;#define LIBXL_SUSPEND_DEBUG 1#define LIBXL_SUSPEND_LIVE 2
tools/libs/light/libxl_dom_save.c +1 -1
@@ -383,7 +383,7 @@ static int libxl__domain_save_precopy_policy(precopy_stats_t stats, void *user) stats.iteration, stats.dirty_count, stats.total_written);if (stats.dirty_count >=0 && stats.dirty_count <LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT ) goto stop_copy; -if (stats.iteration >=LIBXL_XGS_POLICY_MAX_ITERATIONS ) +if (stats.iteration >= dss->max_iters) goto stop_copy;return XGS_POLICY_CONTINUE_PRECOPY ;
tools/libs/light/libxl_domain.c +1 -0
@@ -527,6 +527,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, dss->domid = domid; dss->fd = fd; dss->type = type; + dss->max_iters = props->max_iters ?:LIBXL_XGS_POLICY_MAX_ITERATIONS ; dss->live = props->flags &LIBXL_SUSPEND_LIVE ; dss->debug = props->flags &LIBXL_SUSPEND_DEBUG ; dss->checkpointed_stream =LIBXL_CHECKPOINTED_STREAM_NONE ;
tools/libs/light/libxl_internal.h +1 -0
@@ -3641,6 +3641,7 @@ struct libxl__domain_save_state {int live;int debug;int checkpointed_stream; +uint32_t max_iters;const libxl_domain_remus_info *remus;/* private */ int rc;
tools/xl/xl_cmdtable.c +2 -1
@@ -174,7 +174,8 @@ const struct cmd_spec cmd_table[] = {" of the domain.\n" "--debug Verify transferred domU page data.\n" "-p Do not unpause domain after migrating it.\n" -"-D Preserve the domain id" +"-D Preserve the domain id\n" +"--max_iters N Number of copy iterations before final stop+move" }, { "restore", &main_restore, 0, 1,
tools/xl/xl_migrate.c +9 -1
@@ -178,6 +178,7 @@ static void migrate_do_preamble(int send_fd, int recv_fd, pid_t child,static void migrate_domain (uint32_t domid,int preserve_domid,const char *rune,int debug, +uint32_t max_iters,const char *override_config_file) {pid_t child =-1 ;
@@ -189,6 +190,7 @@ static void migrate_domain(uint32_t domid, int preserve_domid,int config_len;libxl_domain_suspend_props props = { .flags =LIBXL_SUSPEND_LIVE , + .max_iters = max_iters, }; save_domain_core_begin(domid, preserve_domid, override_config_file,
@@ -542,8 +544,10 @@ int main_migrate(int argc, char **argv)char *host;int opt, daemonize =1 , monitor =1 , debug =0 , pause_after_migration =0 ;int preserve_domid =0 ; +uint32_t max_iters =0 ;static struct option opts[] = { {"debug" ,0 ,0 ,0x100 }, + {"max_iters" ,1 ,0 ,0x101 }, {"live" ,0 ,0 ,0x200 },COMMON_LONG_OPTS };
@@ -571,6 +575,9 @@ int main_migrate(int argc, char **argv)case 0x100 :/* --debug */ debug =1 ;break ; +case 0x101 :/* --max_iters */ + max_iters =atoi (optarg); +break ;case 0x200 :/* --live */ /* ignored for compatibility with xm */ break ;
@@ -605,7 +612,8 @@ int main_migrate(int argc, char **argv) pause_after_migration ? " -p" : ""); } -migrate_domain (domid, preserve_domid, rune, debug, config_filename); +migrate_domain (domid, preserve_domid, rune, debug, + max_iters, config_filename);return EXIT_SUCCESS ; }
PATCH v20210601 32/38 tools: add --min_remaining to libxl_domain_suspend
In reply to Olaf HeringThe decision to stop+move a domU to the new host must be based on two factors:
- the available network bandwidth for the migration stream
- the maximum time a workload within a domU can be savely suspended
Both values define how many dirty pages a workload may produce prior the final stop+move.
The default value of 50 pages is much too low with todays network bandwidths. On an idle 1GiB link these 200K will be transferred within ~2ms.
Give the admin a knob to adjust the point when the final stop+move will be done, so he can base this decision on his own needs.
This patch adjusts xl(1) and the libxl API.
External users check LIBXL_HAVE_DOMAIN_SUSPEND_PROPS for the availibility of the new .min_remaining property.
--- docs/man/xl.1.pod.in | 8 ++++++++ tools/include/libxl.h | 1 + tools/libs/light/libxl_dom_save.c | 2 +- tools/libs/light/libxl_domain.c | 1 + tools/libs/light/libxl_internal.h | 1 + tools/xl/xl_cmdtable.c | 23 ++++++++++++----------- tools/xl/xl_migrate.c | 9 ++++++++- 7 files changed, 32 insertions(+), 13 deletions(-)
docs/man/xl.1.pod.in +8 -0
@@ -500,6 +500,14 @@ possible to use this option for a 'localhost' migration. Number of copy iterations before final suspend+move (default: 5) +=item B<--min_remaing> I<pages> + +Number of remaining dirty pages. If the number of dirty pages drops that +low, the guest is suspended and the domU will finally be moved to I<host>. + +This allows the host admin to control for how long the domU will likely +be suspended during transit. + =back =item B<remus> [I<OPTIONS>] I<domain-id> I<host>
tools/include/libxl.h +1 -0
@@ -1715,6 +1715,7 @@ static inline int libxl_retrieve_domain_configuration_0x041200(typedef struct {uint32_t flags;/* LIBXL_SUSPEND_* */ uint32_t max_iters; +uint32_t min_remaining; }libxl_domain_suspend_props ;#define LIBXL_SUSPEND_DEBUG 1#define LIBXL_SUSPEND_LIVE 2
tools/libs/light/libxl_dom_save.c +1 -1
@@ -381,7 +381,7 @@ static int libxl__domain_save_precopy_policy(precopy_stats_t stats, void *user)LOGD (DEBUG , shs->domid,"iteration %u dirty_count %ld total_written %lu" , stats.iteration, stats.dirty_count, stats.total_written); -if (stats.dirty_count >=0 && stats.dirty_count <LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT ) +if (stats.dirty_count >=0 && stats.dirty_count < dss->min_remaining) goto stop_copy;if (stats.iteration >= dss->max_iters) goto stop_copy;
tools/libs/light/libxl_domain.c +1 -0
@@ -528,6 +528,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, dss->fd = fd; dss->type = type; dss->max_iters = props->max_iters ?:LIBXL_XGS_POLICY_MAX_ITERATIONS ; + dss->min_remaining = props->min_remaining ?:LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT ; dss->live = props->flags &LIBXL_SUSPEND_LIVE ; dss->debug = props->flags &LIBXL_SUSPEND_DEBUG ; dss->checkpointed_stream =LIBXL_CHECKPOINTED_STREAM_NONE ;
tools/libs/light/libxl_internal.h +1 -0
@@ -3642,6 +3642,7 @@ struct libxl__domain_save_state {int debug;int checkpointed_stream;uint32_t max_iters; +uint32_t min_remaining;const libxl_domain_remus_info *remus;/* private */ int rc;
tools/xl/xl_cmdtable.c +12 -11
@@ -165,17 +165,18 @@ const struct cmd_spec cmd_table[] = { &main_migrate,0 ,1 ,"Migrate a domain to another host" ,"[options] <Domain> <host>" , -"-h Print this help.\n" -"-C <config> Send <config> instead of config file from creation.\n" -"-s <sshcommand> Use <sshcommand> instead of ssh. String will be passed\n" -" to sh. If empty, run <host> instead of ssh <host> xl\n" -" migrate-receive [-d -e]\n" -"-e Do not wait in the background (on <host>) for the death\n" -" of the domain.\n" -"--debug Verify transferred domU page data.\n" -"-p Do not unpause domain after migrating it.\n" -"-D Preserve the domain id\n" -"--max_iters N Number of copy iterations before final stop+move" +"-h Print this help.\n" +"-C <config> Send <config> instead of config file from creation.\n" +"-s <sshcommand> Use <sshcommand> instead of ssh. String will be passed\n" +" to sh. If empty, run <host> instead of ssh <host> xl\n" +" migrate-receive [-d -e]\n" +"-e Do not wait in the background (on <host>) for the death\n" +" of the domain.\n" +"--debug Verify transferred domU page data.\n" +"-p Do not unpause domain after migrating it.\n" +"-D Preserve the domain id\n" +"--max_iters N Number of copy iterations before final stop+move\n" +"--min_remaining N Number of remaining dirty pages before final stop+move" }, { "restore", &main_restore, 0, 1,
tools/xl/xl_migrate.c +8 -1
@@ -179,6 +179,7 @@ static void migrate_do_preamble(int send_fd, int recv_fd, pid_t child,static void migrate_domain (uint32_t domid,int preserve_domid,const char *rune,int debug,uint32_t max_iters, +uint32_t min_remaining,const char *override_config_file) {pid_t child =-1 ;
@@ -191,6 +192,7 @@ static void migrate_domain(uint32_t domid, int preserve_domid,libxl_domain_suspend_props props = { .flags =LIBXL_SUSPEND_LIVE , .max_iters = max_iters, + .min_remaining = min_remaining, }; save_domain_core_begin(domid, preserve_domid, override_config_file,
@@ -545,9 +547,11 @@ int main_migrate(int argc, char **argv)int opt, daemonize =1 , monitor =1 , debug =0 , pause_after_migration =0 ;int preserve_domid =0 ;uint32_t max_iters =0 ; +uint32_t min_remaining =0 ;static struct option opts[] = { {"debug" ,0 ,0 ,0x100 }, {"max_iters" ,1 ,0 ,0x101 }, + {"min_remaining" ,1 ,0 ,0x102 }, {"live" ,0 ,0 ,0x200 },COMMON_LONG_OPTS };
@@ -578,6 +582,9 @@ int main_migrate(int argc, char **argv)case 0x101 :/* --max_iters */ max_iters =atoi (optarg);break ; +case 0x102 :/* --min_remaining */ + min_remaining =atoi (optarg); +break ;case 0x200 :/* --live */ /* ignored for compatibility with xm */ break ;
@@ -613,7 +620,7 @@ int main_migrate(int argc, char **argv) }migrate_domain (domid, preserve_domid, rune, debug, - max_iters, config_filename); + max_iters, min_remaining, config_filename);return EXIT_SUCCESS ; }
PATCH v20210601 33/38 tools: add --abort_if_busy to libxl_domain_suspend
In reply to Olaf HeringProvide a knob to the host admin to abort the live migration of a running domU if the downtime during final transit will be too long for the workload within domU.
Adjust error reporting. Add ERROR_MIGRATION_ABORTED to allow callers of libxl_domain_suspend to distinguish between errors and the requested constraint.
Adjust precopy_policy to simplify reporting of remaining dirty pages. The loop in send_memory_live populates ->dirty_count in a different place than ->iteration. Let it proceeed one more time to provide the desired information before leaving the loop.
This patch adjusts xl(1) and the libxl API.
External users check LIBXL_HAVE_DOMAIN_SUSPEND_PROPS for the availibility of the new .abort_if_busy property.
--- docs/man/xl.1.pod.in | 8 +++++++ tools/include/libxl.h | 1 + tools/libs/light/libxl_dom_save.c | 7 ++++++- tools/libs/light/libxl_domain.c | 1 + tools/libs/light/libxl_internal.h | 2 ++ tools/libs/light/libxl_stream_write.c | 9 +++++++- tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_cmdtable.c | 6 +++++- tools/xl/xl_migrate.c | 30 ++++++++++++++++++++------- 9 files changed, 55 insertions(+), 10 deletions(-)
docs/man/xl.1.pod.in +8 -0
@@ -508,6 +508,14 @@ low, the guest is suspended and the domU will finally be moved to I<host>. This allows the host admin to control for how long the domU will likely be suspended during transit. +=item B<--abort_if_busy> + +Abort migration instead of doing final suspend/move/resume if the +guest produced more than I<min_remaining> dirty pages during th number +of I<max_iters> iterations. +This avoids long periods of time where the guest is suspended, which +may confuse the workload within domU. + =back =item B<remus> [I<OPTIONS>] I<domain-id> I<host>
tools/include/libxl.h +1 -0
@@ -1719,6 +1719,7 @@ typedef struct { } libxl_domain_suspend_props;#define LIBXL_SUSPEND_DEBUG 1#define LIBXL_SUSPEND_LIVE 2 +#define LIBXL_SUSPEND_ABORT_IF_BUSY 4 int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, libxl_domain_suspend_props *props,
tools/libs/light/libxl_dom_save.c +6 -1
@@ -383,11 +383,16 @@ static int libxl__domain_save_precopy_policy(precopy_stats_t stats, void *user) stats.iteration, stats.dirty_count, stats.total_written);if (stats.dirty_count >=0 && stats.dirty_count < dss->min_remaining) goto stop_copy; -if (stats.iteration >= dss->max_iters) +if (stats.dirty_count >=0 && stats.iteration >= dss->max_iters) goto stop_copy;return XGS_POLICY_CONTINUE_PRECOPY ; stop_copy: +if (dss->abort_if_busy) + { + dss->remaining_dirty_pages = stats.dirty_count; +return XGS_POLICY_ABORT ; + }return XGS_POLICY_STOP_AND_COPY ; }
tools/libs/light/libxl_domain.c +1 -0
@@ -529,6 +529,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, dss->type = type; dss->max_iters = props->max_iters ?:LIBXL_XGS_POLICY_MAX_ITERATIONS ; dss->min_remaining = props->min_remaining ?:LIBXL_XGS_POLICY_TARGET_DIRTY_COUNT ; + dss->abort_if_busy = props->flags &LIBXL_SUSPEND_ABORT_IF_BUSY ; dss->live = props->flags &LIBXL_SUSPEND_LIVE ; dss->debug = props->flags &LIBXL_SUSPEND_DEBUG ; dss->checkpointed_stream =LIBXL_CHECKPOINTED_STREAM_NONE ;
tools/libs/light/libxl_internal.h +2 -0
@@ -3640,9 +3640,11 @@ struct libxl__domain_save_state {libxl_domain_type type;int live;int debug; +int abort_if_busy;int checkpointed_stream;uint32_t max_iters;uint32_t min_remaining; +long remaining_dirty_pages;const libxl_domain_remus_info *remus;/* private */ int rc;
tools/libs/light/libxl_stream_write.c +8 -1
@@ -344,11 +344,18 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void, goto err;if (retval) { +if (dss->remaining_dirty_pages) { +LOGD (NOTICE , dss->domid,"saving domain: aborted," +" %ld remaining dirty pages." , dss->remaining_dirty_pages); + }else {LOGEVD (ERROR , errnoval, dss->domid,"saving domain: %s" , dss->dsps.guest_responded ?"domain responded to suspend request" :"domain did not respond to suspend request" ); -if (!dss->dsps.guest_responded) + } +if (dss->remaining_dirty_pages) + rc =ERROR_MIGRATION_ABORTED ; +else if (!dss->dsps.guest_responded) rc =ERROR_GUEST_TIMEDOUT ;else if (dss->rc) rc = dss->rc;
tools/libs/light/libxl_types.idl +1 -0
@@ -76,6 +76,7 @@ libxl_error = Enumeration("error", [ (-30, "QMP_DEVICE_NOT_ACTIVE"), # a device has failed to be become active (-31, "QMP_DEVICE_NOT_FOUND"), # the requested device has not been found (-32, "QEMU_API"), # QEMU's replies don't contains expected members + (-33, "MIGRATION_ABORTED"), ], value_namespace = "") libxl_domain_type = Enumeration("domain_type", [
tools/xl/xl_cmdtable.c +5 -1
@@ -176,7 +176,11 @@ const struct cmd_spec cmd_table[] = {"-p Do not unpause domain after migrating it.\n" "-D Preserve the domain id\n" "--max_iters N Number of copy iterations before final stop+move\n" -"--min_remaining N Number of remaining dirty pages before final stop+move" +"--min_remaining N Number of remaining dirty pages before final stop+move\n" +"--abort_if_busy Abort migration instead of doing final stop+move,\n" +" if the number of dirty pages is higher than <min_remaining>\n" +" after <max_iters> iterations. Otherwise the amount of memory\n" +" to be transfered would exceed maximum allowed domU downtime." }, { "restore", &main_restore, 0, 1,
tools/xl/xl_migrate.c +23 -7
@@ -177,7 +177,7 @@ static void migrate_do_preamble(int send_fd, int recv_fd, pid_t child, }static void migrate_domain (uint32_t domid,int preserve_domid, -const char *rune,int debug, +const char *rune,int debug,int abort_if_busy,uint32_t max_iters,uint32_t min_remaining,const char *override_config_file)
@@ -213,14 +213,20 @@ static void migrate_domain(uint32_t domid, int preserve_domid,if (debug) props.flags |=LIBXL_SUSPEND_DEBUG ; +if (abort_if_busy) + props.flags |=LIBXL_SUSPEND_ABORT_IF_BUSY ; rc =libxl_domain_suspend (ctx, domid, send_fd, &props,NULL );if (rc) {fprintf (stderr,"migration sender: libxl_domain_suspend failed" " (rc=%d)\n" , rc); -if (rc ==ERROR_GUEST_TIMEDOUT ) - goto failed_suspend; -else - goto failed_resume; +switch (rc) { +case ERROR_GUEST_TIMEDOUT : + goto failed_suspend; +case ERROR_MIGRATION_ABORTED : + goto failed_busy; +default : + goto failed_resume; + } }//fprintf(stderr, "migration sender: Transfer complete.\n");
@@ -302,6 +308,12 @@ static void migrate_domain(uint32_t domid, int preserve_domid,fprintf (stderr,"Migration failed, failed to suspend at sender.\n" );exit (EXIT_FAILURE ); + failed_busy: +close (send_fd); +migration_child_report (recv_fd); +fprintf (stderr,"Migration aborted as requested, domain is too busy.\n" ); +exit (EXIT_FAILURE ); + failed_resume:close (send_fd);migration_child_report (recv_fd);
@@ -545,13 +557,14 @@ int main_migrate(int argc, char **argv)char *rune =NULL ;char *host;int opt, daemonize =1 , monitor =1 , debug =0 , pause_after_migration =0 ; -int preserve_domid =0 ; +int preserve_domid =0 , abort_if_busy =0 ;uint32_t max_iters =0 ;uint32_t min_remaining =0 ;static struct option opts[] = { {"debug" ,0 ,0 ,0x100 }, {"max_iters" ,1 ,0 ,0x101 }, {"min_remaining" ,1 ,0 ,0x102 }, + {"abort_if_busy" ,0 ,0 ,0x103 }, {"live" ,0 ,0 ,0x200 },COMMON_LONG_OPTS };
@@ -585,6 +598,9 @@ int main_migrate(int argc, char **argv)case 0x102 :/* --min_remaining */ min_remaining =atoi (optarg);break ; +case 0x103 :/* --abort_if_busy */ + abort_if_busy =1 ; +break ;case 0x200 :/* --live */ /* ignored for compatibility with xm */ break ;
@@ -619,7 +635,7 @@ int main_migrate(int argc, char **argv) pause_after_migration ? " -p" : ""); } -migrate_domain (domid, preserve_domid, rune, debug, +migrate_domain (domid, preserve_domid, rune, debug, abort_if_busy, max_iters, min_remaining, config_filename);return EXIT_SUCCESS ; }
PATCH v20210601 34/38 tools: add API for expandable bitmaps
In reply to Olaf HeringSince the incoming migration stream lacks info about what the highest pfn will be, data structures can not be allocated upfront.
Add an API for expandable bitmaps, loosely based on pfn_set_populated.
--- tools/libs/saverestore/common.c | 40 ++++++++++++++++++++ tools/libs/saverestore/common.h | 67 +++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+)
tools/libs/saverestore/common.c +40 -0
@@ -163,6 +163,46 @@ static void __attribute__((unused)) build_assertions(void)BUILD_BUG_ON (sizeof (struct xc_sr_rec_hvm_params ) !=8 ); } +/* +* Expand the tracking structures as needed. +* To avoid realloc()ing too excessively, the size increased to the nearest +* power of two large enough to contain the required number of bits. +*/ +bool _xg_sr_bitmap_expand (struct xg_sr_bitmap *bm,unsigned long bits) +{ +size_t new_max; +size_t old_sz, new_sz; +void *p; + +if (bits <= bm->bits) +return true; + +/* Round up to the nearest power of two larger than bit, less 1. */ + new_max = bits; + new_max |= new_max >>1 ; + new_max |= new_max >>2 ; + new_max |= new_max >>4 ; + new_max |= new_max >>8 ; + new_max |= new_max >>16 ; +if (sizeof (unsigned long ) >4 ) + new_max |= new_max >>32 ; + +/* Allocate units of unsigned long */ + new_max = (new_max +BITS_PER_LONG -1 ) & ~(BITS_PER_LONG -1 ); + + old_sz =bitmap_size (bm->bits); + new_sz =bitmap_size (new_max); + p =realloc (bm->p, new_sz); +if (!p) +return false; + +memset (p + old_sz,0 , new_sz - old_sz); + bm->p = p; + bm->bits = new_max; + +return true; +} +/* * Local variables: * mode: C
tools/libs/saverestore/common.h +67 -0
73 lines not on this page · raw message
PATCH v20210601 35/38 tools: use xg_sr_bitmap for populated_pfns
In reply to Olaf Hering--- tools/libs/saverestore/common.h | 21 +++++++- tools/libs/saverestore/restore.c | 69 ------------------------ tools/libs/saverestore/restore_x86_hvm.c | 9 ++++ tools/libs/saverestore/restore_x86_pv.c | 7 +++ 4 files changed, 35 insertions(+), 71 deletions(-)
tools/libs/saverestore/common.h +19 -2
@@ -403,8 +403,7 @@ struct xc_sr_contextuint32_t xenstore_domid, console_domid;/* Bitmap of currently populated PFNs during restore. */ -unsigned long *populated_pfns; -xen_pfn_t max_populated_pfn; +struct xg_sr_bitmap populated_pfns;/* Sender has invoked verify mode on the stream. */ bool verify;
@@ -596,6 +595,24 @@ static inline bool page_type_has_stream_data(uint32_t type) }return ret; } + +static inline bool pfn_is_populated (struct xc_sr_context *ctx,xen_pfn_t pfn) +{ +return xg_sr_test_bit (pfn, &ctx->restore.populated_pfns); +} + +static inline int pfn_set_populated (struct xc_sr_context *ctx,xen_pfn_t pfn) +{ +xc_interface *xch = ctx->xch; + +if (xg_sr_set_bit (pfn, &ctx->restore.populated_pfns) == false ) + { +PERROR ("Failed to realloc populated_pfns bitmap" ); + errno =ENOMEM ; +return -1 ; + } +return 0 ; +}#endif /* * Local variables:
tools/libs/saverestore/restore.c +0 -69
87 lines not on this page · raw message
tools/libs/saverestore/restore_x86_hvm.c +9 -0
@@ -136,6 +136,7 @@ static int x86_hvm_localise_page(struct xc_sr_context *ctx,static int x86_hvm_setup (struct xc_sr_context *ctx) {xc_interface *xch = ctx->xch; +unsigned long max_pfn;if ( ctx->restore.guest_type !=DHDR_TYPE_X86_HVM ) {
@@ -161,6 +162,13 @@ static int x86_hvm_setup(struct xc_sr_context *ctx) }#endif + max_pfn =max (ctx->restore.p2m_size, ctx->dominfo.max_memkb >> (PAGE_SHIFT -10 )); +if ( !xg_sr_bitmap_expand (&ctx->restore.populated_pfns, max_pfn) ) + { +PERROR ("Unable to allocate memory for populated_pfns bitmap" ); +return -1 ; + } +return 0 ; }
@@ -241,6 +249,7 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)static int x86_hvm_cleanup (struct xc_sr_context *ctx) { +xg_sr_bitmap_free (&ctx->restore.populated_pfns);free (ctx->x86.hvm.restore.context.ptr);free (ctx->x86.restore.cpuid.ptr);
tools/libs/saverestore/restore_x86_pv.c +7 -0
@@ -1060,6 +1060,12 @@ static int x86_pv_setup(struct xc_sr_context *ctx)if ( rc )return rc; +if ( !xg_sr_bitmap_expand (&ctx->restore.populated_pfns,32 *1024 /4 ) ) + { +PERROR ("Unable to allocate memory for populated_pfns bitmap" ); +return -1 ; + } + ctx->x86.pv.restore.nr_vcpus = ctx->dominfo.max_vcpu_id +1 ; ctx->x86.pv.restore.vcpus =calloc (sizeof (struct xc_sr_x86_pv_restore_vcpu ), ctx->x86.pv.restore.nr_vcpus);
@@ -1153,6 +1159,7 @@ static int x86_pv_stream_complete(struct xc_sr_context *ctx)*/ static int x86_pv_cleanup (struct xc_sr_context *ctx) { +xg_sr_bitmap_free (&ctx->restore.populated_pfns);free (ctx->x86.pv.p2m);free (ctx->x86.pv.p2m_pfns);
PATCH v20210601 36/38 tools: use superpages during restore of HVM guest
In reply to Olaf HeringDuring creating of a HVM domU meminit_hvm() tries to map superpages. After save/restore or migration this mapping is lost, everything is allocated in single pages. This causes a performance degradation after migration.
Add neccessary code to preallocate a superpage for an incoming chunk of pfns. In case a pfn was not populated on the sending side, it must be freed on the receiving side to avoid over-allocation.
The existing code for x86_pv is moved unmodified into its own file.
--- tools/libs/guest/xg_dom_x86.c | 5 - tools/libs/guest/xg_private.h | 5 + tools/libs/saverestore/common.c | 2 - tools/libs/saverestore/common.h | 29 +- tools/libs/saverestore/restore.c | 62 +--- tools/libs/saverestore/restore_x86_hvm.c | 369 ++++++++++++++++++++++- tools/libs/saverestore/restore_x86_pv.c | 61 +++- 7 files changed, 455 insertions(+), 78 deletions(-)
tools/libs/guest/xg_dom_x86.c +0 -5
@@ -44,11 +44,6 @@#define SUPERPAGE_BATCH_SIZE 512 -#define SUPERPAGE_2MB_SHIFT 9 -#define SUPERPAGE_2MB_NR_PFNS (1UL << SUPERPAGE_2MB_SHIFT) -#define SUPERPAGE_1GB_SHIFT 18 -#define SUPERPAGE_1GB_NR_PFNS (1UL << SUPERPAGE_1GB_SHIFT) -#define X86_CR0_PE 0x01#define X86_CR0_ET 0x10
tools/libs/guest/xg_private.h +5 -0
@@ -154,4 +154,9 @@ int pin_table(xc_interface *xch, unsigned int type, unsigned long mfn,#define M2P_SIZE (_m) ROUNDUP(((_m) * sizeof(xen_pfn_t)), M2P_SHIFT)#define M2P_CHUNKS (_m) (M2P_SIZE((_m)) >> M2P_SHIFT) +#define SUPERPAGE_2MB_SHIFT 9 +#define SUPERPAGE_2MB_NR_PFNS (1UL << SUPERPAGE_2MB_SHIFT) +#define SUPERPAGE_1GB_SHIFT 18 +#define SUPERPAGE_1GB_NR_PFNS (1UL << SUPERPAGE_1GB_SHIFT) +#endif /* XG_PRIVATE_H */
tools/libs/saverestore/common.c +0 -2
@@ -1,5 +1,3 @@ -#include <assert.h> -#include "common.h" #include <xen-tools/libs.h>
tools/libs/saverestore/common.h +21 -8
@@ -3,6 +3,7 @@#include <unistd.h> #include <errno.h> +#include <assert.h> #include <stdbool.h> #include <stdio.h> #include <stdlib.h>
@@ -219,6 +220,16 @@ struct xc_sr_restore_ops*/ int (*setup)(struct xc_sr_context *ctx); +/** +* Populate PFNs +* +* Given a set of pfns, obtain memory from Xen to fill the physmap for the +* unpopulated subset. +*/ +int (*populate_pfns)(struct xc_sr_context *ctx,unsigned count, +const xen_pfn_t *original_pfns,const uint32_t *types); + +/** * Process an individual record from the stream. The caller shall take * care of processing common records (e.g. END, PAGE_DATA).
@@ -366,6 +377,8 @@ struct xc_sr_contextint send_back_fd;unsigned long p2m_size; +unsigned long max_pages; +unsigned long tot_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf;/* From Image Header. */
@@ -503,6 +516,14 @@ struct xc_sr_context {/* HVM context blob. */ struct xc_sr_blob context; + +/* Bitmap of currently allocated PFNs during restore. */ +struct xg_sr_bitmap attempted_1g; +struct xg_sr_bitmap attempted_2m; +struct xg_sr_bitmap allocated_pfns; +xen_pfn_t prev_populated_pfn; +xen_pfn_t iteration_tracker_pfn; +unsigned long iteration; } restore; }; } hvm;
@@ -567,14 +588,6 @@ int read_record_header(struct xc_sr_context *ctx, int fd, struct xc_sr_rhdr *rhdint read_record_data (struct xc_sr_context *ctx,int fd,struct xc_sr_rhdr *rhdr,struct xc_sr_record *rec); -/* -* This would ideally be private in restore.c, but is needed by -* x86_pv_localise_page() if we receive pagetables frames ahead of the -* contents of the frames they point at. -*/ -int populate_pfns (struct xc_sr_context *ctx,unsigned int count, -const xen_pfn_t *original_pfns,const uint32_t *types); -/* Handle a STATIC_DATA_END record. */ int handle_static_data_end (struct xc_sr_context *ctx);
tools/libs/saverestore/restore.c +4 -58
80 lines not on this page · raw message
tools/libs/saverestore/restore_x86_hvm.c +365 -4
401 lines not on this page · raw message
tools/libs/saverestore/restore_x86_pv.c +60 -1
79 lines not on this page · raw message
PATCH v20210601 37/38 tools: remove migration stream verify code
In reply to Olaf HeringThe verify code tries to make sure the transferred pages from the paused domU were received correctly in the previous, final iteration. Every page is transferred once again by the sender, the receiver compares the content with what is already mapped into the not-yet started domU.
This does unfortunately not work because a domU has also a number of pages in its grant table, so that frondend drivers can communicate with the backend drivers. Since the backend drivers are unaware about the ongoing saving of the domU, they will continue to write into the shared pages. As a result the verification of the domU pages will fail.
This is not fixable, so remove the verification code.
The sending side will never send a REC_TYPE_VERIFY record again. It will silently accept, and ignore the debug flag.
The receiving side will abort an almost complete migration in case it sees a REC_TYPE_VERIFY record from an old sender. That record is sent at the very end, it has no way to know in advance that verification was requested.
--- tools/libs/saverestore/common.h | 7 ------ tools/libs/saverestore/restore.c | 43 ++++---------------------------- tools/libs/saverestore/save.c | 43 -------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-)
tools/libs/saverestore/common.h +0 -7
@@ -354,9 +354,6 @@ struct xc_sr_context/* Live migrate vs non live suspend. */ bool live; -/* Further debugging information in the stream. */ -bool debug; -unsigned long p2m_size;size_t pages_sent;size_t overhead_sent;
@@ -418,10 +415,6 @@ struct xc_sr_context/* Bitmap of currently populated PFNs during restore. */ struct xg_sr_bitmap populated_pfns; -/* Sender has invoked verify mode on the stream. */ -bool verify; -void *verify_buf; -struct xc_sr_restore_arrays *m;void *guest_mapping;uint32_t nr_mapped_pages;
tools/libs/saverestore/restore.c +5 -38
67 lines not on this page · raw message
tools/libs/saverestore/save.c +0 -43
61 lines not on this page · raw message
PATCH v20210601 38/38 hotplug/Linux: fix starting of xenstored with restarting systemd
In reply to Olaf HeringA hard to trigger race with another unrelated systemd service and xenstored.service unveiled a bug in the way how xenstored is launched with systemd.
launch-xenstore may start either a daemon or a domain. In case a domain is used, systemd-notify was called. If another service triggered a restart of systemd while xenstored.service was executed, systemd may temporary lose track of services with Type=notify. As a result, xenstored.service would be marked as failed and units that depend on it will not be started. This breaks the enire Xen toolstack.
The chain of events is basically: xenstored.service sends the notification to systemd, this is a one-way event. Then systemd may be restarted by the other unit. During this time, xenstored.service is done and exits. Once systemd is done with its restart, it collects the pending notifications and childs. If it does not find the unit which sent the notification it will declare it as failed.
A workaround for this scenario is to leave the child processes running for a short time after sending the "READY=1" notification. If systemd happens to restart it will still find the unit it launched.
Adjust the callers of launch-xenstore to specifiy the init system:
Do not fork xenstored with systemd, preserve pid. This wil also avoid the need for a sleep because the process which sent the "READY=1" (the previously forked child) is still alive.
Remove the --pid-file in the systemd case because the pid of the child is known, and the file had probably little effect anyway due to lack of PidFile= and Type=forking in the unit file.
Be verbose about xenstored startup only with sysv to avoid interleaved output in systemd journal. Do the same also for domain case, even if is not strictly needed because init-xenstore-domain has no output.
The fix for upstream systemd which is supposed to fix it:575b300b795b6 ("pid1: rework how we dispatch SIGCHLD and other signals")
--
v04:
- do mkdir unconditionally because init-xenstore-domain writes the domid to xenstored.pid
v03:
- remove run_xenstored function, follow style of shell built-in test function v02:
- preserve Type=notify
--- tools/hotplug/Linux/init.d/xencommons.in | 2 +- tools/hotplug/Linux/launch-xenstore.in | 40 ++++++++++++++----- .../Linux/systemd/xenstored.service.in | 2 +- 3 files changed, 31 insertions(+), 13 deletions(-)
tools/hotplug/Linux/init.d/xencommons.in +1 -1
@@ -60,7 +60,7 @@ do_start () { mkdir -m700 -p ${XEN_LOCK_DIR} mkdir -p ${XEN_LOG_DIR} - @XEN_SCRIPT_DIR@/launch-xenstore || exit 1 + @XEN_SCRIPT_DIR@/launch-xenstore 'sysv' || exit 1 echo Setting domain 0 name, domid and JSON config... ${LIBEXEC_BIN}/xen-init-dom0 ${XEN_DOM0_UUID}
tools/hotplug/Linux/launch-xenstore.in +29 -11
@@ -15,6 +15,17 @@ # License along with this library; If not, see <http://www.gnu.org/licenses/>. # +initd=$1 + +case "$initd" in + sysv) nonl='-n' ;; + systemd) nonl= ;; + *) + echo "first argument must be 'sysv' or 'systemd'" + exit 1 + ;; +esac + XENSTORED=@XENSTORED@ . @XEN_SCRIPT_DIR@/hotplugpath.sh
@@ -44,14 +55,16 @@ timeout_xenstore () { return 0 } -test_xenstore && exit 0 +mkdir -p @XEN_RUN_DIR@ + +if test "$initd" = 'sysv' ; then + test_xenstore && exit 0 +fi test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons [ "$XENSTORETYPE" = "" ] && XENSTORETYPE=daemon -/bin/mkdir -p @XEN_RUN_DIR@ - [ "$XENSTORETYPE" = "daemon" ] && { [ -z "$XENSTORED_TRACE" ] || XENSTORED_ARGS="$XENSTORED_ARGS -T @XEN_LOG_DIR@/xenstored-trace.log" [ -z "$XENSTORED" ] && XENSTORED=@XENSTORED@
@@ -59,13 +72,15 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . @CONFIG_DIR@/@CONFIG_LEAF echo "No xenstored found" exit 1 } + [ "$initd" = 'sysv' ] && { + echo $nonl Starting $XENSTORED... + $XENSTORED --pid-file @XEN_RUN_DIR@/xenstored.pid $XENSTORED_ARGS + timeout_xenstore $XENSTORED || exit 1 + exit 0 + } - echo -n Starting $XENSTORED... - $XENSTORED --pid-file @XEN_RUN_DIR@/xenstored.pid $XENSTORED_ARGS - - systemd-notify --booted 2>/dev/null || timeout_xenstore $XENSTORED || exit 1 - - exit 0 + exec $XENSTORED -N $XENSTORED_ARGS + exit 1 } [ "$XENSTORETYPE" = "domain" ] && {
@@ -75,9 +90,12 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . @CONFIG_DIR@/@CONFIG_LEAF XENSTORE_DOMAIN_ARGS="$XENSTORE_DOMAIN_ARGS --memory $XENSTORE_DOMAIN_SIZE" [ -z "$XENSTORE_MAX_DOMAIN_SIZE" ] || XENSTORE_DOMAIN_ARGS="$XENSTORE_DOMAIN_ARGS --maxmem $XENSTORE_MAX_DOMAIN_SIZE" - echo -n Starting $XENSTORE_DOMAIN_KERNEL... + echo $nonl Starting $XENSTORE_DOMAIN_KERNEL... ${LIBEXEC_BIN}/init-xenstore-domain $XENSTORE_DOMAIN_ARGS || exit 1 - systemd-notify --ready 2>/dev/null + [ "$initd" = 'systemd' ] && { + systemd-notify --ready + sleep 9 + } exit 0 }
tools/hotplug/Linux/systemd/xenstored.service.in +1 -1
@@ -11,7 +11,7 @@ Type=notify NotifyAccess=all RemainAfterExit=true ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities -ExecStart=@XEN_SCRIPT_DIR@/launch-xenstore +ExecStart=@XEN_SCRIPT_DIR@/launch-xenstore 'systemd' [Install] WantedBy=multi-user.target
Quoted 1 line · Olaf Hering
Various unreviewed changes, rebase to
57f68dfd2d.
Would it be possible to split this into multiple independent patches/series?
Juergen
Am Wed, 2 Jun 2021 08:10:21 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 2 lines
Would it be possible to split this into multiple independent patches/series?
Sure, I can send each individual part that was already sent over and over again.
Olaf
Quoted 7 lines · Olaf Hering
Am Wed, 2 Jun 2021 08:10:21 +0200
schrieb Juergen Gross <[email protected]>:Quoted 2 lines
Would it be possible to split this into multiple independent patches/series?
Sure, I can send each individual part that was already sent over and over again.
IMO this will make it more probable to get at least parts committed.
Juergen
Am Wed, 2 Jun 2021 09:00:49 +0200
schrieb Juergen Gross <[email protected]>:
Quoted 1 line
IMO this will make it more probable to get at least parts committed.
It depends all on patch #3, so further split can not be done anymore at this point.
Olaf
Quoted 6 lines · Olaf Hering
Am Wed, 2 Jun 2021 09:00:49 +0200
schrieb Juergen Gross <[email protected]>:Quoted 1 line
IMO this will make it more probable to get at least parts committed.
It depends all on patch #3, so further split can not be done anymore at
this point.
I don't see the dependency e.g. in patch 27, 28 and 38.
Juergen