Initial support for multi-process qemu
Subsystems (4)
Re: [PATCH v5 40/50] multi-process/mig: build migration module in the remote process
In reply to Jag RamanQuoted 79 lines · * Jag Raman ([email protected]) wrote:
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 66 lines · * Jagannathan Raman ([email protected]) wrote:
Add
Makefilesupport to enable migration in remote processSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- Makefile.objs | 4 +++- Makefile.target | 1 + migration/Makefile.objs | 13 ++++++++++++- net/Makefile.objs | 2 ++ softmmu/vl.c | 2 -- stubs/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ stubs/net-stub.c | 21 +++++++++++++++++++++ stubs/qapi-misc.c | 2 ++ stubs/replay.c | 8 ++++++++ stubs/vl-stub.c | 24 ++++++++++++++++++++++++ vl-parse.c | 3 +++ 11 files changed, 125 insertions(+), 4 deletions(-)Makefile.objs +3
-1@@ -74,6 +74,8 @@ common-obj-y += qdev-monitor.o device-hotplug.o common-obj-$(CONFIG_WIN32) += os-win32.o common-obj-$(CONFIG_POSIX) += os-posix.o +remote-pci-obj-$(CONFIG_POSIX) += os-posix.o + common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/@@ -104,11 +106,11 @@ common-obj-y += vl-parse.o ####################################################################### # qapi - common-obj-y += qapi/ endif # CONFIG_SOFTMMU +remote-pci-obj-$(CONFIG_MPQEMU) += net/ remote-pci-obj-$(CONFIG_MPQEMU) += qapi/ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev-nbd.o remote-pci-obj-$(CONFIG_MPQEMU) += job-qmp.oMakefile.target +1
-0@@ -240,6 +240,7 @@ all-remote-pci-obj-y += exec.o all-remote-pci-obj-y += exec-vary.o all-remote-pci-obj-y += ioport.o all-remote-pci-obj-y += cpus.o +all-remote-pci-obj-y += migration/ram.o endif remote-pci-obj-y :=migration/Makefile.objs +8
-1@@ -15,4 +15,15 @@ common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o rdma.o-libs := $(RDMA_LIBS) -remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o vmstate.o qjson.o vmstate-types.o +remote-pci-obj-$(CONFIG_MPQEMU) += migration.o socket.o fd.o exec.o +remote-pci-obj-$(CONFIG_MPQEMU) += tls.o channel.o savevm.o +remote-pci-obj-$(CONFIG_MPQEMU) += colo.o colo-failover.o +remote-pci-obj-$(CONFIG_MPQEMU) += vmstate.o vmstate-types.o page_cache.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o global_state.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file-channel.o +remote-pci-obj-$(CONFIG_MPQEMU) += xbzrle.o postcopy-ram.o +remote-pci-obj-$(CONFIG_MPQEMU) += qjson.oQuoted 8 lines
migration/Makefile.objs common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o · line 29
Quoted 4 lines
migration/Makefile.objs +4
-0+remote-pci-obj-$(CONFIG_MPQEMU) += block-dirty-bitmap.o +remote-pci-obj-$(CONFIG_RDMA) += rdma.o +remote-pci-obj-$(CONFIG_MPQEMU) += block.o +remote-pci-obj-$(CONFIG_MPQEMU) += multifd.oHmm, are you really going to want all this lot in your remote process? Assuming it's just devices, I can understand the first line or two, but it seems odd to have all of this.
Quoted 4 lines
Yeah, we ended up needing to compile these in to enable migration. We are only using "fd" to enable migration. Although we don't use tls, xbzrle, rdma, multifd, etc... for example, the migration code does support these protocols and, therefore, we had to compile them in.
But are you even running a migration stream from the remote process? Aren't you just doing vmstate migration of devices; i.e. do you need anything relating to incremental RAM migration (e.g. xbzrle, rdma, postcopy).
Dave
Quoted 219 lines
Thank you!
--
JagQuoted 214 lines
Dave
Quoted 208 lines
net/Makefile.objs +2
-0@@ -30,3 +30,5 @@ common-obj-$(CONFIG_WIN32) += tap-win32.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ + +remote-pci-obj-$(CONFIG_MPQEMU) += announce.osoftmmu/vl.c +0
-2@@ -128,7 +128,6 @@ const char* keyboard_layout = NULL;ram_addr_t ram_size;const char *mem_path =NULL ;int mem_prealloc =0 ;/* force preallocation of physical target memory */ -bool enable_mlock = false;bool enable_cpu_pm = false;int nb_nics;NICInfo nd_table[MAX_NICS ];@@ -168,7 +167,6 @@ const char *prom_envs[MAX_PROM_ENVS];int boot_menu;bool boot_strict;uint8_t *boot_splash_filedata; -int only_migratable;/* turn it off unless user states otherwise */ bool wakeup_suspend_enabled;int icount_align_option;stubs/migration.c +49
-0@@ -6,6 +6,35 @@#include "qapi/qapi-types-migration.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-types-net.h" +#include "net/filter.h" +#include "net/colo-compare.h" + +#pragma weak qmp_query_migrate_capabilities +#pragma weak qmp_query_migrate_parameters +#pragma weak migrate_announce_params +#pragma weak qmp_query_migrate +#pragma weak qmp_migrate_set_capabilities +#pragma weak qmp_migrate_set_parameters +#pragma weak qmp_migrate_incoming +#pragma weak qmp_migrate_recover +#pragma weak qmp_migrate_pause +#pragma weak qmp_migrate +#pragma weak qmp_migrate_cancel +#pragma weak qmp_migrate_continue +#pragma weak qmp_migrate_set_cache_size +#pragma weak qmp_query_migrate_cache_size +#pragma weak qmp_migrate_set_speed +#pragma weak qmp_migrate_set_downtime +#pragma weak qmp_migrate_start_postcopy +#pragma weak migration_global_dump +#pragma weak save_snapshot +#pragma weak qmp_xen_save_devices_state +#pragma weak load_snapshot +#pragma weak qmp_xen_set_replication +#pragma weak qmp_query_xen_replication_status +#pragma weak qmp_xen_colo_do_checkpoint +#pragma weak qmp_query_colo_status +#pragma weak qmp_x_colo_lost_heartbeatMigrationInfo *qmp_query_migrate (Error **errp) {@@ -160,3 +189,23 @@ AnnounceParameters *migrate_announce_params(void)return NULL ; } + +void colo_notify_filters_event (int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_notify_compares_event (void *opaque,int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}stubs/net-stub.c +21
-0@@ -5,6 +5,8 @@#include "qapi/qapi-commands-net.h" #include "qapi/qapi-commands-rocker.h" +#pragma weak qmp_announce_self +int qemu_find_net_clients_except (const char *id,NetClientState **ncs,NetClientDriver type,int max) {@@ -98,3 +100,22 @@ void netdev_add(QemuOpts *opts, Error **errp) {qemu_debug_assert (0 ); } + +NetClientState *qemu_get_queue (NICState *nic) +{ +qemu_debug_assert (0 ); + +return NULL ; +} + +ssize_t qemu_send_packet_raw (NetClientState *nc,const uint8_t *buf,int size) +{ +qemu_debug_assert (0 ); + +return 0 ; +} + +void qemu_foreach_nic (qemu_nic_foreach func,void *opaque) +{ +qemu_debug_assert (0 ); +}stubs/qapi-misc.c +2
-0@@ -5,6 +5,8 @@#include "./qapi/qapi-types-dump.h" #include "qapi/qapi-commands-dump.h" +#pragma weak qmp_xen_load_devices_state + void qmp_dump_guest_memory(bool paging, const char *file, bool has_detach, bool detach, bool has_begin, int64_t begin, bool has_length,stubs/replay.c +8
-0@@ -1,4 +1,5 @@#include "qemu/osdep.h" +#include "qemu-common.h" #include "sysemu/replay.h" ReplayMode replay_mode;@@ -106,3 +107,10 @@ void replay_account_executed_instructions(void)void replay_add_blocker (Error *reason) { } + +bool replay_can_snapshot (void ) +{ +qemu_debug_assert (0 ); + +return false; +}stubs/vl-stub.c +24
-0@@ -14,6 +14,8 @@#include "disas/disas.h" #include "audio/audio.h" +#pragma weak qemu_add_exit_notifier +bool tcg_allowed;bool xen_allowed;bool boot_strict;@@ -169,3 +171,25 @@ int wav_start_capture(AudioState *state, CaptureState *s, const char *path,return -1 ; } + +void qemu_system_killed (int signal,pid_t pid) +{ +qemu_debug_assert (0 ); +} + +void qemu_system_reset (ShutdownCause reason) +{ +qemu_debug_assert (0 ); +} + +bool runstate_store (char *str,size_t size) +{ +qemu_debug_assert (0 ); + +return false; +} + +void qemu_add_exit_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}vl-parse.c +3
-0@@ -27,6 +27,9 @@#include "vl.h" +int only_migratable;/* turn it off unless user states otherwise */ +bool enable_mlock; +/***********************************************************/ /* QEMU Block devices */ Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 40/50] multi-process/mig: build migration module in the remote process
Quoted 85 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Jag Raman ([email protected]) wrote:
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 66 lines · * Jagannathan Raman ([email protected]) wrote:
Add
Makefilesupport to enable migration in remote processSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- Makefile.objs | 4 +++- Makefile.target | 1 + migration/Makefile.objs | 13 ++++++++++++- net/Makefile.objs | 2 ++ softmmu/vl.c | 2 -- stubs/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ stubs/net-stub.c | 21 +++++++++++++++++++++ stubs/qapi-misc.c | 2 ++ stubs/replay.c | 8 ++++++++ stubs/vl-stub.c | 24 ++++++++++++++++++++++++ vl-parse.c | 3 +++ 11 files changed, 125 insertions(+), 4 deletions(-)Makefile.objs +3
-1@@ -74,6 +74,8 @@ common-obj-y += qdev-monitor.o device-hotplug.o common-obj-$(CONFIG_WIN32) += os-win32.o common-obj-$(CONFIG_POSIX) += os-posix.o +remote-pci-obj-$(CONFIG_POSIX) += os-posix.o + common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/@@ -104,11 +106,11 @@ common-obj-y += vl-parse.o ####################################################################### # qapi - common-obj-y += qapi/ endif # CONFIG_SOFTMMU +remote-pci-obj-$(CONFIG_MPQEMU) += net/ remote-pci-obj-$(CONFIG_MPQEMU) += qapi/ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev-nbd.o remote-pci-obj-$(CONFIG_MPQEMU) += job-qmp.oMakefile.target +1
-0@@ -240,6 +240,7 @@ all-remote-pci-obj-y += exec.o all-remote-pci-obj-y += exec-vary.o all-remote-pci-obj-y += ioport.o all-remote-pci-obj-y += cpus.o +all-remote-pci-obj-y += migration/ram.o endif remote-pci-obj-y :=migration/Makefile.objs +8
-1@@ -15,4 +15,15 @@ common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o rdma.o-libs := $(RDMA_LIBS) -remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o vmstate.o qjson.o vmstate-types.o +remote-pci-obj-$(CONFIG_MPQEMU) += migration.o socket.o fd.o exec.o +remote-pci-obj-$(CONFIG_MPQEMU) += tls.o channel.o savevm.o +remote-pci-obj-$(CONFIG_MPQEMU) += colo.o colo-failover.o +remote-pci-obj-$(CONFIG_MPQEMU) += vmstate.o vmstate-types.o page_cache.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o global_state.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file-channel.o +remote-pci-obj-$(CONFIG_MPQEMU) += xbzrle.o postcopy-ram.o +remote-pci-obj-$(CONFIG_MPQEMU) += qjson.oQuoted 8 lines
migration/Makefile.objs common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o · line 29
Quoted 4 lines
migration/Makefile.objs +4
-0+remote-pci-obj-$(CONFIG_MPQEMU) += block-dirty-bitmap.o +remote-pci-obj-$(CONFIG_RDMA) += rdma.o +remote-pci-obj-$(CONFIG_MPQEMU) += block.o +remote-pci-obj-$(CONFIG_MPQEMU) += multifd.oHmm, are you really going to want all this lot in your remote process? Assuming it's just devices, I can understand the first line or two, but it seems odd to have all of this.
Quoted 4 lines
Yeah, we ended up needing to compile these in to enable migration. We are only using "fd" to enable migration. Although we don't use tls, xbzrle, rdma, multifd, etc... for example, the migration code does support these protocols and, therefore, we had to compile them in.
Quoted 4 lines
But are you even running a migration stream from the remote process? Aren't you just doing vmstate migration of devices; i.e. do you need anything relating to incremental RAM migration (e.g. xbzrle, rdma, postcopy).
We are running a migration stream from the remote process. We are only doing the vmstate migration of devices, and not anything incremental related to RAM.
We are using QEMU's existing migration infrastructure (vmstate_save / qemu_loadvm_section_start_full) to move the vmstate. Based on my limited experience with the migration code, I get that it comes as a suite. Without some refactoring, we would need to build all of the files within the migration folder.
The vmstate functions communicate over QEMUFile, which would be using fd, tcp, rdma, etc... at the backend. These functions also need other functions defined in migration.c, which require the building of xbzrle, postcopy, etc...
Thank you!
--
Jag
Quoted 225 lines
Dave
Quoted 219 lines
Thank you!
--
JagQuoted 214 lines
Dave
Quoted 208 lines
net/Makefile.objs +2
-0@@ -30,3 +30,5 @@ common-obj-$(CONFIG_WIN32) += tap-win32.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ + +remote-pci-obj-$(CONFIG_MPQEMU) += announce.osoftmmu/vl.c +0
-2@@ -128,7 +128,6 @@ const char* keyboard_layout = NULL;ram_addr_t ram_size;const char *mem_path =NULL ;int mem_prealloc =0 ;/* force preallocation of physical target memory */ -bool enable_mlock = false;bool enable_cpu_pm = false;int nb_nics;NICInfo nd_table[MAX_NICS ];@@ -168,7 +167,6 @@ const char *prom_envs[MAX_PROM_ENVS];int boot_menu;bool boot_strict;uint8_t *boot_splash_filedata; -int only_migratable;/* turn it off unless user states otherwise */ bool wakeup_suspend_enabled;int icount_align_option;stubs/migration.c +49
-0@@ -6,6 +6,35 @@#include "qapi/qapi-types-migration.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-types-net.h" +#include "net/filter.h" +#include "net/colo-compare.h" + +#pragma weak qmp_query_migrate_capabilities +#pragma weak qmp_query_migrate_parameters +#pragma weak migrate_announce_params +#pragma weak qmp_query_migrate +#pragma weak qmp_migrate_set_capabilities +#pragma weak qmp_migrate_set_parameters +#pragma weak qmp_migrate_incoming +#pragma weak qmp_migrate_recover +#pragma weak qmp_migrate_pause +#pragma weak qmp_migrate +#pragma weak qmp_migrate_cancel +#pragma weak qmp_migrate_continue +#pragma weak qmp_migrate_set_cache_size +#pragma weak qmp_query_migrate_cache_size +#pragma weak qmp_migrate_set_speed +#pragma weak qmp_migrate_set_downtime +#pragma weak qmp_migrate_start_postcopy +#pragma weak migration_global_dump +#pragma weak save_snapshot +#pragma weak qmp_xen_save_devices_state +#pragma weak load_snapshot +#pragma weak qmp_xen_set_replication +#pragma weak qmp_query_xen_replication_status +#pragma weak qmp_xen_colo_do_checkpoint +#pragma weak qmp_query_colo_status +#pragma weak qmp_x_colo_lost_heartbeatMigrationInfo *qmp_query_migrate (Error **errp) {@@ -160,3 +189,23 @@ AnnounceParameters *migrate_announce_params(void)return NULL ; } + +void colo_notify_filters_event (int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_notify_compares_event (void *opaque,int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}stubs/net-stub.c +21
-0@@ -5,6 +5,8 @@#include "qapi/qapi-commands-net.h" #include "qapi/qapi-commands-rocker.h" +#pragma weak qmp_announce_self +int qemu_find_net_clients_except (const char *id,NetClientState **ncs,NetClientDriver type,int max) {@@ -98,3 +100,22 @@ void netdev_add(QemuOpts *opts, Error **errp) {qemu_debug_assert (0 ); } + +NetClientState *qemu_get_queue (NICState *nic) +{ +qemu_debug_assert (0 ); + +return NULL ; +} + +ssize_t qemu_send_packet_raw (NetClientState *nc,const uint8_t *buf,int size) +{ +qemu_debug_assert (0 ); + +return 0 ; +} + +void qemu_foreach_nic (qemu_nic_foreach func,void *opaque) +{ +qemu_debug_assert (0 ); +}stubs/qapi-misc.c +2
-0@@ -5,6 +5,8 @@#include "./qapi/qapi-types-dump.h" #include "qapi/qapi-commands-dump.h" +#pragma weak qmp_xen_load_devices_state + void qmp_dump_guest_memory(bool paging, const char *file, bool has_detach, bool detach, bool has_begin, int64_t begin, bool has_length,stubs/replay.c +8
-0@@ -1,4 +1,5 @@#include "qemu/osdep.h" +#include "qemu-common.h" #include "sysemu/replay.h" ReplayMode replay_mode;@@ -106,3 +107,10 @@ void replay_account_executed_instructions(void)void replay_add_blocker (Error *reason) { } + +bool replay_can_snapshot (void ) +{ +qemu_debug_assert (0 ); + +return false; +}stubs/vl-stub.c +24
-0@@ -14,6 +14,8 @@#include "disas/disas.h" #include "audio/audio.h" +#pragma weak qemu_add_exit_notifier +bool tcg_allowed;bool xen_allowed;bool boot_strict;@@ -169,3 +171,25 @@ int wav_start_capture(AudioState *state, CaptureState *s, const char *path,return -1 ; } + +void qemu_system_killed (int signal,pid_t pid) +{ +qemu_debug_assert (0 ); +} + +void qemu_system_reset (ShutdownCause reason) +{ +qemu_debug_assert (0 ); +} + +bool runstate_store (char *str,size_t size) +{ +qemu_debug_assert (0 ); + +return false; +} + +void qemu_add_exit_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}vl-parse.c +3
-0@@ -27,6 +27,9 @@#include "vl.h" +int only_migratable;/* turn it off unless user states otherwise */ +bool enable_mlock; +/***********************************************************/ /* QEMU Block devices */ Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 40/50] multi-process/mig: build migration module in the remote process
Quoted 103 lines · * Jag Raman ([email protected]) wrote:
Quoted 85 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Jag Raman ([email protected]) wrote:
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 66 lines · * Jagannathan Raman ([email protected]) wrote:
Add
Makefilesupport to enable migration in remote processSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- Makefile.objs | 4 +++- Makefile.target | 1 + migration/Makefile.objs | 13 ++++++++++++- net/Makefile.objs | 2 ++ softmmu/vl.c | 2 -- stubs/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ stubs/net-stub.c | 21 +++++++++++++++++++++ stubs/qapi-misc.c | 2 ++ stubs/replay.c | 8 ++++++++ stubs/vl-stub.c | 24 ++++++++++++++++++++++++ vl-parse.c | 3 +++ 11 files changed, 125 insertions(+), 4 deletions(-)Makefile.objs +3
-1@@ -74,6 +74,8 @@ common-obj-y += qdev-monitor.o device-hotplug.o common-obj-$(CONFIG_WIN32) += os-win32.o common-obj-$(CONFIG_POSIX) += os-posix.o +remote-pci-obj-$(CONFIG_POSIX) += os-posix.o + common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/@@ -104,11 +106,11 @@ common-obj-y += vl-parse.o ####################################################################### # qapi - common-obj-y += qapi/ endif # CONFIG_SOFTMMU +remote-pci-obj-$(CONFIG_MPQEMU) += net/ remote-pci-obj-$(CONFIG_MPQEMU) += qapi/ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev-nbd.o remote-pci-obj-$(CONFIG_MPQEMU) += job-qmp.oMakefile.target +1
-0@@ -240,6 +240,7 @@ all-remote-pci-obj-y += exec.o all-remote-pci-obj-y += exec-vary.o all-remote-pci-obj-y += ioport.o all-remote-pci-obj-y += cpus.o +all-remote-pci-obj-y += migration/ram.o endif remote-pci-obj-y :=migration/Makefile.objs +8
-1@@ -15,4 +15,15 @@ common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o rdma.o-libs := $(RDMA_LIBS) -remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o vmstate.o qjson.o vmstate-types.o +remote-pci-obj-$(CONFIG_MPQEMU) += migration.o socket.o fd.o exec.o +remote-pci-obj-$(CONFIG_MPQEMU) += tls.o channel.o savevm.o +remote-pci-obj-$(CONFIG_MPQEMU) += colo.o colo-failover.o +remote-pci-obj-$(CONFIG_MPQEMU) += vmstate.o vmstate-types.o page_cache.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o global_state.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file-channel.o +remote-pci-obj-$(CONFIG_MPQEMU) += xbzrle.o postcopy-ram.o +remote-pci-obj-$(CONFIG_MPQEMU) += qjson.oQuoted 8 lines
migration/Makefile.objs common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o · line 29
Quoted 4 lines
migration/Makefile.objs +4
-0+remote-pci-obj-$(CONFIG_MPQEMU) += block-dirty-bitmap.o +remote-pci-obj-$(CONFIG_RDMA) += rdma.o +remote-pci-obj-$(CONFIG_MPQEMU) += block.o +remote-pci-obj-$(CONFIG_MPQEMU) += multifd.oHmm, are you really going to want all this lot in your remote process? Assuming it's just devices, I can understand the first line or two, but it seems odd to have all of this.
Quoted 4 lines
Yeah, we ended up needing to compile these in to enable migration. We are only using "fd" to enable migration. Although we don't use tls, xbzrle, rdma, multifd, etc... for example, the migration code does support these protocols and, therefore, we had to compile them in.
Quoted 4 lines
But are you even running a migration stream from the remote process? Aren't you just doing vmstate migration of devices; i.e. do you need anything relating to incremental RAM migration (e.g. xbzrle, rdma, postcopy).
We are running a migration stream from the remote process. We are only doing the vmstate migration of devices, and not anything incremental related to RAM.
We are using QEMU's existing migration infrastructure (
vmstate_save/qemu_loadvm_section_start_full) to move the vmstate. Based on my limited experience with the migration code, I get that it comes as a suite. Without some refactoring, we would need to build all of the files within the migration folder.
Quoted 4 lines
The vmstate functions communicate over QEMUFile, which would be using fd, tcp, rdma, etc... at the backend. These functions also need other functions defined in
migration.c, which require the building of xbzrle, postcopy, etc...
OK, I bet we can refactor some of that to be modular fairly easily; some is harder than others. For example there are different QEMUFile subclasses for RDMA, so you should be able to have just the top level QEMUFile and the implementations you need.
Similarly the vmstate code shouldn't depend on the ram code; we can look at splitting it up some how.
Dave
Quoted 230 lines
Thank you!
--
JagQuoted 225 lines
Dave
Quoted 219 lines
Thank you!
--
JagQuoted 214 lines
Dave
Quoted 208 lines
net/Makefile.objs +2
-0@@ -30,3 +30,5 @@ common-obj-$(CONFIG_WIN32) += tap-win32.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ + +remote-pci-obj-$(CONFIG_MPQEMU) += announce.osoftmmu/vl.c +0
-2@@ -128,7 +128,6 @@ const char* keyboard_layout = NULL;ram_addr_t ram_size;const char *mem_path =NULL ;int mem_prealloc =0 ;/* force preallocation of physical target memory */ -bool enable_mlock = false;bool enable_cpu_pm = false;int nb_nics;NICInfo nd_table[MAX_NICS ];@@ -168,7 +167,6 @@ const char *prom_envs[MAX_PROM_ENVS];int boot_menu;bool boot_strict;uint8_t *boot_splash_filedata; -int only_migratable;/* turn it off unless user states otherwise */ bool wakeup_suspend_enabled;int icount_align_option;stubs/migration.c +49
-0@@ -6,6 +6,35 @@#include "qapi/qapi-types-migration.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-types-net.h" +#include "net/filter.h" +#include "net/colo-compare.h" + +#pragma weak qmp_query_migrate_capabilities +#pragma weak qmp_query_migrate_parameters +#pragma weak migrate_announce_params +#pragma weak qmp_query_migrate +#pragma weak qmp_migrate_set_capabilities +#pragma weak qmp_migrate_set_parameters +#pragma weak qmp_migrate_incoming +#pragma weak qmp_migrate_recover +#pragma weak qmp_migrate_pause +#pragma weak qmp_migrate +#pragma weak qmp_migrate_cancel +#pragma weak qmp_migrate_continue +#pragma weak qmp_migrate_set_cache_size +#pragma weak qmp_query_migrate_cache_size +#pragma weak qmp_migrate_set_speed +#pragma weak qmp_migrate_set_downtime +#pragma weak qmp_migrate_start_postcopy +#pragma weak migration_global_dump +#pragma weak save_snapshot +#pragma weak qmp_xen_save_devices_state +#pragma weak load_snapshot +#pragma weak qmp_xen_set_replication +#pragma weak qmp_query_xen_replication_status +#pragma weak qmp_xen_colo_do_checkpoint +#pragma weak qmp_query_colo_status +#pragma weak qmp_x_colo_lost_heartbeatMigrationInfo *qmp_query_migrate (Error **errp) {@@ -160,3 +189,23 @@ AnnounceParameters *migrate_announce_params(void)return NULL ; } + +void colo_notify_filters_event (int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_notify_compares_event (void *opaque,int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}stubs/net-stub.c +21
-0@@ -5,6 +5,8 @@#include "qapi/qapi-commands-net.h" #include "qapi/qapi-commands-rocker.h" +#pragma weak qmp_announce_self +int qemu_find_net_clients_except (const char *id,NetClientState **ncs,NetClientDriver type,int max) {@@ -98,3 +100,22 @@ void netdev_add(QemuOpts *opts, Error **errp) {qemu_debug_assert (0 ); } + +NetClientState *qemu_get_queue (NICState *nic) +{ +qemu_debug_assert (0 ); + +return NULL ; +} + +ssize_t qemu_send_packet_raw (NetClientState *nc,const uint8_t *buf,int size) +{ +qemu_debug_assert (0 ); + +return 0 ; +} + +void qemu_foreach_nic (qemu_nic_foreach func,void *opaque) +{ +qemu_debug_assert (0 ); +}stubs/qapi-misc.c +2
-0@@ -5,6 +5,8 @@#include "./qapi/qapi-types-dump.h" #include "qapi/qapi-commands-dump.h" +#pragma weak qmp_xen_load_devices_state + void qmp_dump_guest_memory(bool paging, const char *file, bool has_detach, bool detach, bool has_begin, int64_t begin, bool has_length,stubs/replay.c +8
-0@@ -1,4 +1,5 @@#include "qemu/osdep.h" +#include "qemu-common.h" #include "sysemu/replay.h" ReplayMode replay_mode;@@ -106,3 +107,10 @@ void replay_account_executed_instructions(void)void replay_add_blocker (Error *reason) { } + +bool replay_can_snapshot (void ) +{ +qemu_debug_assert (0 ); + +return false; +}stubs/vl-stub.c +24
-0@@ -14,6 +14,8 @@#include "disas/disas.h" #include "audio/audio.h" +#pragma weak qemu_add_exit_notifier +bool tcg_allowed;bool xen_allowed;bool boot_strict;@@ -169,3 +171,25 @@ int wav_start_capture(AudioState *state, CaptureState *s, const char *path,return -1 ; } + +void qemu_system_killed (int signal,pid_t pid) +{ +qemu_debug_assert (0 ); +} + +void qemu_system_reset (ShutdownCause reason) +{ +qemu_debug_assert (0 ); +} + +bool runstate_store (char *str,size_t size) +{ +qemu_debug_assert (0 ); + +return false; +} + +void qemu_add_exit_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}vl-parse.c +3
-0@@ -27,6 +27,9 @@#include "vl.h" +int only_migratable;/* turn it off unless user states otherwise */ +bool enable_mlock; +/***********************************************************/ /* QEMU Block devices */ Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 40/50] multi-process/mig: build migration module in the remote process
Quoted 109 lines · Dr. David Alan Gilbert
Quoted 103 lines · * Jag Raman ([email protected]) wrote:
Quoted 85 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Jag Raman ([email protected]) wrote:
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 66 lines · * Jagannathan Raman ([email protected]) wrote:
Add
Makefilesupport to enable migration in remote processSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- Makefile.objs | 4 +++- Makefile.target | 1 + migration/Makefile.objs | 13 ++++++++++++- net/Makefile.objs | 2 ++ softmmu/vl.c | 2 -- stubs/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ stubs/net-stub.c | 21 +++++++++++++++++++++ stubs/qapi-misc.c | 2 ++ stubs/replay.c | 8 ++++++++ stubs/vl-stub.c | 24 ++++++++++++++++++++++++ vl-parse.c | 3 +++ 11 files changed, 125 insertions(+), 4 deletions(-)Makefile.objs +3
-1@@ -74,6 +74,8 @@ common-obj-y += qdev-monitor.o device-hotplug.o common-obj-$(CONFIG_WIN32) += os-win32.o common-obj-$(CONFIG_POSIX) += os-posix.o +remote-pci-obj-$(CONFIG_POSIX) += os-posix.o + common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/@@ -104,11 +106,11 @@ common-obj-y += vl-parse.o ####################################################################### # qapi - common-obj-y += qapi/ endif # CONFIG_SOFTMMU +remote-pci-obj-$(CONFIG_MPQEMU) += net/ remote-pci-obj-$(CONFIG_MPQEMU) += qapi/ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev-nbd.o remote-pci-obj-$(CONFIG_MPQEMU) += job-qmp.oMakefile.target +1
-0@@ -240,6 +240,7 @@ all-remote-pci-obj-y += exec.o all-remote-pci-obj-y += exec-vary.o all-remote-pci-obj-y += ioport.o all-remote-pci-obj-y += cpus.o +all-remote-pci-obj-y += migration/ram.o endif remote-pci-obj-y :=migration/Makefile.objs +8
-1@@ -15,4 +15,15 @@ common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o rdma.o-libs := $(RDMA_LIBS) -remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o vmstate.o qjson.o vmstate-types.o +remote-pci-obj-$(CONFIG_MPQEMU) += migration.o socket.o fd.o exec.o +remote-pci-obj-$(CONFIG_MPQEMU) += tls.o channel.o savevm.o +remote-pci-obj-$(CONFIG_MPQEMU) += colo.o colo-failover.o +remote-pci-obj-$(CONFIG_MPQEMU) += vmstate.o vmstate-types.o page_cache.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o global_state.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file-channel.o +remote-pci-obj-$(CONFIG_MPQEMU) += xbzrle.o postcopy-ram.o +remote-pci-obj-$(CONFIG_MPQEMU) += qjson.oQuoted 8 lines
migration/Makefile.objs common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o · line 29
Quoted 4 lines
migration/Makefile.objs +4
-0+remote-pci-obj-$(CONFIG_MPQEMU) += block-dirty-bitmap.o +remote-pci-obj-$(CONFIG_RDMA) += rdma.o +remote-pci-obj-$(CONFIG_MPQEMU) += block.o +remote-pci-obj-$(CONFIG_MPQEMU) += multifd.oHmm, are you really going to want all this lot in your remote process? Assuming it's just devices, I can understand the first line or two, but it seems odd to have all of this.
Quoted 4 lines
Yeah, we ended up needing to compile these in to enable migration. We are only using "fd" to enable migration. Although we don't use tls, xbzrle, rdma, multifd, etc... for example, the migration code does support these protocols and, therefore, we had to compile them in.
Quoted 4 lines
But are you even running a migration stream from the remote process? Aren't you just doing vmstate migration of devices; i.e. do you need anything relating to incremental RAM migration (e.g. xbzrle, rdma, postcopy).
We are running a migration stream from the remote process. We are only doing the vmstate migration of devices, and not anything incremental related to RAM.
We are using QEMU's existing migration infrastructure (
vmstate_save/qemu_loadvm_section_start_full) to move the vmstate. Based on my limited experience with the migration code, I get that it comes as a suite. Without some refactoring, we would need to build all of the files within the migration folder.Quoted 4 lines
The vmstate functions communicate over QEMUFile, which would be using fd, tcp, rdma, etc... at the backend. These functions also need other functions defined in
migration.c, which require the building of xbzrle, postcopy, etc...
Quoted 7 lines
OK, I bet we can refactor some of that to be modular fairly easily; some is harder than others. For example there are different QEMUFile subclasses for RDMA, so you should be able to have just the top level QEMUFile and the implementations you need.
Similarly the vmstate code shouldn't depend on the ram code; we can look at splitting it up some how.
Hi Dave
We did initially tried to split some of the vmstate code, and it was a lot of code to add to already big patchset. So we could do this split later on as a separate series. Would that be an option or you would rather see it in this set?
Thank you!
Elena
Quoted 235 lines
Dave
Quoted 230 lines
Thank you!
--
JagQuoted 225 lines
Dave
Quoted 219 lines
Thank you!
--
JagQuoted 214 lines
Dave
Quoted 208 lines
net/Makefile.objs +2
-0@@ -30,3 +30,5 @@ common-obj-$(CONFIG_WIN32) += tap-win32.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ + +remote-pci-obj-$(CONFIG_MPQEMU) += announce.osoftmmu/vl.c +0
-2@@ -128,7 +128,6 @@ const char* keyboard_layout = NULL;ram_addr_t ram_size;const char *mem_path =NULL ;int mem_prealloc =0 ;/* force preallocation of physical target memory */ -bool enable_mlock = false;bool enable_cpu_pm = false;int nb_nics;NICInfo nd_table[MAX_NICS ];@@ -168,7 +167,6 @@ const char *prom_envs[MAX_PROM_ENVS];int boot_menu;bool boot_strict;uint8_t *boot_splash_filedata; -int only_migratable;/* turn it off unless user states otherwise */ bool wakeup_suspend_enabled;int icount_align_option;stubs/migration.c +49
-0@@ -6,6 +6,35 @@#include "qapi/qapi-types-migration.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-types-net.h" +#include "net/filter.h" +#include "net/colo-compare.h" + +#pragma weak qmp_query_migrate_capabilities +#pragma weak qmp_query_migrate_parameters +#pragma weak migrate_announce_params +#pragma weak qmp_query_migrate +#pragma weak qmp_migrate_set_capabilities +#pragma weak qmp_migrate_set_parameters +#pragma weak qmp_migrate_incoming +#pragma weak qmp_migrate_recover +#pragma weak qmp_migrate_pause +#pragma weak qmp_migrate +#pragma weak qmp_migrate_cancel +#pragma weak qmp_migrate_continue +#pragma weak qmp_migrate_set_cache_size +#pragma weak qmp_query_migrate_cache_size +#pragma weak qmp_migrate_set_speed +#pragma weak qmp_migrate_set_downtime +#pragma weak qmp_migrate_start_postcopy +#pragma weak migration_global_dump +#pragma weak save_snapshot +#pragma weak qmp_xen_save_devices_state +#pragma weak load_snapshot +#pragma weak qmp_xen_set_replication +#pragma weak qmp_query_xen_replication_status +#pragma weak qmp_xen_colo_do_checkpoint +#pragma weak qmp_query_colo_status +#pragma weak qmp_x_colo_lost_heartbeatMigrationInfo *qmp_query_migrate (Error **errp) {@@ -160,3 +189,23 @@ AnnounceParameters *migrate_announce_params(void)return NULL ; } + +void colo_notify_filters_event (int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_notify_compares_event (void *opaque,int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}stubs/net-stub.c +21
-0@@ -5,6 +5,8 @@#include "qapi/qapi-commands-net.h" #include "qapi/qapi-commands-rocker.h" +#pragma weak qmp_announce_self +int qemu_find_net_clients_except (const char *id,NetClientState **ncs,NetClientDriver type,int max) {@@ -98,3 +100,22 @@ void netdev_add(QemuOpts *opts, Error **errp) {qemu_debug_assert (0 ); } + +NetClientState *qemu_get_queue (NICState *nic) +{ +qemu_debug_assert (0 ); + +return NULL ; +} + +ssize_t qemu_send_packet_raw (NetClientState *nc,const uint8_t *buf,int size) +{ +qemu_debug_assert (0 ); + +return 0 ; +} + +void qemu_foreach_nic (qemu_nic_foreach func,void *opaque) +{ +qemu_debug_assert (0 ); +}stubs/qapi-misc.c +2
-0@@ -5,6 +5,8 @@#include "./qapi/qapi-types-dump.h" #include "qapi/qapi-commands-dump.h" +#pragma weak qmp_xen_load_devices_state + void qmp_dump_guest_memory(bool paging, const char *file, bool has_detach, bool detach, bool has_begin, int64_t begin, bool has_length,stubs/replay.c +8
-0@@ -1,4 +1,5 @@#include "qemu/osdep.h" +#include "qemu-common.h" #include "sysemu/replay.h" ReplayMode replay_mode;@@ -106,3 +107,10 @@ void replay_account_executed_instructions(void)void replay_add_blocker (Error *reason) { } + +bool replay_can_snapshot (void ) +{ +qemu_debug_assert (0 ); + +return false; +}stubs/vl-stub.c +24
-0@@ -14,6 +14,8 @@#include "disas/disas.h" #include "audio/audio.h" +#pragma weak qemu_add_exit_notifier +bool tcg_allowed;bool xen_allowed;bool boot_strict;@@ -169,3 +171,25 @@ int wav_start_capture(AudioState *state, CaptureState *s, const char *path,return -1 ; } + +void qemu_system_killed (int signal,pid_t pid) +{ +qemu_debug_assert (0 ); +} + +void qemu_system_reset (ShutdownCause reason) +{ +qemu_debug_assert (0 ); +} + +bool runstate_store (char *str,size_t size) +{ +qemu_debug_assert (0 ); + +return false; +} + +void qemu_add_exit_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}vl-parse.c +3
-0@@ -27,6 +27,9 @@#include "vl.h" +int only_migratable;/* turn it off unless user states otherwise */ +bool enable_mlock; +/***********************************************************/ /* QEMU Block devices */ Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 40/50] multi-process/mig: build migration module in the remote process
Quoted 118 lines · * Elena Ufimtseva ([email protected]) wrote:
Quoted 109 lines · Dr. David Alan Gilbert
Quoted 103 lines · * Jag Raman ([email protected]) wrote:
Quoted 85 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Jag Raman ([email protected]) wrote:
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 66 lines · * Jagannathan Raman ([email protected]) wrote:
Add
Makefilesupport to enable migration in remote processSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- Makefile.objs | 4 +++- Makefile.target | 1 + migration/Makefile.objs | 13 ++++++++++++- net/Makefile.objs | 2 ++ softmmu/vl.c | 2 -- stubs/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ stubs/net-stub.c | 21 +++++++++++++++++++++ stubs/qapi-misc.c | 2 ++ stubs/replay.c | 8 ++++++++ stubs/vl-stub.c | 24 ++++++++++++++++++++++++ vl-parse.c | 3 +++ 11 files changed, 125 insertions(+), 4 deletions(-)Makefile.objs +3
-1@@ -74,6 +74,8 @@ common-obj-y += qdev-monitor.o device-hotplug.o common-obj-$(CONFIG_WIN32) += os-win32.o common-obj-$(CONFIG_POSIX) += os-posix.o +remote-pci-obj-$(CONFIG_POSIX) += os-posix.o + common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/@@ -104,11 +106,11 @@ common-obj-y += vl-parse.o ####################################################################### # qapi - common-obj-y += qapi/ endif # CONFIG_SOFTMMU +remote-pci-obj-$(CONFIG_MPQEMU) += net/ remote-pci-obj-$(CONFIG_MPQEMU) += qapi/ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev-nbd.o remote-pci-obj-$(CONFIG_MPQEMU) += job-qmp.oMakefile.target +1
-0@@ -240,6 +240,7 @@ all-remote-pci-obj-y += exec.o all-remote-pci-obj-y += exec-vary.o all-remote-pci-obj-y += ioport.o all-remote-pci-obj-y += cpus.o +all-remote-pci-obj-y += migration/ram.o endif remote-pci-obj-y :=migration/Makefile.objs +8
-1@@ -15,4 +15,15 @@ common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o rdma.o-libs := $(RDMA_LIBS) -remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o vmstate.o qjson.o vmstate-types.o +remote-pci-obj-$(CONFIG_MPQEMU) += migration.o socket.o fd.o exec.o +remote-pci-obj-$(CONFIG_MPQEMU) += tls.o channel.o savevm.o +remote-pci-obj-$(CONFIG_MPQEMU) += colo.o colo-failover.o +remote-pci-obj-$(CONFIG_MPQEMU) += vmstate.o vmstate-types.o page_cache.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file.o global_state.o +remote-pci-obj-$(CONFIG_MPQEMU) += qemu-file-channel.o +remote-pci-obj-$(CONFIG_MPQEMU) += xbzrle.o postcopy-ram.o +remote-pci-obj-$(CONFIG_MPQEMU) += qjson.oQuoted 8 lines
migration/Makefile.objs common-obj-$(CONFIG_LIVE_BLOCK_MIGRATION) += block.o · line 27
Quoted 4 lines
migration/Makefile.objs +4
-0+remote-pci-obj-$(CONFIG_MPQEMU) += block-dirty-bitmap.o +remote-pci-obj-$(CONFIG_RDMA) += rdma.o +remote-pci-obj-$(CONFIG_MPQEMU) += block.o +remote-pci-obj-$(CONFIG_MPQEMU) += multifd.oHmm, are you really going to want all this lot in your remote process? Assuming it's just devices, I can understand the first line or two, but it seems odd to have all of this.
Quoted 4 lines
Yeah, we ended up needing to compile these in to enable migration. We are only using "fd" to enable migration. Although we don't use tls, xbzrle, rdma, multifd, etc... for example, the migration code does support these protocols and, therefore, we had to compile them in.
Quoted 4 lines
But are you even running a migration stream from the remote process? Aren't you just doing vmstate migration of devices; i.e. do you need anything relating to incremental RAM migration (e.g. xbzrle, rdma, postcopy).
We are running a migration stream from the remote process. We are only doing the vmstate migration of devices, and not anything incremental related to RAM.
We are using QEMU's existing migration infrastructure (
vmstate_save/qemu_loadvm_section_start_full) to move the vmstate. Based on my limited experience with the migration code, I get that it comes as a suite. Without some refactoring, we would need to build all of the files within the migration folder.Quoted 4 lines
The vmstate functions communicate over QEMUFile, which would be using fd, tcp, rdma, etc... at the backend. These functions also need other functions defined in
migration.c, which require the building of xbzrle, postcopy, etc...Quoted 7 lines
OK, I bet we can refactor some of that to be modular fairly easily; some is harder than others. For example there are different QEMUFile subclasses for RDMA, so you should be able to have just the top level QEMUFile and the implementations you need.
Similarly the vmstate code shouldn't depend on the ram code; we can look at splitting it up some how.
Quoted 5 lines
Hi Dave
We did initially tried to split some of the vmstate code, and it was a lot of code to add to already big patchset. So we could do this split later on as a separate series. Would that be an option or you would rather see it in this set?
I'd say see if there are any easy things you can split; if there are then lets take those as part of this series.
Dave
Quoted 239 lines
Thank you!
Elena
Quoted 235 lines
Dave
Quoted 230 lines
Thank you!
--
JagQuoted 225 lines
Dave
Quoted 219 lines
Thank you!
--
JagQuoted 214 lines
Dave
Quoted 208 lines
net/Makefile.objs +2
-0@@ -30,3 +30,5 @@ common-obj-$(CONFIG_WIN32) += tap-win32.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ + +remote-pci-obj-$(CONFIG_MPQEMU) += announce.osoftmmu/vl.c +0
-2@@ -128,7 +128,6 @@ const char* keyboard_layout = NULL;ram_addr_t ram_size;const char *mem_path =NULL ;int mem_prealloc =0 ;/* force preallocation of physical target memory */ -bool enable_mlock = false;bool enable_cpu_pm = false;int nb_nics;NICInfo nd_table[MAX_NICS ];@@ -168,7 +167,6 @@ const char *prom_envs[MAX_PROM_ENVS];int boot_menu;bool boot_strict;uint8_t *boot_splash_filedata; -int only_migratable;/* turn it off unless user states otherwise */ bool wakeup_suspend_enabled;int icount_align_option;stubs/migration.c +49
-0@@ -6,6 +6,35 @@#include "qapi/qapi-types-migration.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-types-net.h" +#include "net/filter.h" +#include "net/colo-compare.h" + +#pragma weak qmp_query_migrate_capabilities +#pragma weak qmp_query_migrate_parameters +#pragma weak migrate_announce_params +#pragma weak qmp_query_migrate +#pragma weak qmp_migrate_set_capabilities +#pragma weak qmp_migrate_set_parameters +#pragma weak qmp_migrate_incoming +#pragma weak qmp_migrate_recover +#pragma weak qmp_migrate_pause +#pragma weak qmp_migrate +#pragma weak qmp_migrate_cancel +#pragma weak qmp_migrate_continue +#pragma weak qmp_migrate_set_cache_size +#pragma weak qmp_query_migrate_cache_size +#pragma weak qmp_migrate_set_speed +#pragma weak qmp_migrate_set_downtime +#pragma weak qmp_migrate_start_postcopy +#pragma weak migration_global_dump +#pragma weak save_snapshot +#pragma weak qmp_xen_save_devices_state +#pragma weak load_snapshot +#pragma weak qmp_xen_set_replication +#pragma weak qmp_query_xen_replication_status +#pragma weak qmp_xen_colo_do_checkpoint +#pragma weak qmp_query_colo_status +#pragma weak qmp_x_colo_lost_heartbeatMigrationInfo *qmp_query_migrate (Error **errp) {@@ -160,3 +189,23 @@ AnnounceParameters *migrate_announce_params(void)return NULL ; } + +void colo_notify_filters_event (int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_notify_compares_event (void *opaque,int event,Error **errp) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}stubs/net-stub.c +21
-0@@ -5,6 +5,8 @@#include "qapi/qapi-commands-net.h" #include "qapi/qapi-commands-rocker.h" +#pragma weak qmp_announce_self +int qemu_find_net_clients_except (const char *id,NetClientState **ncs,NetClientDriver type,int max) {@@ -98,3 +100,22 @@ void netdev_add(QemuOpts *opts, Error **errp) {qemu_debug_assert (0 ); } + +NetClientState *qemu_get_queue (NICState *nic) +{ +qemu_debug_assert (0 ); + +return NULL ; +} + +ssize_t qemu_send_packet_raw (NetClientState *nc,const uint8_t *buf,int size) +{ +qemu_debug_assert (0 ); + +return 0 ; +} + +void qemu_foreach_nic (qemu_nic_foreach func,void *opaque) +{ +qemu_debug_assert (0 ); +}stubs/qapi-misc.c +2
-0@@ -5,6 +5,8 @@#include "./qapi/qapi-types-dump.h" #include "qapi/qapi-commands-dump.h" +#pragma weak qmp_xen_load_devices_state + void qmp_dump_guest_memory(bool paging, const char *file, bool has_detach, bool detach, bool has_begin, int64_t begin, bool has_length,stubs/replay.c +8
-0@@ -1,4 +1,5 @@#include "qemu/osdep.h" +#include "qemu-common.h" #include "sysemu/replay.h" ReplayMode replay_mode;@@ -106,3 +107,10 @@ void replay_account_executed_instructions(void)void replay_add_blocker (Error *reason) { } + +bool replay_can_snapshot (void ) +{ +qemu_debug_assert (0 ); + +return false; +}stubs/vl-stub.c +24
-0@@ -14,6 +14,8 @@#include "disas/disas.h" #include "audio/audio.h" +#pragma weak qemu_add_exit_notifier +bool tcg_allowed;bool xen_allowed;bool boot_strict;@@ -169,3 +171,25 @@ int wav_start_capture(AudioState *state, CaptureState *s, const char *path,return -1 ; } + +void qemu_system_killed (int signal,pid_t pid) +{ +qemu_debug_assert (0 ); +} + +void qemu_system_reset (ShutdownCause reason) +{ +qemu_debug_assert (0 ); +} + +bool runstate_store (char *str,size_t size) +{ +qemu_debug_assert (0 ); + +return false; +} + +void qemu_add_exit_notifier (Notifier *notify) +{ +qemu_debug_assert (0 ); +}vl-parse.c +3
-0@@ -27,6 +27,9 @@#include "vl.h" +int only_migratable;/* turn it off unless user states otherwise */ +bool enable_mlock; +/***********************************************************/ /* QEMU Block devices */ Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v5 41/50 multi-process/mig: Enable VMSD save in the Proxy object
In reply to Jagannathan RamanCollect the VMSD from remote process on the source and save it to the channel leading to the destination
---
v4 -> v5:
- Using qemu_file_shutdown() instead of qemu_thread_cancel(). Removed patch which introduced qemu_thread_cancel()
hw/proxy/qemu-proxy.c | 135 ++++++++++++++++++++++++++++++++++++++++++ include/hw/proxy/qemu-proxy.h | 2 + include/io/mpqemu-link.h | 1 + 3 files changed, 138 insertions(+)
hw/proxy/qemu-proxy.c +135 -0
@@ -23,6 +23,14 @@#include "util/event_notifier-posix.c" #include "hw/boards.h" #include "include/qemu/log.h" +#include "io/channel.h" +#include "migration/qemu-file-types.h" +#include "qapi/error.h" +#include "io/channel-util.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "migration/migration.h" +#include "migration/vmstate.h" QEMUTimer *hb_timer;static void pci_proxy_dev_realize (PCIDevice *dev,Error **errp);
@@ -35,6 +43,9 @@ static void broadcast_init(void);static int config_op_send (PCIProxyDev *dev,uint32_t addr,uint32_t *val,int l,unsigned int op); +#define PAGE_SIZE qemu_real_host_page_size +uint8_t *mig_data; +static void childsig_handler (int sig,siginfo_t *siginfo,void *ctx) {/* TODO: Add proper handler. */
@@ -460,6 +471,129 @@ static void pci_proxy_dev_inst_init(Object *obj) dev->mem_init = false; } +typedef struct { +QEMUFile *rem; +PCIProxyDev *dev; +}proxy_mig_data ; + +static void *proxy_mig_out (void *opaque) +{ +proxy_mig_data *data = opaque; +PCIProxyDev *dev = data->dev; +uint8_t byte; +uint64_t data_size =PAGE_SIZE ; + + mig_data =g_malloc (data_size); + +while (true) { + byte =qemu_get_byte (data->rem); + +if (qemu_file_get_error (data->rem)) { +break ; + } + + mig_data[dev->migsize++] = byte; +if (dev->migsize == data_size) { + data_size +=PAGE_SIZE ; + mig_data =g_realloc (mig_data, data_size); + } + } + +return NULL ; +} + +static int proxy_pre_save (void *opaque) +{ +PCIProxyDev *pdev = opaque; +proxy_mig_data *mig_data; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +QemuThread thread; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"PCIProxyDevice-mig" ); + + f_remote =qemu_fopen_channel_input (ioc); + + pdev->migsize =0 ; + + mig_data =g_malloc0 (sizeof (proxy_mig_data)); + mig_data->rem = f_remote; + mig_data->dev = pdev; + +qemu_thread_create (&thread,"Proxy MIG_OUT" , proxy_mig_out, mig_data, +QEMU_THREAD_DETACHED ); + + msg.cmd =START_MIG_OUT ; + msg.bytestream =0 ; + msg.num_fds =2 ; + msg.fds[0 ] = fd[1 ]; + msg.fds[1 ] =GET_REMOTE_WAIT ; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + size =wait_for_remote (msg.fds[1 ]); +PUT_REMOTE_WAIT (msg.fds[1 ]); + +assert (size !=ULLONG_MAX ); + +/* +* migsize is being update by a separate thread. Using volatile to +* instruct the compiler to fetch the value of this variable from +* memory during every read +*/ +while (*((volatile uint64_t *)&pdev->migsize) < size) { + } + +qemu_file_shutdown (f_remote); + +qemu_fclose (f_remote); +close (fd[1 ]); + +return 0 ; +} + +static int proxy_post_save (void *opaque) +{ +MigrationState *ms =migrate_get_current (); +PCIProxyDev *pdev = opaque; +uint64_t pos =0 ; + +while (pos < pdev->migsize) { +qemu_put_byte (ms->to_dst_file, mig_data[pos]); + pos++; + } + +qemu_fflush (ms->to_dst_file); + +return 0 ; +} + +const VMStateDescription vmstate_pci_proxy_device = { + .name ="PCIProxyDevice" , + .version_id =2 , + .minimum_version_id =1 , + .pre_save = proxy_pre_save, + .post_save = proxy_post_save, + .fields = (VMStateField []) { +VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev), +VMSTATE_UINT64 (migsize, PCIProxyDev), +VMSTATE_END_OF_LIST () + } +}; +static void pci_proxy_dev_class_init (ObjectClass *klass,void *data) {PCIDeviceClass *k =PCI_DEVICE_CLASS (klass);
@@ -471,6 +605,7 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data) k->config_write = pci_proxy_write_config; dc->reset = proxy_device_reset; + dc->vmsd = &vmstate_pci_proxy_device; }static const TypeInfo pci_proxy_dev_type_info = {
include/hw/proxy/qemu-proxy.h +2 -0
@@ -75,6 +75,8 @@ struct PCIProxyDev { bool need_spawn, Error **errp);ProxyMemoryRegion region[PCI_NUM_REGIONS ]; + +uint64_t migsize; }; typedef struct PCIProxyDevClass {
include/io/mpqemu-link.h +1 -0
@@ -63,6 +63,7 @@ typedef enum {PROXY_PING ,MMIO_RETURN ,DEVICE_RESET , +START_MIG_OUT ,MAX , } mpqemu_cmd_t;
Signature · 1 line
1.8.3.1
Re: [PATCH v5 41/50] multi-process/mig: Enable VMSD save in the Proxy object
Quoted 100 lines · * Jagannathan Raman ([email protected]) wrote:
Collect the VMSD from remote process on the source and save it to the channel leading to the destination
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
v4 -> v5:
- Usingqemu_file_shutdown()instead ofqemu_thread_cancel(). Removed patch which introducedqemu_thread_cancel()hw/proxy/qemu-proxy.c | 135 ++++++++++++++++++++++++++++++++++++++++++ include/hw/proxy/qemu-proxy.h | 2 + include/io/mpqemu-link.h | 1 + 3 files changed, 138 insertions(+)hw/proxy/qemu-proxy.c +62
-0@@ -23,6 +23,14 @@#include "util/event_notifier-posix.c" #include "hw/boards.h" #include "include/qemu/log.h" +#include "io/channel.h" +#include "migration/qemu-file-types.h" +#include "qapi/error.h" +#include "io/channel-util.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "migration/migration.h" +#include "migration/vmstate.h" QEMUTimer *hb_timer;static void pci_proxy_dev_realize (PCIDevice *dev,Error **errp);@@ -35,6 +43,9 @@ static void broadcast_init(void);static int config_op_send (PCIProxyDev *dev,uint32_t addr,uint32_t *val,int l,unsigned int op); +#define PAGE_SIZE qemu_real_host_page_size +uint8_t *mig_data; +static void childsig_handler (int sig,siginfo_t *siginfo,void *ctx) {/* TODO: Add proper handler. */ @@ -460,6 +471,129 @@ static void pci_proxy_dev_inst_init(Object *obj) dev->mem_init = false; } +typedef struct { +QEMUFile *rem; +PCIProxyDev *dev; +}proxy_mig_data ; + +static void *proxy_mig_out (void *opaque) +{ +proxy_mig_data *data = opaque; +PCIProxyDev *dev = data->dev; +uint8_t byte; +uint64_t data_size =PAGE_SIZE ; + + mig_data =g_malloc (data_size); + +while (true) { + byte =qemu_get_byte (data->rem); + +if (qemu_file_get_error (data->rem)) { +break ; + } + + mig_data[dev->migsize++] = byte; +if (dev->migsize == data_size) { + data_size +=PAGE_SIZE ; + mig_data =g_realloc (mig_data, data_size); + } + } + +return NULL ; +} + +static int proxy_pre_save (void *opaque) +{ +PCIProxyDev *pdev = opaque; +proxy_mig_data *mig_data; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +QemuThread thread; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ;
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 534
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ } + +qio_channel_set_name (QIO_CHANNEL (ioc),"PCIProxyDevice-mig" ); + + f_remote =qemu_fopen_channel_input (ioc); + + pdev->migsize =0 ; + + mig_data =g_malloc0 (sizeof (proxy_mig_data)); + mig_data->rem = f_remote;
This feels way too complicated. Since we know f_remote is always just
a simple fd we're getting we don't need to use qio or qemu_file here; just use the fd - nice and simple.
Then the code to read it can just use read(2) with a sensible size chunk rather than reading a byte at a time.
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 538
Quoted 4 lines
hw/proxy/qemu-proxy.c +4
-0+ mig_data->dev = pdev; + +qemu_thread_create (&thread,"Proxy MIG_OUT" , proxy_mig_out, mig_data, +QEMU_THREAD_DETACHED );
I'm just checking why a thread is necessary; is this because you need to be able to start reading before you block waiting for the remote to tell you the size - worrying that if you don't start reading then the remote might block waiting for us?
Quoted 9 lines
hw/proxy/qemu-proxy.c +9
-0+ msg.cmd =START_MIG_OUT ; + msg.bytestream =0 ; + msg.num_fds =2 ; + msg.fds[0 ] = fd[1 ]; + msg.fds[1 ] =GET_REMOTE_WAIT ; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + size =wait_for_remote (msg.fds[1 ]); +PUT_REMOTE_WAIT (msg.fds[1 ]);
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 558
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ +assert (size !=ULLONG_MAX ); + +/* +* migsize is being update by a separate thread. Using volatile to +* instruct the compiler to fetch the value of this variable from +* memory during every read +*/ +while (*((volatile uint64_t *)&pdev->migsize) < size) { + }
Hmm. I suggest the following:
a) You create a shared 'size' variable; and initialize it to
UINT64_MAX.
b) The thread uses atomic_read(shared_size) to read that value.
c) When you receive the size from the remote you do
atomic_set(&shared_size, size); d) The thread does:
while (received_size < atomic_read(&shared_size))
so the thread will quit either on EOF or it hitting the size.
e) We pthread_join here to wait for the thread
f) We then check a received size to make sure it matches what we
expect.
That removes the tight loop.
Quoted 12 lines
hw/proxy/qemu-proxy.c +12
-0+qemu_file_shutdown (f_remote); + +qemu_fclose (f_remote); +close (fd[1 ]); + +return 0 ; +} + +static int proxy_post_save (void *opaque) +{ +MigrationState *ms =migrate_get_current (); +PCIProxyDev *pdev = opaque;
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 581
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+uint64_t pos =0 ; + +while (pos < pdev->migsize) { +qemu_put_byte (ms->to_dst_file, mig_data[pos]); + pos++; + } + +qemu_fflush (ms->to_dst_file); + +return 0 ;
I don't think you need that.
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 592
Quoted 11 lines
hw/proxy/qemu-proxy.c +11
-0+} + +const VMStateDescription vmstate_pci_proxy_device = { + .name ="PCIProxyDevice" , + .version_id =2 , + .minimum_version_id =1 , + .pre_save = proxy_pre_save, + .post_save = proxy_post_save, + .fields = (VMStateField []) { +VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev), +VMSTATE_UINT64 (migsize, PCIProxyDev),
I think instead you should use a VMSTATE_VBUFFER here to save the mig_data.
What we should do is the post_save should g_free the buffer. (mig_data should be a field in proxy).
Dave
Quoted 43 lines
hw/proxy/qemu-proxy.c +5
-0+ VMSTATE_END_OF_LIST() + } +}; +static void pci_proxy_dev_class_init (ObjectClass *klass,void *data) {PCIDeviceClass *k =PCI_DEVICE_CLASS (klass);@@ -471,6 +605,7 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data) k->config_write = pci_proxy_write_config; dc->reset = proxy_device_reset; + dc->vmsd = &vmstate_pci_proxy_device; }static const TypeInfo pci_proxy_dev_type_info = {include/hw/proxy/qemu-proxy.h +2
-0@@ -75,6 +75,8 @@ struct PCIProxyDev { bool need_spawn, Error **errp);ProxyMemoryRegion region[PCI_NUM_REGIONS ]; + +uint64_t migsize; }; typedef struct PCIProxyDevClass {include/io/mpqemu-link.h +1
-0@@ -63,6 +63,7 @@ typedef enum {PROXY_PING ,MMIO_RETURN ,DEVICE_RESET , +START_MIG_OUT ,MAX , } mpqemu_cmd_t;Signature · 1 line
1.8.3.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 41/50] multi-process/mig: Enable VMSD save in the Proxy object
Quoted 116 lines · Dr. David Alan Gilbert
Quoted 100 lines · * Jagannathan Raman ([email protected]) wrote:
Collect the VMSD from remote process on the source and save it to the channel leading to the destination
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
v4 -> v5:
- Usingqemu_file_shutdown()instead ofqemu_thread_cancel(). Removed patch which introducedqemu_thread_cancel()hw/proxy/qemu-proxy.c | 135 ++++++++++++++++++++++++++++++++++++++++++ include/hw/proxy/qemu-proxy.h | 2 + include/io/mpqemu-link.h | 1 + 3 files changed, 138 insertions(+)hw/proxy/qemu-proxy.c +62
-0@@ -23,6 +23,14 @@#include "util/event_notifier-posix.c" #include "hw/boards.h" #include "include/qemu/log.h" +#include "io/channel.h" +#include "migration/qemu-file-types.h" +#include "qapi/error.h" +#include "io/channel-util.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "migration/migration.h" +#include "migration/vmstate.h" QEMUTimer *hb_timer;static void pci_proxy_dev_realize (PCIDevice *dev,Error **errp);@@ -35,6 +43,9 @@ static void broadcast_init(void);static int config_op_send (PCIProxyDev *dev,uint32_t addr,uint32_t *val,int l,unsigned int op); +#define PAGE_SIZE qemu_real_host_page_size +uint8_t *mig_data; +static void childsig_handler (int sig,siginfo_t *siginfo,void *ctx) {/* TODO: Add proper handler. */ @@ -460,6 +471,129 @@ static void pci_proxy_dev_inst_init(Object *obj) dev->mem_init = false; } +typedef struct { +QEMUFile *rem; +PCIProxyDev *dev; +}proxy_mig_data ; + +static void *proxy_mig_out (void *opaque) +{ +proxy_mig_data *data = opaque; +PCIProxyDev *dev = data->dev; +uint8_t byte; +uint64_t data_size =PAGE_SIZE ; + + mig_data =g_malloc (data_size); + +while (true) { + byte =qemu_get_byte (data->rem); + +if (qemu_file_get_error (data->rem)) { +break ; + } + + mig_data[dev->migsize++] = byte; +if (dev->migsize == data_size) { + data_size +=PAGE_SIZE ; + mig_data =g_realloc (mig_data, data_size); + } + } + +return NULL ; +} + +static int proxy_pre_save (void *opaque) +{ +PCIProxyDev *pdev = opaque; +proxy_mig_data *mig_data; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +QemuThread thread; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ;hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 534
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ } + +qio_channel_set_name (QIO_CHANNEL (ioc),"PCIProxyDevice-mig" ); + + f_remote =qemu_fopen_channel_input (ioc); + + pdev->migsize =0 ; + + mig_data =g_malloc0 (sizeof (proxy_mig_data)); + mig_data->rem = f_remote;This feels way too complicated. Since we know
f_remoteis always just
a simple fd we're getting we don't need to use qio orqemu_filehere; just use the fd - nice and simple.
Quoted 2 lines
Then the code to read it can just use read(2) with a sensible size chunk rather than reading a byte at a time.
Hi Dave,
Upon closer inspection, we found that the migration code on the remote (which uses QEMUFile) could sometimes set an error on the channel using qemu_file_set_error(). We needed to use qemu_file_get_error() to catch these errors and abort migration. So we had to stick with QEMUFile at the receiving end as well.
Quoted 10 lines
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 538
Quoted 5 lines
hw/proxy/qemu-proxy.c +4
-0+ mig_data->dev = pdev; + +qemu_thread_create (&thread,"Proxy MIG_OUT" , proxy_mig_out, mig_data, +QEMU_THREAD_DETACHED );I'm just checking why a thread is necessary; is this because you need to be able to start reading before you block waiting for the remote to tell you the size - worrying that if you don't start reading then the remote might block waiting for us?
Yes, Dave. That is correct.
Quoted 31 lines
Quoted 10 lines
hw/proxy/qemu-proxy.c +9
-0+ msg.cmd =START_MIG_OUT ; + msg.bytestream =0 ; + msg.num_fds =2 ; + msg.fds[0 ] = fd[1 ]; + msg.fds[1 ] =GET_REMOTE_WAIT ; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + size =wait_for_remote (msg.fds[1 ]); +PUT_REMOTE_WAIT (msg.fds[1 ]);hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 558
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ +assert (size !=ULLONG_MAX ); + +/* +* migsize is being update by a separate thread. Using volatile to +* instruct the compiler to fetch the value of this variable from +* memory during every read +*/ +while (*((volatile uint64_t *)&pdev->migsize) < size) { + }Hmm. I suggest the following:
a) You create a shared 'size' variable; and initialize it to
UINT64_MAX. b) The thread uses atomic_read(shared_size) to read that value. c) When you receive the size from the remote you do atomic_set(&shared_size, size);d) The thread does:
while (received_size<atomic_read(&shared_size))
Quoted 7 lines
so the thread will quit either on EOF or it hitting the size.
e) We
pthread_joinhere to wait for the thread
f) We then check a received size to make sure it matches what we
expect.That removes the tight loop.
Sure, will do.
Quoted 26 lines
Quoted 13 lines
hw/proxy/qemu-proxy.c +12
-0+qemu_file_shutdown (f_remote); + +qemu_fclose (f_remote); +close (fd[1 ]); + +return 0 ; +} + +static int proxy_post_save (void *opaque) +{ +MigrationState *ms =migrate_get_current (); +PCIProxyDev *pdev = opaque;hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 581
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+uint64_t pos =0 ; + +while (pos < pdev->migsize) { +qemu_put_byte (ms->to_dst_file, mig_data[pos]); + pos++; + } + +qemu_fflush (ms->to_dst_file); + +return 0 ;I don't think you need that.
Quoted 16 lines
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 592
Quoted 11 lines
hw/proxy/qemu-proxy.c +11
-0+} + +const VMStateDescription vmstate_pci_proxy_device = { + .name ="PCIProxyDevice" , + .version_id =2 , + .minimum_version_id =1 , + .pre_save = proxy_pre_save, + .post_save = proxy_post_save, + .fields = (VMStateField []) { +VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev), +VMSTATE_UINT64 (migsize, PCIProxyDev),I think instead you should use a
VMSTATE_VBUFFERhere to save themig_data.
What we should do is thepost_saveshouldg_freethe buffer. (mig_datashould be a field in proxy).
We noticed that VMSTATE_BUFFER requires that the buffer be part of a "struct" and that the buffer is an array. Since the buffer we're using is neither an array nor part of a "struct", we decided to go with writing the buffer directly to the migration stream in proxy_post_save() instead of using VMSTATE_BUFFER.
I think we should nevertheless g_free this buffer in post_save like you pointed out.
Thank you!
--
Jag
Quoted 50 lines
Dave
Quoted 43 lines
hw/proxy/qemu-proxy.c +5
-0+ VMSTATE_END_OF_LIST() + } +}; +static void pci_proxy_dev_class_init (ObjectClass *klass,void *data) {PCIDeviceClass *k =PCI_DEVICE_CLASS (klass);@@ -471,6 +605,7 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data) k->config_write = pci_proxy_write_config; dc->reset = proxy_device_reset; + dc->vmsd = &vmstate_pci_proxy_device; }static const TypeInfo pci_proxy_dev_type_info = {include/hw/proxy/qemu-proxy.h +2
-0@@ -75,6 +75,8 @@ struct PCIProxyDev { bool need_spawn, Error **errp);ProxyMemoryRegion region[PCI_NUM_REGIONS ]; + +uint64_t migsize; }; typedef struct PCIProxyDevClass {include/io/mpqemu-link.h +1
-0@@ -63,6 +63,7 @@ typedef enum {PROXY_PING ,MMIO_RETURN ,DEVICE_RESET , +START_MIG_OUT ,MAX , } mpqemu_cmd_t;Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 41/50] multi-process/mig: Enable VMSD save in the Proxy object
Quoted 119 lines · * Jag Raman ([email protected]) wrote:
Quoted 113 lines · Dr. David Alan Gilbert
Quoted 97 lines · * Jagannathan Raman ([email protected]) wrote:
Collect the VMSD from remote process on the source and save it to the channel leading to the destination
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
v4 -> v5:
- Usingqemu_file_shutdown()instead ofqemu_thread_cancel(). Removed patch which introducedqemu_thread_cancel()hw/proxy/qemu-proxy.c | 135 ++++++++++++++++++++++++++++++++++++++++++ include/hw/proxy/qemu-proxy.h | 2 + include/io/mpqemu-link.h | 1 + 3 files changed, 138 insertions(+)hw/proxy/qemu-proxy.c +62
-0@@ -23,6 +23,14 @@#include "util/event_notifier-posix.c" #include "hw/boards.h" #include "include/qemu/log.h" +#include "io/channel.h" +#include "migration/qemu-file-types.h" +#include "qapi/error.h" +#include "io/channel-util.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "migration/migration.h" +#include "migration/vmstate.h" QEMUTimer *hb_timer;static void pci_proxy_dev_realize (PCIDevice *dev,Error **errp);@@ -35,6 +43,9 @@ static void broadcast_init(void);static int config_op_send (PCIProxyDev *dev,uint32_t addr,uint32_t *val,int l,unsigned int op); +#define PAGE_SIZE qemu_real_host_page_size +uint8_t *mig_data; +static void childsig_handler (int sig,siginfo_t *siginfo,void *ctx) {/* TODO: Add proper handler. */ @@ -460,6 +471,129 @@ static void pci_proxy_dev_inst_init(Object *obj) dev->mem_init = false; } +typedef struct { +QEMUFile *rem; +PCIProxyDev *dev; +}proxy_mig_data ; + +static void *proxy_mig_out (void *opaque) +{ +proxy_mig_data *data = opaque; +PCIProxyDev *dev = data->dev; +uint8_t byte; +uint64_t data_size =PAGE_SIZE ; + + mig_data =g_malloc (data_size); + +while (true) { + byte =qemu_get_byte (data->rem); + +if (qemu_file_get_error (data->rem)) { +break ; + } + + mig_data[dev->migsize++] = byte; +if (dev->migsize == data_size) { + data_size +=PAGE_SIZE ; + mig_data =g_realloc (mig_data, data_size); + } + } + +return NULL ; +} + +static int proxy_pre_save (void *opaque) +{ +PCIProxyDev *pdev = opaque; +proxy_mig_data *mig_data; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +QemuThread thread; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ;hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 533
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ } + +qio_channel_set_name (QIO_CHANNEL (ioc),"PCIProxyDevice-mig" ); + + f_remote =qemu_fopen_channel_input (ioc); + + pdev->migsize =0 ; + + mig_data =g_malloc0 (sizeof (proxy_mig_data)); + mig_data->rem = f_remote;This feels way too complicated. Since we know
f_remoteis always just
a simple fd we're getting we don't need to use qio orqemu_filehere; just use the fd - nice and simple.Quoted 2 lines
Then the code to read it can just use read(2) with a sensible size chunk rather than reading a byte at a time.
Quoted 7 lines
Hi Dave,
Upon closer inspection, we found that the migration code on the remote (which uses QEMUFile) could sometimes set an error on the channel using
qemu_file_set_error(). We needed to useqemu_file_get_error()to catch these errors and abort migration. So we had to stick with QEMUFile at the receiving end as well.
I realise you need to use a QEMUFile on the part that connects to the Savevm code/vmstate code - but that doesn't mean you need it on the side that just connects between your pipe and the qemu.
Quoted 97 lines
Quoted 10 lines
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 538
Quoted 5 lines
hw/proxy/qemu-proxy.c +4
-0+ mig_data->dev = pdev; + +qemu_thread_create (&thread,"Proxy MIG_OUT" , proxy_mig_out, mig_data, +QEMU_THREAD_DETACHED );I'm just checking why a thread is necessary; is this because you need to be able to start reading before you block waiting for the remote to tell you the size - worrying that if you don't start reading then the remote might block waiting for us?
Yes, Dave. That is correct.
Quoted 31 lines
Quoted 10 lines
hw/proxy/qemu-proxy.c +9
-0+ msg.cmd =START_MIG_OUT ; + msg.bytestream =0 ; + msg.num_fds =2 ; + msg.fds[0 ] = fd[1 ]; + msg.fds[1 ] =GET_REMOTE_WAIT ; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + size =wait_for_remote (msg.fds[1 ]); +PUT_REMOTE_WAIT (msg.fds[1 ]);hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 558
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ +assert (size !=ULLONG_MAX ); + +/* +* migsize is being update by a separate thread. Using volatile to +* instruct the compiler to fetch the value of this variable from +* memory during every read +*/ +while (*((volatile uint64_t *)&pdev->migsize) < size) { + }Hmm. I suggest the following:
a) You create a shared 'size' variable; and initialize it to
UINT64_MAX. b) The thread uses atomic_read(shared_size) to read that value. c) When you receive the size from the remote you do atomic_set(&shared_size, size);d) The thread does:
while (received_size<atomic_read(&shared_size))Quoted 7 lines
so the thread will quit either on EOF or it hitting the size.
e) We
pthread_joinhere to wait for the thread
f) We then check a received size to make sure it matches what we
expect.That removes the tight loop.
Sure, will do.
Quoted 26 lines
Quoted 13 lines
hw/proxy/qemu-proxy.c +12
-0+qemu_file_shutdown (f_remote); + +qemu_fclose (f_remote); +close (fd[1 ]); + +return 0 ; +} + +static int proxy_post_save (void *opaque) +{ +MigrationState *ms =migrate_get_current (); +PCIProxyDev *pdev = opaque;hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 581
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+uint64_t pos =0 ; + +while (pos < pdev->migsize) { +qemu_put_byte (ms->to_dst_file, mig_data[pos]); + pos++; + } + +qemu_fflush (ms->to_dst_file); + +return 0 ;I don't think you need that.
Quoted 16 lines
hw/proxy/qemu-proxy.c static void pci_proxy_dev_inst_init(Object *obj) · line 592
Quoted 11 lines
hw/proxy/qemu-proxy.c +11
-0+} + +const VMStateDescription vmstate_pci_proxy_device = { + .name ="PCIProxyDevice" , + .version_id =2 , + .minimum_version_id =1 , + .pre_save = proxy_pre_save, + .post_save = proxy_post_save, + .fields = (VMStateField []) { +VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev), +VMSTATE_UINT64 (migsize, PCIProxyDev),I think instead you should use a
VMSTATE_VBUFFERhere to save themig_data.
What we should do is thepost_saveshouldg_freethe buffer. (mig_datashould be a field in proxy).
Quoted 8 lines
We noticed that
VMSTATE_BUFFERrequires that the buffer be part of a "struct" and that the buffer is an array. Since the buffer we're using is neither an array nor part of a "struct", we decided to go with writing the buffer directly to the migration stream inproxy_post_save()instead of usingVMSTATE_BUFFER.I think we should nevertheless
g_freethis buffer inpost_savelike you pointed out.
Note I'm suggesting using VMSTATE_VBUFFER, not VMSTATE_BUFFER; VBUFFER is expecting a unsigned char *; it does expect that to be in your structure, so I'd expect your proxy to have a mig_data and uint32_t mig_len fields.
Dave
Quoted 50 lines
Thank you!
--
JagQuoted 45 lines
Dave
Quoted 38 lines
hw/proxy/qemu-proxy.c +5
-0+ VMSTATE_END_OF_LIST() + } +}; +static void pci_proxy_dev_class_init (ObjectClass *klass,void *data) {PCIDeviceClass *k =PCI_DEVICE_CLASS (klass);@@ -471,6 +605,7 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data) k->config_write = pci_proxy_write_config; dc->reset = proxy_device_reset; + dc->vmsd = &vmstate_pci_proxy_device; }static const TypeInfo pci_proxy_dev_type_info = {include/hw/proxy/qemu-proxy.h +2
-0@@ -75,6 +75,8 @@ struct PCIProxyDev { bool need_spawn, Error **errp);ProxyMemoryRegion region[PCI_NUM_REGIONS ]; + +uint64_t migsize; }; typedef struct PCIProxyDevClass {include/io/mpqemu-link.h +1
-0@@ -63,6 +63,7 @@ typedef enum {PROXY_PING ,MMIO_RETURN ,DEVICE_RESET , +START_MIG_OUT ,MAX , } mpqemu_cmd_t;Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v5 42/50 multi-process/mig: Send VMSD of remote to the Proxy object
In reply to Jagannathan RamanThe remote process sends the VMSD to the Proxy object, on the source side
--- migration/savevm.c | 27 +++++++++++++++++++++++++++ migration/savevm.h | 2 ++ remote/remote-main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+)
migration/savevm.c +27 -0
@@ -2942,3 +2942,30 @@ bool vmstate_check_only_migratable(const VMStateDescription *vmsd)return !(vmsd && vmsd->unmigratable); } + +int qemu_remote_savevm (QEMUFile *f) +{ +SaveStateEntry *se; +int ret; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque)) { +continue ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL ); +if (ret) { +qemu_file_set_error (f, ret); +return ret; + } + +save_section_footer (f, se); + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f); + +return 0 ; +}
migration/savevm.h +2 -0
@@ -65,4 +65,6 @@ void qemu_loadvm_state_cleanup(void);int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f); +int qemu_remote_savevm (QEMUFile *f); +#endif
remote/remote-main.c +43 -0
61 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v5 42/50] multi-process/mig: Send VMSD of remote to the Proxy object
Quoted 17 lines · * Jagannathan Raman ([email protected]) wrote:
The remote process sends the VMSD to the Proxy object, on the source side
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- migration/savevm.c | 27 +++++++++++++++++++++++++++ migration/savevm.h | 2 ++ remote/remote-main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+)migration/savevm.c +0
-0
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2945
Quoted 4 lines
migration/savevm.c +1
-0@@ -2942,3 +2942,30 @@ bool vmstate_check_only_migratable(const VMStateDescription *vmsd)return !(vmsd && vmsd->unmigratable); } +
Can we add something here commenting, e.g.
/* Called by the remote process to serialise migration back to the qemu
* */
Quoted 13 lines
migration/savevm.c +13
-0+int qemu_remote_savevm (QEMUFile *f) +{ +SaveStateEntry *se; +int ret; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque)) { +continue ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL );
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2968
Quoted 10 lines
migration/savevm.c +10
-0+if (ret) { +qemu_file_set_error (f, ret); +return ret; + } + +save_section_footer (f, se); + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f);
You have a qemu_fflush in process_start_mig_out just after you call it
- so you don't need both; I suggest you remove this one.
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2970
Quoted 1 line
migration/savevm.c +1
-0+return 0 ;
And make this return qemu_file_get_error(f); just like qemu_save_device_state and then makybe some day we can merge them.
Quoted 48 lines
migration/savevm.c +1
-0+}migration/savevm.h +2
-0@@ -65,4 +65,6 @@ void qemu_loadvm_state_cleanup(void);int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f); +int qemu_remote_savevm (QEMUFile *f); +#endif remote/remote-main.c +19
-0@@ -53,6 +53,16 @@#include "qemu/log.h" #include "qemu/cutils.h" #include "remote-opts.h" +#include "qapi/error.h" +#include "io/channel-util.h" + +#include "io/channel.h" +#include "io/channel-socket.h" +#include "migration/qemu-file-types.h" +#include "migration/savevm.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "monitor/monitor.h" #include "chardev/char.h" #include "sysemu/reset.h" @@ -322,6 +332,36 @@ static int setup_device(MPQemuMsg *msg, Error **errp) } +static void process_start_mig_out (MPQemuMsg *msg) +{ +int wait = msg->fds[1 ]; +Error *err =NULL ; +QIOChannel *ioc; +QEMUFile *f; + + ioc =qio_channel_new_fd (msg->fds[0 ], &err); +if (err) {
remote/remote-main.c static int setup_device(MPQemuMsg *msg, Error **errp) · line 353
Quoted 10 lines
remote/remote-main.c +10
-0+error_report_err (err); +return ; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"remote-migration-channel" ); + + f =qemu_fopen_channel_output (ioc); + +bdrv_drain_all (); + (void )bdrv_flush_all ();
Do remote process always have block code? I mean can't we have a remote process that's just say a NIC ?
remote/remote-main.c static int setup_device(MPQemuMsg *msg, Error **errp) · line 355
Quoted 1 line
remote/remote-main.c +1
-0+ (void )qemu_remote_savevm (f);
It's probably bad to ignore errors here; what you could do is if there's an error, you shoul dprint something and then send a poison value back to the QEMU to let it know that you've failed.
Quoted 23 lines
remote/remote-main.c +11
-0+qemu_fflush (f); + +notify_proxy (wait, (uint64_t )qemu_ftell (f)); +PUT_REMOTE_WAIT (wait); + +qemu_fclose (f); +} +static void process_msg (GIOCondition cond,MPQemuChannel *chan) {MPQemuMsg *msg =NULL ;@@ -411,6 +451,9 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)notify_proxy (msg->fds[0 ],0 ); }break ; +case START_MIG_OUT : +process_start_mig_out (msg); +break ;default :error_setg (&err,"Unknown command" ); goto finalize_loop;Signature · 1 line
1.8.3.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 42/50] multi-process/mig: Send VMSD of remote to the Proxy object
Quoted 19 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Jagannathan Raman ([email protected]) wrote:
The remote process sends the VMSD to the Proxy object, on the source side
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- migration/savevm.c | 27 +++++++++++++++++++++++++++ migration/savevm.h | 2 ++ remote/remote-main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+)migration/savevm.c +0
-0@@ -2942,3 +2942,30 @@ bool vmstate_check_only_migratable(const VMStateDescription *vmsd)
Quoted 7 lines
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2945
Quoted 3 lines
migration/savevm.c +1
-0return !(vmsd && vmsd->unmigratable); } +Can we add something here commenting, e.g.
/* Called by the remote process to serialise migration back to the qemu
* */
Will add this.
Quoted 18 lines
Quoted 18 lines
migration/savevm.c +18
-0+int qemu_remote_savevm (QEMUFile *f) +{ +SaveStateEntry *se; +int ret; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque)) { +continue ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL ); +if (ret) { +qemu_file_set_error (f, ret); +return ret; + } +
Quoted 9 lines
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2968
Quoted 5 lines
migration/savevm.c +5
-0+save_section_footer (f, se); + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f);You have a
qemu_fflushinprocess_start_mig_outjust after you call it
- so you don't need both; I suggest you remove this one.
Ok.
Quoted 5 lines
migration/savevm.c bool vmstate_check_only_migratable(const VMStateDescription *vmsd) · line 2970
Quoted 1 line
migration/savevm.c +1
-0+return 0 ;And make this return
qemu_file_get_error(f); just likeqemu_save_device_stateand then makybe some day we can merge them.
Will do.
Quoted 53 lines
Quoted 53 lines
migration/savevm.c +1
-0+}migration/savevm.h +2
-0@@ -65,4 +65,6 @@ void qemu_loadvm_state_cleanup(void);int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f); +int qemu_remote_savevm (QEMUFile *f); +#endif remote/remote-main.c +24
-0@@ -53,6 +53,16 @@#include "qemu/log.h" #include "qemu/cutils.h" #include "remote-opts.h" +#include "qapi/error.h" +#include "io/channel-util.h" + +#include "io/channel.h" +#include "io/channel-socket.h" +#include "migration/qemu-file-types.h" +#include "migration/savevm.h" +#include "migration/qemu-file-channel.h" +#include "migration/qemu-file.h" +#include "monitor/monitor.h" #include "chardev/char.h" #include "sysemu/reset.h" @@ -322,6 +332,36 @@ static int setup_device(MPQemuMsg *msg, Error **errp) } +static void process_start_mig_out (MPQemuMsg *msg) +{ +int wait = msg->fds[1 ]; +Error *err =NULL ; +QIOChannel *ioc; +QEMUFile *f; + + ioc =qio_channel_new_fd (msg->fds[0 ], &err); +if (err) { +error_report_err (err); +return ; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"remote-migration-channel" );
Quoted 8 lines
remote/remote-main.c static int setup_device(MPQemuMsg *msg, Error **errp) · line 353
Quoted 5 lines
remote/remote-main.c +5
-0+ + f =qemu_fopen_channel_output (ioc); + +bdrv_drain_all (); + (void )bdrv_flush_all ();Do remote process always have block code? I mean can't we have a remote process that's just say a NIC ?
Not always (in the future), we will account for this.
Quoted 7 lines
remote/remote-main.c static int setup_device(MPQemuMsg *msg, Error **errp) · line 355
Quoted 2 lines
remote/remote-main.c +1
-0+ (void )qemu_remote_savevm (f);It's probably bad to ignore errors here; what you could do is if there's an error, you shoul dprint something and then send a poison value back to the QEMU to let it know that you've failed.
Yes, will add this.
Quoted 27 lines
Quoted 23 lines
remote/remote-main.c +11
-0+qemu_fflush (f); + +notify_proxy (wait, (uint64_t )qemu_ftell (f)); +PUT_REMOTE_WAIT (wait); + +qemu_fclose (f); +} +static void process_msg (GIOCondition cond,MPQemuChannel *chan) {MPQemuMsg *msg =NULL ;@@ -411,6 +451,9 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)notify_proxy (msg->fds[0 ],0 ); }break ; +case START_MIG_OUT : +process_start_mig_out (msg); +break ;default :error_setg (&err,"Unknown command" ); goto finalize_loop;Signature · 1 line
1.8.3.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v5 43/50 multi-process/mig: Load VMSD in the proxy object
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
The Proxy object loads the VMSD of remote process in source and send it to the remote process in the destination
--- hw/proxy/qemu-proxy.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ include/io/mpqemu-link.h | 1 + 2 files changed, 51 insertions(+)
hw/proxy/qemu-proxy.c +50 -0
@@ -581,12 +581,62 @@ static int proxy_post_save(void *opaque)return 0 ; } +static int proxy_post_load (void *opaque,int version_id) +{ +MigrationIncomingState *mis =migration_incoming_get_current (); +PCIProxyDev *pdev = opaque; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +uint8_t byte; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"proxy-migration-channel" ); + + f_remote =qemu_fopen_channel_output (ioc); + + msg.cmd =START_MIG_IN ; + msg.bytestream =0 ; + msg.num_fds =1 ; + msg.fds[0 ] = fd[1 ]; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + + size = pdev->migsize; + +while (size) { + byte =qemu_get_byte (mis->from_src_file); +qemu_put_byte (f_remote, byte); + size--; + } + +qemu_fflush (f_remote); +qemu_fclose (f_remote); + +close (fd[1 ]); + +return 0 ; +} +const VMStateDescription vmstate_pci_proxy_device = { .name ="PCIProxyDevice" , .version_id =2 , .minimum_version_id =1 , .pre_save = proxy_pre_save, .post_save = proxy_post_save, + .post_load = proxy_post_load, .fields = (VMStateField []) {VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev),VMSTATE_UINT64 (migsize, PCIProxyDev),
include/io/mpqemu-link.h +1 -0
@@ -64,6 +64,7 @@ typedef enum {MMIO_RETURN ,DEVICE_RESET ,START_MIG_OUT , +START_MIG_IN ,MAX , } mpqemu_cmd_t;
Signature · 1 line
1.8.3.1
Re: [PATCH v5 43/50] multi-process/mig: Load VMSD in the proxy object
Quoted 49 lines · * Jagannathan Raman ([email protected]) wrote:
From: Elena Ufimtseva <[email protected]>
The Proxy object loads the VMSD of remote process in source and send it to the remote process in the destination
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/proxy/qemu-proxy.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ include/io/mpqemu-link.h | 1 + 2 files changed, 51 insertions(+)hw/proxy/qemu-proxy.c +28
-0@@ -581,12 +581,62 @@ static int proxy_post_save(void *opaque)return 0 ; } +static int proxy_post_load (void *opaque,int version_id) +{ +MigrationIncomingState *mis =migration_incoming_get_current (); +PCIProxyDev *pdev = opaque; +QEMUFile *f_remote; +MPQemuMsg msg = {0 }; +Error *err =NULL ; +QIOChannel *ioc; +uint64_t size; +uint8_t byte; +int fd[2 ]; + +if (socketpair (AF_UNIX ,SOCK_STREAM ,0 , fd)) { +return -1 ; + } + + ioc =qio_channel_new_fd (fd[0 ], &err); +if (err) { +error_report_err (err); +return -1 ; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"proxy-migration-channel" ); + + f_remote =qemu_fopen_channel_output (ioc); + + msg.cmd =START_MIG_IN ; + msg.bytestream =0 ;
hw/proxy/qemu-proxy.c static int proxy_post_save(void *opaque) · line 621
Quoted 10 lines
hw/proxy/qemu-proxy.c +10
-0+ msg.num_fds =1 ; + msg.fds[0 ] = fd[1 ]; + +mpqemu_msg_send (&msg, pdev->mpqemu_link->com); + + size = pdev->migsize; + +while (size) { + byte =qemu_get_byte (mis->from_src_file); +qemu_put_byte (f_remote, byte);
My suggestion on the previous patch about using the VMSTATE_BUFFERV means you'd malloc on size, and do the put_byte's to the remote in the post_load.
Quoted 36 lines
hw/proxy/qemu-proxy.c +12
-0+ size--; + } + +qemu_fflush (f_remote); +qemu_fclose (f_remote); + +close (fd[1 ]); + +return 0 ; +} +const VMStateDescription vmstate_pci_proxy_device = { .name ="PCIProxyDevice" , .version_id =2 , .minimum_version_id =1 , .pre_save = proxy_pre_save, .post_save = proxy_post_save, + .post_load = proxy_post_load, .fields = (VMStateField []) {VMSTATE_PCI_DEVICE (parent_dev, PCIProxyDev),VMSTATE_UINT64 (migsize, PCIProxyDev),include/io/mpqemu-link.h +1
-0@@ -64,6 +64,7 @@ typedef enum {MMIO_RETURN ,DEVICE_RESET ,START_MIG_OUT , +START_MIG_IN ,MAX , } mpqemu_cmd_t;Signature · 1 line
1.8.3.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v5 44/50 multi-process/mig: refactor runstate_check into common file
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
runstate_check file is refactored into vl-parse.c
--- Makefile.objs | 2 ++ include/sysemu/runstate.h | 2 ++ runstate.c | 36 ++++++++++++++++++++++++++++++++++++ softmmu/vl.c | 10 ---------- stubs/runstate-check.c | 3 +++ 5 files changed, 43 insertions(+), 10 deletions(-) create mode 100644 runstate.c
Makefile.objs +2 -0
@@ -40,6 +40,7 @@ remote-pci-obj-$(CONFIG_MPQEMU) += blockdev.o remote-pci-obj-$(CONFIG_MPQEMU) += qdev-monitor.o remote-pci-obj-$(CONFIG_MPQEMU) += bootdevice.o remote-pci-obj-$(CONFIG_MPQEMU) += iothread.o +remote-pci-obj-$(CONFIG_MPQEMU) += runstate.o ############################################################## # remote-lsi-obj-y is code used to implement remote LSI device
@@ -103,6 +104,7 @@ qemu-seccomp.o-libs := $(SECCOMP_LIBS) common-obj-$(CONFIG_FDT) += device_tree.o common-obj-y += vl-parse.o +common-obj-y += runstate.o ####################################################################### # qapi
include/sysemu/runstate.h +2 -0
@@ -4,6 +4,8 @@#include "qapi/qapi-types-run-state.h" #include "qemu/notify.h" +extern RunState current_run_state; +bool runstate_check (RunState state);void runstate_set (RunState new_state);int runstate_is_running (void );
runstate.c new file +36 -0
@@ -0,0 +1,36 @@ +/* +* Copyright (c) 2003-2008 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +* THE SOFTWARE. +*/ + + +#include "qemu/osdep.h" +#include "qemu-common.h" +#include "sysemu/runstate.h" + +/***********************************************************/ +/* QEMU state */ + +RunState current_run_state =RUN_STATE_PRECONFIG ; + +bool runstate_check (RunState state) +{ +return current_run_state == state; +}
softmmu/vl.c +0 -10
@@ -620,11 +620,6 @@ static int default_driver_check(void *opaque, QemuOpts *opts, Error **errp)return 0 ; } -/***********************************************************/ -/* QEMU state */ - -static RunState current_run_state =RUN_STATE_PRECONFIG ; -/* We use RUN_STATE__MAX but any invalid value will do */ static RunState vmstop_requested =RUN_STATE__MAX ;static QemuMutex vmstop_lock;
@@ -732,11 +727,6 @@ static const RunStateTransition runstate_transitions_def[] = {static bool runstate_valid_transitions[RUN_STATE__MAX ][RUN_STATE__MAX ]; -bool runstate_check (RunState state) -{ -return current_run_state == state; -} -bool runstate_store (char *str,size_t size) {const char *state =RunState_str (current_run_state);
stubs/runstate-check.c +3 -0
@@ -1,6 +1,9 @@#include "qemu/osdep.h" #include "sysemu/runstate.h" + +#pragma weak runstate_check +bool runstate_check (RunState state) {return state ==RUN_STATE_PRELAUNCH ;
Signature · 1 line
1.8.3.1
PATCH v5 45/50 multi-process/mig: Synchronize runstate of remote process
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Synchronize the runstate of the remote process with that of QEMU
--- hw/proxy/qemu-proxy.c | 26 ++++++++++++++++++++++++++ include/hw/proxy/qemu-proxy.h | 2 ++ include/io/mpqemu-link.h | 8 ++++++++ include/sysemu/runstate.h | 1 + qdev-monitor.c | 11 ++++++++++- remote/remote-main.c | 5 +++++ remote/remote-opts.c | 5 +++++ runstate.c | 5 +++++ 8 files changed, 62 insertions(+), 1 deletion(-)
hw/proxy/qemu-proxy.c +26 -0
@@ -15,6 +15,7 @@#include "hw/pci/pci.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qstring.h" +#include "sysemu/runstate.h" #include "hw/proxy/qemu-proxy.h" #include "hw/proxy/memory-sync.h" #include "qom/object.h"
@@ -765,6 +766,27 @@ static void init_proxy(PCIDevice *dev, char *command, char *exec_name, } } +static void proxy_vm_state_change (void *opaque,int running,RunState state) +{ +PCIProxyDev *dev = opaque; +MPQemuMsg msg = {0 }; +int wait =-1 ; + + msg.cmd =RUNSTATE_SET ; + msg.bytestream =0 ; + msg.size =sizeof (msg.data1); + msg.data1.runstate.state = state; + + wait =eventfd (0 ,EFD_CLOEXEC ); + msg.num_fds =1 ; + msg.fds[0 ] = wait; + +mpqemu_msg_send (&msg, dev->mpqemu_link->com); + +wait_for_remote (wait); +close (wait); +} +static void pci_proxy_dev_realize (PCIDevice *device,Error **errp) {PCIProxyDev *dev =PCI_PROXY_DEV (device);
@@ -782,6 +804,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp) } } + dev->vmcse =qemu_add_vm_change_state_handler (proxy_vm_state_change, dev); + dev->set_proxy_sock = set_proxy_sock; dev->get_proxy_sock = get_proxy_sock; dev->init_proxy = init_proxy;
@@ -807,6 +831,8 @@ static void pci_dev_exit(PCIDevice *pdev)if (!QLIST_EMPTY (&proxy_dev_list.devices)) {start_broadcast_timer (); } + +qemu_del_vm_change_state_handler (dev->vmcse); } static void send_bar_access_msg(PCIProxyDev *dev, MemoryRegion *mr,
include/hw/proxy/qemu-proxy.h +2 -0
@@ -76,6 +76,8 @@ struct PCIProxyDev {ProxyMemoryRegion region[PCI_NUM_REGIONS ]; +VMChangeStateEntry *vmcse; +uint64_t migsize; };
include/io/mpqemu-link.h +8 -0
@@ -20,6 +20,8 @@#include "exec/cpu-common.h" #include "exec/hwaddr.h" +#include "qapi/qapi-types-run-state.h" +#define TYPE_MPQEMU_LINK "mpqemu-link"#define MPQEMU_LINK (obj) \ OBJECT_CHECK(MPQemuLinkState, (obj), TYPE_MPQEMU_LINK)
@@ -65,6 +67,7 @@ typedef enum {DEVICE_RESET ,START_MIG_OUT ,START_MIG_IN , +RUNSTATE_SET ,MAX , } mpqemu_cmd_t;
@@ -115,6 +118,10 @@ typedef struct { } mmio_ret_msg_t;typedef struct { +RunState state; +}runstate_msg_t ; + +typedef struct { mpqemu_cmd_t cmd; int bytestream; size_t size;
@@ -126,6 +133,7 @@ typedef struct {set_irqfd_msg_t set_irqfd;ret_pci_info_msg_t ret_pci_info;mmio_ret_msg_t mmio_ret; +runstate_msg_t runstate; } data1;int fds[REMOTE_MAX_FDS ];
include/sysemu/runstate.h +1 -0
@@ -8,6 +8,7 @@ extern RunState current_run_state;bool runstate_check (RunState state);void runstate_set (RunState new_state); +void remote_runstate_set (RunState state);int runstate_is_running (void );bool runstate_needs_reset (void );bool runstate_store (char *str,size_t size);
qdev-monitor.c +10 -1
@@ -35,6 +35,7 @@#include "qemu/option_int.h" #include "sysemu/block-backend.h" #include "sysemu/sysemu.h" +#include "sysemu/runstate.h" #include "migration/misc.h" #include "migration/migration.h" #include "qemu/cutils.h"
@@ -674,6 +675,7 @@ DeviceState *qdev_proxy_add(const char *rid, const char *id, char *bus,const char *str;bool need_spawn = false;bool remote_exists = false; +char *command =NULL ;if (strlen (rid) >MAX_RID_LENGTH ) {error_setg (errp,"rid %s is too long." , rid);
@@ -743,11 +745,18 @@ DeviceState *qdev_proxy_add(const char *rid, const char *id, char *bus, need_spawn = true; } - pdev->init_proxy (PCI_DEVICE (ds), cmd, exec_name, need_spawn, errp); +if (runstate_check (RUN_STATE_INMIGRATE )) { + command =g_strdup_printf ("%s %s" , cmd,"-incoming defer" ); + }else { + command =g_strdup (cmd); + } + + pdev->init_proxy (PCI_DEVICE (ds), command, exec_name, need_spawn, errp);QLIST_INSERT_HEAD (&proxy_dev_list.devices, pdev, next);qemu_opts_del (proxy_opts); +return ds; }
remote/remote-main.c +5 -0
@@ -31,6 +31,7 @@#include "qemu/main-loop.h" #include "qemu/config-file.h" #include "sysemu/sysemu.h" +#include "sysemu/runstate.h" #include "block/block.h" #include "exec/ramlist.h" #include "exec/memattrs.h"
@@ -454,6 +455,10 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)case START_MIG_OUT :process_start_mig_out (msg);break ; +case RUNSTATE_SET : +remote_runstate_set (msg->data1.runstate.state); +notify_proxy (msg->fds[0 ],0 ); +break ;default :error_setg (&err,"Unknown command" ); goto finalize_loop;
remote/remote-opts.c +5 -0
@@ -40,8 +40,10 @@#include "remote/remote-opts.h" #include "include/qemu-common.h" #include "monitor/monitor.h" +#include "sysemu/runstate.h" #include "vl.h" +/* * In remote process, we parse only subset of options. The code * taken from vl.c to re-use in remote command line parser.
@@ -85,6 +87,9 @@ void parse_cmdline(int argc, char **argv, char **envp)case QEMU_OPTION_qmp:monitor_parse (optarg,"control" , false);break ; +case QEMU_OPTION_incoming: +remote_runstate_set (RUN_STATE_INMIGRATE ); +break ;case QEMU_OPTION_monitor:if (!strncmp (optarg,"stdio" ,5 )) {warn_report ("STDIO not supported in remote process" );
runstate.c +5 -0
@@ -34,3 +34,8 @@ bool runstate_check(RunState state) {return current_run_state == state; } + +void remote_runstate_set (RunState state) +{ + current_run_state = state; +}
Signature · 1 line
1.8.3.1
PATCH v5 46/50 multi-process/mig: Restore the VMSD in remote process
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
The remote process accepts the VMSD from Proxy object and restores it
--- migration/savevm.c | 36 ++++++++++++++++++++++++++++++++++++ migration/savevm.h | 1 + remote/remote-main.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+)
migration/savevm.c +36 -0
@@ -2969,3 +2969,39 @@ int qemu_remote_savevm(QEMUFile *f)return 0 ; } + +int qemu_remote_loadvm (QEMUFile *f) +{ +uint8_t section_type; +int ret =0 ; + +qemu_mutex_lock_iothread (); + +while (true) { + section_type =qemu_get_byte (f); + +if (qemu_file_get_error (f)) { + ret =qemu_file_get_error (f); +break ; + } + +switch (section_type) { +case QEMU_VM_SECTION_FULL : + ret =qemu_loadvm_section_start_full (f,NULL ); +if (ret <0 ) { +break ; + } +break ; +case QEMU_VM_EOF : + goto out; +default : + ret = -EINVAL ; + goto out; + } + } + +out: +qemu_mutex_unlock_iothread (); + +return ret; +}
migration/savevm.h +1 -0
@@ -66,5 +66,6 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_remote_savevm (QEMUFile *f); +int qemu_remote_loadvm (QEMUFile *f);#endif
remote/remote-main.c +33 -0
@@ -68,6 +68,7 @@#include "chardev/char.h" #include "sysemu/reset.h" #include "vl.h" +#include "migration/misc.h" static MPQemuLinkState *mpqemu_link;
@@ -363,6 +364,30 @@ static void process_start_mig_out(MPQemuMsg *msg)qemu_fclose (f); } +static int process_start_mig_in (MPQemuMsg *msg) +{ +Error *err =NULL ; +QIOChannel *ioc; +QEMUFile *f; +int rc = -EINVAL ; + + ioc =qio_channel_new_fd (msg->fds[0 ], &err); +if (err) { +error_report_err (err); +return rc; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"remote-migration-channel" ); + + f =qemu_fopen_channel_input (ioc); + + rc =qemu_remote_loadvm (f); + +qemu_fclose (f); + +return rc; +} +static void process_msg (GIOCondition cond,MPQemuChannel *chan) {MPQemuMsg *msg =NULL ;
@@ -455,6 +480,12 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)case START_MIG_OUT :process_start_mig_out (msg);break ; +case START_MIG_IN : +if (process_start_mig_in (msg)) { +error_setg (&err,"Incoming migration failed." ); + goto finalize_loop; + } +break ;case RUNSTATE_SET :remote_runstate_set (msg->data1.runstate.state);notify_proxy (msg->fds[0 ],0 );
@@ -532,6 +563,8 @@ int main(int argc, char *argv[]) }mpqemu_init_channel (mpqemu_link, &mpqemu_link->mmio, fd); +migration_object_init (); +parse_cmdline (argc -3 , argv +3 ,NULL );mpqemu_link_set_callback (mpqemu_link, process_msg);
Signature · 1 line
1.8.3.1
Re: [PATCH v5 46/50] multi-process/mig: Restore the VMSD in remote process
Quoted 39 lines · * Jagannathan Raman ([email protected]) wrote:
From: Elena Ufimtseva <[email protected]>
The remote process accepts the VMSD from Proxy object and restores it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- migration/savevm.c | 36 ++++++++++++++++++++++++++++++++++++ migration/savevm.h | 1 + remote/remote-main.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+)migration/savevm.c +17
-0@@ -2969,3 +2969,39 @@ int qemu_remote_savevm(QEMUFile *f)return 0 ; } + +int qemu_remote_loadvm (QEMUFile *f) +{ +uint8_t section_type; +int ret =0 ; + +qemu_mutex_lock_iothread (); + +while (true) { + section_type =qemu_get_byte (f); + +if (qemu_file_get_error (f)) { + ret =qemu_file_get_error (f); +break ; + } + +switch (section_type) {
migration/savevm.c int qemu_remote_savevm(QEMUFile *f) · line 2998
Quoted 10 lines
migration/savevm.c +10
-0+case QEMU_VM_SECTION_FULL : + ret =qemu_loadvm_section_start_full (f,NULL ); +if (ret <0 ) { +break ; + } +break ; +case QEMU_VM_EOF : + goto out; +default : + ret = -EINVAL ;
It's good to print an error there - when it goes wrong we'll want to know it.
Quoted 64 lines
migration/savevm.c +9
-0+ goto out; + } + } + +out: +qemu_mutex_unlock_iothread (); + +return ret; +}migration/savevm.h +1
-0@@ -66,5 +66,6 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_remote_savevm (QEMUFile *f); +int qemu_remote_loadvm (QEMUFile *f);#endif remote/remote-main.c +25
-0@@ -68,6 +68,7 @@#include "chardev/char.h" #include "sysemu/reset.h" #include "vl.h" +#include "migration/misc.h" static MPQemuLinkState *mpqemu_link;@@ -363,6 +364,30 @@ static void process_start_mig_out(MPQemuMsg *msg)qemu_fclose (f); } +static int process_start_mig_in (MPQemuMsg *msg) +{ +Error *err =NULL ; +QIOChannel *ioc; +QEMUFile *f; +int rc = -EINVAL ; + + ioc =qio_channel_new_fd (msg->fds[0 ], &err); +if (err) { +error_report_err (err); +return rc; + } + +qio_channel_set_name (QIO_CHANNEL (ioc),"remote-migration-channel" ); + + f =qemu_fopen_channel_input (ioc); + + rc =qemu_remote_loadvm (f); + +qemu_fclose (f); + +return rc; +} +static void process_msg (GIOCondition cond,MPQemuChannel *chan) {MPQemuMsg *msg =NULL ;
remote/remote-main.c static void process_msg(GIOCondition cond, MPQemuChannel *chan) · line 488
Quoted 9 lines
remote/remote-main.c +6
-0@@ -455,6 +480,12 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)case START_MIG_OUT :process_start_mig_out (msg);break ; +case START_MIG_IN : +if (process_start_mig_in (msg)) { +error_setg (&err,"Incoming migration failed." ); + goto finalize_loop; + } +break ;
Do you have any sanity checks to make sure what happens if you were to receive a MIG_IN while you were already running?
Dave
Quoted 15 lines
remote/remote-main.c +2
-0case RUNSTATE_SET :remote_runstate_set (msg->data1.runstate.state);notify_proxy (msg->fds[0 ],0 );@@ -532,6 +563,8 @@ int main(int argc, char *argv[]) }mpqemu_init_channel (mpqemu_link, &mpqemu_link->mmio, fd); +migration_object_init (); +parse_cmdline (argc -3 , argv +3 ,NULL );mpqemu_link_set_callback (mpqemu_link, process_msg);Signature · 1 line
1.8.3.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v5 47/50 multi-process: Enable support for multiple devices in remote
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Add support to allow multiple devices to be configured in the remote process
--- hw/proxy/qemu-proxy.c | 4 ++- include/hw/proxy/qemu-proxy.h | 3 +++ include/io/mpqemu-link.h | 1 + qdev-monitor.c | 2 ++ remote/remote-main.c | 62 +++++++++++++++++++++++++++++++++++-------- 5 files changed, 60 insertions(+), 12 deletions(-)
hw/proxy/qemu-proxy.c +3 -1
@@ -242,7 +242,7 @@ static int set_remote_opts(PCIDevice *dev, QDict *qdict, unsigned int cmd) msg.cmd = cmd; msg.bytestream =1 ; msg.size =qstring_get_length (qstr) +1 ; - + msg.id = pdev->id; wait =eventfd (0 ,EFD_NONBLOCK ); msg.num_fds =1 ;
@@ -404,6 +404,7 @@ static int config_op_send(PCIProxyDev *dev, uint32_t addr, uint32_t *val, int l, msg.size =sizeof (conf_data); msg.cmd = op; msg.bytestream =1 ; + msg.id = dev->id;if (op ==PCI_CONFIG_WRITE ) { msg.num_fds =0 ;
@@ -710,6 +711,7 @@ static void setup_irqfd(PCIProxyDev *dev)memset (&msg,0 ,sizeof (MPQemuMsg)); msg.cmd =SET_IRQFD ; + msg.id = dev->id; msg.num_fds =2 ; msg.fds[0 ] =event_notifier_get_fd (&dev->intr); msg.fds[1 ] =event_notifier_get_fd (&dev->resample);
include/hw/proxy/qemu-proxy.h +3 -0
@@ -43,6 +43,9 @@ extern const MemoryRegionOps proxy_default_ops;struct PCIProxyDev {PCIDevice parent_dev; +uint64_t id; +uint64_t nr_devices; +int n_mr_sections;MemoryRegionSection *mr_sections;
include/io/mpqemu-link.h +1 -0
@@ -124,6 +124,7 @@ typedef struct { typedef struct { mpqemu_cmd_t cmd; int bytestream; + uint64_t id; size_t size; union {
qdev-monitor.c +2 -0
@@ -735,8 +735,10 @@ DeviceState *qdev_proxy_add(const char *rid, const char *id, char *bus, pdev->mmio_sock = old_pdev->mmio_sock; pdev->remote_pid = old_pdev->remote_pid; pdev->mem_init = true; + pdev->id = old_pdev->nr_devices++; }else { pdev->socket = managed ? socket :-1 ; + pdev->id = pdev->nr_devices++; } pdev->managed = managed;
remote/remote-main.c +51 -11
111 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v5 47/50] multi-process: Enable support for multiple devices in remote
Quoted 4 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Add support to allow multiple devices to be configured in the remote process
This patch allows multiple devices to be addressed over a single UNIX domain socket. This could be a scalability/performance bottleneck because an SMP guest can only talk to 1 device at a time.
This approach doesn't address the qemu-storage-daemon use case where one device emulation process provides devices to multiple guests. Multiple UNIX domain sockets are needed for that.
Is multiplexing multiple devices over a single connection is a desirable feature? The alternative of one UNIX domain socket per device instance seems more practical to me because it should perform better and solves the qemu-storage-daemon use case.
Re: [PATCH v5 47/50] multi-process: Enable support for multiple devices in remote
Quoted 10 lines · Stefan Hajnoczi
Quoted 4 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Add support to allow multiple devices to be configured in the remote process
This patch allows multiple devices to be addressed over a single UNIX domain socket. This could be a scalability/performance bottleneck because an SMP guest can only talk to 1 device at a time.
Quoted 8 lines
This approach doesn't address the qemu-storage-daemon use case where one device emulation process provides devices to multiple guests. Multiple UNIX domain sockets are needed for that.
Is multiplexing multiple devices over a single connection is a desirable feature? The alternative of one UNIX domain socket per device instance seems more practical to me because it should perform better and solves the qemu-storage-daemon use case.
Hi Stefan,
We will implement a separate communication channel for each device in the remote process, in the next series.
Thanks!
--
Jag
Quoted 1 line
PATCH v5 48/50 multi-process: Validate incoming commands from Proxy
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Validate the incoming commands to confirm that they would not cause any errors in the remote process.
--- hw/proxy/qemu-proxy.c | 6 +++- include/io/mpqemu-link.h | 2 ++ io/mpqemu-link.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++- remote/remote-main.c | 4 +++ 4 files changed, 85 insertions(+), 2 deletions(-)
hw/proxy/qemu-proxy.c +5 -1
@@ -868,7 +868,11 @@ static void send_bar_access_msg(PCIProxyDev *dev, MemoryRegion *mr,mpqemu_msg_recv (&ret, mpqemu_link->mmio); - *val = ret.data1.mmio_ret.val; +if (!mpqemu_msg_valid (&ret)) { + *val =0 ; + }else { + *val = ret.data1.mmio_ret.val; + } } void proxy_default_bar_write(void *opaque, hwaddr addr, uint64_t val,
include/io/mpqemu-link.h +2 -0
@@ -213,4 +213,6 @@ void mpqemu_start_coms(MPQemuLinkState *s);uint64_t wait_for_remote (int efd);void notify_proxy (int fd,uint64_t val); +bool mpqemu_msg_valid (MPQemuMsg *msg); +#endif
io/mpqemu-link.c +74 -1
84 lines not on this page · raw message
remote/remote-main.c +4 -0
@@ -435,6 +435,10 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)if (msg->id >MAX_REMOTE_DEVICES ) {error_setg (&err,"id of the device is larger than max number of " \"devices per remote process." ); + } + +if (!mpqemu_msg_valid (msg)) { +error_setg (&err,"Message is not valid" ); goto finalize_loop; }
Signature · 1 line
1.8.3.1
Re: [PATCH v5 48/50] multi-process: Validate incoming commands from Proxy
Quoted 8 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Validate the incoming commands to confirm that they would not cause any errors in the remote process.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John G Johnson <[email protected]>
Quoted 6 lines
--- hw/proxy/qemu-proxy.c | 6 +++- include/io/mpqemu-link.h | 2 ++ io/mpqemu-link.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++- remote/remote-main.c | 4 +++ 4 files changed, 85 insertions(+), 2 deletions(-)
Please squash this into the patch(es) that introduced the code.
Reviewers want to see a logical sequence of patches. Introducing unsafe code in an earlier patch and adding checks in a later patch makes it impossible to review the patches in sequence (reviewers would waste time pointing out bugs that end up getting fixed later).
remote/remote-main.c static void process_msg(GIOCondition cond, MPQemuChannel *chan) · line 438
Quoted 9 lines
remote/remote-main.c +1
-0@@ -435,6 +435,10 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)if (msg->id >MAX_REMOTE_DEVICES ) {error_setg (&err,"id of the device is larger than max number of " \"devices per remote process." ); + }
Was goto finalize_loop accidentally dropped?
Re: [PATCH v5 48/50] multi-process: Validate incoming commands from Proxy
Quoted 16 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Validate the incoming commands to confirm that they would not cause any errors in the remote process.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John G Johnson <[email protected]>Quoted 6 lines
--- hw/proxy/qemu-proxy.c | 6 +++- include/io/mpqemu-link.h | 2 ++ io/mpqemu-link.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++- remote/remote-main.c | 4 +++ 4 files changed, 85 insertions(+), 2 deletions(-)
Quoted 7 lines
Please squash this into the patch(es) that introduced the code.
Reviewers want to see a logical sequence of patches. Introducing unsafe code in an earlier patch and adding checks in a later patch makes it impossible to review the patches in sequence (reviewers would waste time pointing out bugs that end up getting fixed later).
Thanks Stefan, will merge that with appropriate patches.
Quoted 11 lines
remote/remote-main.c static void process_msg(GIOCondition cond, MPQemuChannel *chan) · line 438
Quoted 9 lines
remote/remote-main.c +1
-0@@ -435,6 +435,10 @@ static void process_msg(GIOCondition cond, MPQemuChannel *chan)if (msg->id >MAX_REMOTE_DEVICES ) {error_setg (&err,"id of the device is larger than max number of " \"devices per remote process." ); + }Was goto
finalize_loopaccidentally dropped?
Yes, thank you.
Elena
PATCH v5 49/50 multi-process: add the concept description to docs/devel/qemu-multiprocess
In reply to Jagannathan RamanFrom: John G Johnson <[email protected]>
--- docs/devel/index.rst | 1 + docs/devel/qemu-multiprocess.rst | 1102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 1103 insertions(+) create mode 100644 docs/devel/qemu-multiprocess.rst
docs/devel/index.rst +1 -0
@@ -25,3 +25,4 @@ Contents: tcg-plugins bitops reset + multi-process
docs/devel/qemu-multiprocess.rst new file +1102 -0
1,102 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v5 49/50] multi-process: add the concept description to docs/devel/qemu-multiprocess
Quoted 26 lines · Jagannathan Raman
From: John G Johnson <[email protected]>
Signed-off-by John G Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- docs/devel/index.rst | 1 + docs/devel/qemu-multiprocess.rst | 1102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 1103 insertions(+) create mode 100644 docs/devel/qemu-multiprocess.rstdocs/devel/index.rst +1
-0@@ -25,3 +25,4 @@ Contents: tcg-plugins bitops reset + multi-processdocs/devel/qemu-multiprocess.rst new file +0
-0
docs/devel/qemu-multiprocess.rst line 1
Quoted 1 line
docs/devel/qemu-multiprocess.rst new file +1
-0@@ -0,0 +1,1102 @@ +Disaggregating QEMU
Please revise this document and the patch series to use consistent terminology. At least "qemu-multiprocess.rst", "--enable-mpqemu", and "disaggregated QEMU" are used to describe this feature (there are probably more, I have only looked at 2 patches so far).
It's confusing for someone who stumbles across one of these terms and then has to figure out that we're talking about the same thing when encountering other terms later on.
Please use a single name and use it consistently.
docs/devel/qemu-multiprocess.rst line 11
Quoted 10 lines
docs/devel/qemu-multiprocess.rst new file +10
-0+=================== + +QEMU is often used as the hypervisor for virtual machines running in the +Oracle cloud. Since one of the advantages of cloud computing is the +ability to run many VMs from different tenants in the same cloud +infrastructure, a guest that compromised its hypervisor could +potentially use the hypervisor's access privileges to access data it is +not authorized for. + +QEMU can be susceptible to security attack because it is a large,
s/attack/attacks/
docs/devel/qemu-multiprocess.rst line 13
Quoted 2 lines
docs/devel/qemu-multiprocess.rst new file +2
-0+monolithic program that provides many features to the VMs it services. +Many of these feature can be configured out of QEMU, but even a reduced
s/feature/features/
Re: [PATCH v5 49/50] multi-process: add the concept description to docs/devel/qemu-multiprocess
Quoted 34 lines · Stefan Hajnoczi
Quoted 26 lines · Jagannathan Raman
From: John G Johnson <[email protected]>
Signed-off-by John G Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- docs/devel/index.rst | 1 + docs/devel/qemu-multiprocess.rst | 1102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 1103 insertions(+) create mode 100644 docs/devel/qemu-multiprocess.rstdocs/devel/index.rst +1
-0@@ -25,3 +25,4 @@ Contents: tcg-plugins bitops reset + multi-processdocs/devel/qemu-multiprocess.rst new file +0
-0docs/devel/qemu-multiprocess.rst line 1
Quoted 1 line
docs/devel/qemu-multiprocess.rst new file +1
-0@@ -0,0 +1,1102 @@ +Disaggregating QEMUPlease revise this document and the patch series to use consistent terminology. At least "
qemu-multiprocess.rst", "--enable-mpqemu", and "disaggregated QEMU" are used to describe this feature (there are probably more, I have only looked at 2 patches so far).
Quoted 6 lines
It's confusing for someone who stumbles across one of these terms and then has to figure out that we're talking about the same thing when encountering other terms later on.
Please use a single name and use it consistently.
Thanks Stefan, will work on this.
Quoted 17 lines
docs/devel/qemu-multiprocess.rst line 11
Quoted 10 lines
docs/devel/qemu-multiprocess.rst new file +10
-0+=================== + +QEMU is often used as the hypervisor for virtual machines running in the +Oracle cloud. Since one of the advantages of cloud computing is the +ability to run many VMs from different tenants in the same cloud +infrastructure, a guest that compromised its hypervisor could +potentially use the hypervisor's access privileges to access data it is +not authorized for. + +QEMU can be susceptible to security attack because it is a large,s/attack/attacks/
docs/devel/qemu-multiprocess.rst line 13
Quoted 2 lines
docs/devel/qemu-multiprocess.rst new file +2
-0+monolithic program that provides many features to the VMs it services. +Many of these feature can be configured out of QEMU, but even a reduceds/feature/features/
PATCH v5 50/50 multi-process: add configure and usage information
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
--- docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/qemu-multiprocess.txt
docs/qemu-multiprocess.txt new file +86 -0
@@ -0,0 +1,86 @@ +Multi-process QEMU +================== + +This document describes how to configure and use multi-process qemu. +For the design document refer to docs/devel/qemu-multiprocess. + +1) Configuration +---------------- + +To enable support for multi-process add --enable-mpqemu +to the list of options for the "configure" script. + + +2) Usage +-------- + +To start qemu with devices intended to run in a separate emulation +process without libvirtd support, the following should be used on QEMU +command line. As of now, we only support the emulation of lsi53c895a +in a separate process + +* Since parts of the RAM are shared between QEMU & remote process, a + memory-backend-file is required to facilitate this, as follows: + + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on + +* The devices to be emulated in the separate process are defined as + before with addition of "rid" suboption that serves as a remote group + identificator. + + -device <device options>,rid="remote process id" + + For example, for non multi-process qemu: + -device lsi53c895a,id=scsi0 device + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 + -drive id=drive0,file=data-disk.img + + and for multi-process qemu and no libvirt + support (i.e. QEMU forks child processes): + -device lsi53c895a,id=scsi0,rid=0 + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0 + +* The command-line options for the remote process are added to the "command" + suboption of the newly added "-remote" option. + + -remote [socket],rid=0,exec="...",command="..." + + The drives to be emulated by the remote process are specified as part of + this command sub-option. The device to be used to connect to the monitor + is also specified as part of this suboption. + + For example, the following option adds a drive and monitor to the remote + process: + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" + + Note: There's an issue with this "command" sub-option which we are in the + process of fixing. To work around this issue, it requires additional + "comma" characters as illustrated above, and in the example below. + +* Example QEMU command-line to launch lsi53c895a in a remote process + + #/bin/sh + qemu-system-x86_64 \ + -name "OL7.4" \ + -machine q35,accel=kvm \ + -smp sockets=1,cores=1,threads=1 \ + -cpu host \ + -m 2048 \ + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \ + -numa node,memdev=mem \ + -device virtio-scsi-pci,id=virtio_scsi_pci0 \ + -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \ + -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \ + -boot d \ + -monitor stdio \ + -vnc :0 \ + -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \ + -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\ + -remote rid=8,exec="qemu-scsi-dev",command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" + + We could connect to the monitor using the following command: + socat /home/qmp-sock stdio + + After hotplugging disks to the remote process, please execute the + following command in the guest to refresh the list of storage devices: + rescan_scsi_bus.sh -a
Signature · 1 line
1.8.3.1
Re: [PATCH v5 50/50] multi-process: add configure and usage information
Quoted 31 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John G Johnson <[email protected]>--- docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/qemu-multiprocess.txtdocs/qemu-multiprocess.txt new file +15
-0@@ -0,0 +1,86 @@ +Multi-process QEMU +================== + +This document describes how to configure and use multi-process qemu. +For the design document refer to docs/devel/qemu-multiprocess. + +1) Configuration +---------------- + +To enable support for multi-process add --enable-mpqemu +to the list of options for the "configure" script. + + +2) Usage +--------
docs/qemu-multiprocess.txt line 25
Quoted 10 lines
docs/qemu-multiprocess.txt new file +10
-0+ +To start qemu with devices intended to run in a separate emulation +process without libvirtd support, the following should be used on QEMU +command line. As of now, we only support the emulation of lsi53c895a +in a separate process + +* Since parts of the RAM are shared between QEMU & remote process, a + memory-backend-file is required to facilitate this, as follows: + + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
memory-backend-memfd is more convenient. It doesn't require a mem-path and share=on is the default.
Quoted 6 lines
docs/qemu-multiprocess.txt new file +6
-0+ +* The devices to be emulated in the separate process are defined as + before with addition of "rid" suboption that serves as a remote group + identificator. + + -device <device options>,rid="remote process id"
docs/qemu-multiprocess.txt line 41
Quoted 10 lines
docs/qemu-multiprocess.txt new file +10
-0+ + For example, for non multi-process qemu: + -device lsi53c895a,id=scsi0 device + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 + -drive id=drive0,file=data-disk.img + + and for multi-process qemu and no libvirt + support (i.e. QEMU forks child processes): + -device lsi53c895a,id=scsi0,rid=0 + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0
This approach is invasive:
* lsi53c895a should not need to be modified with a new rid= option.
* QEMU should not know about the scsi-hd device or drive0. Only the device emulation process needs to know about scsi-hd.
In order to cleanly separate QEMU and the device emulation process syntax like this is needed:
-object remote-device,id=rid0,...
-device remote-pci-device,id=scsi0,remote-device=rid0
The "remote-device" object could be part of remote-pci-device, but keeping it separate may be useful in the future in order to support things like reconnection.
The generic "remote-pci-device" device handles any remote PCI device, not just the LSI SCSI controller.
Do you agree with this approach?
docs/qemu-multiprocess.txt line 46
Quoted 4 lines
docs/qemu-multiprocess.txt new file +4
-0+* The command-line options for the remote process are added to the "command" + suboption of the newly added "-remote" option. + + -remote [socket],rid=0,exec="...",command="..."
QEMU has been using the -object TYPE syntax instead of adding new -TYPE command-line options. This gives you object_add hotplug for free, for example. I suggest using -object remote-device,id=,exec=,command=, instead of -remote.
docs/qemu-multiprocess.txt line 58
Quoted 12 lines
docs/qemu-multiprocess.txt new file +12
-0+ + The drives to be emulated by the remote process are specified as part of + this command sub-option. The device to be used to connect to the monitor + is also specified as part of this suboption. + + For example, the following option adds a drive and monitor to the remote + process: + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" + + Note: There's an issue with this "command" sub-option which we are in the + process of fixing. To work around this issue, it requires additional + "comma" characters as illustrated above, and in the example below.
command= (which could be called args= for clarity) will be difficult to use not just because of comma escaping but also because of double-quote escaping. How do you pass a command-line argument that contains spaces?
Re: [PATCH v5 50/50] multi-process: add configure and usage information
Quoted 37 lines · Stefan Hajnoczi
Quoted 36 lines · Jagannathan Raman
From: Elena Ufimtseva <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John G Johnson <[email protected]>--- docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/qemu-multiprocess.txtdocs/qemu-multiprocess.txt new file +20
-0@@ -0,0 +1,86 @@ +Multi-process QEMU +================== + +This document describes how to configure and use multi-process qemu. +For the design document refer to docs/devel/qemu-multiprocess. + +1) Configuration +---------------- + +To enable support for multi-process add --enable-mpqemu +to the list of options for the "configure" script. + + +2) Usage +-------- + +To start qemu with devices intended to run in a separate emulation +process without libvirtd support, the following should be used on QEMU +command line. As of now, we only support the emulation of lsi53c895a +in a separate process
Quoted 9 lines
docs/qemu-multiprocess.txt line 25
Quoted 5 lines
docs/qemu-multiprocess.txt new file +5
-0+ +* Since parts of the RAM are shared between QEMU & remote process, a + memory-backend-file is required to facilitate this, as follows: + + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=onmemory-backend-memfd is more convenient. It doesn't require a mem-path and share=on is the default.
Will change this.
Quoted 28 lines
Quoted 6 lines
docs/qemu-multiprocess.txt new file +6
-0+ +* The devices to be emulated in the separate process are defined as + before with addition of "rid" suboption that serves as a remote group + identificator. + + -device <device options>,rid="remote process id"docs/qemu-multiprocess.txt line 41
Quoted 10 lines
docs/qemu-multiprocess.txt new file +10
-0+ + For example, for non multi-process qemu: + -device lsi53c895a,id=scsi0 device + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 + -drive id=drive0,file=data-disk.img + + and for multi-process qemu and no libvirt + support (i.e. QEMU forks child processes): + -device lsi53c895a,id=scsi0,rid=0 + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0This approach is invasive:
* lsi53c895a should not need to be modified with a new rid= option.
* QEMU should not know about the scsi-hd device or drive0. Only the device emulation process needs to know about scsi-hd.In order to cleanly separate QEMU and the device emulation process syntax like this is needed:
-object remote-device,id=rid0,...
-device remote-pci-device,id=scsi0,remote-device=rid0
Quoted 9 lines
The "remote-device" object could be part of remote-pci-device, but keeping it separate may be useful in the future in order to support things like reconnection.
The generic "remote-pci-device" device handles any remote PCI device, not just the LSI SCSI controller.
Do you agree with this approach?
We discussed these changes and they seem to be along the lines with the future work on vfio over socket approach we will be working on later.
Could we for this experimental version have the changes you propose here with one modification - instead of having generic remote-pci-device imply that that is LSI device? And while we work towards vfio over socket this will become any remote PCI device?
Quoted 10 lines
docs/qemu-multiprocess.txt line 46
Quoted 4 lines
docs/qemu-multiprocess.txt new file +4
-0+* The command-line options for the remote process are added to the "command" + suboption of the newly added "-remote" option. + + -remote [socket],rid=0,exec="...",command="..."QEMU has been using the -object TYPE syntax instead of adding new -TYPE command-line options. This gives you
object_addhotplug for free, for example. I suggest using -object remote-device,id=,exec=,command=, instead of -remote.
We will add these changes.
Quoted 8 lines
Quoted 8 lines
docs/qemu-multiprocess.txt new file +8
-0+ + The drives to be emulated by the remote process are specified as part of + this command sub-option. The device to be used to connect to the monitor + is also specified as part of this suboption. + + For example, the following option adds a drive and monitor to the remote + process: + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
Quoted 8 lines
docs/qemu-multiprocess.txt line 58
Quoted 4 lines
docs/qemu-multiprocess.txt new file +4
-0+ + Note: There's an issue with this "command" sub-option which we are in the + process of fixing. To work around this issue, it requires additional + "comma" characters as illustrated above, and in the example below.command= (which could be called args= for clarity) will be difficult to use not just because of comma escaping but also because of double-quote escaping. How do you pass a command-line argument that contains spaces?
Yes, this is not great. And spaces are the problem at the moment. I am looking if the -object has some properties that can allow for arbitrary strings. Maybe such as data for "secret" object would do?
Elena
Re: [PATCH v5 50/50] multi-process: add configure and usage information
Quoted 39 lines · Elena Ufimtseva
Quoted 28 lines · Stefan Hajnoczi
Quoted 5 lines · Jagannathan Raman
docs/qemu-multiprocess.txt new file +5
-0+* The devices to be emulated in the separate process are defined as + before with addition of "rid" suboption that serves as a remote group + identificator. + + -device <device options>,rid="remote process id"docs/qemu-multiprocess.txt line 41
Quoted 10 lines
docs/qemu-multiprocess.txt new file +10
-0+ + For example, for non multi-process qemu: + -device lsi53c895a,id=scsi0 device + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 + -drive id=drive0,file=data-disk.img + + and for multi-process qemu and no libvirt + support (i.e. QEMU forks child processes): + -device lsi53c895a,id=scsi0,rid=0 + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0This approach is invasive:
* lsi53c895a should not need to be modified with a new rid= option.
* QEMU should not know about the scsi-hd device or drive0. Only the device emulation process needs to know about scsi-hd.In order to cleanly separate QEMU and the device emulation process syntax like this is needed:
-object remote-device,id=rid0,...
-device remote-pci-device,id=scsi0,remote-device=rid0Quoted 9 lines
The "remote-device" object could be part of remote-pci-device, but keeping it separate may be useful in the future in order to support things like reconnection.
The generic "remote-pci-device" device handles any remote PCI device, not just the LSI SCSI controller.
Do you agree with this approach?
Quoted 7 lines
We discussed these changes and they seem to be along the lines with the future work on vfio over socket approach we will be working on later.
Could we for this experimental version have the changes you propose here with one modification - instead of having generic remote-pci-device imply that that is LSI device? And while we work towards vfio over socket this will become any remote PCI device?
Yes, that sounds good.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
docs/qemu-multiprocess.txt new file +7
-0+ The drives to be emulated by the remote process are specified as part of + this command sub-option. The device to be used to connect to the monitor + is also specified as part of this suboption. + + For example, the following option adds a drive and monitor to the remote + process: + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"Quoted 8 lines
docs/qemu-multiprocess.txt line 58
Quoted 4 lines
docs/qemu-multiprocess.txt new file +4
-0+ + Note: There's an issue with this "command" sub-option which we are in the + process of fixing. To work around this issue, it requires additional + "comma" characters as illustrated above, and in the example below.command= (which could be called args= for clarity) will be difficult to use not just because of comma escaping but also because of double-quote escaping. How do you pass a command-line argument that contains spaces?
Quoted 3 lines
Yes, this is not great. And spaces are the problem at the moment. I am looking if the -object has some properties that can allow for arbitrary strings. Maybe such as data for "secret" object would do?
I'm not aware of a way to avoid the comma escaping. The space escaping is a question of how the remote process is spawned. If the command-line is processed by a shell like with system(3) then backslash can be used to escape spaces.
Stefan
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series seems to have some coding style problems. See output below for more information:
Subject: [PATCH v5 00/50] Initial support for multi-process qemu
Message-id: [email protected]
Type: series
=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
046649c multi-process: add configure and usage information
7c79419 multi-process: add the concept description to docs/devel/qemu-multiprocess b06e676 multi-process: Validate incoming commands from Proxy
cc1f6b4 multi-process: Enable support for multiple devices in remote 0060f81 multi-process/mig: Restore the VMSD in remote process
f163e9b multi-process/mig: Synchronize runstate of remote process 8bc3796 multi-process/mig: refactor runstate_check into common file 4ebbd74 multi-process/mig: Load VMSD in the proxy object
4b124b2 multi-process/mig: Send VMSD of remote to the Proxy object b8dbad8 multi-process/mig: Enable VMSD save in the Proxy object 3253157 multi-process/mig: build migration module in the remote process 96557d0 multi-process: prevent duplicate memory initialization in remote 9a85af0 multi-process/mon: Initialize QMP module for remote processes ca70d4a multi-process/mon: Refactor monitor/chardev functions out of vl.c dbe3a63 multi-process/mon: enable QMP module support in the remote process a12a802 multi-process/mon: stub functions to enable QMP module for remote process 1e6d3af multi-process/mon: choose HMP commands based on target 888e741 multi-process: perform device reset in the remote process 8f15bd8 multi-process: Use separate MMIO communication channel 64593f5 multi-process: handle heartbeat messages in remote process 93b0f96 multi-process: send heartbeat messages to remote
ad214e9 multi-process: add parse_cmdline in remote process
83bc352 multi-process: add remote options parser
7fcc704 multi-process: add remote option
24ca162 multi-process: refractor vl.c code
20c05d9 multi-process: Introduce build flags to separate remote process code 335f331 multi-process: add processing of remote device command line 413bf28 multi-process: remote: add create_done condition
d0e4f00 multi-process: remote: use fd for socket from parent process 045f8f8 multi-process: remote: add setup_devices msg processing a8ea8aa multi-process: add qdev_proxy_add to create proxy devices 59d0bff multi-process: Retrieve PCI info from remote process
898af59 multi-process: configure remote side devices
b49c9f3 multi-process: create IOHUB object to handle irq
06dc766 multi-process: Synchronize remote memory
a88dcbd multi-process: PCI BAR read/write handling for proxy & remote endpoints 5a3fbbb mutli-process: build remote command line args
c319ce7 multi-process: introduce proxy object
7cb950d multi-process: remote process initialization
e91592e multi-process: setup memory manager for remote device
4f14ba4 multi-process: setup a machine object for remote device process b1e58b1 multi-process: setup PCI host bridge for remote device 2785043 multi-process: add functions to synchronize proxy and remote endpoints f83d3f3 multi-process: define mpqemu-link object
252d736 multi-process: build system for remote device process
1482849 multi-process: Add config option for multi-process QEMU b78f0cc multi-process: Add stub functions to facilate build of multi-process 837f4a6 multi-process: add a command line option for debug file e58fa6d multi-process: Refactor machine_init and exit notifiers bf5b343 multi-process: memory: alloc RAM from file at offset
=== OUTPUT BEGIN ===
1/50 Checking commit bf5b3438f584 (multi-process: memory: alloc RAM from file at offset) 2/50 Checking commit e58fa6db276f (multi-process: Refactor machine_init and exit notifiers) 3/50 Checking commit 837f4a602862 (multi-process: add a command line option for debug file) 4/50 Checking commit b78f0cc3f072 (multi-process: Add stub functions to facilate build of multi-process)
ERROR: suspect code indent for conditional statements (4, 4)
#141: FILE: accel/stubs/tcg-stub.c:109:
+ while (1) {
+ }
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #155:
new file mode 100644
total: 1 errors, 1 warnings, 378 lines checked
Patch 4/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
5/50 Checking commit 1482849e9584 (multi-process: Add config option for multi-process QEMU) 6/50 Checking commit 252d7366e5c1 (multi-process: build system for remote device process)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #311:
new file mode 100644
total: 0 errors, 1 warnings, 250 lines checked
Patch 6/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
7/50 Checking commit f83d3f37432f (multi-process: define mpqemu-link object)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #20:
new file mode 100644
total: 0 errors, 1 warnings, 437 lines checked
Patch 7/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
8/50 Checking commit 278504388c18 (multi-process: add functions to synchronize proxy and remote endpoints) 9/50 Checking commit b1e58b14c4d2 (multi-process: setup PCI host bridge for remote device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#31:
new file mode 100644
total: 0 errors, 1 warnings, 118 lines checked
Patch 9/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
10/50 Checking commit 4f14ba434d5b (multi-process: setup a machine object for remote device process)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#63:
new file mode 100644
total: 0 errors, 1 warnings, 169 lines checked
Patch 10/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
11/50 Checking commit e91592e1d3e5 (multi-process: setup memory manager for remote device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #122:
new file mode 100644
total: 0 errors, 1 warnings, 163 lines checked
Patch 11/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
12/50 Checking commit 7cb950d0b414 (multi-process: remote process initialization) 13/50 Checking commit c319ce7e8e42 (multi-process: introduce proxy object)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #34:
new file mode 100644
WARNING: line over 80 characters
#78: FILE: hw/proxy/qemu-proxy.c:33:
+ error_setg(errp, "Remote processed is managed and launched by external program");
total: 0 errors, 2 warnings, 338 lines checked
Patch 13/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
14/50 Checking commit 5a3fbbb83e89 (mutli-process: build remote command line args)
15/50 Checking commit a88dcbd3b371 (multi-process: PCI BAR read/write handling for proxy & remote endpoints) 16/50 Checking commit 06dc7666cb21 (multi-process: Synchronize remote memory)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#36:
new file mode 100644
total: 0 errors, 1 warnings, 323 lines checked
Patch 16/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/50 Checking commit b49c9f3a3655 (multi-process: create IOHUB object to handle irq)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #43:
deleted file mode 100644
total: 0 errors, 1 warnings, 411 lines checked
Patch 17/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
18/50 Checking commit 898af59b01ef (multi-process: configure remote side devices)
19/50 Checking commit 59d0bff02614 (multi-process: Retrieve PCI info from remote process) 20/50 Checking commit a8ea8aa3417d (multi-process: add qdev_proxy_add to create proxy devices) 21/50 Checking commit 045f8f82de1c (multi-process: remote: add setup_devices msg processing) 22/50 Checking commit d0e4f00c3065 (multi-process: remote: use fd for socket from parent process) 23/50 Checking commit 413bf287c459 (multi-process: remote: add create_done condition) 24/50 Checking commit 335f331eed39 (multi-process: add processing of remote device command line) 25/50 Checking commit 20c05d9deccc (multi-process: Introduce build flags to separate remote process code) 26/50 Checking commit 24ca162587f0 (multi-process: refractor vl.c code)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#189:
new file mode 100644
total: 0 errors, 1 warnings, 326 lines checked
Patch 26/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
27/50 Checking commit 7fcc7047eaa0 (multi-process: add remote option) 28/50 Checking commit 83bc352512c7 (multi-process: add remote options parser) 29/50 Checking commit ad214e916e4d (multi-process: add parse_cmdline in remote process)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #59:
new file mode 100644
ERROR: line over 90 characters
#135: FILE: remote/remote-opts.c:72:
+ error_report("Option not supported for this target, %x arch_mask, %x arch_type",
total: 1 errors, 1 warnings, 148 lines checked
Patch 29/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
30/50 Checking commit 93b0f96ec29b (multi-process: send heartbeat messages to remote)
31/50 Checking commit 64593f51a86c (multi-process: handle heartbeat messages in remote process) 32/50 Checking commit 8f15bd8412a3 (multi-process: Use separate MMIO communication channel) 33/50 Checking commit 888e741a19af (multi-process: perform device reset in the remote process) 34/50 Checking commit 1e6d3af047e6 (multi-process/mon: choose HMP commands based on target) 35/50 Checking commit a12a80202f4e (multi-process/mon: stub functions to enable QMP module for remote process)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#110:
new file mode 100644
ERROR: line over 90 characters
#550: FILE: stubs/qapi-target.c:48:
+#if defined(TARGET_PPC) || defined(TARGET_ARM) || defined(TARGET_I386) || defined(TARGET_S390X) || defined(TARGET_MIPS)
total: 1 errors, 1 warnings, 730 lines checked
Patch 35/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
36/50 Checking commit dbe3a637fac7 (multi-process/mon: enable QMP module support in the remote process) 37/50 Checking commit ca70d4a7859d (multi-process/mon: Refactor monitor/chardev functions out of vl.c) 38/50 Checking commit 9a85af09cf01 (multi-process/mon: Initialize QMP module for remote processes) 39/50 Checking commit 96557d09787c (multi-process: prevent duplicate memory initialization in remote) 40/50 Checking commit 32531572e8c7 (multi-process/mig: build migration module in the remote process) 41/50 Checking commit b8dbad8073b6 (multi-process/mig: Enable VMSD save in the Proxy object)
ERROR: suspect code indent for conditional statements (4, 4)
#131: FILE: hw/proxy/qemu-proxy.c:557:
+ while (*((volatile uint64_t *)&pdev->migsize) < size) {
+ }
total: 1 errors, 0 warnings, 174 lines checked
Patch 41/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
42/50 Checking commit 4b124b25aded (multi-process/mig: Send VMSD of remote to the Proxy object) 43/50 Checking commit 4ebbd744f34b (multi-process/mig: Load VMSD in the proxy object) 44/50 Checking commit 8bc37967dcd8 (multi-process/mig: refactor runstate_check into common file)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#48:
new file mode 100644
total: 0 errors, 1 warnings, 89 lines checked
Patch 44/50 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
45/50 Checking commit f163e9b02e25 (multi-process/mig: Synchronize runstate of remote process) 46/50 Checking commit 0060f814e9c8 (multi-process/mig: Restore the VMSD in remote process) 47/50 Checking commit cc1f6b4970d4 (multi-process: Enable support for multiple devices in remote) 48/50 Checking commit b06e676c2119 (multi-process: Validate incoming commands from Proxy) 49/50 Checking commit 7c794195319d (multi-process: add the concept description to docs/devel/qemu-multiprocess)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#22:
new file mode 100644
total: 0 errors, 1 warnings, 1106 lines checked
Patch 49/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
50/50 Checking commit 046649c71c87 (multi-process: add configure and usage information)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #13:
new file mode 100644
total: 0 errors, 1 warnings, 86 lines checked
Patch 50/50 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===
Test command exited with code: 1
The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===
CC stubs/sysbus.o
CC stubs/tpm.o
Warning, treated as error:
/tmp/qemu-test/src/docs/devel/index.rst:13:toctree contains reference to nonexisting document 'multi-process'
CC stubs/trace-control.o
CC stubs/uuid.o
CC stubs/vm-stop.o
make: *** [Makefile:1063: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
In file included from /tmp/qemu-test/src/include/monitor/qdev.h:4,
from /tmp/qemu-test/src/stubs/monitor.c:14:
/tmp/qemu-test/src/include/hw/proxy/qemu-proxy.h:12:10: fatal error: linux/kvm.h: No such file or directory
#include <linux/kvm.h>
^~~~~~~~~~~~~compilation terminated.
make: *** [/tmp/qemu-test/src/rules.mak:69: stubs/monitor.o] Error 1 Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4f8cf4b3c5d9425cb7ce2e50962601a2', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-gx6fxsj2/src/docker-src.2020-02-24-17.00.12.3070:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=4f8cf4b3c5d9425cb7ce2e50962601a2
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-gx6fxsj2/src'
make: *** [docker-run-test-mingw@fedora] Error 2
real 3m34.769s
user 0m4.558s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
qemu-system-aarch64: falling back to tcg
TEST check-unit: tests/test-char
**
ERROR:/tmp/qemu-test/src/tests/test-char.c:230:char_mux_test: assertion failed (h1.last_event == 42): (3 == 42)
TEST iotest-qcow2: 004
qemu-system-aarch64: -accel kvm: invalid accelerator kvm
qemu-system-aarch64: falling back to tcg
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/test-char.c:230:char_mux_test: assertion failed (h1.last_event == 42): (3 == 42)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
TEST check-qtest-x86_64: tests/qtest/fdc-test
TEST iotest-qcow2: 005
---
Not run: 259
Failures: 030 040 041 071 099 120 184 186 267
Failed 9 of 116 iotests
make: *** [check-tests/check-block.sh] Error 1
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=084f448bfb414d66a22a2ccf133f7163', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-0vhz47_c/src/docker-src.2020-02-24-17.09.16.1174:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=084f448bfb414d66a22a2ccf133f7163
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-0vhz47_c/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 14m39.906s
user 0m8.340s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Quoted 28 lines · Jagannathan Raman
Hello
Started with the presentation in October 2017 made by Marc-Andre (Red Hat) and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018, the multi-process project is now available and presented in this patchset. This first series enables the emulation of lsi53c895a in a separate process.
We posted the Proof Of Concept patches [2] before the BoF session in 2018. Subsequently, we posted RFC v1 [3], RFC v2 [4], RFC v3 [5] and RFC v4 [6].
John & Elena presented the status of this project in KVM Forum 2019. We appreciate the in-person and email feedback we received to improve this patchset. We also received valuable feedback and direction on future improvements from the bi-weekly KVM community conference. We have incorporated all the feedback in the current version of the series, v5.
Following people contributed to this patchset:
John G Johnson <[email protected]>
Jagannathan Raman <[email protected]>
Elena Ufimtseva <[email protected]>
Kanth Ghatraju <[email protected]>
Konrad Wilk <[email protected]>For full concept writeup about QEMU disaggregation, refer to
docs/devel/qemu-multiprocess.rst. Please refer todocs/qemu-multiprocess.txtfor usage information.
Quoted 8 lines
We are planning on making the following improvements in the future to the experimental Qemu multi-process:
- Asynchronous communication channel;
- Performance improvements;
- Libvirt support;
- Enforcement of security policies and privileges control;We welcome all your ideas, concerns, and questions for this patchset.
There seem to be quite a few CI failures with this series applied:
https://travis-ci.org/stsquad/qemu/builds/656432858
https://app.shippable.com/github/stsquad/qemu/runs/1275/summary/console
https://gitlab.com/stsquad/qemu/pipelines/122030403
https://cirrus-ci.com/build/4577637150490624
Signature · 1 line
Alex Bennée
Quoted 38 lines · Alex Bennée
Quoted 35 lines · Jagannathan Raman
Hello
Started with the presentation in October 2017 made by Marc-Andre (Red Hat) and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018, the multi-process project is now available and presented in this patchset. This first series enables the emulation of lsi53c895a in a separate process.
We posted the Proof Of Concept patches [2] before the BoF session in 2018. Subsequently, we posted RFC v1 [3], RFC v2 [4], RFC v3 [5] and RFC v4 [6].
John & Elena presented the status of this project in KVM Forum 2019. We appreciate the in-person and email feedback we received to improve this patchset. We also received valuable feedback and direction on future improvements from the bi-weekly KVM community conference. We have incorporated all the feedback in the current version of the series, v5.
Following people contributed to this patchset:
John G Johnson <[email protected]>
Jagannathan Raman <[email protected]>
Elena Ufimtseva <[email protected]>
Kanth Ghatraju <[email protected]>
Konrad Wilk <[email protected]>For full concept writeup about QEMU disaggregation, refer to
docs/devel/qemu-multiprocess.rst. Please refer todocs/qemu-multiprocess.txtfor usage information.We are planning on making the following improvements in the future to the experimental Qemu multi-process:
- Asynchronous communication channel;
- Performance improvements;
- Libvirt support;
- Enforcement of security policies and privileges control;
Quoted 8 lines
Quoted 1 line
We welcome all your ideas, concerns, and questions for this patchset.
There seem to be quite a few CI failures with this series applied:
https://travis-ci.org/stsquad/qemu/builds/656432858 https://app.shippable.com/github/stsquad/qemu/runs/1275/summary/console https://gitlab.com/stsquad/qemu/pipelines/122030403 https://cirrus-ci.com/build/4577637150490624
Hi Alex,
Thanks for pointing it out.
"Patchew" also identified some errors which we are working on fixing for the next version. Patchew summarized the errors in the following page:
https://patchew.org/QEMU/[email protected]/
To confirm we're compliant with Patchew, we are running docker tests before sending the patches for review next time around.
We'll use the following wiki to trigger "travis-ci" tests before pushing the branch for review next time around:
https://wiki.qemu.org/Testing/CI/Travis#Testing_Changes_to_Travis
Are shippable, stsquad & cirrus-ci redundant if travis-ci & docker tests pass?
Thank you very much!
--
Jag
Quoted 1 line
Quoted 56 lines · Jag Raman
Quoted 37 lines · Alex Bennée
Quoted 35 lines · Jagannathan Raman
Hello
Started with the presentation in October 2017 made by Marc-Andre
(Red Hat)
and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018, the multi-process project is now available and presented in this patchset. This first series enables the emulation of lsi53c895a in a separate process.We posted the Proof Of Concept patches [2] before the BoF session in 2018. Subsequently, we posted RFC v1 [3], RFC v2 [4], RFC v3 [5] and RFC v4 [6].
John & Elena presented the status of this project in KVM Forum 2019. We appreciate the in-person and email feedback we received to improve this patchset. We also received valuable feedback and direction on future improvements from the bi-weekly KVM community conference. We have incorporated all the feedback in the current version of the series, v5.
Following people contributed to this patchset:
John G Johnson <[email protected]>
Jagannathan Raman <[email protected]>
Elena Ufimtseva <[email protected]>
Kanth Ghatraju <[email protected]>
Konrad Wilk <[email protected]>For full concept writeup about QEMU disaggregation, refer to
docs/devel/qemu-multiprocess.rst. Please refer todocs/qemu-multiprocess.txtfor usage information.We are planning on making the following improvements in the future to the experimental Qemu multi-process:
- Asynchronous communication channel;
- Performance improvements;
- Libvirt support;
- Enforcement of security policies and privileges control;Quoted 6 lines
Quoted 1 line
We welcome all your ideas, concerns, and questions for this patchset.
There seem to be quite a few CI failures with this series applied:
https://travis-ci.org/stsquad/qemu/builds/656432858 https://app.shippable.com/github/stsquad/qemu/runs/1275/summary/console https://gitlab.com/stsquad/qemu/pipelines/122030403 https://cirrus-ci.com/build/4577637150490624Hi Alex,
Thanks for pointing it out.
"Patchew" also identified some errors which we are working on fixing for the next version. Patchew summarized the errors in the following page:
https://patchew.org/QEMU/[email protected]/To confirm we're compliant with Patchew, we are running docker tests before sending the patches for review next time around.
Quoted 6 lines
We'll use the following wiki to trigger "travis-ci" tests before pushing the branch for review next time around:
https://wiki.qemu.org/Testing/CI/Travis#Testing_Changes_to_TravisAre shippable, stsquad & cirrus-ci redundant if travis-ci & docker tests
stsquad is just my user account, hopefully I'm not redundant ;-)
They all test slightly different things but you should be able to replicate the tests locally.
Travis basically tests a bunch of different configuration setups on mostly x86 hardware. Unless it's a weird library interaction issue this should replicate in your normal build environment.
Shippable are cross compile tests. They use the existing docker infrastructure to cross compile for various target architectures. See "make docker" and the notes in docs/devel/testing.rst.
CirrusCI tests MacOS and FreeBSD builds. You can build on the BSD's yourself, see "make vm-help". MacOSX is trickier unless you have a Mac yourself of course.
Quoted 3 lines
pass?
Thank you very much!
Signature · 1 line
Alex Bennée
Quoted 59 lines · Alex Bennée
Quoted 56 lines · Jag Raman
Quoted 37 lines · Alex Bennée
Quoted 35 lines · Jagannathan Raman
Hello
Started with the presentation in October 2017 made by Marc-Andre
(Red Hat)
and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018, the multi-process project is now available and presented in this patchset. This first series enables the emulation of lsi53c895a in a separate process.We posted the Proof Of Concept patches [2] before the BoF session in 2018. Subsequently, we posted RFC v1 [3], RFC v2 [4], RFC v3 [5] and RFC v4 [6].
John & Elena presented the status of this project in KVM Forum 2019. We appreciate the in-person and email feedback we received to improve this patchset. We also received valuable feedback and direction on future improvements from the bi-weekly KVM community conference. We have incorporated all the feedback in the current version of the series, v5.
Following people contributed to this patchset:
John G Johnson <[email protected]>
Jagannathan Raman <[email protected]>
Elena Ufimtseva <[email protected]>
Kanth Ghatraju <[email protected]>
Konrad Wilk <[email protected]>For full concept writeup about QEMU disaggregation, refer to
docs/devel/qemu-multiprocess.rst. Please refer todocs/qemu-multiprocess.txtfor usage information.We are planning on making the following improvements in the future to the experimental Qemu multi-process:
- Asynchronous communication channel;
- Performance improvements;
- Libvirt support;
- Enforcement of security policies and privileges control;Quoted 6 lines
Quoted 1 line
We welcome all your ideas, concerns, and questions for this patchset.
There seem to be quite a few CI failures with this series applied:
https://travis-ci.org/stsquad/qemu/builds/656432858 https://app.shippable.com/github/stsquad/qemu/runs/1275/summary/console https://gitlab.com/stsquad/qemu/pipelines/122030403 https://cirrus-ci.com/build/4577637150490624Hi Alex,
Thanks for pointing it out.
"Patchew" also identified some errors which we are working on fixing for the next version. Patchew summarized the errors in the following page:
https://patchew.org/QEMU/[email protected]/To confirm we're compliant with Patchew, we are running docker tests before sending the patches for review next time around.
Quoted 8 lines
Quoted 6 lines
We'll use the following wiki to trigger "travis-ci" tests before pushing the branch for review next time around:
https://wiki.qemu.org/Testing/CI/Travis#Testing_Changes_to_TravisAre shippable, stsquad & cirrus-ci redundant if travis-ci & docker tests
stsquad is just my user account, hopefully I'm not redundant ;-)
Sorry I didn't realize it's your account. :)
Quoted 7 lines
They all test slightly different things but you should be able to replicate the tests locally.
Travis basically tests a bunch of different configuration setups on mostly x86 hardware. Unless it's a weird library interaction issue this should replicate in your normal build environment.
We'll run Travis CI also before submitting patches next time. The qemu wiki page for TravisCI says it tests MacOS. So this test should be sufficient to cover MacOS build tests I suppose.
https://wiki.qemu.org/Testing/CI/Travis#Testing_Changes_to_Travis
Quoted 4 lines
Shippable are cross compile tests. They use the existing docker infrastructure to cross compile for various target architectures. See "make docker" and the notes in
docs/devel/testing.rst.
We're running these docker tests locally.
Thank you very much!
--
Jag
Quoted 10 lines
CirrusCI tests MacOS and FreeBSD builds. You can build on the BSD's yourself, see "make vm-help". MacOSX is trickier unless you have a Mac yourself of course.
Quoted 3 lines
pass?
Thank you very much!