migration: Synchronize CPUs sooner in postcopy switchover phase
RFC PATCH 1/1 migration: Synchronize CPUs sooner in postcopy switchover phase
From: Juraj Marcin <[email protected]>
Previously, the post init CPU synchronization with the accelerator, like KVM, was performed in the bottom half of the POSTCOPY_RUN command handler. However, this causes several problems.
First issue is that if CPU synchronization fails, the destination QEMU crashes. However, it is too late to recover the source side as the response to special PING has been already sent, and both sides are already in the POSTCOPY_ACTIVE state.
By moving synchronization before responding, if the machine crashes, the response is never sent and the source side can resume from the POSTCOPY_DEVICE state.
Second issue is caused when migration is paused due to a network failure or user command right after transitioning to the POSTCOPY_ACTIVE state and the CPU synchronization causes a page fault. This page fault blocks the CPU and the main QEMU threads and cannot be resolved until postcopy migration is recovered. However, as libvirt also tries to execute 'cont' QMP command at this time (destination side transitions from POSTCOPY_DEVICE to POSTCOPY_ACTIVE), it will also halt waiting for the response from the blocked main QEMU thread, unaware of the fact that the migration is paused and needs to be recovered. Thus, it will wait indefinitely and never report postcopy migration error.
When the CPU synchronization happens sooner, and the network fails during it, the source side can transition from POSTCOPY_DEVICE to FAILED state and resume safely. If migration is paused later, the main thread won't be blocked by CPU synchronization and can respond to libvirt.
---
I am posting this as RFC to discuss the point at which the CPU synchronization should happen.
For the POSTCOPY_DEVICE state to be effective, this synchronization must happen before the destination machine responds to the special PING command. This leaves us with 2 options:
1) In the PING command handler before responding to the specific request, as proposed in this patch.
2) After loading CPU VMSD, for example in the post_load hook.
The first solution limits the number of places where synchronization needs to happen, however, having it in the PING command handler feels somewhat hacky.
I have also tested the second solution, and while it seems natural to synchronize the CPU after its data is loaded in the post_load, there are multiple CPU types and VMSDs and each one would need to call synchronization in its post_load hook. This would basically revert Jan Kiszka's commit [1] which refactored CPU synchronization and united it to one place.
Kind of middle ground idea I have, is to add a special flag to VMSDs that contain CPU data. After loading such VMSD the loadvm core would then run CPU synchronization for that specific CPU. This solution would bring the synchronization close to the actual load from the migration stream but also keep it in one place.
[1]: ea375f9ab8c7 ("KVM: Rework VCPU state writeback API")
--- migration/migration.c | 1 + migration/migration.h | 7 +++++++ migration/savevm.c | 16 +++++++++++++--- 3 files changed, 21 insertions(+), 3 deletions(-)
migration/migration.c +1 -0
@@ -320,6 +320,7 @@ void migration_object_init(void) current_incoming->page_requested =g_tree_new (page_request_addr_cmp); current_incoming->exit_on_error =INMIGRATE_DEFAULT_EXIT_ON_ERROR ; + current_incoming->postcopy_device_cpu_synchronized = false;migration_object_check (current_migration, &error_fatal);
migration/migration.h +7 -0
@@ -210,6 +210,13 @@ struct MigrationIncomingState {*/ QemuSemaphore postcopy_pause_sem_fast_load; +/* +* CPUs have been synchronized during POSTCOPY_DEVICE state before +* responding to a special PING to source. This means, synchronization is +* not required later during loadvm_postcopy_handle_run_bh(). +*/ +bool postcopy_device_cpu_synchronized; +/* List of listening socket addresses */ SocketAddressList *socket_address_list;
migration/savevm.c +13 -3
@@ -2143,9 +2143,10 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)/* TODO we should move all of this lot into postcopy_ram.c or a shared code * in migration.c */ -cpu_synchronize_all_post_init (); - -trace_vmstate_downtime_checkpoint ("dst-postcopy-bh-cpu-synced" ); +if (!mis->postcopy_device_cpu_synchronized) { +cpu_synchronize_all_post_init (); +trace_vmstate_downtime_checkpoint ("dst-postcopy-bh-cpu-synced" ); + }qemu_announce_self (&mis->announce_timer,migrate_announce_params ());
@@ -2510,6 +2511,15 @@ static int loadvm_process_command(QEMUFile *f, Error **errp) tmp32);return -1 ; } +if (tmp32 ==QEMU_VM_PING_PACKAGED_LOADED ) { +/* +* Try synchronizing CPU before responding. If it fails, QEMU exits +* and source side can resume. +*/ +cpu_synchronize_all_post_init (); + mis->postcopy_device_cpu_synchronized = true; +trace_vmstate_downtime_checkpoint ("dst-postcopy-bh-cpu-synced" ); + }migrate_send_rp_pong (mis, tmp32);return 0 ;
Signature · 1 line
2.53.0
Quoted 40 lines · Juraj Marcin
From: Juraj Marcin <[email protected]>
Previously, the post init CPU synchronization with the accelerator, like KVM, was performed in the bottom half of the
POSTCOPY_RUNcommand handler. However, this causes several problems.First issue is that if CPU synchronization fails, the destination QEMU crashes. However, it is too late to recover the source side as the response to special PING has been already sent, and both sides are already in the
POSTCOPY_ACTIVEstate.By moving synchronization before responding, if the machine crashes, the response is never sent and the source side can resume from the
POSTCOPY_DEVICEstate.Second issue is caused when migration is paused due to a network failure or user command right after transitioning to the POSTCOPY_ACTIVE state and the CPU synchronization causes a page fault. This page fault blocks the CPU and the main QEMU threads and cannot be resolved until postcopy migration is recovered. However, as libvirt also tries to execute 'cont' QMP command at this time (destination side transitions from
POSTCOPY_DEVICEtoPOSTCOPY_ACTIVE), it will also halt waiting for the response from the blocked main QEMU thread, unaware of the fact that the migration is paused and needs to be recovered. Thus, it will wait indefinitely and never report postcopy migration error.When the CPU synchronization happens sooner, and the network fails during it, the source side can transition from
POSTCOPY_DEVICEto FAILED state and resume safely. If migration is paused later, the main thread won't be blocked by CPU synchronization and can respond to libvirt.Signed-off-by Juraj Marcin <[email protected]>---
I am posting this as RFC to discuss the point at which the CPU synchronization should happen.For the
POSTCOPY_DEVICEstate to be effective, this synchronization must happen before the destination machine responds to the special PING command. This leaves us with 2 options:
Quoted 8 lines
1) In the PING command handler before responding to the specific request, as proposed in this patch.
2) After loading CPU VMSD, for example in the
post_loadhook.The first solution limits the number of places where synchronization needs to happen, however, having it in the PING command handler feels somewhat hacky.
Yes.
Quoted 7 lines
I have also tested the second solution, and while it seems natural to synchronize the CPU after its data is loaded in the
post_load, there are multiple CPU types and VMSDs and each one would need to call synchronization in itspost_loadhook. This would basically revert Jan Kiszka's commit [1] which refactored CPU synchronization and united it to one place.
What I was thinking is not reverting all of it, but only removing the migration relevant paths for post_init().
For example, we have different reasons to sync CPU, and we have two directions to do that (from/to kernel, in KVM's context). For migration, what used to be confusing is why we need to have CPU specific post_init() hooks when each CPU has VMSD and post_load() on its own.
The other one is on savevm side (cpu_synchronize_all_states()) and we can at least keep it as-is for now, simplify the discussion.
So if we want to move this into any of such post_load(), it means removal of three call sites of post_init() only in migration/savevm.c, then do per-CPU's post_init() in its post_load() hook.
I don't know the real answer of this one; I recall you tried it out but hit some ARM specific issue. Just to want to make sure we're on the same page on what you have experimented.
Said that, I do see some complexity over such a change already. For example, cpu_vmstate_register() seems to be able to register more than one VMSDs for each vCPU. I don't know if it means at least we can't simply put it into post_load() of vmstate_cpu_common, as when reaching there it is not guaranteed that all vCPUs' registers are uptodate..
Besides..
I do have another thought, though, to avoid the hacky part of this patch and looks pretty safe: dest QEMU does not reply directly to the QEMU_VM_PING_PACKAGED_LOADED ping, instead it sets a flag. Then we move the conditional PONG into loadvm_postcopy_handle_run().
With that, we can move post_init() into loadvm_postcopy_handle_run() altogether.
loadvm_postcopy_handle_run():
...
cpu_synchronize_all_post_init();
if (package_loaded_ping_received) {
send_pong();
}
migrate_set_state(POSTCOPY_DEVICE, POSTCOPY_ACTIVE);
migration_bh_schedule(mis);Would this work?
Signature · 1 line
Peter Xu
Hi Peter,
Quoted 71 lines · Peter Xu
Quoted 40 lines · Juraj Marcin
From: Juraj Marcin <[email protected]>
Previously, the post init CPU synchronization with the accelerator, like KVM, was performed in the bottom half of the
POSTCOPY_RUNcommand handler. However, this causes several problems.First issue is that if CPU synchronization fails, the destination QEMU crashes. However, it is too late to recover the source side as the response to special PING has been already sent, and both sides are already in the
POSTCOPY_ACTIVEstate.By moving synchronization before responding, if the machine crashes, the response is never sent and the source side can resume from the
POSTCOPY_DEVICEstate.Second issue is caused when migration is paused due to a network failure or user command right after transitioning to the POSTCOPY_ACTIVE state and the CPU synchronization causes a page fault. This page fault blocks the CPU and the main QEMU threads and cannot be resolved until postcopy migration is recovered. However, as libvirt also tries to execute 'cont' QMP command at this time (destination side transitions from
POSTCOPY_DEVICEtoPOSTCOPY_ACTIVE), it will also halt waiting for the response from the blocked main QEMU thread, unaware of the fact that the migration is paused and needs to be recovered. Thus, it will wait indefinitely and never report postcopy migration error.When the CPU synchronization happens sooner, and the network fails during it, the source side can transition from
POSTCOPY_DEVICEto FAILED state and resume safely. If migration is paused later, the main thread won't be blocked by CPU synchronization and can respond to libvirt.Signed-off-by Juraj Marcin <[email protected]>---
I am posting this as RFC to discuss the point at which the CPU synchronization should happen.For the
POSTCOPY_DEVICEstate to be effective, this synchronization must happen before the destination machine responds to the special PING command. This leaves us with 2 options:Quoted 8 lines
1) In the PING command handler before responding to the specific request, as proposed in this patch.
2) After loading CPU VMSD, for example in the
post_loadhook.The first solution limits the number of places where synchronization needs to happen, however, having it in the PING command handler feels somewhat hacky.
Yes.
Quoted 7 lines
I have also tested the second solution, and while it seems natural to synchronize the CPU after its data is loaded in the
post_load, there are multiple CPU types and VMSDs and each one would need to call synchronization in itspost_loadhook. This would basically revert Jan Kiszka's commit [1] which refactored CPU synchronization and united it to one place.What I was thinking is not reverting all of it, but only removing the migration relevant paths for
post_init().For example, we have different reasons to sync CPU, and we have two directions to do that (from/to kernel, in KVM's context). For migration, what used to be confusing is why we need to have CPU specific
post_init()hooks when each CPU has VMSD andpost_load()on its own.The other one is on savevm side (
cpu_synchronize_all_states()) and we can at least keep it as-is for now, simplify the discussion.
Quoted 7 lines
So if we want to move this into any of such
post_load(), it means removal of three call sites ofpost_init()only inmigration/savevm.c, then do per-CPU'spost_init()in itspost_load()hook.I don't know the real answer of this one; I recall you tried it out but hit some ARM specific issue. Just to want to make sure we're on the same page on what you have experimented.
I thought I did, however, that ARM issue in the CI turned out to be unrelated.
Quoted 26 lines
Said that, I do see some complexity over such a change already. For example,
cpu_vmstate_register()seems to be able to register more than one VMSDs for each vCPU. I don't know if it means at least we can't simply put it intopost_load()ofvmstate_cpu_common, as when reaching there it is not guaranteed that all vCPUs' registers are uptodate..Besides..
I do have another thought, though, to avoid the hacky part of this patch and looks pretty safe: dest QEMU does not reply directly to the
QEMU_VM_PING_PACKAGED_LOADEDping, instead it sets a flag. Then we move the conditional PONG intoloadvm_postcopy_handle_run().With that, we can move
post_init()intoloadvm_postcopy_handle_run()altogether.loadvm_postcopy_handle_run(): ... cpu_synchronize_all_post_init(); if (package_loaded_ping_received) { send_pong(); } migrate_set_state(POSTCOPY_DEVICE, POSTCOPY_ACTIVE); migration_bh_schedule(mis);
Quoted 1 line
Would this work?
Interesting idea, I think it should work, but I am a bit unsure about delaying the PONG response in such way, if it could potentially break anything.
Thank you!
Best regards,
Juraj Marcin
Quoted 4 lines
Signature · 2 lines
-- Peter Xu