migration: Notifier fixes for 11.0
cover migration: Notifier fixes for 11.0
CI: https://gitlab.com/peterx/qemu/-/pipelines/2280356561
Two major goals for this small series:
- Fix postcopy issue where DONE and FAILED notifiers will be invoked twice
- Move FAILED notifier to be before vm_start() if the failure happens during switchover (where we will stop the VM first)
The 2nd goal will be needed by Stefan's ongoing work on block persistent reservations, where a fallback should be required on src to happen before vm_start(). Instead of introducing another FAILED_BEFORE_START, this patchset should make FAILED work instead.
Patch 1 adds a tracepoint for me to verify this fix.
Patch 2-3 are the real changes of above two.
Patch 3-4 are some cleanups alone the context that we can do, hence attached at the end.
More details in commit logs individually. Comments welcomed, thanks.
Peter Xu (5):
migration: Add a tracepoint for invoking migration notifiers
migration: Fix double notification of DONE/FAIL for postcopy
migration: Notify migration FAILED before starting VM
migration: Drop explicit block activation in postcopy fail path
migration: Rename MIG_EVENT_PRECOPY_* to MIG_EVENT_*
include/migration/misc.h | 15 ++++++++------- hw/intc/arm_gicv3_kvm.c | 2 +- hw/net/virtio-net.c | 4 ++-- hw/vfio/cpr-legacy.c | 2 +- hw/vfio/cpr.c | 8 ++++---- hw/vfio/migration.c | 4 ++-- migration/cpr-exec.c | 6 +++--- migration/migration.c | 30 +++++++++++++++++++++--------- net/vhost-vdpa.c | 4 ++-- ui/spice-core.c | 7 ++++--- migration/trace-events | 1 + 11 files changed, 49 insertions(+), 34 deletions(-)
Signature · 1 line
2.50.1
PATCH 1/5 migration: Add a tracepoint for invoking migration notifiers
--- migration/migration.c | 2 ++ migration/trace-events | 1 + 2 files changed, 3 insertions(+)
migration/migration.c +2 -0
@@ -1706,6 +1706,8 @@ int migration_call_notifiers(MigrationState *s, MigrationEventType type,GSList *elem, *next;int ret; +trace_migration_call_notifiers (type); + e.type = type;for (elem = migration_state_notifiers[mode]; elem; elem = next) {
migration/trace-events +1 -0
@@ -198,6 +198,7 @@ process_incoming_migration_co_end(int ret) "ret=%d" process_incoming_migration_co_postcopy_end_main(void) "" postcopy_preempt_enabled(bool value) "%d" migration_precopy_complete(void) "" +migration_call_notifiers(int type) "type=%d" # migration-stats migration_transferred_bytes(uint64_t qemu_file, uint64_t multifd, uint64_t rdma) "qemu_file %" PRIu64 " multifd %" PRIu64 " RDMA %" PRIu64
Signature · 1 line
2.50.1
Re: [PATCH 1/5] migration: Add a tracepoint for invoking migration notifiers
Quoted 24 lines · Peter Xu
Signed-off-by Peter Xu <[email protected]>--- migration/migration.c | 2 ++ migration/trace-events | 1 + 2 files changed, 3 insertions(+)migration/migration.c +2
-0@@ -1706,6 +1706,8 @@ int migration_call_notifiers(MigrationState *s, MigrationEventType type,GSList *elem, *next;int ret; +trace_migration_call_notifiers (type); + e.type = type;for (elem = migration_state_notifiers[mode]; elem; elem = next) {migration/trace-events +0
-0
migration/trace-events process_incoming_migration_co_end(int ret) "ret=%d" · line 204
Quoted 7 lines
migration/trace-events +1
-0@@ -198,6 +198,7 @@ process_incoming_migration_co_end(int ret) "ret=%d" process_incoming_migration_co_postcopy_end_main(void) "" postcopy_preempt_enabled(bool value) "%d" migration_precopy_complete(void) "" +migration_call_notifiers(int type) "type=%d" # migration-stats migration_transferred_bytes(uint64_t qemu_file, uint64_t multifd, uint64_t rdma) "qemu_file %" PRIu64 " multifd %" PRIu64 " RDMA %" PRIu64
PATCH 2/5 migration: Fix double notification of DONE/FAIL for postcopy
In reply to Peter XuMigration notifiers will notify at any of three places: (1) SETUP phase, (2) migration completes, (3) migration fails.
There's actually a special case for spice: one can refer to b82fc321bf ("Postcopy+spice: Pass spice migration data earlier"). It doesn't need another 4th event because in commit 9d9babf78d ("migration:
MigrationEvent for notifiers") we merged it together with the DONE event.
The merge makes some sense if we treat "switchover" of postcopy as "DONE", however that also means for postcopy we'll notify DONE twice.. The other one at the end of postcopy when migration_cleanup().
In reality, the current code base will also notify FAILED for postcopy twice. It's because an (maybe accidental) change in commit 4af667f87c ("migration: notifier error checking").
First of all, we still need that notification when switchover as stated in Dave's commit, however that's only needed for spice. To fix it, introduce POSTCOPY_START event to differenciate it from DONE. Use that instead in postcopy_start(). Then spice will need to capture this event too.
Then we remove the extra FAILED notification in postcopy_start().
If one wonder if other DONE users should also monitor POSTCOPY_START event.. We have two more DONE users:
- kvm_arm_gicv3_notifier
- cpr_exec_notifier
Both of them do not need a notification for POSTCOPY_START, but only when migration completed. Actually, both of them are used in CPR, which doesn't support postcopy.
I didn't attach Fixes: because I am not aware of any real bug on such double reporting. I'm wildly guessing the 2nd notify might be silently ignored in many cases. However this is still worth fixing.
--- include/migration/misc.h | 1 + migration/migration.c | 3 +-- ui/spice-core.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-)
include/migration/misc.h +1 -0
@@ -63,6 +63,7 @@ typedef enum MigrationEventType {MIG_EVENT_PRECOPY_SETUP ,MIG_EVENT_PRECOPY_DONE ,MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_POSTCOPY_START , MIG_EVENT_MAX } MigrationEventType;
migration/migration.c +1 -2
@@ -2857,7 +2857,7 @@ static int postcopy_start(MigrationState *ms, Error **errp)* at the transition to postcopy and after the device state; in particular * spice needs to trigger a transition now */ -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_DONE ,NULL ); +migration_call_notifiers (ms,MIG_EVENT_POSTCOPY_START ,NULL );migration_downtime_end (ms);
@@ -2906,7 +2906,6 @@ fail:migrate_set_state (&ms->state, ms->state,MIGRATION_STATUS_FAILED ); }migration_block_activate (NULL ); -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_FAILED ,NULL );bql_unlock ();return -1 ; }
ui/spice-core.c +2 -1
@@ -585,7 +585,8 @@ static int migration_state_notifier(NotifierWithReturn *notifier,if (e->type ==MIG_EVENT_PRECOPY_SETUP ) {spice_server_migrate_start (spice_server); - }else if (e->type ==MIG_EVENT_PRECOPY_DONE ) { + }else if (e->type ==MIG_EVENT_PRECOPY_DONE || + e->type ==MIG_EVENT_POSTCOPY_START ) {spice_server_migrate_end (spice_server, true); spice_have_target_host = false; }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) {
Signature · 1 line
2.50.1
Re: [PATCH 2/5] migration: Fix double notification of DONE/FAIL for postcopy
Quoted 51 lines · Peter Xu
Migration notifiers will notify at any of three places: (1) SETUP phase, (2) migration completes, (3) migration fails.
There's actually a special case for spice: one can refer to
b82fc321bf("Postcopy+spice: Pass spice migration data earlier"). It doesn't need another 4th event because in commit9d9babf78d("migration:
MigrationEvent for notifiers") we merged it together with the DONE event.The merge makes some sense if we treat "switchover" of postcopy as "DONE", however that also means for postcopy we'll notify DONE twice.. The other one at the end of postcopy when
migration_cleanup().In reality, the current code base will also notify FAILED for postcopy twice. It's because an (maybe accidental) change in commit
4af667f87c("migration: notifier error checking").First of all, we still need that notification when switchover as stated in Dave's commit, however that's only needed for spice. To fix it, introduce
POSTCOPY_STARTevent to differenciate it from DONE. Use that instead inpostcopy_start(). Then spice will need to capture this event too.Then we remove the extra FAILED notification in
postcopy_start().If one wonder if other DONE users should also monitor
POSTCOPY_STARTevent.. We have two more DONE users:-
kvm_arm_gicv3_notifier
-cpr_exec_notifierBoth of them do not need a notification for
POSTCOPY_START, but only when migration completed. Actually, both of them are used in CPR, which doesn't support postcopy.I didn't attach Fixes: because I am not aware of any real bug on such double reporting. I'm wildly guessing the 2nd notify might be silently ignored in many cases. However this is still worth fixing.
Cc Marc-André Lureau <[email protected]>Cc Dr. David Alan Gilbert <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/migration/misc.h | 1 + migration/migration.c | 3 +-- ui/spice-core.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-)include/migration/misc.h +0
-0
include/migration/misc.h typedef enum MigrationEventType { · line 68
Quoted 6 lines
include/migration/misc.h +1
-0@@ -63,6 +63,7 @@ typedef enum MigrationEventType {MIG_EVENT_PRECOPY_SETUP ,MIG_EVENT_PRECOPY_DONE ,MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_POSTCOPY_START , MIG_EVENT_MAX } MigrationEventType;
With the new state there's a doc text to update at migration_add_notifier below.
Quoted 15 lines
migration/migration.c +1
-1@@ -2857,7 +2857,7 @@ static int postcopy_start(MigrationState *ms, Error **errp)* at the transition to postcopy and after the device state; in particular * spice needs to trigger a transition now */ -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_DONE ,NULL ); +migration_call_notifiers (ms,MIG_EVENT_POSTCOPY_START ,NULL );migration_downtime_end (ms);
migration/migration.c fail: · line 2909
Quoted 4 lines
migration/migration.c +0
-1@@ -2906,7 +2906,6 @@ fail:migrate_set_state (&ms->state, ms->state,MIGRATION_STATUS_FAILED ); }migration_block_activate (NULL ); -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_FAILED ,NULL );
Does anything consuming the FAILED event expects it to be issued before the vm_start() at migration_iteration_finish?
migration_iteration_finish:
bql_lock();
switch (s->state) {
case MIGRATION_STATUS_FAILED:
if (!migration_block_activate(&local_err)) {
error_free(local_err);
break;
}
if (runstate_is_live(s->vm_old_state)) {
if (!runstate_check(RUN_STATE_SHUTDOWN)) {
vm_start();
}
} else {
if (runstate_check(RUN_STATE_FINISH_MIGRATE)) {
runstate_set(s->vm_old_state);
}
}The extra migration_block_activate() in postcopy_start() also seems wrong, no?
Quoted 17 lines
migration/migration.c +0
-0bql_unlock ();return -1 ; }ui/spice-core.c +2
-1@@ -585,7 +585,8 @@ static int migration_state_notifier(NotifierWithReturn *notifier,if (e->type ==MIG_EVENT_PRECOPY_SETUP ) {spice_server_migrate_start (spice_server); - }else if (e->type ==MIG_EVENT_PRECOPY_DONE ) { + }else if (e->type ==MIG_EVENT_PRECOPY_DONE || + e->type ==MIG_EVENT_POSTCOPY_START ) {spice_server_migrate_end (spice_server, true); spice_have_target_host = false; }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) {
Re: [PATCH 2/5] migration: Fix double notification of DONE/FAIL for postcopy
Quoted 105 lines · Fabiano Rosas
Quoted 51 lines · Peter Xu
Migration notifiers will notify at any of three places: (1) SETUP phase, (2) migration completes, (3) migration fails.
There's actually a special case for spice: one can refer to
b82fc321bf("Postcopy+spice: Pass spice migration data earlier"). It doesn't need another 4th event because in commit9d9babf78d("migration:
MigrationEvent for notifiers") we merged it together with the DONE event.The merge makes some sense if we treat "switchover" of postcopy as "DONE", however that also means for postcopy we'll notify DONE twice.. The other one at the end of postcopy when
migration_cleanup().In reality, the current code base will also notify FAILED for postcopy twice. It's because an (maybe accidental) change in commit
4af667f87c("migration: notifier error checking").First of all, we still need that notification when switchover as stated in Dave's commit, however that's only needed for spice. To fix it, introduce
POSTCOPY_STARTevent to differenciate it from DONE. Use that instead inpostcopy_start(). Then spice will need to capture this event too.Then we remove the extra FAILED notification in
postcopy_start().If one wonder if other DONE users should also monitor
POSTCOPY_STARTevent.. We have two more DONE users:-
kvm_arm_gicv3_notifier
-cpr_exec_notifierBoth of them do not need a notification for
POSTCOPY_START, but only when migration completed. Actually, both of them are used in CPR, which doesn't support postcopy.I didn't attach Fixes: because I am not aware of any real bug on such double reporting. I'm wildly guessing the 2nd notify might be silently ignored in many cases. However this is still worth fixing.
Cc Marc-André Lureau <[email protected]>Cc Dr. David Alan Gilbert <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/migration/misc.h | 1 + migration/migration.c | 3 +-- ui/spice-core.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-)include/migration/misc.h +0
-0include/migration/misc.h typedef enum MigrationEventType { · line 68
Quoted 6 lines
include/migration/misc.h +1
-0@@ -63,6 +63,7 @@ typedef enum MigrationEventType {MIG_EVENT_PRECOPY_SETUP ,MIG_EVENT_PRECOPY_DONE ,MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_POSTCOPY_START , MIG_EVENT_MAX } MigrationEventType;With the new state there's a doc text to update at
migration_add_notifierbelow.Quoted 15 lines
migration/migration.c +1
-1@@ -2857,7 +2857,7 @@ static int postcopy_start(MigrationState *ms, Error **errp)* at the transition to postcopy and after the device state; in particular * spice needs to trigger a transition now */ -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_DONE ,NULL ); +migration_call_notifiers (ms,MIG_EVENT_POSTCOPY_START ,NULL );migration_downtime_end (ms);migration/migration.c fail: · line 2909
Quoted 4 lines
migration/migration.c +0
-1@@ -2906,7 +2906,6 @@ fail:migrate_set_state (&ms->state, ms->state,MIGRATION_STATUS_FAILED ); }migration_block_activate (NULL ); -migration_call_notifiers (ms,MIG_EVENT_PRECOPY_FAILED ,NULL );Does anything consuming the FAILED event expects it to be issued before the
vm_start()atmigration_iteration_finish?
migration_iteration_finish:
bql_lock();
switch (s->state) {
caseMIGRATION_STATUS_FAILED:if (!migration_block_activate(&local_err)) { error_free(local_err); break; } if (runstate_is_live(s->vm_old_state)) { if (!runstate_check(RUN_STATE_SHUTDOWN)) { vm_start(); } } else { if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { runstate_set(s->vm_old_state); } }
Quoted 3 lines
The extra
migration_block_activate()inpostcopy_start()also seems wrong, no?
Heh, nevermind...
Quoted 17 lines
Quoted 17 lines
migration/migration.c +0
-0bql_unlock ();return -1 ; }ui/spice-core.c +2
-1@@ -585,7 +585,8 @@ static int migration_state_notifier(NotifierWithReturn *notifier,if (e->type ==MIG_EVENT_PRECOPY_SETUP ) {spice_server_migrate_start (spice_server); - }else if (e->type ==MIG_EVENT_PRECOPY_DONE ) { + }else if (e->type ==MIG_EVENT_PRECOPY_DONE || + e->type ==MIG_EVENT_POSTCOPY_START ) {spice_server_migrate_end (spice_server, true); spice_have_target_host = false; }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) {
Re: [PATCH 2/5] migration: Fix double notification of DONE/FAIL for postcopy
In reply to Fabiano RosasQuoted 48 lines · Fabiano Rosas
Quoted 46 lines · Peter Xu
Migration notifiers will notify at any of three places: (1) SETUP phase, (2) migration completes, (3) migration fails.
There's actually a special case for spice: one can refer to
b82fc321bf("Postcopy+spice: Pass spice migration data earlier"). It doesn't need another 4th event because in commit9d9babf78d("migration:
MigrationEvent for notifiers") we merged it together with the DONE event.The merge makes some sense if we treat "switchover" of postcopy as "DONE", however that also means for postcopy we'll notify DONE twice.. The other one at the end of postcopy when
migration_cleanup().In reality, the current code base will also notify FAILED for postcopy twice. It's because an (maybe accidental) change in commit
4af667f87c("migration: notifier error checking").First of all, we still need that notification when switchover as stated in Dave's commit, however that's only needed for spice. To fix it, introduce
POSTCOPY_STARTevent to differenciate it from DONE. Use that instead inpostcopy_start(). Then spice will need to capture this event too.Then we remove the extra FAILED notification in
postcopy_start().If one wonder if other DONE users should also monitor
POSTCOPY_STARTevent.. We have two more DONE users:-
kvm_arm_gicv3_notifier
-cpr_exec_notifierBoth of them do not need a notification for
POSTCOPY_START, but only when migration completed. Actually, both of them are used in CPR, which doesn't support postcopy.I didn't attach Fixes: because I am not aware of any real bug on such double reporting. I'm wildly guessing the 2nd notify might be silently ignored in many cases. However this is still worth fixing.
Cc Marc-André Lureau <[email protected]>Cc Dr. David Alan Gilbert <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/migration/misc.h | 1 + migration/migration.c | 3 +-- ui/spice-core.c | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-)
Quoted 14 lines
include/migration/misc.h typedef enum MigrationEventType { · line 68
Quoted 11 lines
include/migration/misc.h +1
-0@@ -63,6 +63,7 @@ typedef enum MigrationEventType {MIG_EVENT_PRECOPY_SETUP ,MIG_EVENT_PRECOPY_DONE ,MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_POSTCOPY_START , MIG_EVENT_MAX } MigrationEventType;With the new state there's a doc text to update at
migration_add_notifierbelow.
Yeh I noticed it, I didn't do it as I found POSTCOPY_START is special in this case marking an optional phase for postcopy-only.
I can still update it, though.. Then it'll look like:
- MIG_EVENT_SETUP [-> MIG_EVENT_POSTCOPY_START] -> MIG_EVENT_DONE
- MIG_EVENT_SETUP [-> MIG_EVENT_POSTCOPY_START] -> MIG_EVENT_FAILED
- MIG_EVENT_FAILED
I'll also move it above the enum instead if no objections.
Signature · 1 line
Peter Xu
PATCH 3/5 migration: Notify migration FAILED before starting VM
In reply to Peter XuDevices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.
This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
After walking over all existing FAILED notifier users, I got the conclusion that this should also be a cleaner approach at least from design POV.
We have these notifier users, where the first two do not need to trap FAILED:
|----------------------------+-------------------------------------+---------------------| | device | handler | events needed | |----------------------------+-------------------------------------+---------------------| | gicv3 | kvm_arm_gicv3_notifier | DONE | | vfio_iommufd / vfio_legacy | vfio_cpr_reboot_notifier | SETUP | | cpr-exec | cpr_exec_notifier | FAILED, DONE | | virtio-net | virtio_net_migration_state_notifier | SETUP, FAILED | | vfio | vfio_migration_state_notifier | FAILED | | vdpa | vdpa_net_migration_state_notifier | SETUP, FAILED | | spice [*] | migration_state_notifier | SETUP, FAILED, DONE | |----------------------------+-------------------------------------+---------------------|
For cpr-exec, it tries to cleanup some cpr-exec specific fd or env variables. This should be fine either way, as long as before migration_cleanup().
For virtio-net, we need to re-plug the primary device back to guest in the failover mode. Likely benign.
VFIO needs to re-start the device if FAILED. IIUC it should do it before vm_start(), if the VFIO device can be put into a STOPed state due to migration, we should logically make it running again before vCPUs run.
VDPA will disable SVQ when migration is FAILED. Likely benign too, but looks better if we can do it before resuming vCPUs.
For spice, we should rely on "spice_server_migrate_end(false)" to retake the ownership. Benign, but looks more reasonable if the spice client does it before VM runs again.
Note that this change may introduce slightly more downtime, if the migration failed exactly at the switchover phase. But that's very rare, and even if it happens, none of above expects a long delay, but a short one, likely will be buried in the total downtime even if failed.
--- migration/migration.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
migration/migration.c +16 -4
@@ -1481,7 +1481,6 @@ static void migration_cleanup_json_writer(MigrationState *s)static void migration_cleanup (MigrationState *s) { -MigrationEventType type;QEMUFile *tmp =NULL ;trace_migration_cleanup ();
@@ -1535,9 +1534,15 @@ static void migration_cleanup(MigrationState *s)/* It is used on info migrate. We can't free it */ error_report_err (error_copy (s->error)); } - type =migration_has_failed (s) ?MIG_EVENT_PRECOPY_FAILED : -MIG_EVENT_PRECOPY_DONE ; -migration_call_notifiers (s, type,NULL ); + +/* +* FAILED notification should have already happened. Notify DONE if +* migration completed successfully. +*/ +if (!migration_has_failed (s)) { +migration_call_notifiers (s,MIG_EVENT_PRECOPY_DONE ,NULL ); + } +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); }
@@ -3589,6 +3594,13 @@ static void migration_iteration_finish(MigrationState *s)error_free (local_err);break ; } + +/* +* Notify FAILED before starting VM, so that devices can invoke +* necessary fallbacks before vCPUs run again. +*/ +migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +if (runstate_is_live (s->vm_old_state)) {if (!runstate_check (RUN_STATE_SHUTDOWN )) {vm_start ();
Signature · 1 line
2.50.1
Re: [PATCH 3/5] migration: Notify migration FAILED before starting VM
Quoted 8 lines · Peter Xu
Devices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in
migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
The change to FAILED in patch 2 should come to this patch to avoid having a window where the notification only happens at the end.
Quoted 93 lines
After walking over all existing FAILED notifier users, I got the conclusion that this should also be a cleaner approach at least from design POV.
We have these notifier users, where the first two do not need to trap FAILED:
|----------------------------+-------------------------------------+---------------------| | device | handler | events needed | |----------------------------+-------------------------------------+---------------------| | gicv3 | kvm_arm_gicv3_notifier | DONE | | vfio_iommufd / vfio_legacy | vfio_cpr_reboot_notifier | SETUP | | cpr-exec | cpr_exec_notifier | FAILED, DONE | | virtio-net | virtio_net_migration_state_notifier | SETUP, FAILED | | vfio | vfio_migration_state_notifier | FAILED | | vdpa | vdpa_net_migration_state_notifier | SETUP, FAILED | | spice [*] | migration_state_notifier | SETUP, FAILED, DONE | |----------------------------+-------------------------------------+---------------------|For cpr-exec, it tries to cleanup some cpr-exec specific fd or env variables. This should be fine either way, as long as before
migration_cleanup().For virtio-net, we need to re-plug the primary device back to guest in the failover mode. Likely benign.
VFIO needs to re-start the device if FAILED. IIUC it should do it before
vm_start(), if the VFIO device can be put into a STOPed state due to migration, we should logically make it running again before vCPUs run.VDPA will disable SVQ when migration is FAILED. Likely benign too, but looks better if we can do it before resuming vCPUs.
For spice, we should rely on "
spice_server_migrate_end(false)" to retake the ownership. Benign, but looks more reasonable if the spice client does it before VM runs again.Note that this change may introduce slightly more downtime, if the migration failed exactly at the switchover phase. But that's very rare, and even if it happens, none of above expects a long delay, but a short one, likely will be buried in the total downtime even if failed.
Cc Cédric Le Goater <[email protected]>Cc Marc-André Lureau <[email protected]>Signed-off-by Peter Xu <[email protected]>--- migration/migration.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)migration/migration.c +16
-4@@ -1481,7 +1481,6 @@ static void migration_cleanup_json_writer(MigrationState *s)static void migration_cleanup (MigrationState *s) { -MigrationEventType type;QEMUFile *tmp =NULL ;trace_migration_cleanup ();@@ -1535,9 +1534,15 @@ static void migration_cleanup(MigrationState *s)/* It is used on info migrate. We can't free it */ error_report_err (error_copy (s->error)); } - type =migration_has_failed (s) ?MIG_EVENT_PRECOPY_FAILED : -MIG_EVENT_PRECOPY_DONE ; -migration_call_notifiers (s, type,NULL ); + +/* +* FAILED notification should have already happened. Notify DONE if +* migration completed successfully. +*/ +if (!migration_has_failed (s)) { +migration_call_notifiers (s,MIG_EVENT_PRECOPY_DONE ,NULL ); + } +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); }@@ -3589,6 +3594,13 @@ static void migration_iteration_finish(MigrationState *s)error_free (local_err);break ; } + +/* +* Notify FAILED before starting VM, so that devices can invoke +* necessary fallbacks before vCPUs run again. +*/ +migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +if (runstate_is_live (s->vm_old_state)) {if (!runstate_check (RUN_STATE_SHUTDOWN )) {vm_start ();
Re: [PATCH 3/5] migration: Notify migration FAILED before starting VM
Quoted 7 lines · Fabiano Rosas
Quoted 5 lines · Peter Xu
Devices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in
migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.
Quoted 6 lines
Quoted 3 lines
This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
The change to FAILED in patch 2 should come to this patch to avoid having a window where the notification only happens at the end.
Hmm.. Isn't that expected? Even after patch 2, we still notify FAILED at the end for precopy. It's the same for postcopy.
For a failed postcopy we have following behavior:
Before patch 2
==============
- notify FAILED (during switchover)
- vm_start()
- notify FAILED (during migration_cleanup)
After patch 2
=============
- vm_start()
- notify FAILED (during migration_cleanup)
So patch 2 fixes the duplicate issue, and only fixes that.
After patch 3
=============
- notify FAILED (during migration_iteration_finish)
- vm_start()
Patch 3 changes the place of FAILED notification so that it happens always before vm_start(), for both precopy and postcopy.
Signature · 1 line
Peter Xu
Re: [PATCH 3/5] migration: Notify migration FAILED before starting VM
Quoted 12 lines · Peter Xu
Quoted 11 lines · Fabiano Rosas
Quoted 8 lines · Peter Xu
Devices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in
migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
Quoted 6 lines
Quoted 2 lines
The change to FAILED in patch 2 should come to this patch to avoid having a window where the notification only happens at the end.
Hmm.. Isn't that expected? Even after patch 2, we still notify FAILED at the end for precopy. It's the same for postcopy.
Sorry, I meant: s/at the end/after vm_start/.
Quoted 17 lines
For a failed postcopy we have following behavior:
Before patch 2
==============- notify FAILED (during switchover)
-vm_start()
- notify FAILED (duringmigration_cleanup)After patch 2
=============-
vm_start()
- notify FAILED (duringmigration_cleanup)So patch 2 fixes the duplicate issue, and only fixes that.
Quoted 8 lines
After patch 3
=============- notify FAILED (during
migration_iteration_finish)
-vm_start()Patch 3 changes the place of FAILED notification so that it happens always before
vm_start(), for both precopy and postcopy.
Right, my point is that with patch 3 we're establishing that the correct place to notify is before vm_start(). But after patch 2, *if* any driver actually depends on being informed of failure *before* starting the VM, that will not happen. I think both changes could be made at once so that this intermediate state never exists.
Re: [PATCH 3/5] migration: Notify migration FAILED before starting VM
Quoted 43 lines · Fabiano Rosas
Quoted 12 lines · Peter Xu
Quoted 11 lines · Fabiano Rosas
Quoted 8 lines · Peter Xu
Devices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in
migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
Quoted 6 lines
Quoted 2 lines
The change to FAILED in patch 2 should come to this patch to avoid having a window where the notification only happens at the end.
Hmm.. Isn't that expected? Even after patch 2, we still notify FAILED at the end for precopy. It's the same for postcopy.
Sorry, I meant: s/at the end/after vm_start/.
Quoted 20 lines
For a failed postcopy we have following behavior:
Before patch 2
==============- notify FAILED (during switchover)
-vm_start()
- notify FAILED (duringmigration_cleanup)After patch 2
=============-
vm_start()
- notify FAILED (duringmigration_cleanup)So patch 2 fixes the duplicate issue, and only fixes that.
After patch 3
=============
Quoted 8 lines
Quoted 5 lines
- notify FAILED (during
migration_iteration_finish)
-vm_start()Patch 3 changes the place of FAILED notification so that it happens always before
vm_start(), for both precopy and postcopy.Right, my point is that with patch 3 we're establishing that the correct place to notify is before
vm_start().
Yep, likely not strictly correctness in terms of current notifiers, but since Stefan may have yet another use case that may require a notifier to be done before vm_start(), it makes more sense for us to move, IMHO.
Quoted 4 lines
But after patch 2, *if* any driver actually depends on being informed of failure *before* starting the VM, that will not happen. I think both changes could be made at once so that this intermediate state never exists.
I see what you meant. I think there should have no such user.
It's because we always notify FAILED at migration_cleanup() for precopy, or even postcopy before the cpr-exec work (before QEMU 9.0).
That behavior of "notify FAILED before vm_start() for postcopy" is very specific and only added after commit 4af667f87c ("migration: notifier error checking"). IOW, before QEMU 9.0, for both precopy and postcopy we always notify FAILED in migration_cleanup(), never before vm_start().
I mentioned this in the commit log of previous patch too, where I bet the additional FAILED notification added in 4af667f87c for postcopy path is an accident (to make it pairing with the "reused DONE", however it turns out we likely shouldn't do either of them..). So I don't expect anything will depend on that behavior, and only for postcopy.
The benefit of splitting this patch and previous one is, the previous one is a "fix" of duplicated notifications, hence if we need a backport that can be done without this one. Said that, I don't think one should need it.. It should also make each commits slightly easier to follow, because they're fundamentally two changes.
Let me know what you think after reading my explanations above. I prefer the split like as-is, but I can still squash it to close the trivially small window that you described. I'll make sure if merged the commit message contains separate discussions on two problems.
Signature · 1 line
Peter Xu
Fabiano Rosas
Reviewed-byRe: [PATCH 3/5] migration: Notify migration FAILED before starting VM
Quoted 84 lines · Peter Xu
Quoted 43 lines · Fabiano Rosas
Quoted 12 lines · Peter Xu
Quoted 11 lines · Fabiano Rosas
Quoted 8 lines · Peter Xu
Devices may opt-in migration FAILED notifiers to be invoked when migration fails. Currently, the notifications happen in
migration_cleanup(). It is normally fine, but maybe not ideal if there's dependency of the fallback v.s. VM starts.This patch moves the FAILED notification earlier, so that if the failure happened during switchover, it'll notify before VM restart.
Quoted 6 lines
Quoted 2 lines
The change to FAILED in patch 2 should come to this patch to avoid having a window where the notification only happens at the end.
Hmm.. Isn't that expected? Even after patch 2, we still notify FAILED at the end for precopy. It's the same for postcopy.
Sorry, I meant: s/at the end/after vm_start/.
Quoted 20 lines
For a failed postcopy we have following behavior:
Before patch 2
==============- notify FAILED (during switchover)
-vm_start()
- notify FAILED (duringmigration_cleanup)After patch 2
=============-
vm_start()
- notify FAILED (duringmigration_cleanup)So patch 2 fixes the duplicate issue, and only fixes that.
After patch 3
=============Quoted 8 lines
Quoted 5 lines
- notify FAILED (during
migration_iteration_finish)
-vm_start()Patch 3 changes the place of FAILED notification so that it happens always before
vm_start(), for both precopy and postcopy.Right, my point is that with patch 3 we're establishing that the correct place to notify is before
vm_start().Yep, likely not strictly correctness in terms of current notifiers, but since Stefan may have yet another use case that may require a notifier to be done before
vm_start(), it makes more sense for us to move, IMHO.Quoted 4 lines
But after patch 2, *if* any driver actually depends on being informed of failure *before* starting the VM, that will not happen. I think both changes could be made at once so that this intermediate state never exists.
I see what you meant. I think there should have no such user.
It's because we always notify FAILED at
migration_cleanup()for precopy, or even postcopy before the cpr-exec work (before QEMU 9.0).That behavior of "notify FAILED before
vm_start()for postcopy" is very specific and only added after commit4af667f87c("migration: notifier error checking"). IOW, before QEMU 9.0, for both precopy and postcopy we always notify FAILED inmigration_cleanup(), never beforevm_start().I mentioned this in the commit log of previous patch too, where I bet the additional FAILED notification added in
4af667f87cfor postcopy path is an accident (to make it pairing with the "reused DONE", however it turns out we likely shouldn't do either of them..). So I don't expect anything will depend on that behavior, and only for postcopy.The benefit of splitting this patch and previous one is, the previous one is a "fix" of duplicated notifications, hence if we need a backport that can be done without this one. Said that, I don't think one should need it.. It should also make each commits slightly easier to follow, because they're fundamentally two changes.
Quoted 4 lines
Let me know what you think after reading my explanations above. I prefer the split like as-is, but I can still squash it to close the trivially small window that you described. I'll make sure if merged the commit message contains separate discussions on two problems.
Ok, fair points.
PATCH 4/5 migration: Drop explicit block activation in postcopy fail path
In reply to Peter XuPostcopy (in failure path) should share with precopy on disk reactivations. Explicit activiation should used to be fine even if called twice, but after 26f65c01ed ("migration: Do not try to start VM if disk activation fails") we may want to avoid it and always capture failure when reactivation happens (even if we do not expect the failure to happen). Remove this redundant call.
--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)
migration/migration.c +0 -1
@@ -2910,7 +2910,6 @@ fail:if (ms->state !=MIGRATION_STATUS_CANCELLING ) {migrate_set_state (&ms->state, ms->state,MIGRATION_STATUS_FAILED ); } -migration_block_activate (NULL );bql_unlock ();return -1 ; }
Signature · 1 line
2.50.1
Re: [PATCH 4/5] migration: Drop explicit block activation in postcopy fail path
Quoted 17 lines · Peter Xu
Postcopy (in failure path) should share with precopy on disk reactivations. Explicit activiation should used to be fine even if called twice, but after
26f65c01ed("migration: Do not try to start VM if disk activation fails") we may want to avoid it and always capture failure when reactivation happens (even if we do not expect the failure to happen). Remove this redundant call.Signed-off-by Peter Xu <[email protected]>--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)migration/migration.c +0
-0
migration/migration.c fail: · line 2915
Quoted 7 lines
migration/migration.c +0
-1@@ -2910,7 +2910,6 @@ fail:if (ms->state !=MIGRATION_STATUS_CANCELLING ) {migrate_set_state (&ms->state, ms->state,MIGRATION_STATUS_FAILED ); } -migration_block_activate (NULL );bql_unlock ();return -1 ; }
PATCH 5/5 migration: Rename MIG_EVENT_PRECOPY_* to MIG_EVENT_*
In reply to Peter XuAll three events are shared between precopy and postcopy, rather than precopy specific.
For example, both precopy and postcopy will go through a SETUP process.
Meanwhile, both FAILED and DONE notifiers will be notified for either precopy or postcopy on completions / failures.
Rename them to make them match what they do, and shorter.
--- include/migration/misc.h | 14 +++++++------- hw/intc/arm_gicv3_kvm.c | 2 +- hw/net/virtio-net.c | 4 ++-- hw/vfio/cpr-legacy.c | 2 +- hw/vfio/cpr.c | 8 ++++---- hw/vfio/migration.c | 4 ++-- migration/cpr-exec.c | 6 +++--- migration/migration.c | 8 ++++---- net/vhost-vdpa.c | 4 ++-- ui/spice-core.c | 6 +++--- 10 files changed, 29 insertions(+), 29 deletions(-)
include/migration/misc.h +7 -7
@@ -60,10 +60,10 @@ bool migration_is_running(void);bool migration_thread_is_self (void );typedef enum MigrationEventType { -MIG_EVENT_PRECOPY_SETUP , -MIG_EVENT_PRECOPY_DONE , -MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_SETUP ,MIG_EVENT_POSTCOPY_START , +MIG_EVENT_DONE , +MIG_EVENT_FAILED ,MIG_EVENT_MAX }MigrationEventType ;
@@ -73,7 +73,7 @@ typedef struct MigrationEvent {/* * A MigrationNotifyFunc may return an error code and an Error object, -* but only when @e->type is MIG_EVENT_PRECOPY_SETUP. The code is an int +* but only when @e->type is MIG_EVENT_SETUP. The code is an int * to allow for different failure modes and recovery actions. */ typedef int (*MigrationNotifyFunc)(NotifierWithReturn *notify,
@@ -83,9 +83,9 @@ typedef int (*MigrationNotifyFunc)(NotifierWithReturn *notify,* Register the notifier @notify to be called when a migration event occurs * for MIG_MODE_NORMAL, as specified by the MigrationEvent passed to @func. * Notifiers may receive events in any of the following orders: -* - MIG_EVENT_PRECOPY_SETUP -> MIG_EVENT_PRECOPY_DONE -* - MIG_EVENT_PRECOPY_SETUP -> MIG_EVENT_PRECOPY_FAILED -* - MIG_EVENT_PRECOPY_FAILED +* - MIG_EVENT_SETUP -> MIG_EVENT_DONE +* - MIG_EVENT_SETUP -> MIG_EVENT_FAILED +* - MIG_EVENT_FAILED */ void migration_add_notifier (NotifierWithReturn *notify,MigrationNotifyFunc func);
hw/intc/arm_gicv3_kvm.c +1 -1
@@ -774,7 +774,7 @@ static void vm_change_state_handler(void *opaque, bool running,static int kvm_arm_gicv3_notifier (NotifierWithReturn *notifier,MigrationEvent *e,Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {GICv3State *s =container_of (notifier, GICv3State, cpr_notifier); return kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES,
hw/net/virtio-net.c +2 -2
@@ -3786,7 +3786,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationEvent *e) should_be_hidden =qatomic_read (&n->failover_primary_hidden); -if (e->type ==MIG_EVENT_PRECOPY_SETUP && !should_be_hidden) { +if (e->type ==MIG_EVENT_SETUP && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev);qapi_event_send_unplug_primary (dev->id);
@@ -3794,7 +3794,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationEvent *e) }else {warn_report ("couldn't unplug primary device" ); } - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {/* We already unplugged the device let's plug it back */ if (!failover_replug_primary (n, dev, &err)) {if (err) {
hw/vfio/cpr-legacy.c +1 -1
@@ -137,7 +137,7 @@ static int vfio_cpr_fail_notifier(NotifierWithReturn *notifier,container_of (notifier, VFIOLegacyContainer, cpr.transfer_notifier);VFIOContainer *bcontainer =VFIO_IOMMU (container); -if (e->type !=MIG_EVENT_PRECOPY_FAILED ) { +if (e->type !=MIG_EVENT_FAILED ) {return 0 ; }
hw/vfio/cpr.c +4 -4
@@ -18,7 +18,7 @@int vfio_cpr_reboot_notifier (NotifierWithReturn *notifier,MigrationEvent *e,Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_SETUP && +if (e->type ==MIG_EVENT_SETUP && !runstate_check (RUN_STATE_SUSPENDED ) && !vm_get_suspended ()) { error_setg(errp,
@@ -186,7 +186,7 @@ static int vfio_cpr_kvm_close_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {vfio_kvm_device_close (); }return 0 ;
@@ -272,9 +272,9 @@ static int vfio_cpr_pci_notifier(NotifierWithReturn *notifier,VFIOPCIDevice *vdev =container_of (notifier, VFIOPCIDevice, cpr.transfer_notifier); -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {return vfio_cpr_set_msi_virq (vdev, errp, false); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {return vfio_cpr_set_msi_virq (vdev, errp, true); }return 0 ;
hw/vfio/migration.c +2 -2
@@ -917,10 +917,10 @@ static int vfio_migration_state_notifier(NotifierWithReturn *notifier,trace_vfio_migration_state_notifier (vbasedev->name, e->type); -if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { +if (e->type ==MIG_EVENT_FAILED ) {/* * MigrationNotifyFunc may not return an error code and an Error -* object for MIG_EVENT_PRECOPY_FAILED. Hence, report the error +* object for MIG_EVENT_FAILED. Hence, report the error * locally and ignore the errp argument. */ ret = vfio_migration_set_state_or_reset(vbasedev,
migration/cpr-exec.c +3 -3
@@ -164,7 +164,7 @@ static void cpr_exec_cb(void *opaque) err =NULL ;/* Note, we can go from state COMPLETED to FAILED */ -migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +migration_call_notifiers (s,MIG_EVENT_FAILED ,NULL );if (!migration_block_activate (&err)) {/* error was already reported */
@@ -182,12 +182,12 @@ static int cpr_exec_notifier(NotifierWithReturn *notifier, MigrationEvent *e, {MigrationState *s =migrate_get_current (); -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {QEMUBH *cpr_exec_bh =qemu_bh_new (cpr_exec_cb,NULL );assert (s->state ==MIGRATION_STATUS_COMPLETED );qemu_bh_schedule (cpr_exec_bh);qemu_notify_event (); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {cpr_exec_unpersist_state (); }return 0 ;
migration/migration.c +4 -4
@@ -1540,7 +1540,7 @@ static void migration_cleanup(MigrationState *s)* migration completed successfully. */ if (!migration_has_failed (s)) { -migration_call_notifiers (s,MIG_EVENT_PRECOPY_DONE ,NULL ); +migration_call_notifiers (s,MIG_EVENT_DONE ,NULL ); }yank_unregister_instance (MIGRATION_YANK_INSTANCE );
@@ -1720,7 +1720,7 @@ int migration_call_notifiers(MigrationState *s, MigrationEventType type, notifier = (NotifierWithReturn *)elem->data; ret = notifier->notify (notifier, &e, errp);if (ret) { -assert (type ==MIG_EVENT_PRECOPY_SETUP ); +assert (type ==MIG_EVENT_SETUP );return ret; } }
@@ -3598,7 +3598,7 @@ static void migration_iteration_finish(MigrationState *s)* Notify FAILED before starting VM, so that devices can invoke * necessary fallbacks before vCPUs run again. */ -migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +migration_call_notifiers (s,MIG_EVENT_FAILED ,NULL );if (runstate_is_live (s->vm_old_state)) {if (!runstate_check (RUN_STATE_SHUTDOWN )) {
@@ -4064,7 +4064,7 @@ void migration_connect(MigrationState *s, Error *error_in) rate_limit =migrate_max_bandwidth ();/* Notify before starting migration thread */ -if (migration_call_notifiers (s,MIG_EVENT_PRECOPY_SETUP , &local_err)) { +if (migration_call_notifiers (s,MIG_EVENT_SETUP , &local_err)) { goto fail; } }
net/vhost-vdpa.c +2 -2
@@ -378,9 +378,9 @@ static int vdpa_net_migration_state_notifier(NotifierWithReturn *notifier, {VhostVDPAState *s =container_of (notifier, VhostVDPAState, migration_state); -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {vhost_vdpa_net_log_global_enable (s, true); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {vhost_vdpa_net_log_global_enable (s, false); }return 0 ;
ui/spice-core.c +3 -3
@@ -583,13 +583,13 @@ static int migration_state_notifier(NotifierWithReturn *notifier,return 0 ; } -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {spice_server_migrate_start (spice_server); - }else if (e->type ==MIG_EVENT_PRECOPY_DONE || + }else if (e->type ==MIG_EVENT_DONE || e->type ==MIG_EVENT_POSTCOPY_START ) {spice_server_migrate_end (spice_server, true); spice_have_target_host = false; - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {spice_server_migrate_end (spice_server, false); spice_have_target_host = false; }
Signature · 1 line
2.50.1
Re: [PATCH 5/5] migration: Rename MIG_EVENT_PRECOPY_* to MIG_EVENT_*
Quoted 258 lines · Peter Xu
All three events are shared between precopy and postcopy, rather than precopy specific.
For example, both precopy and postcopy will go through a SETUP process.
Meanwhile, both FAILED and DONE notifiers will be notified for either precopy or postcopy on completions / failures.
Rename them to make them match what they do, and shorter.
Signed-off-by Peter Xu <[email protected]>--- include/migration/misc.h | 14 +++++++------- hw/intc/arm_gicv3_kvm.c | 2 +- hw/net/virtio-net.c | 4 ++-- hw/vfio/cpr-legacy.c | 2 +- hw/vfio/cpr.c | 8 ++++---- hw/vfio/migration.c | 4 ++-- migration/cpr-exec.c | 6 +++--- migration/migration.c | 8 ++++---- net/vhost-vdpa.c | 4 ++-- ui/spice-core.c | 6 +++--- 10 files changed, 29 insertions(+), 29 deletions(-)include/migration/misc.h +7
-7@@ -60,10 +60,10 @@ bool migration_is_running(void);bool migration_thread_is_self (void );typedef enum MigrationEventType { -MIG_EVENT_PRECOPY_SETUP , -MIG_EVENT_PRECOPY_DONE , -MIG_EVENT_PRECOPY_FAILED , +MIG_EVENT_SETUP ,MIG_EVENT_POSTCOPY_START , +MIG_EVENT_DONE , +MIG_EVENT_FAILED ,MIG_EVENT_MAX }MigrationEventType ;@@ -73,7 +73,7 @@ typedef struct MigrationEvent {/* * A MigrationNotifyFunc may return an error code and an Error object, -* but only when @e->type is MIG_EVENT_PRECOPY_SETUP. The code is an int +* but only when @e->type is MIG_EVENT_SETUP. The code is an int * to allow for different failure modes and recovery actions. */ typedef int (*MigrationNotifyFunc)(NotifierWithReturn *notify,@@ -83,9 +83,9 @@ typedef int (*MigrationNotifyFunc)(NotifierWithReturn *notify,* Register the notifier @notify to be called when a migration event occurs * for MIG_MODE_NORMAL, as specified by the MigrationEvent passed to @func. * Notifiers may receive events in any of the following orders: -* - MIG_EVENT_PRECOPY_SETUP -> MIG_EVENT_PRECOPY_DONE -* - MIG_EVENT_PRECOPY_SETUP -> MIG_EVENT_PRECOPY_FAILED -* - MIG_EVENT_PRECOPY_FAILED +* - MIG_EVENT_SETUP -> MIG_EVENT_DONE +* - MIG_EVENT_SETUP -> MIG_EVENT_FAILED +* - MIG_EVENT_FAILED */ void migration_add_notifier (NotifierWithReturn *notify,MigrationNotifyFunc func);hw/intc/arm_gicv3_kvm.c +1
-1@@ -774,7 +774,7 @@ static void vm_change_state_handler(void *opaque, bool running,static int kvm_arm_gicv3_notifier (NotifierWithReturn *notifier,MigrationEvent *e,Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {GICv3State *s =container_of (notifier, GICv3State, cpr_notifier); return kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES,hw/net/virtio-net.c +2
-2@@ -3786,7 +3786,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationEvent *e) should_be_hidden =qatomic_read (&n->failover_primary_hidden); -if (e->type ==MIG_EVENT_PRECOPY_SETUP && !should_be_hidden) { +if (e->type ==MIG_EVENT_SETUP && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev);qapi_event_send_unplug_primary (dev->id);@@ -3794,7 +3794,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationEvent *e) }else {warn_report ("couldn't unplug primary device" ); } - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {/* We already unplugged the device let's plug it back */ if (!failover_replug_primary (n, dev, &err)) {if (err) {hw/vfio/cpr-legacy.c +1
-1@@ -137,7 +137,7 @@ static int vfio_cpr_fail_notifier(NotifierWithReturn *notifier,container_of (notifier, VFIOLegacyContainer, cpr.transfer_notifier);VFIOContainer *bcontainer =VFIO_IOMMU (container); -if (e->type !=MIG_EVENT_PRECOPY_FAILED ) { +if (e->type !=MIG_EVENT_FAILED ) {return 0 ; }hw/vfio/cpr.c +4
-4@@ -18,7 +18,7 @@int vfio_cpr_reboot_notifier (NotifierWithReturn *notifier,MigrationEvent *e,Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_SETUP && +if (e->type ==MIG_EVENT_SETUP && !runstate_check (RUN_STATE_SUSPENDED ) && !vm_get_suspended ()) { error_setg(errp,@@ -186,7 +186,7 @@ static int vfio_cpr_kvm_close_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {vfio_kvm_device_close (); }return 0 ;@@ -272,9 +272,9 @@ static int vfio_cpr_pci_notifier(NotifierWithReturn *notifier,VFIOPCIDevice *vdev =container_of (notifier, VFIOPCIDevice, cpr.transfer_notifier); -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {return vfio_cpr_set_msi_virq (vdev, errp, false); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {return vfio_cpr_set_msi_virq (vdev, errp, true); }return 0 ;hw/vfio/migration.c +2
-2@@ -917,10 +917,10 @@ static int vfio_migration_state_notifier(NotifierWithReturn *notifier,trace_vfio_migration_state_notifier (vbasedev->name, e->type); -if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { +if (e->type ==MIG_EVENT_FAILED ) {/* * MigrationNotifyFunc may not return an error code and an Error -* object for MIG_EVENT_PRECOPY_FAILED. Hence, report the error +* object for MIG_EVENT_FAILED. Hence, report the error * locally and ignore the errp argument. */ ret = vfio_migration_set_state_or_reset(vbasedev,migration/cpr-exec.c +3
-3@@ -164,7 +164,7 @@ static void cpr_exec_cb(void *opaque) err =NULL ;/* Note, we can go from state COMPLETED to FAILED */ -migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +migration_call_notifiers (s,MIG_EVENT_FAILED ,NULL );if (!migration_block_activate (&err)) {/* error was already reported */ @@ -182,12 +182,12 @@ static int cpr_exec_notifier(NotifierWithReturn *notifier, MigrationEvent *e, {MigrationState *s =migrate_get_current (); -if (e->type ==MIG_EVENT_PRECOPY_DONE ) { +if (e->type ==MIG_EVENT_DONE ) {QEMUBH *cpr_exec_bh =qemu_bh_new (cpr_exec_cb,NULL );assert (s->state ==MIGRATION_STATUS_COMPLETED );qemu_bh_schedule (cpr_exec_bh);qemu_notify_event (); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {cpr_exec_unpersist_state (); }return 0 ;migration/migration.c +4
-4@@ -1540,7 +1540,7 @@ static void migration_cleanup(MigrationState *s)* migration completed successfully. */ if (!migration_has_failed (s)) { -migration_call_notifiers (s,MIG_EVENT_PRECOPY_DONE ,NULL ); +migration_call_notifiers (s,MIG_EVENT_DONE ,NULL ); }yank_unregister_instance (MIGRATION_YANK_INSTANCE );@@ -1720,7 +1720,7 @@ int migration_call_notifiers(MigrationState *s, MigrationEventType type, notifier = (NotifierWithReturn *)elem->data; ret = notifier->notify (notifier, &e, errp);if (ret) { -assert (type ==MIG_EVENT_PRECOPY_SETUP ); +assert (type ==MIG_EVENT_SETUP );return ret; } }@@ -3598,7 +3598,7 @@ static void migration_iteration_finish(MigrationState *s)* Notify FAILED before starting VM, so that devices can invoke * necessary fallbacks before vCPUs run again. */ -migration_call_notifiers (s,MIG_EVENT_PRECOPY_FAILED ,NULL ); +migration_call_notifiers (s,MIG_EVENT_FAILED ,NULL );if (runstate_is_live (s->vm_old_state)) {if (!runstate_check (RUN_STATE_SHUTDOWN )) {@@ -4064,7 +4064,7 @@ void migration_connect(MigrationState *s, Error *error_in) rate_limit =migrate_max_bandwidth ();/* Notify before starting migration thread */ -if (migration_call_notifiers (s,MIG_EVENT_PRECOPY_SETUP , &local_err)) { +if (migration_call_notifiers (s,MIG_EVENT_SETUP , &local_err)) { goto fail; } }net/vhost-vdpa.c +2
-2@@ -378,9 +378,9 @@ static int vdpa_net_migration_state_notifier(NotifierWithReturn *notifier, {VhostVDPAState *s =container_of (notifier, VhostVDPAState, migration_state); -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {vhost_vdpa_net_log_global_enable (s, true); - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {vhost_vdpa_net_log_global_enable (s, false); }return 0 ;ui/spice-core.c +1
-1@@ -583,13 +583,13 @@ static int migration_state_notifier(NotifierWithReturn *notifier,return 0 ; } -if (e->type ==MIG_EVENT_PRECOPY_SETUP ) { +if (e->type ==MIG_EVENT_SETUP ) {spice_server_migrate_start (spice_server);
ui/spice-core.c static int migration_state_notifier(NotifierWithReturn *notifier, · line 595
Quoted 10 lines
ui/spice-core.c +2
-2- }else if (e->type ==MIG_EVENT_PRECOPY_DONE || + }else if (e->type ==MIG_EVENT_DONE || e->type ==MIG_EVENT_POSTCOPY_START ) {spice_server_migrate_end (spice_server, true); spice_have_target_host = false; - }else if (e->type ==MIG_EVENT_PRECOPY_FAILED ) { + }else if (e->type ==MIG_EVENT_FAILED ) {spice_server_migrate_end (spice_server, false); spice_have_target_host = false; }
Quoted 30 lines · Peter Xu
CI: https://gitlab.com/peterx/qemu/-/pipelines/2280356561
Two major goals for this small series:
- Fix postcopy issue where DONE and FAILED notifiers will be invoked twice
- Move FAILED notifier to be before
vm_start()if the failure happens during switchover (where we will stop the VM first)The 2nd goal will be needed by Stefan's ongoing work on block persistent reservations, where a fallback should be required on src to happen before
vm_start(). Instead of introducing anotherFAILED_BEFORE_START, this patchset should make FAILED work instead.Patch 1 adds a tracepoint for me to verify this fix.
Patch 2-3 are the real changes of above two.
Patch 3-4 are some cleanups alone the context that we can do, hence attached at the end.
More details in commit logs individually. Comments welcomed, thanks.
Peter Xu (5):
migration: Add a tracepoint for invoking migration notifiers
migration: Fix double notification of DONE/FAIL for postcopy
migration: Notify migration FAILED before starting VM
migration: Drop explicit block activation in postcopy fail path
migration: Rename MIG_EVENT_PRECOPY_* to MIG_EVENT_*
Quoted 16 lines
include/migration/misc.h | 15 ++++++++------- hw/intc/arm_gicv3_kvm.c | 2 +- hw/net/virtio-net.c | 4 ++-- hw/vfio/cpr-legacy.c | 2 +- hw/vfio/cpr.c | 8 ++++---- hw/vfio/migration.c | 4 ++-- migration/cpr-exec.c | 6 +++--- migration/migration.c | 30 +++++++++++++++++++++--------- net/vhost-vdpa.c | 4 ++-- ui/spice-core.c | 7 ++++--- migration/trace-events | 1 + 11 files changed, 49 insertions(+), 34 deletions(-)Signature · 1 line
2.50.1
Thank you!