Skip to contentSkip to conversation

migration: Fix blocking in POSTCOPY_DEVICE during package load

PATCH In review Pranav Tyagi 3 participants 6 messages Apr 23 – Apr 24

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 migration: Fix blocking in POSTCOPY_DEVICE during package load

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side.

Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

---
V1: https://lore.kernel.org/all/[email protected]/

changed in v2:
- removed postcopy_package_loaded_event and using rp_sem to kick the migration thread
- using migration_rp_wait() in place of qemu_event_wait() in the migration thread

 migration/migration.c | 48 ++++++++++++++++++++++++++++---------------
 migration/migration.h |  1 -
 2 files changed, 31 insertions(+), 18 deletions(-)
migration/migration.c +31 -17
index 5c9aaa6e58..6e4988a590 100644
@@ -1661,7 +1661,6 @@ int migrate_init(MigrationState *s, Error **errp)
     migration_reset_vfio_bytes_transferred();
 
     s->postcopy_package_loaded = false;
-    qemu_event_reset(&s->postcopy_package_loaded_event);
 
     return 0;
 }
@@ -2317,7 +2316,7 @@ static void *source_return_path_thread(void *opaque)
             if (tmp32 == QEMU_VM_PING_PACKAGED_LOADED) {
                 trace_source_return_path_thread_postcopy_package_loaded();
                 ms->postcopy_package_loaded = true;
-                qemu_event_set(&ms->postcopy_package_loaded_event);
+                migration_rp_kick(ms);
             }
             break;
 
@@ -2388,16 +2387,21 @@ out:
         trace_source_return_path_thread_bad_end();
     }
 
-    if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
+    if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER ||
+        ms->state == MIGRATION_STATUS_POSTCOPY_DEVICE) {
         /*
-         * this will be extremely unlikely: that we got yet another network
-         * issue during recovering of the 1st network failure.. during this
-         * period the main migration thread can be waiting on rp_sem for
-         * this thread to sync with the other side.
+         * The migration thread can get stuck waiting for rp_sem if the
+         * return path fails to sync with the destination. This handles
+         * two specific cases:
          *
-         * When this happens, explicitly kick the migration thread out of
-         * RECOVER stage and back to PAUSED, so the admin can try
-         * everything again.
+         * POSTCOPY_RECOVER: A failure occurs during a recovery attempt.
+         * We kick the migration thread back to PAUSED so the admin can
+         * retry.
+         *
+         * POSTCOPY_DEVICE: The MIG_RP_MSG_PONG is lost due to a
+         * network failure or destination crash. We kick the migration
+         * thread out of its wait so it can fail the migration and safely
+         * resume the VM on the source.
          */
         migration_rp_kick(ms);
     }
@@ -3226,12 +3230,24 @@ static MigIterateState migration_iteration_run(MigrationState *s)
         if (s->state == MIGRATION_STATUS_POSTCOPY_DEVICE &&
             (s->postcopy_package_loaded || complete_ready)) {
             /*
-             * If package has been loaded, the event is set and we will
-             * immediatelly transition to POSTCOPY_ACTIVE. If we are ready for
-             * completion, we need to wait for destination to load the postcopy
-             * package before actually completing.
+             * We will immediately transition to POSTCOPY_ACTIVE.
+             * If we are ready for completion, we need to wait for
+             * destination to load the postcopy package before actually
+             * completing.
              */
-            qemu_event_wait(&s->postcopy_package_loaded_event);
+            while (!s->postcopy_package_loaded) {
+                if (migration_rp_wait(s)) {
+                    /*
+                     * Error happened. Migration thread was stuck waiting in
+                     * POSTCOPY_DEVICE for rp_sem which was never set.
+                     */
+                    migrate_set_state(&s->state,
+                                    MIGRATION_STATUS_POSTCOPY_DEVICE,
+                                    MIGRATION_STATUS_FAILING);
+                    return MIG_ITERATE_BREAK;
+                }
+            }
+            /* Acknowledgement received from the destination */
             migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                               MIGRATION_STATUS_POSTCOPY_ACTIVE);
         }
@@ -3863,7 +3879,6 @@ static void migration_instance_finalize(Object *obj)
     qemu_sem_destroy(&ms->rp_state.rp_pong_acks);
     qemu_sem_destroy(&ms->postcopy_qemufile_src_sem);
     error_free(ms->error);
-    qemu_event_destroy(&ms->postcopy_package_loaded_event);
 }
 
 static void migration_instance_init(Object *obj)
@@ -3885,7 +3900,6 @@ static void migration_instance_init(Object *obj)
     qemu_sem_init(&ms->wait_unplug_sem, 0);
     qemu_sem_init(&ms->postcopy_qemufile_src_sem, 0);
     qemu_mutex_init(&ms->qemu_file_lock);
-    qemu_event_init(&ms->postcopy_package_loaded_event, 0);
 }
 
 /*
migration/migration.h +0 -1
index b6888daced..9081e6a612 100644
@@ -512,7 +512,6 @@ struct MigrationState {
     bool rdma_migration;
 
     bool postcopy_package_loaded;
-    QemuEvent postcopy_package_loaded_event;
 
     GSource *hup_source;
 
Signature · 1 line
2.53.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 16 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side.

Quoted 2 lines
Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

I assume Juraj has looked at this already internally, in that case you can always attach his R-b directly when post / repost.

If not, then it becomes a sincere request.. :-D

Thanks!

Signature · 1 line
Peter Xu
#2 lore raw
Quoted 25 lines · Peter Xu
Quoted 16 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side.

Quoted 2 lines
Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

I assume Juraj has looked at this already internally, in that case you can always attach his R-b directly when post / repost.

Quoted 8 lines

If not, then it becomes a sincere request.. :-D

Thanks!

--
Peter Xu

Hello Peter, thanks for the review. Juraj had already reviewed the patch.

In such a case, I'll remember to attach the R-b tag directly from next time.

Regards
Pranav Tyagi

#3 lore raw
In reply to Peter Xu

Hi Peter,

Quoted 17 lines · Peter Xu
Quoted 16 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side.

Quoted 7 lines
Quoted 2 lines
Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

I assume Juraj has looked at this already internally, in that case you can always attach his R-b directly when post / repost.

I did indeed check it, you can include by R-b!

Quoted 8 lines

If not, then it becomes a sincere request.. :-D

Thanks!

Signature · 1 line
Peter Xu
Reviewed-by on v2 1/1
#4 lore raw
In reply to Pranav Tyagi
Quoted 27 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch removes the package_loaded event and uses rp_sem, instead of kicking multiple events. The error is detected in case of network failure or destination crash and rp_sem is set in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for rp_sem. The migration thread then fails early and breaks from the migration loop to resume the vm on the source side.

Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

---
V1: https://lore.kernel.org/all/[email protected]/

changed in v2:
- removed postcopy_package_loaded_event and using rp_sem to kick the migration thread
- using migration_rp_wait() in place of qemu_event_wait() in the migration thread

Quoted 3 lines
 migration/migration.c | 48 ++++++++++++++++++++++++++++---------------
 migration/migration.h |  1 -
 2 files changed, 31 insertions(+), 18 deletions(-)