Skip to contentSkip to conversation

migration: Fix blocking in POSTCOPY_DEVICE during package load

PATCH Superseded Pranav Tyagi 2 participants 3 messages Apr 21 – Apr 22

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 migration: Fix blocking in POSTCOPY_DEVICE during package load

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch uses the error detected in case of network failure or destination crash to set the package_loaded event in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for the package_loaded event. The migration thread then fails early and breaks from the migration loop to resume the VM on the source side.

Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")
---
 migration/migration.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
migration/migration.c +20 -0
index 5c9aaa6e58..1656c1203c 100644
@@ -2386,6 +2386,15 @@ out:
     if (err) {
         migrate_error_propagate(ms, err);
         trace_source_return_path_thread_bad_end();
+        if (ms->state == MIGRATION_STATUS_POSTCOPY_DEVICE) {
+            /*
+             * Kick the migration thread if it gets stuck in
+             * POSTCOPY_DEVICE state waiting for
+             * postcopy_package_loaded_event. The event will never be
+             * set as MIG_RP_MSG_PONG from the destination is lost.
+             */
+            qemu_event_set(&ms->postcopy_package_loaded_event);
+        }
     }
 
     if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
@@ -3232,6 +3241,17 @@ static MigIterateState migration_iteration_run(MigrationState *s)
              * package before actually completing.
              */
             qemu_event_wait(&s->postcopy_package_loaded_event);
+            /*
+             * Check for errors in case the migration thread was stuck in
+             * POSTCOPY_DEVICE state waiting for the
+             * postcopy_package_loaded_event which was never set.
+             * If so, fail now and break out of the iteration.
+             */
+            if (migrate_has_error(s)) {
+                migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
+                                  MIGRATION_STATUS_FAILING);
+                return MIG_ITERATE_BREAK;
+            }
             migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                               MIGRATION_STATUS_POSTCOPY_ACTIVE);
         }
Signature · 1 line
2.53.0
#1 lore raw
Quoted 15 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch uses the error detected in case of network failure or destination crash to set the package_loaded event in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for the package_loaded event. The migration thread then fails early and breaks from the migration loop to resume the VM on the source side.

Quoted 2 lines
Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

Ah I see.. thanks for figuring this out.

Quoted 11 lines
---
 migration/migration.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
migration/migration.c +0 -0
index 5c9aaa6e58..1656c1203c 100644
@@ -2386,6 +2386,15 @@ out:
     if (err) {
         migrate_error_propagate(ms, err);

migration/migration.c out: · line 2397

Quoted 10 lines
migration/migration.c +9 -0
         trace_source_return_path_thread_bad_end();
+        if (ms->state == MIGRATION_STATUS_POSTCOPY_DEVICE) {
+            /*
+             * Kick the migration thread if it gets stuck in
+             * POSTCOPY_DEVICE state waiting for
+             * postcopy_package_loaded_event. The event will never be
+             * set as MIG_RP_MSG_PONG from the destination is lost.
+             */
+            qemu_event_set(&ms->postcopy_package_loaded_event);
+        }

This makes sense. Said that, we have another similar case right below for the postcopy recover path:

if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
/*
* this will be extremely unlikely: that we got yet another network
* issue during recovering of the 1st network failure.. during this
* period the main migration thread can be waiting on rp_sem for
* this thread to sync with the other side.
*
* When this happens, explicitly kick the migration thread out of
* RECOVER stage and back to PAUSED, so the admin can try
* everything again.
*/
migration_rp_kick(ms);
}

It achieves some similar goal, where the migration thread is waiting for some event from return path thread, and when the channel is broken we want to kick it out.

I'm thinking if we can reuse that, instead of kicking multiple events.

Say, we can do migration_rp_kick() for both POSTCOPY_RECOVER and POSTCOPY_DEVICE states in the rp thread. Meanwhile at below..

Quoted 8 lines
migration/migration.c +1 -0
     }
 
     if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
@@ -3232,6 +3241,17 @@ static MigIterateState migration_iteration_run(MigrationState *s)
              * package before actually completing.
              */
             qemu_event_wait(&s->postcopy_package_loaded_event);
+            /*

migration/migration.c static MigIterateState migration_iteration_run(MigrationState *s) · line 3254

Quoted 10 lines
migration/migration.c +10 -0
+             * Check for errors in case the migration thread was stuck in
+             * POSTCOPY_DEVICE state waiting for the
+             * postcopy_package_loaded_event which was never set.
+             * If so, fail now and break out of the iteration.
+             */
+            if (migrate_has_error(s)) {
+                migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
+                                  MIGRATION_STATUS_FAILING);
+                return MIG_ITERATE_BREAK;
+            }

... instead of waiting on postcopy_package_loaded_event, we can do:

    while (!s->postcopy_package_loaded) {
        if (migration_rp_wait(s)) {
            /* Error happened */
            migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                              MIGRATION_STATUS_FAILING);
            return MIG_ITERATE_BREAK;
        }
    }
    /* Acknowledgement received from dest */
    migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                      MIGRATION_STATUS_POSTCOPY_ACTIVE);

PS: I think event also works in this case, so likely either one shared sem or event should work; it's just that it was a sem before, and rp_sem has a name that is generic enough to apply directly to the postcopy_package_loaded_event use case.

Thanks,

Quoted 6 lines
migration/migration.c +0 -0
             migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                               MIGRATION_STATUS_POSTCOPY_ACTIVE);
         }
Signature · 1 line
2.53.0
Signature · 1 line
Peter Xu
#2 lore raw
Quoted 114 lines · Peter Xu
Quoted 15 lines · Pranav Tyagi

The package_loaded event is not set in case MIG_RP_MSG_PONG does not arrive on the source from the destination in the return path thread. The migration thread would then be blocked waiting for package_loaded event indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the source VM can safely resume as the destination has not yet started. The pong message can get lost in case of a network failure or destination crash before sending the pong.

This patch uses the error detected in case of network failure or destination crash to set the package_loaded event in the out path of the return path thread. This will kick the migration thread out from a condition of indefinitely waiting for the package_loaded event. The migration thread then fails early and breaks from the migration loop to resume the VM on the source side.

Quoted 2 lines
Fixes 7b842fe354c6 ("migration: Introduce POSTCOPY_DEVICE state")

Ah I see.. thanks for figuring this out.

Quoted 11 lines
---
 migration/migration.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
migration/migration.c +0 -0
index 5c9aaa6e58..1656c1203c 100644
@@ -2386,6 +2386,15 @@ out:
     if (err) {
         migrate_error_propagate(ms, err);

migration/migration.c out: · line 2397

Quoted 10 lines
migration/migration.c +9 -0
         trace_source_return_path_thread_bad_end();
+        if (ms->state == MIGRATION_STATUS_POSTCOPY_DEVICE) {
+            /*
+             * Kick the migration thread if it gets stuck in
+             * POSTCOPY_DEVICE state waiting for
+             * postcopy_package_loaded_event. The event will never be
+             * set as MIG_RP_MSG_PONG from the destination is lost.
+             */
+            qemu_event_set(&ms->postcopy_package_loaded_event);
+        }

This makes sense. Said that, we have another similar case right below for the postcopy recover path:

if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
/*
* this will be extremely unlikely: that we got yet another network
* issue during recovering of the 1st network failure.. during this
* period the main migration thread can be waiting on rp_sem for
* this thread to sync with the other side.
*
* When this happens, explicitly kick the migration thread out of
* RECOVER stage and back to PAUSED, so the admin can try
* everything again.
*/
migration_rp_kick(ms);
}

It achieves some similar goal, where the migration thread is waiting for some event from return path thread, and when the channel is broken we want to kick it out.

I'm thinking if we can reuse that, instead of kicking multiple events.

Say, we can do migration_rp_kick() for both POSTCOPY_RECOVER and POSTCOPY_DEVICE states in the rp thread. Meanwhile at below..

migration/migration.c static MigIterateState

Quoted 4 lines
migration/migration.c +0 -0
     }
 
     if (ms->state == MIGRATION_STATUS_POSTCOPY_RECOVER) {
@@ -3232,6 +3241,17 @@ static MigIterateState

migration_iteration_run(MigrationState *s)

migration/migration.c static MigIterateState migration_iteration_run(MigrationState *s)

Quoted 5 lines
migration/migration.c +2 -0
              * package before actually completing.
              */
             qemu_event_wait(&s->postcopy_package_loaded_event);
+            /*
+             * Check for errors in case the migration thread was stuck

in

migration/migration.c static MigIterateState migration_iteration_run(MigrationState *s)

Quoted 6 lines
migration/migration.c +6 -0
+             * POSTCOPY_DEVICE state waiting for the
+             * postcopy_package_loaded_event which was never set.
+             * If so, fail now and break out of the iteration.
+             */
+            if (migrate_has_error(s)) {
+                migrate_set_state(&s->state,

MIGRATION_STATUS_POSTCOPY_DEVICE,

migration/migration.c static MigIterateState migration_iteration_run(MigrationState *s) · line 3254

Quoted 3 lines
migration/migration.c +3 -0
+                                  MIGRATION_STATUS_FAILING);
+                return MIG_ITERATE_BREAK;
+            }

... instead of waiting on postcopy_package_loaded_event, we can do:

    while (!s->postcopy_package_loaded) {
        if (migration_rp_wait(s)) {
            /* Error happened */
            migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                              MIGRATION_STATUS_FAILING);
            return MIG_ITERATE_BREAK;
        }
    }
    /* Acknowledgement received from dest */
    migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
                      MIGRATION_STATUS_POSTCOPY_ACTIVE);

PS: I think event also works in this case, so likely either one shared sem or event should work; it's just that it was a sem before, and rp_sem has a name that is generic enough to apply directly to the postcopy_package_loaded_event use case.

Thanks,

migration/migration.c static MigIterateState · line 3255

Quoted 1 line
migration/migration.c +0 -0
             migrate_set_state(&s->state,

MIGRATION_STATUS_POSTCOPY_DEVICE,

Quoted 10 lines
Quoted 5 lines
migration/migration.c +0 -1
                               MIGRATION_STATUS_POSTCOPY_ACTIVE);
         }
--

2.53.0

--
Peter Xu

Thanks for the feedback Peter. I will update the patch accordingly and send it again.

Regards
Pranav Tyagi