Skip to contentSkip to conversation

migration/postcopy: ensure preempt channel is ready before loading states

PATCH Superseded Wei Wang 3 participants 8 messages Apr 4, 2024 – Apr 5, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Wei Wang

author
#0 lore raw

PATCH 1/1 migration/postcopy: ensure preempt channel is ready before loading states

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the preempt channel. So yield to the main thread in the case that the channel create event has been dispatched.

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c +17 -0
index 388d7af7cd..fbc9f2bdd4 100644
@@ -2342,6 +2342,23 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel has
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt channel.
+     * So yield to the main thread in the case that the channel create event
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(), qemu_coroutine_self());
+        qemu_coroutine_yield();
+    } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 1));
+
     ret = qemu_loadvm_state_main(packf, mis);
     trace_loadvm_handle_cmd_packaged_main(ret);
     qemu_fclose(packf);
Signature · 1 line
2.27.0
#1 lore raw
Quoted 29 lines · Wei Wang

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the preempt channel. So yield to the main thread in the case that the channel create event has been dispatched.

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c +6 -0
index 388d7af7cd..fbc9f2bdd4 100644
@@ -2342,6 +2342,23 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel has
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt channel.
+     * So yield to the main thread in the case that the channel create event
+     * has been dispatched.

migration/savevm.c static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis) · line 2360

Quoted 10 lines
migration/savevm.c +10 -0
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(), qemu_coroutine_self());
+        qemu_coroutine_yield();
+    } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 1));

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

Besides, this patch kept the sem_wait() in postcopy_preempt_thread() so it will wait() on this sem again. If this qemu_sem_timedwait() accidentally consumed the sem count then I think the other thread can hang forever?

That's why I put the wait before creation of the preempt thread (in postcopy_ram_incoming_setup()), as we can only consume the sem once, so we must prepare the qemufile when the thread is created.

Thanks,

Quoted 7 lines
migration/savevm.c +1 -0
+
     ret = qemu_loadvm_state_main(packf, mis);
     trace_loadvm_handle_cmd_packaged_main(ret);
     qemu_fclose(packf);
Signature · 1 line
2.27.0
Signature · 1 line
Peter Xu
#2 lore raw
Quoted 42 lines · Peter Xu
Quoted 30 lines · Wei Wang

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the preempt channel. So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced249@intel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +4 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel has
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt channel.

migration/savevm.c static int · line 2359

Quoted 10 lines
migration/savevm.c +10 -0
+     * So yield to the main thread in the case that the channel create event
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),

qemu_coroutine_self());

Quoted 6 lines

migration/savevm.c static int · line 2362

Quoted 3 lines
migration/savevm.c +3 -0
+        qemu_coroutine_yield();
+    } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

No. qemu_sem_timedwait returns false when timeout, which means sem isn’t posted yet. So it needs to go back to the loop. (the patch was tested)

Quoted 4 lines

Besides, this patch kept the sem_wait() in postcopy_preempt_thread() so it will wait() on this sem again. If this qemu_sem_timedwait() accidentally consumed the sem count then I think the other thread can hang forever?

I can get the issue you mentioned, and seems better to be placed before the creation of the preempt thread. Then we probably don’t need to wait_sem in the preempt thread, as the channel is guaranteed to be ready when it runs?

Update will be:

migration/postcopy-ram.c +9 -5
index eccff499cb..5a70ce4f23 100644
@@ -1254,6 +1254,15 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
     }
 
     if (migrate_postcopy_preempt()) {
+        do {
+            if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+                mis->postcopy_qemufile_dst) {
+                break;
+            }
+            aio_co_schedule(qemu_get_current_aio_context(), qemu_coroutine_self());
+            qemu_coroutine_yield();
+        } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 1));
+
         /*
          * This thread needs to be created after the temp pages because
          * it'll fetch RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately.
@@ -1743,12 +1752,6 @@ void *postcopy_preempt_thread(void *opaque)
 
     qemu_sem_post(&mis->thread_sync_sem);
 
-    /*
-     * The preempt channel is established in asynchronous way.  Wait
-     * for its completion.
-     */
-    qemu_sem_wait(&mis->postcopy_qemufile_dst_done);
 
 
 
#3 lore raw
Quoted 46 lines · Peter Xu
Quoted 46 lines · Wei Wang
Quoted 30 lines

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the preempt channel. So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced249@intel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +4 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel has
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt channel.

migration/savevm.c static int · line 2359

Quoted 10 lines
migration/savevm.c +10 -0
+     * So yield to the main thread in the case that the channel create event
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),

qemu_coroutine_self());

Quoted 3 lines
migration/savevm.c +3 -0
+        qemu_coroutine_yield();
+    } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));
Quoted 5 lines
Quoted 2 lines

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

No. qemu_sem_timedwait returns false when timeout, which means sem isn’t posted yet. So it needs to go back to the loop. (the patch was tested)

When timeout, qemu_sem_timedwait() will return -1. I think the patch test passed may because you will always have at least one yield (the first yield in the do ...while ...) when loadvm_handle_cmd_packaged()?

Quoted 48 lines
Quoted 5 lines

Besides, this patch kept the sem_wait() in postcopy_preempt_thread() so it will wait() on this sem again. If this qemu_sem_timedwait() accidentally consumed the sem count then I think the other thread can hang forever?

I can get the issue you mentioned, and seems better to be placed before the creation of the preempt thread. Then we probably don’t need to wait_sem in the preempt thread, as the channel is guaranteed to be ready when it runs?

Update will be:

migration/postcopy-ram.c +9 -5
index eccff499cb..5a70ce4f23 100644
@@ -1254,6 +1254,15 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
     }
 
     if (migrate_postcopy_preempt()) {
+        do {
+            if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+                mis->postcopy_qemufile_dst) {
+                break;
+            }
+            aio_co_schedule(qemu_get_current_aio_context(), qemu_coroutine_self());
+            qemu_coroutine_yield();
+        } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 1));
+
         /*
          * This thread needs to be created after the temp pages because
          * it'll fetch RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately.
@@ -1743,12 +1752,6 @@ void *postcopy_preempt_thread(void *opaque)
 
     qemu_sem_post(&mis->thread_sync_sem);
 
-    /*
-     * The preempt channel is established in asynchronous way.  Wait
-     * for its completion.
-     */
-    qemu_sem_wait(&mis->postcopy_qemufile_dst_done);
 
 
 
#4 lore raw
Quoted 51 lines · Lei
Quoted 49 lines · Peter Xu
Quoted 48 lines · Wei Wang
Quoted 30 lines

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the preempt channel. So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced249@intel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +4 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel has
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt channel.

migration/savevm.c static int · line 2359

Quoted 10 lines
migration/savevm.c +10 -0
+     * So yield to the main thread in the case that the channel create event
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),

qemu_coroutine_self());

migration/savevm.c static int · line 2362

Quoted 3 lines
migration/savevm.c +3 -0
+        qemu_coroutine_yield();
+    } while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

Quoted 6 lines
Quoted 2 lines

No. qemu_sem_timedwait returns false when timeout, which means sem isn’t posted yet. So it needs to go back to the loop. (the patch was tested)

When timeout, qemu_sem_timedwait() will return -1. I think the patch test passed may because you will always have at least one yield (the first yield in the do ...while ...) when loadvm_handle_cmd_packaged()?

My guess is that here the kick will work and qemu_sem_timedwait() later will ETIMEOUT -> qemu_sem_timedwait() returns -1, then the loop just broke. That aio schedule should make sure anyway that the file is ready; the preempt thread must run before this to not hang that thread.

I think it more kind of justifies that the retval needs to be properly defined. :( It's confusion is on top of when I know libpthread returns positive error codes.

Thans,

Signature · 1 line
Peter Xu
#5 lore raw
Quoted 62 lines · Peter Xu
Quoted 6 lines · Lei
Quoted 4 lines · Peter Xu
Quoted 4 lines · Wei Wang
Quoted 3 lines

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the

preempt channel.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced249@i
ntel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +0 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

migration/savevm.c static int · line 2347

Quoted 4 lines
migration/savevm.c +2 -0
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel

has

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

migration/savevm.c static int · line 2349

Quoted 2 lines
migration/savevm.c +2 -0
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt

channel.

Quoted 1 line
Quoted 1 line
Quoted 1 line

migration/savevm.c static int · line 2350

Quoted 1 line
migration/savevm.c +1 -0
+     * So yield to the main thread in the case that the channel create

event

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

migration/savevm.c static int · line 2359

Quoted 9 lines
migration/savevm.c +9 -0
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),

qemu_coroutine_self());

Quoted 4 lines
migration/savevm.c +4 -0
+        qemu_coroutine_yield();
+    } while
+ (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

No. qemu_sem_timedwait returns false when timeout, which means sem

isn’t posted yet.

Quoted 2 lines
Quoted 1 line

So it needs to go back to the loop. (the patch was tested)

Quoted 8 lines
Quoted 3 lines

When timeout, qemu_sem_timedwait() will return -1. I think the patch test passed may because you will always have at least one yield (the first yield in the do ...while ...) when loadvm_handle_cmd_packaged()?

My guess is that here the kick will work and qemu_sem_timedwait() later will ETIMEOUT -> qemu_sem_timedwait() returns -1, then the loop just broke. That aio schedule should make sure anyway that the file is ready; the preempt thread must run before this to not hang that thread.

Yes, misread of the return value. It still worked because the loop broke at the "if (mis->postcopy_qemufile_dst)" check.

Even below will work:

do {
    if (mis->postcopy_qemufile_dst) {
        break;
     }

...
} while (1);

I still don’t see the value of using postcopy_qemufile_dst_done sem in the code though It simplify blocks the main thread from creating the preempt channel for 1ms (regardless of the possibility about whether the sem has been be posted or not. We add it for the case it is not posted and need to go back to the loop).

#6 lore raw
Quoted 80 lines · Wei W
Quoted 62 lines · Peter Xu
Quoted 6 lines · Lei
Quoted 4 lines · Peter Xu
Quoted 4 lines · Wei Wang
Quoted 3 lines

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the

preempt channel.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced249@i
ntel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +0 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

migration/savevm.c static int · line 2347

Quoted 4 lines
migration/savevm.c +2 -0
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the preempt channel

has

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

migration/savevm.c static int · line 2349

Quoted 2 lines
migration/savevm.c +2 -0
+     * been ready to use, as some of the states (e.g. via virtio_load) might
+     * trigger page faults that will be handled through the preempt

channel.

Quoted 1 line
Quoted 1 line
Quoted 1 line

migration/savevm.c static int · line 2350

Quoted 1 line
migration/savevm.c +1 -0
+     * So yield to the main thread in the case that the channel create

event

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

migration/savevm.c static int · line 2359

Quoted 9 lines
migration/savevm.c +9 -0
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),

qemu_coroutine_self());

Quoted 4 lines
migration/savevm.c +4 -0
+        qemu_coroutine_yield();
+    } while
+ (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some point later..

No. qemu_sem_timedwait returns false when timeout, which means sem

isn’t posted yet.

Quoted 2 lines
Quoted 1 line

So it needs to go back to the loop. (the patch was tested)

Quoted 8 lines
Quoted 3 lines

When timeout, qemu_sem_timedwait() will return -1. I think the patch test passed may because you will always have at least one yield (the first yield in the do ...while ...) when loadvm_handle_cmd_packaged()?

My guess is that here the kick will work and qemu_sem_timedwait() later will ETIMEOUT -> qemu_sem_timedwait() returns -1, then the loop just broke. That aio schedule should make sure anyway that the file is ready; the preempt thread must run before this to not hang that thread.

Yes, misread of the return value. It still worked because the loop broke at the "if (mis->postcopy_qemufile_dst)" check.

Even below will work:

do {
    if (mis->postcopy_qemufile_dst) {
        break;
     }
Quoted 7 lines

...
} while (1);

I still don’t see the value of using postcopy_qemufile_dst_done sem in the code though It simplify blocks the main thread from creating the preempt channel for 1ms (regardless of the possibility about whether the sem has been be posted or not. We add it for the case it is not posted and need to go back to the loop).

I think it used to only wait() in the preempt thread, so that is needed. It's also needed when postcopy is interrupted and need a recover, see loadvm_postcopy_handle_resume(), in that case it's the postcopy ram load thread that waits for it rather than the main thread or preempt thread.

Indeed if we move channel creation out of the preempt thread then it seems we don't need the sem in this path. However the other path will still need it, then when the new channel created (postcopy_preempt_new_channel) we'll need to identify a "switch to postcopy" case or "postcopy recovery" case, only post the sem when the former. I think it might complicate the code, I'll think again tomorrow after a sleep so my brain will work better, but I doubt this is what we want to do at rc3.

If you feel comfortable, please feel free to send a version that you think is the most correct so far (if you prefer no timedwait it's fine), and make sure the test works the best on your side. Then I'll smoke it a bit during weekends. Please always keep that in mind if that will be for rc3 it should be non-intrusive change, or we keep it slow for 9.1 and we don't need to rush.

Thanks,

Signature · 1 line
Peter Xu
#7 lore raw
Quoted 105 lines · Peter Xu
Quoted 33 lines · Wei W
Quoted 32 lines · Peter Xu
Quoted 7 lines · Lei
Quoted 5 lines · Peter Xu
Quoted 5 lines · Wei Wang
Quoted 4 lines

Before loading the guest states, ensure that the preempt channel has been ready to use, as some of the states (e.g. via virtio_load) might trigger page faults that will be handled through the

preempt channel.

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines

So yield to the main thread in the case that the channel create event has been dispatched.

Link:
https://lore.kernel.org/all/9aa5d1be-7801-40dd-83fd-f7e041ced2
49@i
ntel
.com/T/

---
 migration/savevm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
migration/savevm.c index renamed from migration/savevm.c +0 -0

388d7af7cd..fbc9f2bdd4 100644

migration/savevm.c +0 -0
@@ -2342,6 +2342,23 @@ static int
 loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

migration/savevm.c static int · line 2348

Quoted 5 lines
migration/savevm.c +3 -0
     QEMUFile *packf = qemu_file_new_input(QIO_CHANNEL(bioc));
 
+    /*
+     * Before loading the guest states, ensure that the
+ preempt channel

has

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
migration/savevm.c +1 -0
+     * been ready to use, as some of the states (e.g. via virtio_load)

might

Quoted 15 lines
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

migration/savevm.c static int · line 2351

Quoted 2 lines
migration/savevm.c +2 -0
+     * trigger page faults that will be handled through the
+ preempt

channel.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

migration/savevm.c static int · line 2353

Quoted 2 lines
migration/savevm.c +2 -0
+     * So yield to the main thread in the case that the
+ channel create

event

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
migration/savevm.c +9 -0
+     * has been dispatched.
+     */
+    do {
+        if (!migrate_postcopy_preempt() || !qemu_in_coroutine() ||
+            mis->postcopy_qemufile_dst) {
+            break;
+        }
+
+        aio_co_schedule(qemu_get_current_aio_context(),
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

qemu_coroutine_self());

migration/savevm.c static int

Quoted 4 lines
migration/savevm.c +4 -0
+        qemu_coroutine_yield();
+    } while
+ (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
+ 1));

I think we need s/!// here, so the same mistake I made? I think we need to rework the retval of qemu_sem_timedwait() at some

point later..

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

No. qemu_sem_timedwait returns false when timeout, which means sem

isn’t posted yet.

Quoted 5 lines
Quoted 1 line

So it needs to go back to the loop. (the patch was tested)

When timeout, qemu_sem_timedwait() will return -1. I think the patch test passed may because you will always have at least one yield (the first yield in the do ...while ...) when

loadvm_handle_cmd_packaged()?

Quoted 3 lines
Quoted 3 lines

My guess is that here the kick will work and qemu_sem_timedwait() later will ETIMEOUT -> qemu_sem_timedwait() returns -1, then the loop

just broke.

Quoted 11 lines
Quoted 2 lines

That aio schedule should make sure anyway that the file is ready; the preempt thread must run before this to not hang that thread.

Yes, misread of the return value. It still worked because the loop broke at the "if (mis->postcopy_qemufile_dst)" check.

Even below will work:

do {
    if (mis->postcopy_qemufile_dst) {
        break;
     }
Quoted 7 lines

...
} while (1);

I still don’t see the value of using postcopy_qemufile_dst_done sem in the code though It simplify blocks the main thread from creating the preempt channel for 1ms (regardless of the possibility about whether the sem has been be posted or not. We add it for the case it is not posted

and need to go back to the loop).

I think it used to only wait() in the preempt thread, so that is needed. It's also needed when postcopy is interrupted and need a recover, see loadvm_postcopy_handle_resume(), in that case it's the postcopy ram load thread that waits for it rather than the main thread or preempt thread.

Quoted 7 lines

Indeed if we move channel creation out of the preempt thread then it seems we don't need the sem in this path. However the other path will still need it, then when the new channel created (postcopy_preempt_new_channel) we'll need to identify a "switch to postcopy" case or "postcopy recovery" case, only post the sem when the former. I think it might complicate the code, I'll think again tomorrow after a sleep so my brain will work better, but I doubt this is what we want to do at rc3.

Yes, it's a bit rushed (no need to remove it completely at rc3).

Quoted 7 lines

If you feel comfortable, please feel free to send a version that you think is the most correct so far (if you prefer no timedwait it's fine), and make sure the test works the best on your side. Then I'll smoke it a bit during weekends. Please always keep that in mind if that will be for rc3 it should be non-intrusive change, or we keep it slow for 9.1 and we don't need to rush.

Sounds good.