Skip to contentSkip to conversation

virtio-balloon: always indicate S_DONE when migration fails

PATCH In review David Hildenbrand 3 participants 7 messages Jun 29, 2020 – Jul 23, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 virtio-balloon: always indicate S_DONE when migration fails

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>
---
 hw/virtio/virtio-balloon.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
hw/virtio/virtio-balloon.c +20 -6
index 10507b2a43..8a84718490 100644
@@ -628,8 +628,13 @@ static void virtio_balloon_free_page_done(VirtIOBalloon *s)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(s);
 
-    s->free_page_report_status = FREE_PAGE_REPORT_S_DONE;
-    virtio_notify_config(vdev);
+    if (s->free_page_report_status != FREE_PAGE_REPORT_S_DONE) {
+        /* See virtio_balloon_free_page_stop() */
+        qemu_mutex_lock(&s->free_page_lock);
+        s->free_page_report_status = FREE_PAGE_REPORT_S_DONE;
+        qemu_mutex_unlock(&s->free_page_lock);
+        virtio_notify_config(vdev);
+    }
 }
 
 static int
@@ -653,17 +658,26 @@ virtio_balloon_free_page_report_notify(NotifierWithReturn *n, void *data)
     case PRECOPY_NOTIFY_SETUP:
         precopy_enable_free_page_optimization();
         break;
-    case PRECOPY_NOTIFY_COMPLETE:
-    case PRECOPY_NOTIFY_CLEANUP:
     case PRECOPY_NOTIFY_BEFORE_BITMAP_SYNC:
         virtio_balloon_free_page_stop(dev);
         break;
     case PRECOPY_NOTIFY_AFTER_BITMAP_SYNC:
         if (vdev->vm_running) {
             virtio_balloon_free_page_start(dev);
-        } else {
-            virtio_balloon_free_page_done(dev);
+            break;
         }
+        /*
+         * Set S_DONE before migrating the vmstate, so the guest will reuse
+         * all hinted pages once running on the destination. Fall through.
+         */
+    case PRECOPY_NOTIFY_CLEANUP:
+        /*
+         * Especially, if something goes wrong during precopy or if migration
+         * is canceled, we have to properly communicate S_DONE to the VM.
+         */
+        virtio_balloon_free_page_done(dev);
+        break;
+    case PRECOPY_NOTIFY_COMPLETE:
         break;
     default:
         virtio_error(vdev, "%s: %d reason unknown", __func__, pnd->reason);
Signature · 6 lines
2.26.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#1 lore raw
Quoted 14 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

Quoted 8 lines

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>

IIUC this is superceded by Alexander's patches right? If not pls rebase ...

Quoted 57 lines
---
 hw/virtio/virtio-balloon.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
hw/virtio/virtio-balloon.c +20 -6
index 10507b2a43..8a84718490 100644
@@ -628,8 +628,13 @@ static void virtio_balloon_free_page_done(VirtIOBalloon *s)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(s);
 
-    s->free_page_report_status = FREE_PAGE_REPORT_S_DONE;
-    virtio_notify_config(vdev);
+    if (s->free_page_report_status != FREE_PAGE_REPORT_S_DONE) {
+        /* See virtio_balloon_free_page_stop() */
+        qemu_mutex_lock(&s->free_page_lock);
+        s->free_page_report_status = FREE_PAGE_REPORT_S_DONE;
+        qemu_mutex_unlock(&s->free_page_lock);
+        virtio_notify_config(vdev);
+    }
 }
 
 static int
@@ -653,17 +658,26 @@ virtio_balloon_free_page_report_notify(NotifierWithReturn *n, void *data)
     case PRECOPY_NOTIFY_SETUP:
         precopy_enable_free_page_optimization();
         break;
-    case PRECOPY_NOTIFY_COMPLETE:
-    case PRECOPY_NOTIFY_CLEANUP:
     case PRECOPY_NOTIFY_BEFORE_BITMAP_SYNC:
         virtio_balloon_free_page_stop(dev);
         break;
     case PRECOPY_NOTIFY_AFTER_BITMAP_SYNC:
         if (vdev->vm_running) {
             virtio_balloon_free_page_start(dev);
-        } else {
-            virtio_balloon_free_page_done(dev);
+            break;
         }
+        /*
+         * Set S_DONE before migrating the vmstate, so the guest will reuse
+         * all hinted pages once running on the destination. Fall through.
+         */
+    case PRECOPY_NOTIFY_CLEANUP:
+        /*
+         * Especially, if something goes wrong during precopy or if migration
+         * is canceled, we have to properly communicate S_DONE to the VM.
+         */
+        virtio_balloon_free_page_done(dev);
+        break;
+    case PRECOPY_NOTIFY_COMPLETE:
         break;
     default:
         virtio_error(vdev, "%s: %d reason unknown", __func__, pnd->reason);
Signature · 1 line
2.26.2

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#2 lore raw
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Quoted 7 lines
Quoted 5 lines
Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>

IIUC this is superceded by Alexander's patches right?

Not that I know ... @Alex?

Quoted 2 lines

If not pls rebase ...

Signature · 8 lines
Thanks,

David / dhildenb


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#3 lore raw
Quoted 25 lines · David Hildenbrand
Quoted 24 lines · Michael S. Tsirkin
Quoted 22 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>
Quoted 4 lines
Quoted 1 line

IIUC this is superceded by Alexander's patches right?

Not that I know ... @Alex?

Okay, I'm confused, that patch is already upstream (via your tree)?

dd8eeb9671fc ("virtio-balloon: always indicate S_DONE when migration fails")

Did you stumble over this mail by mistake again?

Signature · 8 lines
Thanks,

David / dhildenb


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#4 lore raw
Quoted 29 lines · David Hildenbrand
Quoted 27 lines · David Hildenbrand
Quoted 25 lines · Michael S. Tsirkin
Quoted 14 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

Quoted 8 lines

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>

IIUC this is superceded by Alexander's patches right?

Quoted 8 lines
Quoted 2 lines

Not that I know ... @Alex?

Okay, I'm confused, that patch is already upstream (via your tree)?

dd8eeb9671fc ("virtio-balloon: always indicate S_DONE when migration fails")

Did you stumble over this mail by mistake again?

I agree. David's patches should be upstream, and if they aren't they should be applied before mine. Mine are meant to be a follow-on as I end the set with the "report"->"hint" rename for several fields. The idea is the fixes go first and the rename comes last.

Thanks.

- Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#5 lore raw
In reply to David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 27 lines · David Hildenbrand
Quoted 25 lines · Michael S. Tsirkin
Quoted 14 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

Quoted 8 lines

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>

IIUC this is superceded by Alexander's patches right?

Quoted 13 lines
Quoted 2 lines

Not that I know ... @Alex?

Okay, I'm confused, that patch is already upstream (via your tree)?

dd8eeb9671fc ("virtio-balloon: always indicate S_DONE when migration fails")

Did you stumble over this mail by mistake again?

Signature · 3 lines
Thanks,

David / dhildenb

Oh. I guess that's what happened. I saw the code in the tree and thought it came in from Alex's patch.
Sorry about the noise.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#6 lore raw
In reply to David Hildenbrand
Quoted 25 lines · David Hildenbrand
Quoted 24 lines · Michael S. Tsirkin
Quoted 22 lines · David Hildenbrand

If something goes wrong during precopy, before stopping the VM, we will never send a S_DONE indication to the VM, resulting in the hinted pages not getting released to be used by the guest OS (e.g., Linux).

Easy to reproduce:
1. Start migration (e.g., HMP "migrate -d 'exec:gzip -c > STATEFILE.gz'")
2. Cancel migration (e.g., HMP "migrate_cancel")
3. Oberve in the guest (e.g., cat /proc/meminfo) that there is basically no free memory left.

While at it, add similar locking to virtio_balloon_free_page_done() as done in virtio_balloon_free_page_stop. Locking is still weird, but that has to be sorted out separately.

There is nothing to do in the PRECOPY_NOTIFY_COMPLETE case. Add some comments regarding S_DONE handling.

Fixes c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
Cc Wei Wang <[email protected]>
Cc Alexander Duyck <[email protected]>
Quoted 6 lines
Quoted 1 line

IIUC this is superceded by Alexander's patches right?

Not that I know ... @Alex?

Quoted 2 lines

If not pls rebase ...

OK then I guess I was confused. This is older so I guess I should have applied this and asked Alex to rebase his patches, but I did the reverse.., Sorry about that. Could you rebase on top of the pci tree pls?

Thanks and sorry for messing up.

Quoted 6 lines
Signature · 4 lines
-- 
Thanks,

David / dhildenb

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]