Skip to contentSkip to conversation

virtio-mem: Support "x-ignore-shared" migration

PATCH Superseded David Hildenbrand 4 participants 15 messages Jun 20, 2023 – Jul 6, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover virtio-mem: Support "x-ignore-shared" migration

Stumbling over "x-ignore-shared" migration support for virtio-mem on my todo list, I remember talking to Dave G. a while ago about how ram_block_discard_range() in MAP_PIRVATE file mappings is possibly harmful when the file is used somewhere else -- for example, with VM templating in multiple VMs.

This series adds a warning to ram_block_discard_range() in that problematic case and adds "x-ignore-shared" migration support for virtio-mem, which is pretty straight-forward. The last patch also documents how VM templating interacts with virtio-mem.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Juan Quintela <[email protected]>
Cc Peter Xu <[email protected]>
Cc Leonardo Bras <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Philippe Mathieu-Daudé" <[email protected]>
Cc Peng Tao <[email protected]>

David Hildenbrand (4):
softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE
file mapping
virtio-mem: Skip most of virtio_mem_unplug_all() without plugged
memory
migration/ram: Expose ramblock_is_ignored() as
migrate_ram_is_ignored()
virtio-mem: Support "x-ignore-shared" migration

 hw/virtio/virtio-mem.c   | 67 ++++++++++++++++++++++++++++------------
 include/migration/misc.h |  1 +
 migration/postcopy-ram.c |  2 +-
 migration/ram.c          | 14 ++++-----
 migration/ram.h          |  3 +-
 softmmu/physmem.c        | 18 +++++++++++
 6 files changed, 76 insertions(+), 29 deletions(-)
Signature · 1 line
2.40.1
#8 lore raw

PATCH 3/4 migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()

virtio-mem wants to know whether it should not mess with the RAMBlock content (e.g., discard RAM, preallocate memory) on incoming migration.

So let's expose that function as migrate_ram_is_ignored() in migration/misc.h

---
 include/migration/misc.h |  1 +
 migration/postcopy-ram.c |  2 +-
 migration/ram.c          | 14 +++++++-------
 migration/ram.h          |  3 +--
 4 files changed, 10 insertions(+), 10 deletions(-)
include/migration/misc.h +1 -0
index 5ebe13b4b9..7dcc0b5c2c 100644
@@ -40,6 +40,7 @@ int precopy_notify(PrecopyNotifyReason reason, Error **errp);
 
 void ram_mig_init(void);
 void qemu_guest_free_page_hint(void *addr, size_t len);
+bool migrate_ram_is_ignored(RAMBlock *block);
 
 /* migration/block.c */
 
migration/postcopy-ram.c +1 -1
index 5615ec29eb..29aea9456d 100644
@@ -408,7 +408,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis, Error **errp)
     /*
      * We don't support postcopy with some type of ramblocks.
      *
-     * NOTE: we explicitly ignored ramblock_is_ignored() instead we checked
+     * NOTE: we explicitly ignored migrate_ram_is_ignored() instead we checked
      * all possible ramblocks.  This is because this function can be called
      * when creating the migration object, during the phase RAM_MIGRATABLE
      * is not even properly set for all the ramblocks.
migration/ram.c +7 -7
index 5283a75f02..0ada6477e8 100644
@@ -194,7 +194,7 @@ static bool postcopy_preempt_active(void)
     return migrate_postcopy_preempt() && migration_in_postcopy();
 }
 
-bool ramblock_is_ignored(RAMBlock *block)
+bool migrate_ram_is_ignored(RAMBlock *block)
 {
     return !qemu_ram_is_migratable(block) ||
            (migrate_ignore_shared() && qemu_ram_is_shared(block)
@@ -696,7 +696,7 @@ static void pss_find_next_dirty(PageSearchStatus *pss)
     unsigned long size = rb->used_length >> TARGET_PAGE_BITS;
     unsigned long *bitmap = rb->bmap;
 
-    if (ramblock_is_ignored(rb)) {
+    if (migrate_ram_is_ignored(rb)) {
         /* Points directly to the end, so we know no dirty page */
         pss->page = size;
         return;
@@ -780,7 +780,7 @@ unsigned long colo_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
 
     *num = 0;
 
-    if (ramblock_is_ignored(rb)) {
+    if (migrate_ram_is_ignored(rb)) {
         return size;
     }
 
@@ -2260,7 +2260,7 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss)
     unsigned long start_page = pss->page;
     int res;
 
-    if (ramblock_is_ignored(pss->block)) {
+    if (migrate_ram_is_ignored(pss->block)) {
         error_report("block %s should not be migrated !", pss->block->idstr);
         return 0;
     }
@@ -3347,7 +3347,7 @@ static inline RAMBlock *ram_block_from_stream(MigrationIncomingState *mis,
         return NULL;
     }
 
-    if (ramblock_is_ignored(block)) {
+    if (migrate_ram_is_ignored(block)) {
         error_report("block %s should not be migrated !", id);
         return NULL;
     }
@@ -3958,7 +3958,7 @@ static int ram_load_precopy(QEMUFile *f)
                     }
                     if (migrate_ignore_shared()) {
                         hwaddr addr = qemu_get_be64(f);
-                        if (ramblock_is_ignored(block) &&
+                        if (migrate_ram_is_ignored(block) &&
                             block->mr->addr != addr) {
                             error_report("Mismatched GPAs for block %s "
                                          "%" PRId64 "!= %" PRId64,
@@ -4254,7 +4254,7 @@ static void ram_mig_ram_block_resized(RAMBlockNotifier *n, void *host,
     RAMBlock *rb = qemu_ram_block_from_host(host, false, &offset);
     Error *err = NULL;
 
-    if (ramblock_is_ignored(rb)) {
+    if (migrate_ram_is_ignored(rb)) {
         return;
     }
 
migration/ram.h +1 -2
index ea1f3c25b5..145c915ca7 100644
@@ -36,11 +36,10 @@
 extern XBZRLECacheStats xbzrle_counters;
 extern CompressionStats compression_counters;
 
-bool ramblock_is_ignored(RAMBlock *block);
 /* Should be holding either ram_list.mutex, or the RCU lock. */
 #define RAMBLOCK_FOREACH_NOT_IGNORED(block)            \
     INTERNAL_RAMBLOCK_FOREACH(block)                   \
-        if (ramblock_is_ignored(block)) {} else
+        if (migrate_ram_is_ignored(block)) {} else
 
 #define RAMBLOCK_FOREACH_MIGRATABLE(block)             \
     INTERNAL_RAMBLOCK_FOREACH(block)                   \
Signature · 1 line
2.40.1
#10 lore raw

PATCH 4/4 virtio-mem: Support "x-ignore-shared" migration

In reply to David Hildenbrand

To achieve desired "x-ignore-shared" functionality, we should not discard all RAM when realizing the device and not mess with preallocation/postcopy when loading device state. In essence, we should not touch RAM content.

As "x-ignore-shared" gets set after realizing the device, we cannot rely on that. Let's simply skip discarding of RAM on incoming migration. Note that virtio_mem_post_load() will call virtio_mem_restore_unplugged() -- unless "x-ignore-shared" is set. So once migration finished we'll have a consistent state.

The initial system reset will also not discard any RAM, because virtio_mem_unplug_all() will not call virtio_mem_unplug_all() when no memory is plugged (which is the case before loading the device state).

Note that something like VM templating -- see commit b17fbbe55cba ("migration: allow private destination ram with x-ignore-shared") -- is currently incompatible with virtio-mem and ram_block_discard_range() will warn in case a private file mapping is supplied by virtio-mem.

For VM templating with virtio-mem, it makes more sense to either
(a) Create the template without the virtio-mem device and hotplug a virtio-mem device to the new VM instances using proper own memory backend.
(b) Use a virtio-mem device that doesn't provide any memory in the template (requested-size=0) and use private anonymous memory.

---
 hw/virtio/virtio-mem.c | 47 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 9 deletions(-)
hw/virtio/virtio-mem.c +38 -9
index 9f6169af32..b013dfbaf0 100644
@@ -18,6 +18,7 @@
 #include "sysemu/numa.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/reset.h"
+#include "sysemu/runstate.h"
 #include "hw/virtio/virtio.h"
 #include "hw/virtio/virtio-bus.h"
 #include "hw/virtio/virtio-access.h"
@@ -886,11 +887,23 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb));
-    if (ret) {
-        error_setg_errno(errp, -ret, "Unexpected error discarding RAM");
-        ram_block_coordinated_discard_require(false);
-        return;
+    /*
+     * We don't know at this point whether shared RAM is migrated using
+     * QEMU or migrated using the file content. "x-ignore-shared" will be
+     * configurated after realizing the device. So in case we have an
+     * incoming migration, simply always skip the discard step.
+     *
+     * Otherwise, make sure that we start with a clean slate: either the
+     * memory backend might get reused or the shared file might still have
+     * memory allocated.
+     */
+    if (!runstate_check(RUN_STATE_INMIGRATE)) {
+        ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb));
+        if (ret) {
+            error_setg_errno(errp, -ret, "Unexpected error discarding RAM");
+            ram_block_coordinated_discard_require(false);
+            return;
+        }
     }
 
     virtio_mem_resize_usable_region(vmem, vmem->requested_size, true);
@@ -962,10 +975,6 @@ static int virtio_mem_post_load(void *opaque, int version_id)
     RamDiscardListener *rdl;
     int ret;
 
-    if (vmem->prealloc && !vmem->early_migration) {
-        warn_report("Proper preallocation with migration requires a newer QEMU machine");
-    }
-
     /*
      * We started out with all memory discarded and our memory region is mapped
      * into an address space. Replay, now that we updated the bitmap.
@@ -978,6 +987,18 @@ static int virtio_mem_post_load(void *opaque, int version_id)
         }
     }
 
+    /*
+     * If shared RAM is migrated using the file content and not using QEMU,
+     * don't mess with preallocation and postcopy.
+     */
+    if (migrate_ram_is_ignored(vmem->memdev->mr.ram_block)) {
+        return 0;
+    }
+
+    if (vmem->prealloc && !vmem->early_migration) {
+        warn_report("Proper preallocation with migration requires a newer QEMU machine");
+    }
+
     if (migration_in_incoming_postcopy()) {
         return 0;
     }
@@ -1010,6 +1031,14 @@ static int virtio_mem_post_load_early(void *opaque, int version_id)
         return 0;
     }
 
+    /*
+     * If shared RAM is migrated using the file content and not using QEMU,
+     * don't mess with preallocation and postcopy.
+     */
+    if (migrate_ram_is_ignored(rb)) {
+        return 0;
+    }
+
     /*
      * We restored the bitmap and verified that the basic properties
      * match on source and destination, so we can go ahead and preallocate
Signature · 1 line
2.40.1
#1 lore raw

PATCH 1/4 softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

In reply to David Hildenbrand

ram_block_discard_range() cannot possibly do the right thing in MAP_PRIVATE file mappings in the general case.

To achieve the documented semantics, we also have to punch a hole into the file, possibly messing with other MAP_PRIVATE/MAP_SHARED mappings of such a file.

For example, using VM templating -- see commit b17fbbe55cba ("migration:
allow private destination ram with x-ignore-shared") -- in combination with any mechanism that relies on discarding of RAM is problematic. This includes:
* Postcopy live migration
* virtio-balloon inflation/deflation or free-page-reporting
* virtio-mem

So at least warn that there is something possibly dangerous is going on when using ram_block_discard_range() in these cases.

---
 softmmu/physmem.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
softmmu/physmem.c +18 -0
index 6bdd944fe8..27c7219c82 100644
@@ -3451,6 +3451,24 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
              * so a userfault will trigger.
              */
 #ifdef CONFIG_FALLOCATE_PUNCH_HOLE
+            /*
+             * We'll discard data from the actual file, even though we only
+             * have a MAP_PRIVATE mapping, possibly messing with other
+             * MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to
+             * change that behavior whithout violating the promised
+             * semantics of ram_block_discard_range().
+             *
+             * Only warn, because it work as long as nobody else uses that
+             * file.
+             */
+            if (!qemu_ram_is_shared(rb)) {
+                warn_report_once("ram_block_discard_range: Discarding RAM"
+                                 " in private file mappings is possibly"
+                                 " dangerous, because it will modify the"
+                                 " underlying file and will affect other"
+                                 " users of the file");
+            }
+
             ret = fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
                             start, length);
             if (ret) {
Signature · 1 line
2.40.1
#7 lore raw

PATCH 2/4 virtio-mem: Skip most of virtio_mem_unplug_all() without plugged memory

In reply to David Hildenbrand

Already when starting QEMU we perform one system reset that ends up triggering virtio_mem_unplug_all() with no actual memory plugged yet. That, in turn will trigger ram_block_discard_range() and perform some other actions that are not required in that case.

Let's optimize virtio_mem_unplug_all() for the case that no memory is plugged. This will be beneficial for x-ignore-shared support as well.

---
 hw/virtio/virtio-mem.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
hw/virtio/virtio-mem.c +10 -10
index 538b695c29..9f6169af32 100644
@@ -606,20 +606,20 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem)
 {
     RAMBlock *rb = vmem->memdev->mr.ram_block;
 
-    if (virtio_mem_is_busy()) {
-        return -EBUSY;
-    }
-
-    if (ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb))) {
-        return -EBUSY;
-    }
-    virtio_mem_notify_unplug_all(vmem);
-
-    bitmap_clear(vmem->bitmap, 0, vmem->bitmap_size);
     if (vmem->size) {
+        if (virtio_mem_is_busy()) {
+            return -EBUSY;
+        }
+        if (ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb))) {
+            return -EBUSY;
+        }
+        virtio_mem_notify_unplug_all(vmem);
+
+        bitmap_clear(vmem->bitmap, 0, vmem->bitmap_size);
         vmem->size = 0;
         notifier_list_notify(&vmem->size_change_notifiers, &vmem->size);
     }
+
     trace_virtio_mem_unplugged_all();
     virtio_mem_resize_usable_region(vmem, vmem->requested_size, true);
     return 0;
Signature · 1 line
2.40.1
#11 lore raw
In reply to David Hildenbrand
Quoted 47 lines · David Hildenbrand

To achieve desired "x-ignore-shared" functionality, we should not discard all RAM when realizing the device and not mess with preallocation/postcopy when loading device state. In essence, we should not touch RAM content.

As "x-ignore-shared" gets set after realizing the device, we cannot rely on that. Let's simply skip discarding of RAM on incoming migration. Note that virtio_mem_post_load() will call virtio_mem_restore_unplugged() -- unless "x-ignore-shared" is set. So once migration finished we'll have a consistent state.

The initial system reset will also not discard any RAM, because virtio_mem_unplug_all() will not call virtio_mem_unplug_all() when no memory is plugged (which is the case before loading the device state).

Note that something like VM templating -- see commit b17fbbe55cba ("migration: allow private destination ram with x-ignore-shared") -- is currently incompatible with virtio-mem and ram_block_discard_range() will warn in case a private file mapping is supplied by virtio-mem.

For VM templating with virtio-mem, it makes more sense to either
(a) Create the template without the virtio-mem device and hotplug a virtio-mem device to the new VM instances using proper own memory backend.
(b) Use a virtio-mem device that doesn't provide any memory in the template (requested-size=0) and use private anonymous memory.

---
 hw/virtio/virtio-mem.c | 47 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 9 deletions(-)
hw/virtio/virtio-mem.c +1 -0
index 9f6169af32..b013dfbaf0 100644
@@ -18,6 +18,7 @@
 #include "sysemu/numa.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/reset.h"
+#include "sysemu/runstate.h"
 #include "hw/virtio/virtio.h"
 #include "hw/virtio/virtio-bus.h"
 #include "hw/virtio/virtio-access.h"
@@ -886,11 +887,23 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp)
         return;
     }

hw/virtio/virtio-mem.c static void virtio_mem_device_realize(DeviceState *dev, Error **errp) · line 893

Quoted 10 lines
hw/virtio/virtio-mem.c +4 -5
 
-    ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb));
-    if (ret) {
-        error_setg_errno(errp, -ret, "Unexpected error discarding RAM");
-        ram_block_coordinated_discard_require(false);
-        return;
+    /*
+     * We don't know at this point whether shared RAM is migrated using
+     * QEMU or migrated using the file content. "x-ignore-shared" will be
+     * configurated

configurated == configured?

Quoted 64 lines

after realizing the device. So in case we have an

hw/virtio/virtio-mem.c +33 -4
+     * incoming migration, simply always skip the discard step.
+     *
+     * Otherwise, make sure that we start with a clean slate: either the
+     * memory backend might get reused or the shared file might still have
+     * memory allocated.
+     */
+    if (!runstate_check(RUN_STATE_INMIGRATE)) {
+        ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb));
+        if (ret) {
+            error_setg_errno(errp, -ret, "Unexpected error discarding RAM");
+            ram_block_coordinated_discard_require(false);
+            return;
+        }
     }
 
     virtio_mem_resize_usable_region(vmem, vmem->requested_size, true);
@@ -962,10 +975,6 @@ static int virtio_mem_post_load(void *opaque, int version_id)
     RamDiscardListener *rdl;
     int ret;
 
-    if (vmem->prealloc && !vmem->early_migration) {
-        warn_report("Proper preallocation with migration requires a newer QEMU machine");
-    }
-
     /*
      * We started out with all memory discarded and our memory region is mapped
      * into an address space. Replay, now that we updated the bitmap.
@@ -978,6 +987,18 @@ static int virtio_mem_post_load(void *opaque, int version_id)
         }
     }
 
+    /*
+     * If shared RAM is migrated using the file content and not using QEMU,
+     * don't mess with preallocation and postcopy.
+     */
+    if (migrate_ram_is_ignored(vmem->memdev->mr.ram_block)) {
+        return 0;
+    }
+
+    if (vmem->prealloc && !vmem->early_migration) {
+        warn_report("Proper preallocation with migration requires a newer QEMU machine");
+    }
+
     if (migration_in_incoming_postcopy()) {
         return 0;
     }
@@ -1010,6 +1031,14 @@ static int virtio_mem_post_load_early(void *opaque, int version_id)
         return 0;
     }
 
+    /*
+     * If shared RAM is migrated using the file content and not using QEMU,
+     * don't mess with preallocation and postcopy.
+     */
+    if (migrate_ram_is_ignored(rb)) {
+        return 0;
+    }
+
     /*
      * We restored the bitmap and verified that the basic properties
      * match on source and destination, so we can go ahead and preallocate
Signature · 1 line
2.40.1
#12 lore raw
Quoted 49 lines · Michael S. Tsirkin
Quoted 48 lines · David Hildenbrand

To achieve desired "x-ignore-shared" functionality, we should not discard all RAM when realizing the device and not mess with preallocation/postcopy when loading device state. In essence, we should not touch RAM content.

As "x-ignore-shared" gets set after realizing the device, we cannot rely on that. Let's simply skip discarding of RAM on incoming migration. Note that virtio_mem_post_load() will call virtio_mem_restore_unplugged() -- unless "x-ignore-shared" is set. So once migration finished we'll have a consistent state.

The initial system reset will also not discard any RAM, because virtio_mem_unplug_all() will not call virtio_mem_unplug_all() when no memory is plugged (which is the case before loading the device state).

Note that something like VM templating -- see commit b17fbbe55cba ("migration: allow private destination ram with x-ignore-shared") -- is currently incompatible with virtio-mem and ram_block_discard_range() will warn in case a private file mapping is supplied by virtio-mem.

For VM templating with virtio-mem, it makes more sense to either
(a) Create the template without the virtio-mem device and hotplug a virtio-mem device to the new VM instances using proper own memory backend.
(b) Use a virtio-mem device that doesn't provide any memory in the template (requested-size=0) and use private anonymous memory.

---
  hw/virtio/virtio-mem.c | 47 ++++++++++++++++++++++++++++++++++--------
  1 file changed, 38 insertions(+), 9 deletions(-)
hw/virtio/virtio-mem.c +1 -0
index 9f6169af32..b013dfbaf0 100644
@@ -18,6 +18,7 @@
  #include "sysemu/numa.h"
  #include "sysemu/sysemu.h"
  #include "sysemu/reset.h"
+#include "sysemu/runstate.h"
  #include "hw/virtio/virtio.h"
  #include "hw/virtio/virtio-bus.h"
  #include "hw/virtio/virtio-access.h"
@@ -886,11 +887,23 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp)
          return;
      }
 
Quoted 11 lines

hw/virtio/virtio-mem.c static void virtio_mem_device_realize(DeviceState *dev, Error **errp)

Quoted 9 lines
hw/virtio/virtio-mem.c +4 -5
-    ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb));
-    if (ret) {
-        error_setg_errno(errp, -ret, "Unexpected error discarding RAM");
-        ram_block_coordinated_discard_require(false);
-        return;
+    /*
+     * We don't know at this point whether shared RAM is migrated using
+     * QEMU or migrated using the file content. "x-ignore-shared" will be
+     * configurated

configurated == configured?

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
Acked-by on v1 1/4
#2 lore raw

Re: [PATCH v1 1/4] softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

In reply to David Hildenbrand
Quoted 15 lines · David Hildenbrand

ram_block_discard_range() cannot possibly do the right thing in MAP_PRIVATE file mappings in the general case.

To achieve the documented semantics, we also have to punch a hole into the file, possibly messing with other MAP_PRIVATE/MAP_SHARED mappings of such a file.

For example, using VM templating -- see commit b17fbbe55cba ("migration:
allow private destination ram with x-ignore-shared") -- in combination with any mechanism that relies on discarding of RAM is problematic. This includes:
* Postcopy live migration
* virtio-balloon inflation/deflation or free-page-reporting
* virtio-mem

Quoted 2 lines

So at least warn that there is something possibly dangerous is going on when using ram_block_discard_range() in these cases.

The issue is probably valid.

One thing I worry is when the user (or, qemu instance) exclusively owns the file, just forgot to attach share=on, where it used to work perfectly then it'll show this warning. But I agree maybe it's good to remind them just to attach the share=on.

For real private mem users, the warning can of real help, one should probably leverage things like file snapshot provided by modern file systems, so each VM should just have its own snapshot ram file to use then map it share=on I suppose.

For the long term, maybe we should simply support private mem here simply by a MADV_DONTNEED. I assume that's the right semantics for postcopy (just need to support MINOR faults, though; MISSING faults definitely will stop working.. but for all the rest framework shouldn't need much change), and I hope that's also the semantics that balloon/virtio-mem wants here. Not sure whether/when that's strongly needed, assuming the corner case above can still be work arounded properly by other means.

For now, a warning looks all sane.

Quoted 2 lines
Quoted 36 lines
---
 softmmu/physmem.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
softmmu/physmem.c +18 -0
index 6bdd944fe8..27c7219c82 100644
@@ -3451,6 +3451,24 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
              * so a userfault will trigger.
              */
 #ifdef CONFIG_FALLOCATE_PUNCH_HOLE
+            /*
+             * We'll discard data from the actual file, even though we only
+             * have a MAP_PRIVATE mapping, possibly messing with other
+             * MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to
+             * change that behavior whithout violating the promised
+             * semantics of ram_block_discard_range().
+             *
+             * Only warn, because it work as long as nobody else uses that
+             * file.
+             */
+            if (!qemu_ram_is_shared(rb)) {
+                warn_report_once("ram_block_discard_range: Discarding RAM"
+                                 " in private file mappings is possibly"
+                                 " dangerous, because it will modify the"
+                                 " underlying file and will affect other"
+                                 " users of the file");
+            }
+
             ret = fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
                             start, length);
             if (ret) {
Signature · 1 line
2.40.1
Signature · 1 line
Peter Xu
Acked-by on v1 3/4
#9 lore raw

Re: [PATCH v1 3/4] migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()

In reply to David Hildenbrand
Quoted 7 lines · David Hildenbrand

virtio-mem wants to know whether it should not mess with the RAMBlock content (e.g., discard RAM, preallocate memory) on incoming migration.

So let's expose that function as migrate_ram_is_ignored() in migration/misc.h

Signature · 1 line
Peter Xu
#3 lore raw

Re: [PATCH v1 1/4] softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

In reply to Peter Xu
Quoted 19 lines · Peter Xu
Quoted 15 lines · David Hildenbrand

ram_block_discard_range() cannot possibly do the right thing in MAP_PRIVATE file mappings in the general case.

To achieve the documented semantics, we also have to punch a hole into the file, possibly messing with other MAP_PRIVATE/MAP_SHARED mappings of such a file.

For example, using VM templating -- see commit b17fbbe55cba ("migration:
allow private destination ram with x-ignore-shared") -- in combination with any mechanism that relies on discarding of RAM is problematic. This includes:
* Postcopy live migration
* virtio-balloon inflation/deflation or free-page-reporting
* virtio-mem

Quoted 2 lines

So at least warn that there is something possibly dangerous is going on when using ram_block_discard_range() in these cases.

Quoted 6 lines

The issue is probably valid.

One thing I worry is when the user (or, qemu instance) exclusively owns the file, just forgot to attach share=on, where it used to work perfectly then it'll show this warning. But I agree maybe it's good to remind them just to attach the share=on.

For memory-backend-memfd "share=on" is fortunately the default. For memory-backend-file it isn't (and in most cases you do want share=on, like for hugetlbfs or tmpfs).

Missing the "share=on" for memory-backend-file can have sane use cases, but for the common /dev/shm/ case it even results in an undesired double-memory consumption (just like memory-backend-memfd,share=off).

Quoted 5 lines

For real private mem users, the warning can of real help, one should probably leverage things like file snapshot provided by modern file systems, so each VM should just have its own snapshot ram file to use then map it share=on I suppose.

Yes, I agree. Although we recently learned that fs-backed VM RAM (SSD) performs poorly and will severely wear your SSD severly :(

Quoted 8 lines

For the long term, maybe we should simply support private mem here simply by a MADV_DONTNEED. I assume that's the right semantics for postcopy (just need to support MINOR faults, though; MISSING faults definitely will stop working.. but for all the rest framework shouldn't need much change), and I hope that's also the semantics that balloon/virtio-mem wants here. Not sure whether/when that's strongly needed, assuming the corner case above can still be work arounded properly by other means.

I briefly thought about that but came to the conclusion that fixing it is not that easy. So I went with the warn.

As documented, ram_block_discard_range() guarantees two things

a) Read 0 after discarding succeeded
b) Make postcopy work by triggering a fault on next access

And if we'd simply want to drop the FALLOC_FL_PUNCH_HOLE:

1) For hugetlb, only newer kernels support MADV_DONTNEED. So there is no way to just discard in a private mapping here that works for kernels we still care about.

2) free-page-reporting wants to read 0's when re-accessing discarded memory. If there is still something there in the file, that won't work.

3) Regarding postcopy on MAP_PRIVATE shmem, I am not sure if it will actually do what you want if the pagecache holds a page. Maybe it works, but I am not so sure. Needs investigation.

Quoted 7 lines

For now, a warning looks all sane.

Quoted 2 lines

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw

Re: [PATCH v1 1/4] softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

Quoted 5 lines · David Hildenbrand

As documented, ram_block_discard_range() guarantees two things

a) Read 0 after discarding succeeded
b) Make postcopy work by triggering a fault on next access

Quoted 8 lines

And if we'd simply want to drop the FALLOC_FL_PUNCH_HOLE:

1) For hugetlb, only newer kernels support MADV_DONTNEED. So there is no way to just discard in a private mapping here that works for kernels we still care about.

2) free-page-reporting wants to read 0's when re-accessing discarded memory. If there is still something there in the file, that won't work.

Ah right. The semantics is indeed slightly different..

IMHO, ideally here we need a zero page installed as private, ignoring the page cache underneath, freeing any possible private page. But I just don't know how to do that easily with current default mm infrastructures, or free-page-reporting over private mem seems just won't really work at all, it seems to me.

Maybe.. UFFDIO_ZEROPAGE would work? We need uffd registered by default, but that's slightly tricky.

Quoted 4 lines

3) Regarding postcopy on MAP_PRIVATE shmem, I am not sure if it will actually do what you want if the pagecache holds a page. Maybe it works, but I am not so sure. Needs investigation.

For MINOR I think it will. I actually already implemented some of that (I think, all of that is required) in the HGM qemu rfc series, and smoked it a bit without any known issue yet with the HGM kernel.

IIUC we can work on MINOR support without HGM; I can separate it out. It's really a matter of whether it'll be worthwhile the effort and time.

Thanks,

Signature · 1 line
Peter Xu
#5 lore raw

Re: [PATCH v1 1/4] softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

Quoted 17 lines · Peter Xu
Quoted 5 lines · David Hildenbrand

As documented, ram_block_discard_range() guarantees two things

a) Read 0 after discarding succeeded
b) Make postcopy work by triggering a fault on next access

Quoted 8 lines

And if we'd simply want to drop the FALLOC_FL_PUNCH_HOLE:

1) For hugetlb, only newer kernels support MADV_DONTNEED. So there is no way to just discard in a private mapping here that works for kernels we still care about.

2) free-page-reporting wants to read 0's when re-accessing discarded memory. If there is still something there in the file, that won't work.

Ah right. The semantics is indeed slightly different..

Quoted 8 lines

IMHO, ideally here we need a zero page installed as private, ignoring the page cache underneath, freeing any possible private page. But I just don't know how to do that easily with current default mm infrastructures, or free-page-reporting over private mem seems just won't really work at all, it seems to me.

Maybe.. UFFDIO_ZEROPAGE would work? We need uffd registered by default, but that's slightly tricky.

Maybe ... depends also on the uffd semantics as in 3).

Quoted 6 lines
Quoted 5 lines

3) Regarding postcopy on MAP_PRIVATE shmem, I am not sure if it will actually do what you want if the pagecache holds a page. Maybe it works, but I am not so sure. Needs investigation.

Quoted 6 lines

For MINOR I think it will. I actually already implemented some of that (I think, all of that is required) in the HGM qemu rfc series, and smoked it a bit without any known issue yet with the HGM kernel.

IIUC we can work on MINOR support without HGM; I can separate it out. It's really a matter of whether it'll be worthwhile the effort and time.

Yes, MINOR might work. But especially postcopy doesn't make too much sense targeting a private mapping that has some other pages in there already ... so it might not be worth the trouble I guess.

Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw

Re: [PATCH v1 1/4] softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping

Quoted 1 line · David Hildenbrand

Maybe ... depends also on the uffd semantics as in 3).

UFFDIO_COPY|ZEROPAGE bypasses page cache for private file mappings, afaict. We'll still got a limit on the inode size (so we can't COPY|ZEROPAGE over that offset of vma) but the rest should be all fine.

Feel free to have a quick skim over 5b51072e97d5 ("userfaultfd: shmem:
allocate anonymous memory for MAP_PRIVATE shmem").

Thanks,

Signature · 1 line
Peter Xu
Tested-by on the v1 series
#13 lore raw
In reply to David Hildenbrand

This series has been tested successfully by QE. Start a VM with a 8G virtio-mem device and start memtester on it. Enable x-ignore-shared capability and then do migration. Migration was successful and virtio-mem can be resized as usual.

BR,
Mario

Quoted 41 lines · David Hildenbrand

Stumbling over "x-ignore-shared" migration support for virtio-mem on my todo list, I remember talking to Dave G. a while ago about how ram_block_discard_range() in MAP_PIRVATE file mappings is possibly harmful when the file is used somewhere else -- for example, with VM templating in multiple VMs.

This series adds a warning to ram_block_discard_range() in that problematic case and adds "x-ignore-shared" migration support for virtio-mem, which is pretty straight-forward. The last patch also documents how VM templating interacts with virtio-mem.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Juan Quintela <[email protected]>
Cc Peter Xu <[email protected]>
Cc Leonardo Bras <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Philippe Mathieu-Daudé" <[email protected]>
Cc Peng Tao <[email protected]>

David Hildenbrand (4):
softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE
file mapping
virtio-mem: Skip most of virtio_mem_unplug_all() without plugged
memory
migration/ram: Expose ramblock_is_ignored() as
migrate_ram_is_ignored()
virtio-mem: Support "x-ignore-shared" migration

 hw/virtio/virtio-mem.c   | 67 ++++++++++++++++++++++++++++------------
 include/migration/misc.h |  1 +
 migration/postcopy-ram.c |  2 +-
 migration/ram.c          | 14 ++++-----
 migration/ram.h          |  3 +-
 softmmu/physmem.c        | 18 +++++++++++
 6 files changed, 76 insertions(+), 29 deletions(-)

--
2.40.1

#14 lore raw
Quoted 7 lines · Mario Casquero

This series has been tested successfully by QE. Start a VM with a 8G virtio-mem device and start memtester on it. Enable x-ignore-shared capability and then do migration. Migration was successful and virtio-mem can be resized as usual.

Thanks a lot Mario!

Signature · 3 lines
Cheers,

David / dhildenb