Skip to contentSkip to conversation

migration: fast snapshot load

PATCH Superseded Aadeshveer Singh 2 participants 29 messages Jul 14 – Jul 22

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover migration: fast snapshot load

This series implements a "fast snapshot load" mechanism to significantly reduce the perceived resume time of a VM from a snapshot file.

Currently, resuming a VM from a snapshot file requires loading all RAM pages into the QEMU instance before execution begins. This extension allows the user to run the VM nearly instantly by loading only the required device states up front and loading RAM pages lazily, by trapping access to pages that have not yet been loaded.

Using the Linux userfaultfd syscall, a fault thread catches all page faults caused by the guest and loads in the pages required to keep the VM running. Concurrently, an eager background thread iteratively loads all remaining pages into RAM so the guest does not have to depend on the fault thread indefinitely.

Much of code is reused from postcopy for fault handling and precopy for reading mapped ram file. Implementation revolves around two threads named the fault thread and eager load thread. Fault thread as name suggests catches page faults by the guest and serves them using userfaultfd. Postcopy fault thread is reused but instead of requesting source for a page it loads the page directly by reading from file. In order to remove the dependency of guest on fault thread indefinitely the eager load thread loads in the entire RAM sequentially, and after iterating through the entire RAM signals fault thread to exit and calls cleanup.

In order to prevent the case of a page being loaded twice(in the case when eager load thread is loading it and fault thread also tries to serve fault on same page) a bitmap called pending_bmap is used to track pages which are pending and not being loaded by any thread. Atomic operations on this bitmap allows coordination between threads to prevent any unwanted behaviours

Future direction:
- Add support for multifd
- Add support for vhost-user

---
v1 -> v2

- Structure Change: Expanded from 7 to 11 patches
- Move test removal patch from first to post implementation as suggested by Peter
- Patch 1 to 5 are meant to be non functional changes meant to modernize parts of code and prepare for feature Implementation
- Patch 10 performs a smoke test for implemented feature
- Patch 11 adds a documentation page for fast snapshot load
- Added explicit fails for features not supported/incompatible with fast snapshot load in compatibility check in options.c as suggested by Peter
- Passed errp object to qemu_get_buffer_at() to improve on error reporting as suggested by Peter
- Size of pending_bmap is updated to have a bit for every guest page according to guest page size and not host page size as pointed by Peter
- New function(ramblock_file_bitmap_page_is_nonzero()) has been introduced to test for zero pages while loading essentially enableing support for hugepages
- Updated the documentation comment over postcopy_mapped_ram_load_page() to add errp as pointed by Peter
- Added a new function try_mark_postcopy_blocktime_begin() to call mark_postcopy_blocktime_begin() taking care of assumptions like skipping if page is already received to improve on the readability as pointed by Peter
- Fixed bug, that sent non page alined addresses to mark_postcopy_blocktime_begin() as pointed by Peter
- Separate the postcopy_listen_thread_bh() rename patch as suggested by Peter
- Add comment explaining changes in process_incoming_migration_co() as suggested by Peter
- Make qemu_loadvm_run_fast_snapshot_load() and postcopy_ram_eager_load_setup() return void as these functions never fail as pointed by Peter

Aadeshveer Singh (11):
migration: Propagate error in postcopy setup functions
migration: Extract blocktime marking helper
migration: Rename postcopy_listen_thread_bh
migration: Use file_bmap for RAMBlock during incoming file load
migration: Make qemu_get_buffer_at() thread-safe
migration: add RAMBlock field and helper for fast snapshot load
migration: add support for fault thread to load pages from disk
migration: add eager load thread and setup for fast snapshot load migration/tests: remove capability conflict test
postcopy-ram+mapped-ram
migration/tests: Add test for fast snapshot load docs/migration: Add documentation for fast snapshot load feature

 docs/devel/migration/fast-snapshot-load.rst |  81 ++++++
 docs/devel/migration/features.rst           |   1 +
 include/system/ramblock.h                   |   6 +
 migration/migration.c                       |  60 +++--
 migration/migration.h                       |   5 +
 migration/options.c                         |  14 +-
 migration/postcopy-ram.c                    | 266 ++++++++++++++++----
 migration/postcopy-ram.h                    |   7 +-
 migration/qemu-file.c                       |   9 +-
 migration/qemu-file.h                       |   4 +-
 migration/ram.c                             | 103 +++++++-
 migration/ram.h                             |   2 +
 migration/savevm.c                          |  16 ++
 migration/savevm.h                          |   2 +
 migration/trace-events                      |   2 +
 tests/qtest/migration/file-tests.c          |  24 ++
 tests/qtest/migration/misc-tests.c          |  52 ----
 17 files changed, 511 insertions(+), 143 deletions(-)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
Signature · 1 line
2.55.0
#1 lore raw

PATCH v3 1/11 migration: Propagate error in postcopy setup functions

Modernize error handling in postcopy_ram_incoming_setup() and postcopy_temp_pages_setup() by replacing error_reports and local error handling with standard Error propagation.

Replace use of strerror() on errno with error_setg_errno() for modular handling of errors and change return values to -1 on failure as no caller checks the actual return value.

---
 migration/postcopy-ram.c | 41 +++++++++++++++++-----------------------
 migration/postcopy-ram.h |  2 +-
 2 files changed, 18 insertions(+), 25 deletions(-)
migration/postcopy-ram.c +17 -24
index f5ef93f193..96a65aa976 100644
@@ -1464,10 +1464,9 @@ retry:
     return NULL;
 }
 
-static int postcopy_temp_pages_setup(MigrationIncomingState *mis)
+static int postcopy_temp_pages_setup(MigrationIncomingState *mis, Error **errp)
 {
     PostcopyTmpPage *tmp_page;
-    int err;
     unsigned i, channels;
     void *temp_page;
 
@@ -1487,11 +1486,11 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis)
         temp_page = mmap(NULL, mis->largest_page_size, PROT_READ | PROT_WRITE,
                          MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
         if (temp_page == MAP_FAILED) {
-            err = errno;
-            error_report("%s: Failed to map postcopy_tmp_pages[%d]: %s",
-                         __func__, i, strerror(err));
+            error_setg_errno(errp, errno,
+                             "%s: Failed to map postcopy_tmp_pages[%d]",
+                             __func__, i);
             /* Clean up will be done later */
-            return -err;
+            return -1;
         }
         tmp_page->tmp_huge_page = temp_page;
         /* Initialize default states for each tmp page */
@@ -1505,11 +1504,10 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis)
                                        PROT_READ | PROT_WRITE,
                                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
     if (mis->postcopy_tmp_zero_page == MAP_FAILED) {
-        err = errno;
         mis->postcopy_tmp_zero_page = NULL;
-        error_report("%s: Failed to map large zero page %s",
-                     __func__, strerror(err));
-        return -err;
+        error_setg_errno(errp, errno, "%s: Failed to map large zero page",
+                         __func__);
+        return -1;
     }
 
     memset(mis->postcopy_tmp_zero_page, '\0', mis->largest_page_size);
@@ -1517,15 +1515,13 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis)
     return 0;
 }
 
-int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis, Error **errp)
 {
-    Error *local_err = NULL;
-
     /* Open the fd for the kernel to give us userfaults */
     mis->userfault_fd = uffd_open(O_CLOEXEC | O_NONBLOCK);
     if (mis->userfault_fd == -1) {
-        error_report("%s: Failed to open userfault fd: %s", __func__,
-                     strerror(errno));
+        error_setg_errno(errp, errno, "%s: Failed to open userfault fd",
+                         __func__);
         return -1;
     }
 
@@ -1533,8 +1529,7 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
      * Although the host check already tested the API, we need to
      * do the check again as an ABI handshake on the new fd.
      */
-    if (!ufd_check_and_apply(mis->userfault_fd, mis, &local_err)) {
-        error_report_err(local_err);
+    if (!ufd_check_and_apply(mis->userfault_fd, mis, errp)) {
         return -1;
     }
 
@@ -1546,8 +1541,8 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
     /* Now an eventfd we use to tell the fault-thread to quit */
     mis->userfault_event_fd = eventfd(0, EFD_CLOEXEC);
     if (mis->userfault_event_fd == -1) {
-        error_report("%s: Opening userfault_event_fd: %s", __func__,
-                     strerror(errno));
+        error_setg_errno(errp, errno, "%s: Opening userfault_event_fd",
+                         __func__);
         close(mis->userfault_fd);
         return -1;
     }
@@ -1559,12 +1554,11 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 
     /* Mark so that we get notified of accesses to unwritten areas */
     if (foreach_not_ignored_block(ram_block_enable_notify, mis)) {
-        error_report("ram_block_enable_notify failed");
+        error_setg(errp, "ram_block_enable_notify failed");
         return -1;
     }
 
-    if (postcopy_temp_pages_setup(mis)) {
-        /* Error dumped in the sub-function */
+    if (postcopy_temp_pages_setup(mis, errp)) {
         return -1;
     }
 
@@ -2201,9 +2195,8 @@ int postcopy_incoming_setup(MigrationIncomingState *mis, Error **errp)
      * shouldn't be doing anything yet so don't actually expect requests
      */
     if (migrate_postcopy_ram()) {
-        if (postcopy_ram_incoming_setup(mis)) {
+        if (postcopy_ram_incoming_setup(mis, errp)) {
             postcopy_ram_incoming_cleanup(mis);
-            error_setg(errp, "Failed to setup incoming postcopy RAM blocks");
             return -1;
         }
     }
migration/postcopy-ram.h +1 -1
index a080dd65a7..98d918713f 100644
@@ -23,7 +23,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis,
  * Make all of RAM sensitive to accesses to areas that haven't yet been written
  * and wire up anything necessary to deal with it.
  */
-int postcopy_ram_incoming_setup(MigrationIncomingState *mis);
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis, Error **errp);
 
 /*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
Signature · 1 line
2.55.0
Reviewed-by on v3 1/11
#2 lore raw

Re: [PATCH v3 01/11] migration: Propagate error in postcopy setup functions

Quoted 9 lines · Aadeshveer Singh

Modernize error handling in postcopy_ram_incoming_setup() and postcopy_temp_pages_setup() by replacing error_reports and local error handling with standard Error propagation.

Replace use of strerror() on errno with error_setg_errno() for modular handling of errors and change return values to -1 on failure as no caller checks the actual return value.

Signature · 1 line
Peter Xu
#3 lore raw

PATCH v3 2/11 migration: Extract blocktime marking helper

In reply to Aadeshveer Singh

Use new function try_mark_postcopy_blocktime_begin() to call mark_postcopy_blocktime_begin if page was not received and return if it actually marked it.

This will help in having a cleaner API to call mark_postcopy_blocktime_begin as it requires the page to not be received by asserting on it's existance in RAMBlock->receivedmap.

---
 migration/migration.c    | 24 ++----------------------
 migration/postcopy-ram.c | 31 +++++++++++++++++++++++++++++++
 migration/postcopy-ram.h |  3 +++
 3 files changed, 36 insertions(+), 22 deletions(-)
migration/migration.c +2 -22
index 074d3f2c69..07394d8eea 100644
@@ -577,32 +577,12 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis,
                               RAMBlock *rb, ram_addr_t start, uint64_t haddr,
                               uint32_t tid)
 {
-    void *aligned = (void *)(uintptr_t)ROUND_DOWN(haddr, qemu_ram_pagesize(rb));
-    bool received = false;
-
-    WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex) {
-        received = ramblock_recv_bitmap_test_byte_offset(rb, start);
-        if (!received) {
-            if (!g_tree_lookup(mis->page_requested, aligned)) {
-                /*
-                 * The page has not been received, and it's not yet in the
-                 * page request list.  Queue it.  Set the value of element
-                 * to 1, so that things like g_tree_lookup() will return
-                 * TRUE (1) when found.
-                 */
-                g_tree_insert(mis->page_requested, aligned, (gpointer)1);
-                qatomic_inc(&mis->page_requested_count);
-                trace_postcopy_page_req_add(aligned, mis->page_requested_count);
-            }
-            mark_postcopy_blocktime_begin(haddr, tid, rb);
-        }
-    }
-
     /*
+     * If not able to mark page for blocktime it must already be present.
      * If the page is there, skip sending the message.  We don't even need the
      * lock because as long as the page arrived, it'll be there forever.
      */
-    if (received) {
+    if (!try_mark_postcopy_blocktime_begin(mis, rb, start, haddr, tid)) {
         return 0;
     }
 
migration/postcopy-ram.c +31 -0
index 96a65aa976..4b8f882e1e 100644
@@ -1057,6 +1057,37 @@ static void blocktime_fault_inject(PostcopyBlocktimeContext *ctx,
     trace_postcopy_blocktime_begin(addr, time, cpu, !!head);
 }
 
+/*
+ * Take @page_request_mutex and try marking postcopy blocktime begin.
+ * Return true if marking is successful and false if page alredy exists.
+ */
+bool try_mark_postcopy_blocktime_begin(MigrationIncomingState *mis,
+                                       RAMBlock *rb, ram_addr_t start,
+                                       uint64_t haddr, uint32_t tid)
+{
+    bool received = false;
+    void *aligned = (void *)(uintptr_t)ROUND_DOWN(haddr, qemu_ram_pagesize(rb));
+
+    WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex) {
+        received = ramblock_recv_bitmap_test_byte_offset(rb, start);
+        if (!received) {
+            if (!g_tree_lookup(mis->page_requested, aligned)) {
+                /*
+                 * The page has not been received, and it's not yet in the
+                 * page request list.  Queue it.  Set the value of element
+                 * to 1, so that things like g_tree_lookup() will return
+                 * TRUE (1) when found.
+                 */
+                g_tree_insert(mis->page_requested, aligned, (gpointer)1);
+                qatomic_inc(&mis->page_requested_count);
+                trace_postcopy_page_req_add(aligned, mis->page_requested_count);
+            }
+            mark_postcopy_blocktime_begin((uint64_t)aligned, tid, rb);
+        }
+    }
+    return !received;
+}
+
 /*
  * This function is being called when pagefault occurs. It tracks down vCPU
  * blocking time.  It's protected by @page_request_mutex.
migration/postcopy-ram.h +3 -0
index 98d918713f..ea879f0bc6 100644
@@ -196,6 +196,9 @@ void postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file);
 void postcopy_preempt_setup(MigrationState *s);
 int postcopy_preempt_establish_channel(MigrationState *s);
 bool postcopy_is_paused(MigrationStatus status);
+bool try_mark_postcopy_blocktime_begin(MigrationIncomingState *mis,
+                                       RAMBlock *rb, ram_addr_t start,
+                                       uint64_t haddr, uint32_t tid);
 void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
                                    RAMBlock *rb);
 
Signature · 1 line
2.55.0
Reviewed-by on v3 2/11
#4 lore raw

Re: [PATCH v3 02/11] migration: Extract blocktime marking helper

Quoted 9 lines · Aadeshveer Singh

Use new function try_mark_postcopy_blocktime_begin() to call mark_postcopy_blocktime_begin if page was not received and return if it actually marked it.

This will help in having a cleaner API to call mark_postcopy_blocktime_begin as it requires the page to not be received by asserting on it's existance in RAMBlock->receivedmap.

Signature · 1 line
Peter Xu
#5 lore raw

PATCH v3 3/11 migration: Rename postcopy_listen_thread_bh

In reply to Aadeshveer Singh

Rename postcopy_listen_thread_bh to postcopy_incoming_complete_bh for a more robust naming scheme. This will allow fast snapshot load to use this bh funciton directly to cleanup the incoming state.

---
 migration/postcopy-ram.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
migration/postcopy-ram.c +2 -2
index 4b8f882e1e..2e2c9fae10 100644
@@ -2103,7 +2103,7 @@ bool postcopy_is_paused(MigrationStatus status)
         status == MIGRATION_STATUS_POSTCOPY_RECOVER_SETUP;
 }
 
-static void postcopy_listen_thread_bh(void *opaque)
+static void postcopy_incoming_complete_bh(void *opaque)
 {
     MigrationState *s = migrate_get_current();
     MigrationIncomingState *mis = migration_incoming_get_current();
@@ -2211,7 +2211,7 @@ out:
     rcu_unregister_thread();
     postcopy_state_set(POSTCOPY_INCOMING_END);
 
-    migration_bh_schedule(postcopy_listen_thread_bh, NULL);
+    migration_bh_schedule(postcopy_incoming_complete_bh, NULL);
 
     object_unref(OBJECT(migr));
 
Signature · 1 line
2.55.0
Reviewed-by on v3 3/11
#6 lore raw

Re: [PATCH v3 03/11] migration: Rename postcopy_listen_thread_bh

Quoted 5 lines · Aadeshveer Singh

Rename postcopy_listen_thread_bh to postcopy_incoming_complete_bh for a more robust naming scheme. This will allow fast snapshot load to use this bh funciton directly to cleanup the incoming state.

Signature · 1 line
Peter Xu
#7 lore raw

PATCH v3 4/11 migration: Use file_bmap for RAMBlock during incoming file load

In reply to Aadeshveer Singh

Replace the temporary bitmap with the existing file_bmap attribute of the RAMBlock. This acts as a preparatory change for the upcoming fast snapshot load feature.

Reusing this bitmap allows the destination to track page types during a postcopy load, enabling faster, direct placement of zero pages.

Since file_bmap is currently only utilized during the migration save phase, it can be safely repurposed during the load phase without introducing conflicts.

---
 migration/ram.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
migration/ram.c +27 -6
index fc38ffbf8a..4728f14d73 100644
@@ -252,6 +252,17 @@ int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque)
     return ret;
 }
 
+static void ramblock_file_bmap_init(void)
+{
+    RAMBlock *rb;
+
+    RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
+        assert(!rb->file_bmap);
+        size_t size = rb->max_length >> qemu_target_page_bits();
+        rb->file_bmap = bitmap_new(size);
+    }
+}
+
 static void ramblock_recv_map_init(void)
 {
     RAMBlock *rb;
@@ -3749,6 +3760,9 @@ static int ram_load_setup(QEMUFile *f, void *opaque, Error **errp)
 {
     xbzrle_load_setup();
     ramblock_recv_map_init();
+    if (migrate_mapped_ram()) {
+        ramblock_file_bmap_init();
+    }
 
     return 0;
 }
@@ -3766,8 +3780,8 @@ static int ram_load_cleanup(void *opaque)
     xbzrle_load_cleanup();
 
     RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
-        g_free(rb->receivedmap);
-        rb->receivedmap = NULL;
+        g_clear_pointer(&rb->receivedmap, g_free);
+        g_clear_pointer(&rb->file_bmap, g_free);
     }
 
     return 0;
@@ -4142,11 +4156,18 @@ err:
 static void parse_ramblock_mapped_ram(QEMUFile *f, RAMBlock *block,
                                       ram_addr_t length, Error **errp)
 {
-    g_autofree unsigned long *bitmap = NULL;
     MappedRamHeader header;
     size_t bitmap_size;
     long num_pages;
 
+    if (length > block->max_length) {
+        error_setg(errp,
+                   "mapped-ram header length %" PRIu64 " exceeds "
+                   "RAMBlock(\"%s\") max_length %" PRIu64,
+                   (uint64_t)length, block->idstr, (uint64_t)block->max_length);
+        return;
+    }
+
     if (!mapped_ram_read_header(f, &header, errp)) {
         return;
     }
@@ -4174,14 +4195,14 @@ static void parse_ramblock_mapped_ram(QEMUFile *f, RAMBlock *block,
     num_pages = length / header.page_size;
     bitmap_size = BITS_TO_LONGS(num_pages) * sizeof(unsigned long);
 
-    bitmap = g_malloc0(bitmap_size);
-    if (qemu_get_buffer_at(f, (uint8_t *)bitmap, bitmap_size,
+    if (qemu_get_buffer_at(f, (uint8_t *)block->file_bmap, bitmap_size,
                            header.bitmap_offset) != bitmap_size) {
         error_setg(errp, "Error reading dirty bitmap");
         return;
     }
 
-    if (!read_ramblock_mapped_ram(f, block, num_pages, bitmap, errp)) {
+    if (!read_ramblock_mapped_ram(f, block, num_pages, block->file_bmap,
+                                  errp)) {
         return;
     }
 
Signature · 1 line
2.55.0
Reviewed-by on v3 4/11
#8 lore raw

Re: [PATCH v3 04/11] migration: Use file_bmap for RAMBlock during incoming file load

Quoted 4 lines · Aadeshveer Singh

Replace the temporary bitmap with the existing file_bmap attribute of the RAMBlock. This acts as a preparatory change for the upcoming fast snapshot load feature.

Quoted 8 lines

Reusing this bitmap allows the destination to track page types during a postcopy load, enabling faster, direct placement of zero pages.

Since file_bmap is currently only utilized during the migration save phase, it can be safely repurposed during the load phase without introducing conflicts.

Signature · 1 line
Peter Xu
#9 lore raw

PATCH v3 5/11 migration: Make qemu_get_buffer_at() thread-safe

In reply to Aadeshveer Singh

Remove the internal QEMUFile error state modification from qemu_get_buffer_at(). This function is called by two functions, both of which already check for unexpected return values and handle their own error reporting.

Removing this shared state modification makes qemu_get_buffer_at() strictly thread-safe for concurrent disk reads, serving as a preparatory change for the upcoming fast snapshot load feature.

Removed local error by passing errp to improve on error handling, consequently also change caller to use error_prevent and not error_setg.

---
 migration/qemu-file.c | 9 +++------
 migration/qemu-file.h | 4 ++--
 migration/ram.c       | 6 +++---
 3 files changed, 8 insertions(+), 11 deletions(-)
migration/qemu-file.c +3 -6
index d5a48115bd..0c28c878a9 100644
@@ -550,17 +550,14 @@ void qemu_put_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen,
 }
 
 
-size_t qemu_get_buffer_at(QEMUFile *f, uint8_t *buf, size_t buflen,
-                          off_t pos)
+size_t qemu_get_buffer_at(QEMUFile *f, uint8_t *buf, size_t buflen, off_t pos,
+                          Error **errp)
 {
-    Error *err = NULL;
-
     if (f->last_error) {
         return 0;
     }
 
-    if (qio_channel_pread_all(f->ioc, buf, buflen, pos, &err) < 0) {
-        qemu_file_set_error_obj(f, -EIO, err);
+    if (qio_channel_pread_all(f->ioc, buf, buflen, pos, errp) < 0) {
         return 0;
     }
 
migration/qemu-file.h +2 -2
index 8f824c124d..966766788d 100644
@@ -76,8 +76,8 @@ void qemu_set_offset(QEMUFile *f, off_t off, int whence);
 off_t qemu_get_offset(QEMUFile *f);
 void qemu_put_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen,
                         off_t pos);
-size_t qemu_get_buffer_at(QEMUFile *f, uint8_t *buf, size_t buflen,
-                          off_t pos);
+size_t qemu_get_buffer_at(QEMUFile *f, uint8_t *buf, size_t buflen, off_t pos,
+                          Error **errp);
 
 QIOChannel *qemu_file_get_ioc(QEMUFile *file);
 int qemu_file_put_fd(QEMUFile *f, int fd);
migration/ram.c +3 -3
index 4728f14d73..967db7c0db 100644
@@ -4127,7 +4127,7 @@ static bool read_ramblock_mapped_ram(QEMUFile *f, RAMBlock *block,
                                               block->pages_offset + offset);
             } else {
                 read = qemu_get_buffer_at(f, host, size,
-                                          block->pages_offset + offset);
+                                          block->pages_offset + offset, errp);
             }
 
             if (!read) {
@@ -4196,8 +4196,8 @@ static void parse_ramblock_mapped_ram(QEMUFile *f, RAMBlock *block,
     bitmap_size = BITS_TO_LONGS(num_pages) * sizeof(unsigned long);
 
     if (qemu_get_buffer_at(f, (uint8_t *)block->file_bmap, bitmap_size,
-                           header.bitmap_offset) != bitmap_size) {
-        error_setg(errp, "Error reading dirty bitmap");
+                           header.bitmap_offset, errp) != bitmap_size) {
+        error_prepend(errp, "Error reading dirty bitmap");
         return;
     }
 
Signature · 1 line
2.55.0
Reviewed-by on v3 5/11
#10 lore raw

Re: [PATCH v3 05/11] migration: Make qemu_get_buffer_at() thread-safe

Quoted 5 lines · Aadeshveer Singh

Remove the internal QEMUFile error state modification from qemu_get_buffer_at(). This function is called by two functions, both of which already check for unexpected return values and handle their own error reporting.

Quoted 8 lines

Removing this shared state modification makes qemu_get_buffer_at() strictly thread-safe for concurrent disk reads, serving as a preparatory change for the upcoming fast snapshot load feature.

Removed local error by passing errp to improve on error handling, consequently also change caller to use error_prevent and not error_setg.

Signature · 1 line
Peter Xu
#11 lore raw

PATCH v3 6/11 migration: add RAMBlock field and helper for fast snapshot load

In reply to Aadeshveer Singh

Add pending_bmap field per RAMBlock which is a Bitmap to store internal state of which pages have been read by some thread to ensure coordination between fault thread and eager load thread.

Modify parse_ramblock_mapped_ram(), to not load the actual RAMBlocks data in postcopy case as that will be loaded by fault thread and eager thread after the VM starts running.

Change ram_load() to use new function ram_should_load_postcopy_pages() to decide how to load/read RAM.

---
 include/system/ramblock.h |  6 ++++
 migration/ram.c           | 59 +++++++++++++++++++++++++++++++++++----
 2 files changed, 59 insertions(+), 6 deletions(-)
include/system/ramblock.h +6 -0
index 4435f8d55f..83187bf44c 100644
@@ -60,6 +60,12 @@ struct RAMBlock {
 
     /* Bitmap of already received pages.  Only used on destination side. */
     unsigned long *receivedmap;
+    /*
+     * Bitmap for pages that are yet to be read from disk. It is required for
+     * fault thread and eager thread to keep note of which pages are currently
+     * being read. Used by fast snapshot load.
+     */
+    unsigned long *pending_bmap;
 
     /*
      * bitmap to track already cleared dirty bitmap.  When the bit is
migration/ram.c +53 -6

102 lines not on this page · raw message

Signature · 1 line
2.55.0
Reviewed-by on v3 6/11
#12 lore raw

Re: [PATCH v3 06/11] migration: add RAMBlock field and helper for fast snapshot load

Quoted 3 lines · Aadeshveer Singh

Add pending_bmap field per RAMBlock which is a Bitmap to store internal state of which pages have been read by some thread to ensure coordination between fault thread and eager load thread.

The other way to do this is squashing this three core patches (starting from this one) into one big patch implementing the idea. Otherwise it might be hard to review a newly added bitmap when it got allocated but never used.

But it's still fine when implementing a major feature like this - people may have different opinion on how to split patches, you can keep it like this if nobody else complains.

Quoted 9 lines

Modify parse_ramblock_mapped_ram(), to not load the actual RAMBlocks data in postcopy case as that will be loaded by fault thread and eager thread after the VM starts running.

Change ram_load() to use new function ram_should_load_postcopy_pages() to decide how to load/read RAM.

Signature · 1 line
Peter Xu
#13 lore raw

Re: [PATCH v3 06/11] migration: add RAMBlock field and helper for fast snapshot load

In reply to Aadeshveer Singh

migration/ram.c static void ramblock_file_bmap_init(void) · line 272

Quoted 7 lines · Aadeshveer Singh
migration/ram.c +7 -0
+static void ramblock_pending_bmap_init(void)
+{
+    RAMBlock *rb;
+
+    RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
+        assert(!rb->pending_bmap);
+        size_t size = rb->max_length >> qemu_target_page_bits();

I forgot one thing here: I think it should use qemu_ram_pagesize(), for huge pages when the fault thread or eager load thread want to figure how who to do the work, it should always be in host psize, not guest-psize, because that's the granule userfaultfd's UFFDIO_COPY/ZEROPAGE uses.

Quoted 4 lines
migration/ram.c +4 -0
+        rb->pending_bmap = bitmap_new(size);
+        bitmap_set(rb->pending_bmap, 0, size);
+    }
+}
Signature · 1 line
Peter Xu
#14 lore raw

Re: [PATCH v3 06/11] migration: add RAMBlock field and helper for fast snapshot load

Quoted 15 lines · Peter Xu

migration/ram.c static void ramblock_file_bmap_init(void) · line 272

Quoted 7 lines · Aadeshveer Singh
migration/ram.c +7 -0
+static void ramblock_pending_bmap_init(void)
+{
+    RAMBlock *rb;
+
+    RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
+        assert(!rb->pending_bmap);
+        size_t size = rb->max_length >> qemu_target_page_bits();

I forgot one thing here: I think it should use qemu_ram_pagesize(), for huge pages when the fault thread or eager load thread want to figure how who to do the work, it should always be in host psize, not guest-psize, because that's the granule userfaultfd's UFFDIO_COPY/ZEROPAGE uses.

Hi Peter,

Thanks for pointing this out; I think I noted that but missed changing it. I will address this in the next version.

Quoted 8 lines

migration/ram.c static void ramblock_file_bmap_init(void) · line 276

Quoted 4 lines
migration/ram.c +4 -0
+        rb->pending_bmap = bitmap_new(size);
+        bitmap_set(rb->pending_bmap, 0, size);
+    }
+}

--
Peter Xu

#15 lore raw

PATCH v3 7/11 migration: add support for fault thread to load pages from disk

In reply to Aadeshveer Singh

In fast snapshot load, we would like to serve faults as soon as possible hence loading pages directly instead of requesting a source

Add postcopy_mapped_ram_load_page() function which serves single page fault by reading the snapshot file. It uses bitmap_test_and_clear_atomic on pending_bmap to coordinate between threads so each page is loaded exactly once. Non-zero pages are read using qemu_get_buffer_at into a temporary page (for loading page atomically), which is then placed using postcopy_place_page. Zero pages are placed directly using postcopy_place_page_zero.

Update postcopy_ram_fault_thread to call postcopy_mapped_ram_load_page instead of requesting source in case of fast snapshot load. to_src_file check is bypassed in fast snapshot load case as there is no source.

Call try_mark_postcopy_blocktime_begin on every page fault to support postcopy-blocktime.

Allocate another channel in postcopy_temp_pages_setup(like the preempt case), for both the fault thread and eager thread to load pages independently.

Add function ramblock_file_bitmap_page_is_nonzero() which searches a range of bits corresponding to a page for a set bit. This is just a bit check for normal pages but for hugepages it checks the range to see if any part of page is non zero.

---
 migration/postcopy-ram.c | 121 +++++++++++++++++++++++++++++++++------
 migration/ram.c          |  11 +++-
 migration/ram.h          |   2 +
 3 files changed, 116 insertions(+), 18 deletions(-)
migration/postcopy-ram.c +104 -17
index 2e2c9fae10..723070b5cd 100644
@@ -949,6 +949,68 @@ int postcopy_wake_shared(struct PostCopyFD *pcfd,
                        pagesize);
 }
 
+/**
+ * postcopy_mapped_ram_load_page() - Load a page to given host address.
+ * @mis: Migration Incoming State.
+ * @rb: RAMBlock from where page is loaded.
+ * @rb_offset: Offset of page in RAMBlock.
+ * @haddr: Base of page where to load in page.
+ * @channel: Used to identify between threads and use corresponding temp.
+ * @errp: Set error in case of failure
+ *
+ * Load a page from RAMBlock at offset at given host address. Used by postcopy
+ * ram fault thread and eager thread in fast snapshot load case.
+ *
+ * Return: True on success.
+ */
+static bool postcopy_mapped_ram_load_page(MigrationIncomingState *mis,
+                                          RAMBlock *rb, ram_addr_t rb_offset,
+                                          uint64_t haddr, int channel,
+                                          Error **errp)
+{
+    void *place_source = mis->postcopy_tmp_pages[channel].tmp_huge_page;
+    size_t page;
+    size_t read;
+
+    page = rb_offset / qemu_ram_pagesize(rb);
+
+    if (bitmap_test_and_clear_atomic(rb->pending_bmap, page, 1)) {
+        if (ramblock_file_bitmap_page_is_nonzero(rb, page)) {
+            /*
+             * This can happen concurrently, but it's thread-safe because
+             * qemu_get_buffer_at() is thread-safe, and the caller will be using
+             * different temporary buffers.
+             */
+            read = qemu_get_buffer_at(mis->from_src_file, place_source,
+                                      qemu_ram_pagesize(rb),
+                                      rb->pages_offset + rb_offset, errp);
+
+            if (read != qemu_ram_pagesize(rb)) {
+                error_prepend(errp, "Could not read page %zu from RAM Block %s",
+                              page, rb->idstr);
+                return false;
+            }
+
+            if (postcopy_place_page(mis, (void *)haddr, place_source, rb)) {
+                error_setg(errp,
+                           "Failed to place page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        } else {
+            if (postcopy_place_page_zero(mis, (void *)haddr, rb)) {
+                error_setg(errp,
+                           "Failed to place zero page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
 /*
  * NOTE: @tid is only used when postcopy-blocktime feature is enabled, and
  * also optional: when zero is provided, the fault accounting will be ignored.
@@ -1310,6 +1372,7 @@ static void *postcopy_ram_fault_thread(void *opaque)
     int ret;
     size_t index;
     RAMBlock *rb = NULL;
+    Error *local_err = NULL;
 
     trace_postcopy_ram_fault_thread_entry();
     rcu_register_thread();
@@ -1351,11 +1414,13 @@ static void *postcopy_ram_fault_thread(void *opaque)
             break;
         }
 
-        if (!mis->to_src_file) {
+        if (!migrate_mapped_ram() && !mis->to_src_file) {
             /*
-             * Possibly someone tells us that the return path is
-             * broken already using the event. We should hold until
-             * the channel is rebuilt.
+             * Possibly someone tells us that the return path is broken already
+             * using the event. We should hold until the channel is rebuilt.
+             * Fast snapshot load doesn't support pause and recover, because
+             * it's not necessary: we can fail right away when QEMU just booted
+             * with nothing to lose.
              */
             postcopy_pause_fault_thread(mis);
         }
@@ -1418,18 +1483,37 @@ static void *postcopy_ram_fault_thread(void *opaque)
                                                 qemu_ram_get_idstr(rb),
                                                 rb_offset,
                                                 msg.arg.pagefault.feat.ptid);
+
+            if (migrate_mapped_ram()) {
+                /* Load page directly in case of fast snapshot load */
+
+                uintptr_t aligned = (uintptr_t)ROUND_DOWN(
+                    msg.arg.pagefault.address, qemu_ram_pagesize(rb));
+
+                if (try_mark_postcopy_blocktime_begin(
+                        mis, rb, rb_offset, (uintptr_t)aligned,
+                        msg.arg.pagefault.feat.ptid)) {
+                    if (!postcopy_mapped_ram_load_page(
+                            mis, rb, rb_offset, aligned, RAM_CHANNEL_POSTCOPY,
+                            &local_err)) {
+                        error_report_err(local_err);
+                        break;
+                    }
+                }
+            } else {
 retry:
-            /*
-             * Send the request to the source - we want to request one
-             * of our host page sizes (which is >= TPS)
-             */
-            ret = postcopy_request_page(mis, rb, rb_offset,
-                                        msg.arg.pagefault.address,
-                                        msg.arg.pagefault.feat.ptid);
-            if (ret) {
-                /* May be network failure, try to wait for recovery */
-                postcopy_pause_fault_thread(mis);
-                goto retry;
+                /*
+                 * Send the request to the source - we want to request one
+                 * of our host page sizes (which is >= TPS)
+                 */
+                ret = postcopy_request_page(mis, rb, rb_offset,
+                                            msg.arg.pagefault.address,
+                                            msg.arg.pagefault.feat.ptid);
+                if (ret) {
+                    /* May be network failure, try to wait for recovery */
+                    postcopy_pause_fault_thread(mis);
+                    goto retry;
+                }
             }
         }
 
@@ -1501,8 +1585,11 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis, Error **errp)
     unsigned i, channels;
     void *temp_page;
 
-    if (migrate_postcopy_preempt()) {
-        /* If preemption enabled, need extra channel for urgent requests */
+    if (migrate_postcopy_preempt() || migrate_mapped_ram()) {
+        /*
+         * If preemption enabled or it is fast snapshot load, need extra channel
+         * for urgent requests/faults
+         */
         mis->postcopy_channels = RAM_CHANNEL_MAX;
     } else {
         /* Both precopy/postcopy on the same channel */
migration/ram.c +10 -1
index 330fceaa43..4ab8e0e750 100644
@@ -269,12 +269,21 @@ static void ramblock_pending_bmap_init(void)
 
     RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
         assert(!rb->pending_bmap);
-        size_t size = rb->max_length >> qemu_target_page_bits();
+        size_t size = rb->max_length / qemu_ram_pagesize(rb);
         rb->pending_bmap = bitmap_new(size);
         bitmap_set(rb->pending_bmap, 0, size);
     }
 }
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx)
+{
+    int page_bits = qemu_ram_pagesize(rb) / qemu_target_page_size();
+    uint64_t bmap_page_start = page_idx * page_bits;
+    uint64_t bmap_page_end = bmap_page_start + page_bits;
+    return find_next_bit(rb->file_bmap, bmap_page_end, bmap_page_start) !=
+           bmap_page_end;
+}
+
 static void ramblock_recv_map_init(void)
 {
     RAMBlock *rb;
migration/ram.h +2 -0
index 41697a7599..7e2eac58d3 100644
@@ -95,6 +95,8 @@ void ram_handle_zero(void *host, uint64_t size);
 void ram_transferred_add(uint64_t bytes);
 void ram_release_page(const char *rbname, uint64_t offset);
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx);
+
 int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_addr);
 bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_offset);
 void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr);
Signature · 1 line
2.55.0
#16 lore raw

Re: [PATCH v3 07/11] migration: add support for fault thread to load pages from disk

Quoted 59 lines · Aadeshveer Singh

In fast snapshot load, we would like to serve faults as soon as possible hence loading pages directly instead of requesting a source

Add postcopy_mapped_ram_load_page() function which serves single page fault by reading the snapshot file. It uses bitmap_test_and_clear_atomic on pending_bmap to coordinate between threads so each page is loaded exactly once. Non-zero pages are read using qemu_get_buffer_at into a temporary page (for loading page atomically), which is then placed using postcopy_place_page. Zero pages are placed directly using postcopy_place_page_zero.

Update postcopy_ram_fault_thread to call postcopy_mapped_ram_load_page instead of requesting source in case of fast snapshot load. to_src_file check is bypassed in fast snapshot load case as there is no source.

Call try_mark_postcopy_blocktime_begin on every page fault to support postcopy-blocktime.

Allocate another channel in postcopy_temp_pages_setup(like the preempt case), for both the fault thread and eager thread to load pages independently.

Add function ramblock_file_bitmap_page_is_nonzero() which searches a range of bits corresponding to a page for a set bit. This is just a bit check for normal pages but for hugepages it checks the range to see if any part of page is non zero.

---
 migration/postcopy-ram.c | 121 +++++++++++++++++++++++++++++++++------
 migration/ram.c          |  11 +++-
 migration/ram.h          |   2 +
 3 files changed, 116 insertions(+), 18 deletions(-)
migration/postcopy-ram.c +17 -0
index 2e2c9fae10..723070b5cd 100644
@@ -949,6 +949,68 @@ int postcopy_wake_shared(struct PostCopyFD *pcfd,
                        pagesize);
 }
 
+/**
+ * postcopy_mapped_ram_load_page() - Load a page to given host address.
+ * @mis: Migration Incoming State.
+ * @rb: RAMBlock from where page is loaded.
+ * @rb_offset: Offset of page in RAMBlock.
+ * @haddr: Base of page where to load in page.
+ * @channel: Used to identify between threads and use corresponding temp.
+ * @errp: Set error in case of failure
+ *
+ * Load a page from RAMBlock at offset at given host address. Used by postcopy
+ * ram fault thread and eager thread in fast snapshot load case.
+ *
+ * Return: True on success.
+ */
+static bool postcopy_mapped_ram_load_page(MigrationIncomingState *mis,
+                                          RAMBlock *rb, ram_addr_t rb_offset,
+                                          uint64_t haddr, int channel,

migration/postcopy-ram.c int postcopy_wake_shared(struct PostCopyFD *pcfd, · line 978

Quoted 10 lines
migration/postcopy-ram.c +10 -0
+                                          Error **errp)
+{
+    void *place_source = mis->postcopy_tmp_pages[channel].tmp_huge_page;
+    size_t page;
+    size_t read;
+
+    page = rb_offset / qemu_ram_pagesize(rb);
+
+    if (bitmap_test_and_clear_atomic(rb->pending_bmap, page, 1)) {
+        if (ramblock_file_bitmap_page_is_nonzero(rb, page)) {

I discussed this part previously I think, but maybe not with details. This is fine but it is less optimal in two things:

- It relies on the mapped-ram pre-fill all zero pages to be zeros in the image, otherwise if you see a huge page having some but not all non-zero page, it'll fallback to always copy from image, then we start to read data from "zero page" ranges, which we never do before (in vanilla mapped-ram logic, we always skip ranges that represent zero pages).

- It is slightly less efficient, because we need to copy the zeros from the image even if we know they're zeros

IIUC it's better to loop over all guest pages on this specific host page when preparing the place_source buffer, then do a one-shot place page.

NOTE: I believe in all paths you touched, you assumed host psize is always

Quoted 1 line

= guest psize. Just to mention it's actually possible to have host psize

< guest psize. One case is ARM64 using 4K host psize to emulate 64K guest psize. But that's less of a problem; normally production users rarely use such setup, IIUC, so I think we can put it aside for now (however I believe migration core should work with such setup right now in general). But if you can take that into account it'll be even better.

Quoted 132 lines
migration/postcopy-ram.c +77 -17
+            /*
+             * This can happen concurrently, but it's thread-safe because
+             * qemu_get_buffer_at() is thread-safe, and the caller will be using
+             * different temporary buffers.
+             */
+            read = qemu_get_buffer_at(mis->from_src_file, place_source,
+                                      qemu_ram_pagesize(rb),
+                                      rb->pages_offset + rb_offset, errp);
+
+            if (read != qemu_ram_pagesize(rb)) {
+                error_prepend(errp, "Could not read page %zu from RAM Block %s",
+                              page, rb->idstr);
+                return false;
+            }
+
+            if (postcopy_place_page(mis, (void *)haddr, place_source, rb)) {
+                error_setg(errp,
+                           "Failed to place page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        } else {
+            if (postcopy_place_page_zero(mis, (void *)haddr, rb)) {
+                error_setg(errp,
+                           "Failed to place zero page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
 /*
  * NOTE: @tid is only used when postcopy-blocktime feature is enabled, and
  * also optional: when zero is provided, the fault accounting will be ignored.
@@ -1310,6 +1372,7 @@ static void *postcopy_ram_fault_thread(void *opaque)
     int ret;
     size_t index;
     RAMBlock *rb = NULL;
+    Error *local_err = NULL;
 
     trace_postcopy_ram_fault_thread_entry();
     rcu_register_thread();
@@ -1351,11 +1414,13 @@ static void *postcopy_ram_fault_thread(void *opaque)
             break;
         }
 
-        if (!mis->to_src_file) {
+        if (!migrate_mapped_ram() && !mis->to_src_file) {
             /*
-             * Possibly someone tells us that the return path is
-             * broken already using the event. We should hold until
-             * the channel is rebuilt.
+             * Possibly someone tells us that the return path is broken already
+             * using the event. We should hold until the channel is rebuilt.
+             * Fast snapshot load doesn't support pause and recover, because
+             * it's not necessary: we can fail right away when QEMU just booted
+             * with nothing to lose.
              */
             postcopy_pause_fault_thread(mis);
         }
@@ -1418,18 +1483,37 @@ static void *postcopy_ram_fault_thread(void *opaque)
                                                 qemu_ram_get_idstr(rb),
                                                 rb_offset,
                                                 msg.arg.pagefault.feat.ptid);
+
+            if (migrate_mapped_ram()) {
+                /* Load page directly in case of fast snapshot load */
+
+                uintptr_t aligned = (uintptr_t)ROUND_DOWN(
+                    msg.arg.pagefault.address, qemu_ram_pagesize(rb));
+
+                if (try_mark_postcopy_blocktime_begin(
+                        mis, rb, rb_offset, (uintptr_t)aligned,
+                        msg.arg.pagefault.feat.ptid)) {
+                    if (!postcopy_mapped_ram_load_page(
+                            mis, rb, rb_offset, aligned, RAM_CHANNEL_POSTCOPY,
+                            &local_err)) {
+                        error_report_err(local_err);
+                        break;
+                    }
+                }
+            } else {
 retry:
-            /*
-             * Send the request to the source - we want to request one
-             * of our host page sizes (which is >= TPS)
-             */
-            ret = postcopy_request_page(mis, rb, rb_offset,
-                                        msg.arg.pagefault.address,
-                                        msg.arg.pagefault.feat.ptid);
-            if (ret) {
-                /* May be network failure, try to wait for recovery */
-                postcopy_pause_fault_thread(mis);
-                goto retry;
+                /*
+                 * Send the request to the source - we want to request one
+                 * of our host page sizes (which is >= TPS)
+                 */
+                ret = postcopy_request_page(mis, rb, rb_offset,
+                                            msg.arg.pagefault.address,
+                                            msg.arg.pagefault.feat.ptid);
+                if (ret) {
+                    /* May be network failure, try to wait for recovery */
+                    postcopy_pause_fault_thread(mis);
+                    goto retry;
+                }
             }
         }
 
@@ -1501,8 +1585,11 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis, Error **errp)
     unsigned i, channels;
     void *temp_page;
 
-    if (migrate_postcopy_preempt()) {
-        /* If preemption enabled, need extra channel for urgent requests */
+    if (migrate_postcopy_preempt() || migrate_mapped_ram()) {
+        /*
+         * If preemption enabled or it is fast snapshot load, need extra channel
+         * for urgent requests/faults
+         */
         mis->postcopy_channels = RAM_CHANNEL_MAX;
     } else {
         /* Both precopy/postcopy on the same channel */
migration/ram.c +0 -0
index 330fceaa43..4ab8e0e750 100644

migration/ram.c static void ramblock_pending_bmap_init(void) · line 272

Quoted 5 lines
migration/ram.c +1 -1
@@ -269,12 +269,21 @@ static void ramblock_pending_bmap_init(void)
 
     RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
         assert(!rb->pending_bmap);
-        size_t size = rb->max_length >> qemu_target_page_bits();
+        size_t size = rb->max_length / qemu_ram_pagesize(rb);

Oh here is the change.. let's move this to that patch directly (or squash three patches).

Other than that this patch looks good.

Thanks,

Quoted 33 lines
migration/ram.c +9 -0
         rb->pending_bmap = bitmap_new(size);
         bitmap_set(rb->pending_bmap, 0, size);
     }
 }
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx)
+{
+    int page_bits = qemu_ram_pagesize(rb) / qemu_target_page_size();
+    uint64_t bmap_page_start = page_idx * page_bits;
+    uint64_t bmap_page_end = bmap_page_start + page_bits;
+    return find_next_bit(rb->file_bmap, bmap_page_end, bmap_page_start) !=
+           bmap_page_end;
+}
+
 static void ramblock_recv_map_init(void)
 {
     RAMBlock *rb;
migration/ram.h +2 -0
index 41697a7599..7e2eac58d3 100644
@@ -95,6 +95,8 @@ void ram_handle_zero(void *host, uint64_t size);
 void ram_transferred_add(uint64_t bytes);
 void ram_release_page(const char *rbname, uint64_t offset);
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx);
+
 int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_addr);
 bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_offset);
 void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr);
Signature · 1 line
2.55.0
Signature · 1 line
Peter Xu
#17 lore raw

Re: [PATCH v3 07/11] migration: add support for fault thread to load pages from disk

Quoted 87 lines · Peter Xu
Quoted 59 lines · Aadeshveer Singh

In fast snapshot load, we would like to serve faults as soon as possible hence loading pages directly instead of requesting a source

Add postcopy_mapped_ram_load_page() function which serves single page fault by reading the snapshot file. It uses bitmap_test_and_clear_atomic on pending_bmap to coordinate between threads so each page is loaded exactly once. Non-zero pages are read using qemu_get_buffer_at into a temporary page (for loading page atomically), which is then placed using postcopy_place_page. Zero pages are placed directly using postcopy_place_page_zero.

Update postcopy_ram_fault_thread to call postcopy_mapped_ram_load_page instead of requesting source in case of fast snapshot load. to_src_file check is bypassed in fast snapshot load case as there is no source.

Call try_mark_postcopy_blocktime_begin on every page fault to support postcopy-blocktime.

Allocate another channel in postcopy_temp_pages_setup(like the preempt case), for both the fault thread and eager thread to load pages independently.

Add function ramblock_file_bitmap_page_is_nonzero() which searches a range of bits corresponding to a page for a set bit. This is just a bit check for normal pages but for hugepages it checks the range to see if any part of page is non zero.

---
 migration/postcopy-ram.c | 121 +++++++++++++++++++++++++++++++++------
 migration/ram.c          |  11 +++-
 migration/ram.h          |   2 +
 3 files changed, 116 insertions(+), 18 deletions(-)
migration/postcopy-ram.c +17 -0
index 2e2c9fae10..723070b5cd 100644
@@ -949,6 +949,68 @@ int postcopy_wake_shared(struct PostCopyFD *pcfd,
                        pagesize);
 }
 
+/**
+ * postcopy_mapped_ram_load_page() - Load a page to given host address.
+ * @mis: Migration Incoming State.
+ * @rb: RAMBlock from where page is loaded.
+ * @rb_offset: Offset of page in RAMBlock.
+ * @haddr: Base of page where to load in page.
+ * @channel: Used to identify between threads and use corresponding temp.
+ * @errp: Set error in case of failure
+ *
+ * Load a page from RAMBlock at offset at given host address. Used by postcopy
+ * ram fault thread and eager thread in fast snapshot load case.
+ *
+ * Return: True on success.
+ */
+static bool postcopy_mapped_ram_load_page(MigrationIncomingState *mis,
+                                          RAMBlock *rb, ram_addr_t rb_offset,
+                                          uint64_t haddr, int channel,

migration/postcopy-ram.c int postcopy_wake_shared(struct PostCopyFD *pcfd, · line 978

Quoted 10 lines
migration/postcopy-ram.c +10 -0
+                                          Error **errp)
+{
+    void *place_source = mis->postcopy_tmp_pages[channel].tmp_huge_page;
+    size_t page;
+    size_t read;
+
+    page = rb_offset / qemu_ram_pagesize(rb);
+
+    if (bitmap_test_and_clear_atomic(rb->pending_bmap, page, 1)) {
+        if (ramblock_file_bitmap_page_is_nonzero(rb, page)) {

I discussed this part previously I think, but maybe not with details. This is fine but it is less optimal in two things:

- It relies on the mapped-ram pre-fill all zero pages to be zeros in the image, otherwise if you see a huge page having some but not all non-zero page, it'll fallback to always copy from image, then we start to read data from "zero page" ranges, which we never do before (in vanilla mapped-ram logic, we always skip ranges that represent zero pages).

- It is slightly less efficient, because we need to copy the zeros from the image even if we know they're zeros

IIUC it's better to loop over all guest pages on this specific host page when preparing the place_source buffer, then do a one-shot place page.

Quoted 8 lines

NOTE: I believe in all paths you touched, you assumed host psize is always

Quoted 1 line

= guest psize. Just to mention it's actually possible to have host psize

< guest psize. One case is ARM64 using 4K host psize to emulate 64K guest psize. But that's less of a problem; normally production users rarely use such setup, IIUC, so I think we can put it aside for now (however I believe migration core should work with such setup right now in general). But if you can take that into account it'll be even better.

I agree with the looping over and loading pages into the buffer. That would be more efficient and would get rid of the assumption I had that zero ranges in hugepages should always be zero. I will look into how to generalize this for differences between host and guest page sizes. That depends on how mapped-ram handles those cases. I might have to look into that.

Quoted 133 lines
Quoted 133 lines
migration/postcopy-ram.c +77 -17
+            /*
+             * This can happen concurrently, but it's thread-safe because
+             * qemu_get_buffer_at() is thread-safe, and the caller will be using
+             * different temporary buffers.
+             */
+            read = qemu_get_buffer_at(mis->from_src_file, place_source,
+                                      qemu_ram_pagesize(rb),
+                                      rb->pages_offset + rb_offset, errp);
+
+            if (read != qemu_ram_pagesize(rb)) {
+                error_prepend(errp, "Could not read page %zu from RAM Block %s",
+                              page, rb->idstr);
+                return false;
+            }
+
+            if (postcopy_place_page(mis, (void *)haddr, place_source, rb)) {
+                error_setg(errp,
+                           "Failed to place page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        } else {
+            if (postcopy_place_page_zero(mis, (void *)haddr, rb)) {
+                error_setg(errp,
+                           "Failed to place zero page %zu from RAM Block %s at "
+                           "address %" PRIu64,
+                           page, rb->idstr, haddr);
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
 /*
  * NOTE: @tid is only used when postcopy-blocktime feature is enabled, and
  * also optional: when zero is provided, the fault accounting will be ignored.
@@ -1310,6 +1372,7 @@ static void *postcopy_ram_fault_thread(void *opaque)
     int ret;
     size_t index;
     RAMBlock *rb = NULL;
+    Error *local_err = NULL;
 
     trace_postcopy_ram_fault_thread_entry();
     rcu_register_thread();
@@ -1351,11 +1414,13 @@ static void *postcopy_ram_fault_thread(void *opaque)
             break;
         }
 
-        if (!mis->to_src_file) {
+        if (!migrate_mapped_ram() && !mis->to_src_file) {
             /*
-             * Possibly someone tells us that the return path is
-             * broken already using the event. We should hold until
-             * the channel is rebuilt.
+             * Possibly someone tells us that the return path is broken already
+             * using the event. We should hold until the channel is rebuilt.
+             * Fast snapshot load doesn't support pause and recover, because
+             * it's not necessary: we can fail right away when QEMU just booted
+             * with nothing to lose.
              */
             postcopy_pause_fault_thread(mis);
         }
@@ -1418,18 +1483,37 @@ static void *postcopy_ram_fault_thread(void *opaque)
                                                 qemu_ram_get_idstr(rb),
                                                 rb_offset,
                                                 msg.arg.pagefault.feat.ptid);
+
+            if (migrate_mapped_ram()) {
+                /* Load page directly in case of fast snapshot load */
+
+                uintptr_t aligned = (uintptr_t)ROUND_DOWN(
+                    msg.arg.pagefault.address, qemu_ram_pagesize(rb));
+
+                if (try_mark_postcopy_blocktime_begin(
+                        mis, rb, rb_offset, (uintptr_t)aligned,
+                        msg.arg.pagefault.feat.ptid)) {
+                    if (!postcopy_mapped_ram_load_page(
+                            mis, rb, rb_offset, aligned, RAM_CHANNEL_POSTCOPY,
+                            &local_err)) {
+                        error_report_err(local_err);
+                        break;
+                    }
+                }
+            } else {
 retry:
-            /*
-             * Send the request to the source - we want to request one
-             * of our host page sizes (which is >= TPS)
-             */
-            ret = postcopy_request_page(mis, rb, rb_offset,
-                                        msg.arg.pagefault.address,
-                                        msg.arg.pagefault.feat.ptid);
-            if (ret) {
-                /* May be network failure, try to wait for recovery */
-                postcopy_pause_fault_thread(mis);
-                goto retry;
+                /*
+                 * Send the request to the source - we want to request one
+                 * of our host page sizes (which is >= TPS)
+                 */
+                ret = postcopy_request_page(mis, rb, rb_offset,
+                                            msg.arg.pagefault.address,
+                                            msg.arg.pagefault.feat.ptid);
+                if (ret) {
+                    /* May be network failure, try to wait for recovery */
+                    postcopy_pause_fault_thread(mis);
+                    goto retry;
+                }
             }
         }
 
@@ -1501,8 +1585,11 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis, Error **errp)
     unsigned i, channels;
     void *temp_page;
 
-    if (migrate_postcopy_preempt()) {
-        /* If preemption enabled, need extra channel for urgent requests */
+    if (migrate_postcopy_preempt() || migrate_mapped_ram()) {
+        /*
+         * If preemption enabled or it is fast snapshot load, need extra channel
+         * for urgent requests/faults
+         */
         mis->postcopy_channels = RAM_CHANNEL_MAX;
     } else {
         /* Both precopy/postcopy on the same channel */
migration/ram.c +0 -0
index 330fceaa43..4ab8e0e750 100644
@@ -269,12 +269,21 @@ static void ramblock_pending_bmap_init(void)
 
Quoted 7 lines

migration/ram.c static void ramblock_pending_bmap_init(void) · line 272

Quoted 4 lines
migration/ram.c +1 -1
     RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
         assert(!rb->pending_bmap);
-        size_t size = rb->max_length >> qemu_target_page_bits();
+        size_t size = rb->max_length / qemu_ram_pagesize(rb);

Oh here is the change.. let's move this to that patch directly (or squash three patches).

I think I messed up this change. Moved it to the previous patch for following version.

Quoted 42 lines

Other than that this patch looks good.

Thanks,

Quoted 31 lines
migration/ram.c +9 -0
         rb->pending_bmap = bitmap_new(size);
         bitmap_set(rb->pending_bmap, 0, size);
     }
 }
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx)
+{
+    int page_bits = qemu_ram_pagesize(rb) / qemu_target_page_size();
+    uint64_t bmap_page_start = page_idx * page_bits;
+    uint64_t bmap_page_end = bmap_page_start + page_bits;
+    return find_next_bit(rb->file_bmap, bmap_page_end, bmap_page_start) !=
+           bmap_page_end;
+}
+
 static void ramblock_recv_map_init(void)
 {
     RAMBlock *rb;
migration/ram.h +2 -1
index 41697a7599..7e2eac58d3 100644
@@ -95,6 +95,8 @@ void ram_handle_zero(void *host, uint64_t size);
 void ram_transferred_add(uint64_t bytes);
 void ram_release_page(const char *rbname, uint64_t offset);
 
+bool ramblock_file_bitmap_page_is_nonzero(RAMBlock *rb, uint64_t page_idx);
+
 int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_addr);
 bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_offset);
 void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr);
--
Quoted 2 lines

2.55.0

--
Peter Xu

#18 lore raw

Re: [PATCH v3 07/11] migration: add support for fault thread to load pages from disk

Quoted 9 lines · Aadeshveer Singh
Quoted 8 lines

NOTE: I believe in all paths you touched, you assumed host psize is always

Quoted 1 line

= guest psize. Just to mention it's actually possible to have host psize

< guest psize. One case is ARM64 using 4K host psize to emulate 64K guest psize. But that's less of a problem; normally production users rarely use such setup, IIUC, so I think we can put it aside for now (however I believe migration core should work with such setup right now in general). But if you can take that into account it'll be even better.

Quoted 6 lines

I agree with the looping over and loading pages into the buffer. That would be more efficient and would get rid of the assumption I had that zero ranges in hugepages should always be zero. I will look into how to generalize this for differences between host and guest page sizes. That depends on how mapped-ram handles those cases. I might have to look into that.

For host/guest psize, it's actually easier for vanilla mapped-ram, and your feature is slightly more complex.

It used to be non-issue because mapped-ram only cares about guest psize. It does not care about the size of host pages, being huge or not.

IOW, mapped-ram sync mode directly manipuates the HVA pointer, ignoring what is underneath, when requesting IOs from qemu_get_buffer_at() from the snapshot images. Since VM isn't running, destination QEMU doesn't care about e.g. prepopulation of a huge page with partially filled data when the first HVA access happens.

Your work will need to consider this though, because in async faults your feature will depend on userfaultfd, which is host-page aware, and the VM is running, so we can't populate some host page partially, we can only populate it always atomically. This is the same to network-based postcopy.

When I said on normal migration code will work with various guest/host page setups, I meant more on the iteration logic of when we iterate over RAMs. For example, on the source / saving side (where your series didn't yet touch, but you will need to start looking into if you want to enable multifd + postcopy..), we have a loop structure like this:

ram_save_iterate() (NOTE, for complete(), it's the same)

    ram_find_and_save_block()
      ram_save_host_page()               <--------------          [1]
        ram_save_target_page()           <--------------          [2]

So someone who read the code first may think that QEMU only supports host psize >= guest psize, because loop [1] is higher in stack, looping over smaller guest psizes when they don't match. But in reality what happens is, in loop [2] when guest psize is even larger than host psize, what will happen is (take example of host psize 4K, guest psize 64K):

- Loop [1], find a host page (e.g. start at address 1M, which is 4K on

    the host), try to migrate the "page" (pointed by pss->page)

- At [2], migrate the whole guest "page" (pointed by pss->page), which is

    64K (> 4K) return to loop [1].

- Loop [1], found that after sending the guest page, we've already covered the host psize range (4K), fallbck to ram_find_and_save_block() to find another dirty host page

- In ram_find_and_save_block(), it'll find the next dirty (e.g. 1M+64K,

    rather than 1M+4K, because ramblock->bmap is guest psize based).

You can take time to reference to above (also comments in pss_host_page_prepare()).

Thanks,

Signature · 1 line
Peter Xu
#19 lore raw

PATCH v3 8/11 migration: add eager load thread and setup for fast snapshot load

In reply to Aadeshveer Singh

In fast snapshot load a thread is needed for actively loading in pages along with the fault path so that the guest is not dependent on fault thread indefinitely. Considering the difference from usual network postcopy where major chunk of RAM is already loaded here entire RAM needs to be loaded later. Existance of background pages which are not really accessed by the guest might never be loaded and system will be locked in migration for indefinite time. As there should be no assumption about how guest accesses memory, the load times can be indefinite.

Add postcopy_ram_eager_load_thread(), for the eager thread which iterates over all non ignored blocks calling ram_block_load_eager() on each. ram_block_load_eager then iterates to load in all pages using postcopy_mapped_ram_load_page(), with a different channel, which takes care of not loading in pages already loaded by fault thread. On completion the thread schedules postcopy_incoming_complete_bh() to destroy the incoming migration state.

Add postcopy_ram_eager_load_setup() to create the thread. Added joining logic in postcopy_incoming_cleanup().

Add tracepoints for entry and exit to eager load thread.

When both mapped-ram and postcopy-ram are set, divert from qemu_loadvm_state to run fast snapshot load

Initialize postcopy RAM state and register RAM Blocks with userfaultfd via ram_postcopy_incoming_init() and postcopy_ram_incoming_setup() in process_incoming_migration_co(). Fault thread needs to be launched before VM to serve faults for some hardwares emulation that need to read RAM (like vapic devices). Populate bitmaps and offset tables while reading file in qemu_loadvm_state_main.

Add function qemu_loadvm_run_fast_snapshot_load() which starts the VM using loadvm_postcopy_handle_run_bh() and launches eager load thread.

Skip scheduling process_incoming_migration_bh() in process_incoming_migration_co(), for fast snapshot load as the state cleanup is managed by eager load thread on completion.

---
 migration/migration.c    | 36 +++++++++++++++++++--
 migration/migration.h    |  5 +++
 migration/postcopy-ram.c | 69 ++++++++++++++++++++++++++++++++++++++++
 migration/postcopy-ram.h |  2 ++
 migration/savevm.c       | 16 ++++++++++
 migration/savevm.h       |  2 ++
 migration/trace-events   |  2 ++
 7 files changed, 130 insertions(+), 2 deletions(-)
migration/migration.c +34 -2
index 07394d8eea..e9c4eee6b9 100644
@@ -710,6 +710,11 @@ static void process_incoming_migration_bh(void *opaque)
     migration_incoming_state_destroy();
 }
 
+static bool migration_incoming_has_postcopy_thread(MigrationIncomingState *mis)
+{
+    return mis->have_listen_thread || mis->have_eager_load_thread;
+}
+
 static void coroutine_fn
 process_incoming_migration_co(void *opaque)
 {
@@ -739,17 +744,44 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    /*
+     * When loading snapshot with postcopy enabled, setup the postcopy
+     * infrastructure before loading the major part of device states.
+     * It's required because qemu_loadvm_state() may access guest memory
+     * while loading device states, which can cause page faults already.
+     */
+    if (migrate_postcopy_ram() && migrate_mapped_ram()) {
+        migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+                          MIGRATION_STATUS_POSTCOPY_DEVICE);
+
+        if (ram_postcopy_incoming_init(mis, &local_err)) {
+            goto fail;
+        }
+
+        postcopy_state_set(POSTCOPY_INCOMING_LISTENING);
+        if (postcopy_ram_incoming_setup(mis, &local_err)) {
+            goto fail;
+        }
+    }
+
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
+    if (ret < 0) {
+        goto fail;
+    }
+
+    if (migrate_postcopy_ram() && migrate_mapped_ram()) {
+        qemu_loadvm_run_fast_snapshot_load(mis->from_src_file, mis);
+    }
 
     trace_vmstate_downtime_checkpoint("dst-precopy-loadvm-completed");
 
     trace_process_incoming_migration_co_end(ret);
-    if (mis->have_listen_thread) {
+    if (migration_incoming_has_postcopy_thread(mis)) {
         /*
          * Postcopy was started, cleanup should happen at the end of the
-         * postcopy listen thread.
+         * postcopy listen thread or eager load thread.
          */
         trace_process_incoming_migration_co_postcopy_end_main();
         goto out;
migration/migration.h +5 -0
index 841f49b215..540124bc27 100644
@@ -42,6 +42,7 @@
 #define  MIGRATION_THREAD_DST_FAULT         "mig/dst/fault"
 #define  MIGRATION_THREAD_DST_LISTEN        "mig/dst/listen"
 #define  MIGRATION_THREAD_DST_PREEMPT       "mig/dst/preempt"
+#define  MIGRATION_THREAD_DST_SNAPSHOT_LOAD "mig/dst/snapshot_load"
 
 struct PostcopyBlocktimeContext;
 typedef struct ThreadPool ThreadPool;
@@ -120,6 +121,10 @@ struct MigrationIncomingState {
     bool           have_listen_thread;
     QemuThread     listen_thread;
 
+    /* Thread to load pages eagerly in fast snapshot load case */
+    bool have_eager_load_thread;
+    QemuThread eager_load_thread;
+
     /* For the kernel to send us notifications */
     int       userfault_fd;
     /* To notify the fault_thread to wake, e.g., when need to quit */
migration/postcopy-ram.c +69 -0

78 lines not on this page · raw message

migration/postcopy-ram.h +2 -0
index ea879f0bc6..c58a60bc8a 100644
@@ -205,4 +205,6 @@ void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
 int postcopy_incoming_setup(MigrationIncomingState *mis, Error **errp);
 int postcopy_incoming_cleanup(MigrationIncomingState *mis);
 
+void postcopy_ram_eager_load_setup(MigrationIncomingState *mis);
+
 #endif
migration/savevm.c +16 -0
index 23adaf9dd9..08f614349b 100644
@@ -2959,6 +2959,22 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis)
     return true;
 }
 
+/*
+ * Starts the VM and launches the eager thread for fast snapshot load
+ */
+void qemu_loadvm_run_fast_snapshot_load(QEMUFile *f,
+                                        MigrationIncomingState *mis)
+{
+    postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+
+    migration_bh_schedule(loadvm_postcopy_handle_run_bh, mis);
+
+    migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
+                      MIGRATION_STATUS_POSTCOPY_ACTIVE);
+
+    postcopy_ram_eager_load_setup(mis);
+}
+
 int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis,
                            Error **errp)
 {
migration/savevm.h +2 -0
index 96fdf96d4e..27c6becbc3 100644
@@ -67,6 +67,8 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
 int qemu_save_device_state(QEMUFile *f, Error **errp);
 int qemu_loadvm_state(QEMUFile *f, Error **errp);
 void qemu_loadvm_state_cleanup(MigrationIncomingState *mis);
+void qemu_loadvm_run_fast_snapshot_load(QEMUFile *f,
+                                        MigrationIncomingState *mis);
 int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis,
                            Error **errp);
 int qemu_load_device_state(QEMUFile *f, Error **errp);
migration/trace-events +2 -0
index de99d976ab..38f11e1e9f 100644
@@ -314,6 +314,8 @@ postcopy_blocktime_tid_cpu_map(int cpu, uint32_t tid) "cpu: %d, tid: %u"
 postcopy_blocktime_begin(uint64_t addr, uint64_t time, int cpu, bool exists) "addr: 0x%" PRIx64 ", time: %" PRIu64 ", cpu: %d, exist: %d"
 postcopy_blocktime_end(uint64_t addr, uint64_t time, int affected_cpu, int affected_non_cpus) "addr: 0x%" PRIx64 ", time: %" PRIu64 ", affected_cpus: %d, affected_non_cpus: %d"
 postcopy_blocktime_end_one(int cpu, uint8_t left_faults) "cpu: %d, left_faults: %" PRIu8
+postcopy_ram_eager_load_thread_entry(void) ""
+postcopy_ram_eager_load_thread_exit(void) ""
 
 # exec.c
 migration_exec_outgoing(const char *cmd) "cmd=%s"
Signature · 1 line
2.55.0
Reviewed-by on v3 8/11
#20 lore raw

Re: [PATCH v3 08/11] migration: add eager load thread and setup for fast snapshot load

Quoted 33 lines · Aadeshveer Singh

In fast snapshot load a thread is needed for actively loading in pages along with the fault path so that the guest is not dependent on fault thread indefinitely. Considering the difference from usual network postcopy where major chunk of RAM is already loaded here entire RAM needs to be loaded later. Existance of background pages which are not really accessed by the guest might never be loaded and system will be locked in migration for indefinite time. As there should be no assumption about how guest accesses memory, the load times can be indefinite.

Add postcopy_ram_eager_load_thread(), for the eager thread which iterates over all non ignored blocks calling ram_block_load_eager() on each. ram_block_load_eager then iterates to load in all pages using postcopy_mapped_ram_load_page(), with a different channel, which takes care of not loading in pages already loaded by fault thread. On completion the thread schedules postcopy_incoming_complete_bh() to destroy the incoming migration state.

Add postcopy_ram_eager_load_setup() to create the thread. Added joining logic in postcopy_incoming_cleanup().

Add tracepoints for entry and exit to eager load thread.

When both mapped-ram and postcopy-ram are set, divert from qemu_loadvm_state to run fast snapshot load

Initialize postcopy RAM state and register RAM Blocks with userfaultfd via ram_postcopy_incoming_init() and postcopy_ram_incoming_setup() in process_incoming_migration_co(). Fault thread needs to be launched before VM to serve faults for some hardwares emulation that need to read RAM (like vapic devices). Populate bitmaps and offset tables while reading file in qemu_loadvm_state_main.

Quoted 8 lines

Add function qemu_loadvm_run_fast_snapshot_load() which starts the VM using loadvm_postcopy_handle_run_bh() and launches eager load thread.

Skip scheduling process_incoming_migration_bh() in process_incoming_migration_co(), for fast snapshot load as the state cleanup is managed by eager load thread on completion.

One nitpick below:

Quoted 151 lines
---
 migration/migration.c    | 36 +++++++++++++++++++--
 migration/migration.h    |  5 +++
 migration/postcopy-ram.c | 69 ++++++++++++++++++++++++++++++++++++++++
 migration/postcopy-ram.h |  2 ++
 migration/savevm.c       | 16 ++++++++++
 migration/savevm.h       |  2 ++
 migration/trace-events   |  2 ++
 7 files changed, 130 insertions(+), 2 deletions(-)
migration/migration.c +34 -2
index 07394d8eea..e9c4eee6b9 100644
@@ -710,6 +710,11 @@ static void process_incoming_migration_bh(void *opaque)
     migration_incoming_state_destroy();
 }
 
+static bool migration_incoming_has_postcopy_thread(MigrationIncomingState *mis)
+{
+    return mis->have_listen_thread || mis->have_eager_load_thread;
+}
+
 static void coroutine_fn
 process_incoming_migration_co(void *opaque)
 {
@@ -739,17 +744,44 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    /*
+     * When loading snapshot with postcopy enabled, setup the postcopy
+     * infrastructure before loading the major part of device states.
+     * It's required because qemu_loadvm_state() may access guest memory
+     * while loading device states, which can cause page faults already.
+     */
+    if (migrate_postcopy_ram() && migrate_mapped_ram()) {
+        migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+                          MIGRATION_STATUS_POSTCOPY_DEVICE);
+
+        if (ram_postcopy_incoming_init(mis, &local_err)) {
+            goto fail;
+        }
+
+        postcopy_state_set(POSTCOPY_INCOMING_LISTENING);
+        if (postcopy_ram_incoming_setup(mis, &local_err)) {
+            goto fail;
+        }
+    }
+
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
+    if (ret < 0) {
+        goto fail;
+    }
+
+    if (migrate_postcopy_ram() && migrate_mapped_ram()) {
+        qemu_loadvm_run_fast_snapshot_load(mis->from_src_file, mis);
+    }
 
     trace_vmstate_downtime_checkpoint("dst-precopy-loadvm-completed");
 
     trace_process_incoming_migration_co_end(ret);
-    if (mis->have_listen_thread) {
+    if (migration_incoming_has_postcopy_thread(mis)) {
         /*
          * Postcopy was started, cleanup should happen at the end of the
-         * postcopy listen thread.
+         * postcopy listen thread or eager load thread.
          */
         trace_process_incoming_migration_co_postcopy_end_main();
         goto out;
migration/migration.h +5 -0
index 841f49b215..540124bc27 100644
@@ -42,6 +42,7 @@
 #define  MIGRATION_THREAD_DST_FAULT         "mig/dst/fault"
 #define  MIGRATION_THREAD_DST_LISTEN        "mig/dst/listen"
 #define  MIGRATION_THREAD_DST_PREEMPT       "mig/dst/preempt"
+#define  MIGRATION_THREAD_DST_SNAPSHOT_LOAD "mig/dst/snapshot_load"
 
 struct PostcopyBlocktimeContext;
 typedef struct ThreadPool ThreadPool;
@@ -120,6 +121,10 @@ struct MigrationIncomingState {
     bool           have_listen_thread;
     QemuThread     listen_thread;
 
+    /* Thread to load pages eagerly in fast snapshot load case */
+    bool have_eager_load_thread;
+    QemuThread eager_load_thread;
+
     /* For the kernel to send us notifications */
     int       userfault_fd;
     /* To notify the fault_thread to wake, e.g., when need to quit */
migration/postcopy-ram.c +41 -0
index 723070b5cd..37eb98b5b1 100644
@@ -2342,9 +2342,78 @@ int postcopy_incoming_cleanup(MigrationIncomingState *mis)
         mis->have_listen_thread = false;
     }
 
+    if (mis->have_eager_load_thread) {
+        qemu_thread_join(&mis->eager_load_thread);
+        mis->have_eager_load_thread = false;
+    }
+
     if (migrate_postcopy_ram()) {
         rc = postcopy_ram_incoming_cleanup(mis);
     }
 
     return rc;
 }
+
+/*
+ * Called by postcopy_ram_eager_load_thread over all blocks to load in all the
+ * pending pages of given ram block
+ */
+static int ram_block_load_eager(RAMBlock *rb, void *opaque)
+{
+    MigrationIncomingState *mis = migration_incoming_get_current();
+    MigrationState *s = migrate_get_current();
+    Error *errp = NULL;
+    void *host = qemu_ram_get_host_addr(rb);
+    void *target;
+
+    for (ram_addr_t page_loc = 0; page_loc < rb->used_length;
+         page_loc += qemu_ram_pagesize(rb)) {
+        target = (uint8_t *)host + page_loc;
+        if (!postcopy_mapped_ram_load_page(mis, rb, page_loc, (uint64_t)target,
+                                           RAM_CHANNEL_PRECOPY, &errp)) {
+            migrate_error_propagate(s, errp);
+            return -1;
+        }
+    }
+    return 0;
+}
+
+/*
+ * Used by fast snapshot load to eagerly load in all pages of RAM and schedule
+ * cleanup after entire RAM is loaded
+ */
+static void *postcopy_ram_eager_load_thread(void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+
+    trace_postcopy_ram_eager_load_thread_entry();
+    rcu_register_thread();
+    qemu_event_set(&mis->thread_sync_event);

migration/postcopy-ram.c int postcopy_incoming_cleanup(MigrationIncomingState *mis) · line 2401

Quoted 10 lines
migration/postcopy-ram.c +10 -0
+
+    if (foreach_not_ignored_block(ram_block_load_eager, NULL)) {
+        migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
+                          MIGRATION_STATUS_FAILED);
+        goto out;
+    }
+    migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
+                      MIGRATION_STATUS_COMPLETED);
+
+out:

The label isn't needed.

Meanwhile we can still pass "mis" as opaque, even if migrate_get_current() is still needed.. but slightly cleaner.

So it can be:

  if (foreach_not_ignored_block(ram_block_load_eager, mis)) {
      next_state = MIGRATION_STATUS_FAILED;
  } else {
      next_state = MIGRATION_STATUS_COMPLETED;
  }

  migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_ACTIVE, next_state);
Quoted 85 lines
migration/postcopy-ram.c +18 -0
+    postcopy_state_set(POSTCOPY_INCOMING_END);
+    migration_bh_schedule(postcopy_incoming_complete_bh, mis);
+
+    rcu_unregister_thread();
+    trace_postcopy_ram_eager_load_thread_exit();
+    return NULL;
+}
+
+/*
+ * Create thread for eager loading in fast snapshot load case
+ */
+void postcopy_ram_eager_load_setup(MigrationIncomingState *mis)
+{
+    postcopy_thread_create(
+        mis, &mis->eager_load_thread, MIGRATION_THREAD_DST_SNAPSHOT_LOAD,
+        postcopy_ram_eager_load_thread, QEMU_THREAD_JOINABLE);
+    mis->have_eager_load_thread = true;
+}
migration/postcopy-ram.h +2 -0
index ea879f0bc6..c58a60bc8a 100644
@@ -205,4 +205,6 @@ void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
 int postcopy_incoming_setup(MigrationIncomingState *mis, Error **errp);
 int postcopy_incoming_cleanup(MigrationIncomingState *mis);
 
+void postcopy_ram_eager_load_setup(MigrationIncomingState *mis);
+
 #endif
migration/savevm.c +16 -0
index 23adaf9dd9..08f614349b 100644
@@ -2959,6 +2959,22 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis)
     return true;
 }
 
+/*
+ * Starts the VM and launches the eager thread for fast snapshot load
+ */
+void qemu_loadvm_run_fast_snapshot_load(QEMUFile *f,
+                                        MigrationIncomingState *mis)
+{
+    postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+
+    migration_bh_schedule(loadvm_postcopy_handle_run_bh, mis);
+
+    migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_DEVICE,
+                      MIGRATION_STATUS_POSTCOPY_ACTIVE);
+
+    postcopy_ram_eager_load_setup(mis);
+}
+
 int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis,
                            Error **errp)
 {
migration/savevm.h +2 -0
index 96fdf96d4e..27c6becbc3 100644
@@ -67,6 +67,8 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
 int qemu_save_device_state(QEMUFile *f, Error **errp);
 int qemu_loadvm_state(QEMUFile *f, Error **errp);
 void qemu_loadvm_state_cleanup(MigrationIncomingState *mis);
+void qemu_loadvm_run_fast_snapshot_load(QEMUFile *f,
+                                        MigrationIncomingState *mis);
 int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis,
                            Error **errp);
 int qemu_load_device_state(QEMUFile *f, Error **errp);
migration/trace-events +2 -0
index de99d976ab..38f11e1e9f 100644
@@ -314,6 +314,8 @@ postcopy_blocktime_tid_cpu_map(int cpu, uint32_t tid) "cpu: %d, tid: %u"
 postcopy_blocktime_begin(uint64_t addr, uint64_t time, int cpu, bool exists) "addr: 0x%" PRIx64 ", time: %" PRIu64 ", cpu: %d, exist: %d"
 postcopy_blocktime_end(uint64_t addr, uint64_t time, int affected_cpu, int affected_non_cpus) "addr: 0x%" PRIx64 ", time: %" PRIu64 ", affected_cpus: %d, affected_non_cpus: %d"
 postcopy_blocktime_end_one(int cpu, uint8_t left_faults) "cpu: %d, left_faults: %" PRIu8
+postcopy_ram_eager_load_thread_entry(void) ""
+postcopy_ram_eager_load_thread_exit(void) ""
 
 # exec.c
 migration_exec_outgoing(const char *cmd) "cmd=%s"
Signature · 1 line
2.55.0
Signature · 1 line
Peter Xu
#21 lore raw

PATCH v3 9/11 migration/tests: remove capability conflict test postcopy-ram+mapped-ram

In reply to Aadeshveer Singh

Remove the test test_validate_caps_pair, which asserted postcopy-ram and mapped-ram capabilities cannot be active together. The new fast snapshot load feature is exactly this pair of capabilities active together, with the following patches in this series, this combination will now supported and be functional.

Remove the capability check that rejected mapped-ram and postcopy-ram being set simultaneously, as this combination now corresponds to fast snapshot load.

Add a capability check against setting multifd when configured for fast snapshot load as it is not supported yet. Also add capability check against setting postcopy-preempt as it is incompatible.

A smoke test exercising this feature has been added further in this series.

---
 migration/options.c                | 14 ++++++--
 tests/qtest/migration/misc-tests.c | 52 ------------------------------
 2 files changed, 12 insertions(+), 54 deletions(-)
migration/options.c +12 -2
index 5cbfd29099..d61a10f1c4 100644
@@ -727,10 +727,20 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
                        "Mapped-ram migration is incompatible with xbzrle");
             return false;
         }
+    }
 
-        if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
+    if (new_caps[MIGRATION_CAPABILITY_MAPPED_RAM] &&
+        new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
+        if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
             error_setg(errp,
-                       "Mapped-ram migration is incompatible with postcopy");
+                       "Multifd is not supported with fast snapshot load");
+            return false;
+        }
+
+        if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT]) {
+            error_setg(
+                errp,
+                "Postcopy Preempt is incompatible with fast snapshot load");
             return false;
         }
     }
tests/qtest/migration/misc-tests.c +0 -52

62 lines not on this page · raw message

Signature · 1 line
2.55.0
Reviewed-by on v3 9/11
#22 lore raw

Re: [PATCH v3 09/11] migration/tests: remove capability conflict test postcopy-ram+mapped-ram

Quoted 10 lines · Aadeshveer Singh

Remove the test test_validate_caps_pair, which asserted postcopy-ram and mapped-ram capabilities cannot be active together. The new fast snapshot load feature is exactly this pair of capabilities active together, with the following patches in this series, this combination will now supported and be functional.

Remove the capability check that rejected mapped-ram and postcopy-ram being set simultaneously, as this combination now corresponds to fast snapshot load.

Quoted 8 lines

Add a capability check against setting multifd when configured for fast snapshot load as it is not supported yet. Also add capability check against setting postcopy-preempt as it is incompatible.

A smoke test exercising this feature has been added further in this series.

The subject with "migration/tests:" may imply this is a pure test change. Let's rename it to something like "migration: Enable mapped-ram with postcopy" to say this is a functional change. It's normally fine to have test changes with such functional change, not vice versa (or people may ignore this patch when backporting the feature only).

After rename, feel free to take:

Quoted 103 lines
---
 migration/options.c                | 14 ++++++--
 tests/qtest/migration/misc-tests.c | 52 ------------------------------
 2 files changed, 12 insertions(+), 54 deletions(-)
migration/options.c +12 -2
index 5cbfd29099..d61a10f1c4 100644
@@ -727,10 +727,20 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
                        "Mapped-ram migration is incompatible with xbzrle");
             return false;
         }
+    }
 
-        if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
+    if (new_caps[MIGRATION_CAPABILITY_MAPPED_RAM] &&
+        new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
+        if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
             error_setg(errp,
-                       "Mapped-ram migration is incompatible with postcopy");
+                       "Multifd is not supported with fast snapshot load");
+            return false;
+        }
+
+        if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT]) {
+            error_setg(
+                errp,
+                "Postcopy Preempt is incompatible with fast snapshot load");
             return false;
         }
     }
tests/qtest/migration/misc-tests.c +0 -52
index ec6d438cdc..4e0deb7f18 100644
@@ -201,55 +201,6 @@ static void do_test_validate_uri_channel(MigrateCommon *args)
     migrate_end(from, to, false);
 }
 
-static void validate_caps_pair(QTestState *from,
-                               const char *first_capability,
-                               const char *second_capability,
-                               const char *expected_error)
-{
-    QDict *rsp;
-    const char *error_desc;
-
-    migrate_set_capability(from, first_capability, true);
-
-    rsp = qtest_qmp_assert_failure_ref(
-        from,
-        "{ 'execute': 'migrate-set-capabilities',"
-        "  'arguments': { 'capabilities': [ { "
-        "      'capability': %s, 'state': true } ] } }",
-        second_capability);
-
-    error_desc = qdict_get_str(rsp, "desc");
-    g_assert_cmpstr(error_desc, ==, expected_error);
-    qobject_unref(rsp);
-
-    migrate_set_capability(from, first_capability, false);
-}
-
-static void test_validate_caps_pair(char *test_path, MigrateCommon *args)
-{
-    g_autofree char *serial_path = g_strconcat(tmpfs, "/src_serial", NULL);
-    g_autofree char *cap_pair = g_path_get_basename(test_path);
-    QTestState *from, *to;
-
-    args->start.hide_stderr = true;
-    args->start.only_source = true;
-
-    if (migrate_start(&from, &to, &args->start)) {
-        return;
-    }
-
-    if (g_str_equal(cap_pair, "mapped_ram_postcopy")) {
-        const char *error =
-            "Mapped-ram migration is incompatible with postcopy";
-
-        validate_caps_pair(from, "mapped-ram", "postcopy-ram", error);
-        validate_caps_pair(from, "postcopy-ram", "mapped-ram", error);
-    }
-
-    qtest_quit(from);
-    unlink(serial_path);
-}
-
 static void test_validate_uri_channels_both_set(char *name, MigrateCommon *args)
 {
     args->uri = "tcp:127.0.0.1:0",
@@ -309,7 +260,4 @@ void migration_test_add_misc(MigrationTestEnv *env)
                        test_validate_uri_channels_both_set);
     migration_test_add("/migration/validate_uri/channels/none_set",
                        test_validate_uri_channels_none_set);
-    migration_test_add_suffix("/migration/validate_caps/",
-                              "mapped_ram_postcopy",
-                              test_validate_caps_pair);
 }
Signature · 1 line
2.55.0
Signature · 1 line
Peter Xu
#23 lore raw

PATCH v3 10/11 migration/tests: Add test for fast snapshot load

In reply to Aadeshveer Singh

Add test_postcopy_file_mapped_ram() to test for saving a snapshot in mapped ram format and loading it using postcopy/fast snapshot load.

Test is similar to test_precopy_file_mapped_ram() with additional start_hook to set postcopy capabilities on the receiving end.

---
 tests/qtest/migration/file-tests.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tests/qtest/migration/file-tests.c +24 -0
index 5118d9dec9..c7e4b05c61 100644
@@ -190,6 +190,25 @@ static void test_multifd_file_mapped_ram_dio(char *name, MigrateCommon *args)
     test_file_common(args, true);
 }
 
+static void *migrate_hook_start_postcopy_mapped_ram(QTestState *from,
+                                                    QTestState *to)
+{
+    migrate_set_capability(
+        to, MigrationCapability_lookup.array[MIGRATION_CAPABILITY_POSTCOPY_RAM],
+        true);
+
+    return NULL;
+}
+
+static void test_postcopy_file_mapped_ram(char *name, MigrateCommon *args)
+{
+    args->start.caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true;
+
+    args->start_hook = migrate_hook_start_postcopy_mapped_ram;
+
+    test_file_common(args, false);
+}
+
 #ifndef _WIN32
 static void migrate_hook_end_multifd_mapped_ram_fdset(QTestState *from,
                                                       QTestState *to,
@@ -330,6 +349,11 @@ void migration_test_add_file(MigrationTestEnv *env)
     migration_test_add("/migration/multifd/file/mapped-ram/live",
                        test_multifd_file_mapped_ram_live);
 
+    if (env->has_uffd) {
+        migration_test_add("/migration/postcopy/file/mapped-ram",
+                           test_postcopy_file_mapped_ram);
+    }
+
 #ifndef _WIN32
     migration_test_add("/migration/multifd/file/mapped-ram/fdset",
                        test_multifd_file_mapped_ram_fdset);
Signature · 1 line
2.55.0
Reviewed-by on v3 10/11
#24 lore raw

Re: [PATCH v3 10/11] migration/tests: Add test for fast snapshot load

Quoted 7 lines · Aadeshveer Singh

Add test_postcopy_file_mapped_ram() to test for saving a snapshot in mapped ram format and loading it using postcopy/fast snapshot load.

Test is similar to test_precopy_file_mapped_ram() with additional start_hook to set postcopy capabilities on the receiving end.

Signature · 1 line
Peter Xu
#25 lore raw

PATCH v3 11/11 docs/migration: Add documentation for fast snapshot load feature

In reply to Aadeshveer Singh

Add documenatation for fast snapshot load covering an overview, the architecture, usage and limitations.

---
 docs/devel/migration/fast-snapshot-load.rst | 81 +++++++++++++++++++++
 docs/devel/migration/features.rst           |  1 +
 2 files changed, 82 insertions(+)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
docs/devel/migration/fast-snapshot-load.rst new file +81 -0
new file mode 100644
index 0000000000..0c0dc676fb
@@ -0,0 +1,81 @@
+==================
+Fast Snapshot Load
+==================
+
+Overview
+========
+Fast snapshot load is an extension of the postcopy migration feature
+to disk loads.
+
+Unlike a usual snapshot load, which requires all VM data (RAM as well
+as device states) to be loaded into host RAM from the snapshot file
+for the guest to run, fast snapshot load uses postcopy infrastructure
+to load in only the required device states and load RAM pages on
+demand. The idea is to start the guest and serve its page faults on
+the go, reducing the perceived resume time for large snapshots.
+
+Architecture
+============
+This feature combines postcopy migration and mapped-ram capabilities
+to load RAM pages on demand. It is done by catching guest faults using
+Linux ``userfaultfd`` and loading the page by calculating the offset
+of its location in the snapshot file using mapped-ram capabilities.
+
+Fault Thread
+------------
+The fault thread uses Linux ``userfaultfd`` to catch page faults caused
+by guest and directly load the page from the snapshot file. It is
+very similar to network postcopy fault thread, with primary difference
+being it loads pages directly by reading from the snapshot file.
+
+Eager Thread
+------------
+Eager thread iterates over all pages in RAM and loads each page not
+yet loaded by fault thread. It is required as unlike network postcopy
+where majority of RAM has already been loaded via precopy, here entire
+RAM is waiting to be loaded. If there is no eager loading thread each
+page will only be loaded when it is required by guest. In case there
+are some background pages that are never/rarely accessed by guest,
+the system will be locked in migration state indefinitely.
+
+Synchronization
+---------------
+In order to make sure both of these threads do not load the same page
+twice potentially overwriting and corrupting user RAM, a bitmap is
+used (``RAMBlock->pending_bmap``) which tracks the pages claimed to
+be loaded by threads. This prevents race condition when one thread
+is loading the page and other one tries to do the same.
+
+Usage
+=====
+
+Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on
+the destination:
+
+.. code-block:: text
+
+    migrate_set_capability mapped-ram on
+    migrate_set_capability postcopy-ram on
+
+Use a ``file:`` URI for migration:
+
+.. code-block:: text
+
+    migrate_incoming file:/path/to/snapshot/file
+
+Limitations
+===========
+
+ - Multifd
+    Fast snapshot load is currently incompatible with ``multifd``
+    capability. While ``mapped-ram`` allows for parallel disk I/O,
+    coupling it with ``postcopy`` capability requires additional
+    infrastructure.
+
+ - Host OS support
+    Because this feautre essentially depends on ``userfaultfd``
+    to trap page faults, it is supported only on Linux hosts.
+
+ - vhost-user
+    Fast snapshot load does not currently support ``vhost-user``
+    backends.
docs/devel/migration/features.rst +1 -0
index 9aef79e7fa..23c2a93173 100644
@@ -11,6 +11,7 @@ Migration has plenty of features to support different use cases.
    vfio
    virtio
    mapped-ram
+   fast-snapshot-load
    CPR
    qpl-compression
    uadk-compression
Signature · 1 line
2.55.0
#26 lore raw

Re: [PATCH v3 11/11] docs/migration: Add documentation for fast snapshot load feature

Quoted 20 lines · Aadeshveer Singh

Add documenatation for fast snapshot load covering an overview, the architecture, usage and limitations.

---
 docs/devel/migration/fast-snapshot-load.rst | 81 +++++++++++++++++++++
 docs/devel/migration/features.rst           |  1 +
 2 files changed, 82 insertions(+)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
docs/devel/migration/fast-snapshot-load.rst new file +4 -0
new file mode 100644
index 0000000000..0c0dc676fb
@@ -0,0 +1,81 @@
+==================
+Fast Snapshot Load
+==================
+

docs/devel/migration/fast-snapshot-load.rst line 14

Quoted 10 lines
docs/devel/migration/fast-snapshot-load.rst new file +10 -0
+Overview
+========
+Fast snapshot load is an extension of the postcopy migration feature
+to disk loads.
+
+Unlike a usual snapshot load, which requires all VM data (RAM as well
+as device states) to be loaded into host RAM from the snapshot file
+for the guest to run, fast snapshot load uses postcopy infrastructure
+to load in only the required device states and load RAM pages on
+demand. The idea is to start the guest and serve its page faults on

Suggest to replace "load RAM pages on demand" to something more general, like, "allows the RAM pages to be loaded after VM starts". E.g. the eager load thread work isn't on-demand.

Quoted 57 lines
docs/devel/migration/fast-snapshot-load.rst new file +57 -0
+the go, reducing the perceived resume time for large snapshots.
+
+Architecture
+============
+This feature combines postcopy migration and mapped-ram capabilities
+to load RAM pages on demand. It is done by catching guest faults using
+Linux ``userfaultfd`` and loading the page by calculating the offset
+of its location in the snapshot file using mapped-ram capabilities.
+
+Fault Thread
+------------
+The fault thread uses Linux ``userfaultfd`` to catch page faults caused
+by guest and directly load the page from the snapshot file. It is
+very similar to network postcopy fault thread, with primary difference
+being it loads pages directly by reading from the snapshot file.
+
+Eager Thread
+------------
+Eager thread iterates over all pages in RAM and loads each page not
+yet loaded by fault thread. It is required as unlike network postcopy
+where majority of RAM has already been loaded via precopy, here entire
+RAM is waiting to be loaded. If there is no eager loading thread each
+page will only be loaded when it is required by guest. In case there
+are some background pages that are never/rarely accessed by guest,
+the system will be locked in migration state indefinitely.
+
+Synchronization
+---------------
+In order to make sure both of these threads do not load the same page
+twice potentially overwriting and corrupting user RAM, a bitmap is
+used (``RAMBlock->pending_bmap``) which tracks the pages claimed to
+be loaded by threads. This prevents race condition when one thread
+is loading the page and other one tries to do the same.
+
+Usage
+=====
+
+Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on
+the destination:
+
+.. code-block:: text
+
+    migrate_set_capability mapped-ram on
+    migrate_set_capability postcopy-ram on
+
+Use a ``file:`` URI for migration:
+
+.. code-block:: text
+
+    migrate_incoming file:/path/to/snapshot/file
+
+Limitations
+===========
+
+ - Multifd
+    Fast snapshot load is currently incompatible with ``multifd``
+    capability. While ``mapped-ram`` allows for parallel disk I/O,

docs/devel/migration/fast-snapshot-load.rst line 81

Quoted 10 lines
docs/devel/migration/fast-snapshot-load.rst new file +10 -0
+    coupling it with ``postcopy`` capability requires additional
+    infrastructure.
+
+ - Host OS support
+    Because this feautre essentially depends on ``userfaultfd``
+    to trap page faults, it is supported only on Linux hosts.
+
+ - vhost-user
+    Fast snapshot load does not currently support ``vhost-user``
+    backends.

For this one you haven't added a check in the other patch to detect it happening, you can add one check to see if postcopy_notifier_list is empty:
anything registered implies vhost-user enabled.

Thanks,

Quoted 15 lines
docs/devel/migration/features.rst +1 -0
index 9aef79e7fa..23c2a93173 100644
@@ -11,6 +11,7 @@ Migration has plenty of features to support different use cases.
    vfio
    virtio
    mapped-ram
+   fast-snapshot-load
    CPR
    qpl-compression
    uadk-compression
Signature · 1 line
2.55.0
Signature · 1 line
Peter Xu
#27 lore raw

Re: [PATCH v3 11/11] docs/migration: Add documentation for fast snapshot load feature

Quoted 28 lines · Peter Xu
Quoted 26 lines · Aadeshveer Singh

Add documenatation for fast snapshot load covering an overview, the architecture, usage and limitations.

---
 docs/devel/migration/fast-snapshot-load.rst | 81 +++++++++++++++++++++
 docs/devel/migration/features.rst           |  1 +
 2 files changed, 82 insertions(+)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
docs/devel/migration/fast-snapshot-load.rst new file +10 -0
new file mode 100644
index 0000000000..0c0dc676fb
@@ -0,0 +1,81 @@
+==================
+Fast Snapshot Load
+==================
+
+Overview
+========
+Fast snapshot load is an extension of the postcopy migration feature
+to disk loads.
+
+Unlike a usual snapshot load, which requires all VM data (RAM as well
Quoted 8 lines

docs/devel/migration/fast-snapshot-load.rst line 14

Quoted 4 lines
docs/devel/migration/fast-snapshot-load.rst new file +4 -0
+as device states) to be loaded into host RAM from the snapshot file
+for the guest to run, fast snapshot load uses postcopy infrastructure
+to load in only the required device states and load RAM pages on
+demand. The idea is to start the guest and serve its page faults on

Suggest to replace "load RAM pages on demand" to something more general, like, "allows the RAM pages to be loaded after VM starts". E.g. the eager load thread work isn't on-demand.

replacing with a more generalized line does make sense. I will add it in next version.

Quoted 64 lines
Quoted 64 lines
docs/devel/migration/fast-snapshot-load.rst new file +63 -0
+the go, reducing the perceived resume time for large snapshots.
+
+Architecture
+============
+This feature combines postcopy migration and mapped-ram capabilities
+to load RAM pages on demand. It is done by catching guest faults using
+Linux ``userfaultfd`` and loading the page by calculating the offset
+of its location in the snapshot file using mapped-ram capabilities.
+
+Fault Thread
+------------
+The fault thread uses Linux ``userfaultfd`` to catch page faults caused
+by guest and directly load the page from the snapshot file. It is
+very similar to network postcopy fault thread, with primary difference
+being it loads pages directly by reading from the snapshot file.
+
+Eager Thread
+------------
+Eager thread iterates over all pages in RAM and loads each page not
+yet loaded by fault thread. It is required as unlike network postcopy
+where majority of RAM has already been loaded via precopy, here entire
+RAM is waiting to be loaded. If there is no eager loading thread each
+page will only be loaded when it is required by guest. In case there
+are some background pages that are never/rarely accessed by guest,
+the system will be locked in migration state indefinitely.
+
+Synchronization
+---------------
+In order to make sure both of these threads do not load the same page
+twice potentially overwriting and corrupting user RAM, a bitmap is
+used (``RAMBlock->pending_bmap``) which tracks the pages claimed to
+be loaded by threads. This prevents race condition when one thread
+is loading the page and other one tries to do the same.
+
+Usage
+=====
+
+Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on
+the destination:
+
+.. code-block:: text
+
+    migrate_set_capability mapped-ram on
+    migrate_set_capability postcopy-ram on
+
+Use a ``file:`` URI for migration:
+
+.. code-block:: text
+
+    migrate_incoming file:/path/to/snapshot/file
+
+Limitations
+===========
+
+ - Multifd
+    Fast snapshot load is currently incompatible with ``multifd``
+    capability. While ``mapped-ram`` allows for parallel disk I/O,
+    coupling it with ``postcopy`` capability requires additional
+    infrastructure.
+
+ - Host OS support
+    Because this feautre essentially depends on ``userfaultfd``
+    to trap page faults, it is supported only on Linux hosts.
Quoted 8 lines

docs/devel/migration/fast-snapshot-load.rst line 81

Quoted 4 lines
docs/devel/migration/fast-snapshot-load.rst new file +4 -0
+
+ - vhost-user
+    Fast snapshot load does not currently support ``vhost-user``
+    backends.

For this one you haven't added a check in the other patch to detect it happening, you can add one check to see if postcopy_notifier_list is empty:
anything registered implies vhost-user enabled.

Thanks for pointing out the use of postcopy_notifier_list, will add it in next patch.

Quoted 22 lines

Thanks,

Quoted 13 lines
docs/devel/migration/features.rst +1 -1
index 9aef79e7fa..23c2a93173 100644
@@ -11,6 +11,7 @@ Migration has plenty of features to support different use cases.
    vfio
    virtio
    mapped-ram
+   fast-snapshot-load
    CPR
    qpl-compression
    uadk-compression
--
Quoted 2 lines

2.55.0

--
Peter Xu

#28 lore raw
In reply to Aadeshveer Singh
Quoted 34 lines · Aadeshveer Singh

This series implements a "fast snapshot load" mechanism to significantly reduce the perceived resume time of a VM from a snapshot file.

Currently, resuming a VM from a snapshot file requires loading all RAM pages into the QEMU instance before execution begins. This extension allows the user to run the VM nearly instantly by loading only the required device states up front and loading RAM pages lazily, by trapping access to pages that have not yet been loaded.

Using the Linux userfaultfd syscall, a fault thread catches all page faults caused by the guest and loads in the pages required to keep the VM running. Concurrently, an eager background thread iteratively loads all remaining pages into RAM so the guest does not have to depend on the fault thread indefinitely.

Much of code is reused from postcopy for fault handling and precopy for reading mapped ram file. Implementation revolves around two threads named the fault thread and eager load thread. Fault thread as name suggests catches page faults by the guest and serves them using userfaultfd. Postcopy fault thread is reused but instead of requesting source for a page it loads the page directly by reading from file. In order to remove the dependency of guest on fault thread indefinitely the eager load thread loads in the entire RAM sequentially, and after iterating through the entire RAM signals fault thread to exit and calls cleanup.

In order to prevent the case of a page being loaded twice(in the case when eager load thread is loading it and fault thread also tries to serve fault on same page) a bitmap called pending_bmap is used to track pages which are pending and not being loaded by any thread. Atomic operations on this bitmap allows coordination between threads to prevent any unwanted behaviours

Quoted 3 lines

Future direction:
- Add support for multifd
- Add support for vhost-user

Please always still at least have a section for testing. It's an important piece of information on what you have tested (and what you have not, which can also be included when it may matter).

For example, I know you should have tested at least normal and some huge page setup, you can list them there. You can also mention if you only tested x86_64, or if you have played with anything else; logically this should also work elsewhere whenever all facilities are available. It's fine to only test on x86_64, though.

Thanks,

Quoted 75 lines

---
v1 -> v2

- Structure Change: Expanded from 7 to 11 patches
- Move test removal patch from first to post implementation as suggested by Peter
- Patch 1 to 5 are meant to be non functional changes meant to modernize parts of code and prepare for feature Implementation
- Patch 10 performs a smoke test for implemented feature
- Patch 11 adds a documentation page for fast snapshot load
- Added explicit fails for features not supported/incompatible with fast snapshot load in compatibility check in options.c as suggested by Peter
- Passed errp object to qemu_get_buffer_at() to improve on error reporting as suggested by Peter
- Size of pending_bmap is updated to have a bit for every guest page according to guest page size and not host page size as pointed by Peter
- New function(ramblock_file_bitmap_page_is_nonzero()) has been introduced to test for zero pages while loading essentially enableing support for hugepages
- Updated the documentation comment over postcopy_mapped_ram_load_page() to add errp as pointed by Peter
- Added a new function try_mark_postcopy_blocktime_begin() to call mark_postcopy_blocktime_begin() taking care of assumptions like skipping if page is already received to improve on the readability as pointed by Peter
- Fixed bug, that sent non page alined addresses to mark_postcopy_blocktime_begin() as pointed by Peter
- Separate the postcopy_listen_thread_bh() rename patch as suggested by Peter
- Add comment explaining changes in process_incoming_migration_co() as suggested by Peter
- Make qemu_loadvm_run_fast_snapshot_load() and postcopy_ram_eager_load_setup() return void as these functions never fail as pointed by Peter

Aadeshveer Singh (11):
migration: Propagate error in postcopy setup functions
migration: Extract blocktime marking helper
migration: Rename postcopy_listen_thread_bh
migration: Use file_bmap for RAMBlock during incoming file load
migration: Make qemu_get_buffer_at() thread-safe
migration: add RAMBlock field and helper for fast snapshot load
migration: add support for fault thread to load pages from disk
migration: add eager load thread and setup for fast snapshot load migration/tests: remove capability conflict test
postcopy-ram+mapped-ram
migration/tests: Add test for fast snapshot load docs/migration: Add documentation for fast snapshot load feature

 docs/devel/migration/fast-snapshot-load.rst |  81 ++++++
 docs/devel/migration/features.rst           |   1 +
 include/system/ramblock.h                   |   6 +
 migration/migration.c                       |  60 +++--
 migration/migration.h                       |   5 +
 migration/options.c                         |  14 +-
 migration/postcopy-ram.c                    | 266 ++++++++++++++++----
 migration/postcopy-ram.h                    |   7 +-
 migration/qemu-file.c                       |   9 +-
 migration/qemu-file.h                       |   4 +-
 migration/ram.c                             | 103 +++++++-
 migration/ram.h                             |   2 +
 migration/savevm.c                          |  16 ++
 migration/savevm.h                          |   2 +
 migration/trace-events                      |   2 +
 tests/qtest/migration/file-tests.c          |  24 ++
 tests/qtest/migration/misc-tests.c          |  52 ----
 17 files changed, 511 insertions(+), 143 deletions(-)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
Signature · 1 line
2.55.0
Signature · 1 line
Peter Xu