Skip to content

migration: fast snapshot load

Aadeshveer Singh qemu-devel 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest11 patches 4 repliesIn review
  2. v411 patches 18 repliesSuperseded
  3. v311 patches 17 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 RFC7 patches13 repliesSuperseded
v1 RFC5 patches18 repliesSuperseded

Changes in v5 from the cover letter

- Modify pending_bmap(Patch 6) to use variable size for better efficient atomic state management
- Added a comment(Patch 7) explaining postcopy_mapped_ram_load_page explaining why cases where guest page is larger than host page are disabled following what remote postcopy follows
- Add a new check(Patch 9) in migrate_prepare to bail out if migrate was used with fast snapshot load enabled, as suggested by Peter

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11migration: Propagate error in postcopy setup functionsmigration/postcopy-ram.c migration/postcopy-ram.h+18 -250 replies
2/11migration: Extract blocktime marking helpermigration/migration.c migration/postcopy-ram.c migration/postcopy-ram.h+36 -220 replies
3/11migration: Rename postcopy_listen_thread_bhmigration/postcopy-ram.c+2 -20 replies
4/11migration: Use file_bmap for RAMBlock during incoming file loadmigration/ram.c+27 -60 replies
5/11migration: Make qemu_get_buffer_at() thread-safemigration/qemu-file.c migration/qemu-file.h migration/ram.c+10 -110 replies
6/11migration: add RAMBlock field and helper for fast snapshot loadinclude/system/ramblock.h migration/ram.c+66 -60 replies
7/11migration: add support for fault thread to load pages from diskmigration/postcopy-ram.c+166 -170 replies
8/11migration: add eager load thread and setup for fast snapshot loadmigration/migration.c migration/migration.h migration/postcopy-ram.c +4 more+130 -20 replies
9/11migration: update capability conflict test for postcopy-ram+mapped-raminclude/qemu/notify.h migration/migration.c migration/options.c +4 more+37 -540 replies
10/11migration/tests: Add test for fast snapshot loadtests/qtest/migration/file-tests.c+24 -00 replies
11/11docs/migration: Add documentation for fast snapshot load featuredocs/devel/migration/fast-snapshot-load.rst docs/devel/migration/features.rst+83 -00 replies

Reviews trailers per patch

8 of 11 patches have a Reviewed-by or Acked-by.

PatchPeter XuJuraj Marcin
1/11R-b R-b
2/11R-b R-b
3/11R-b R-b
4/11R-b R-b
5/11R-b R-b
6/11
7/11
8/11R-b R-b
9/11
10/11R-b R-b
11/11R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 00/11] migration: fast snapshot load

This series implements a "fast snapshot load" mechanism to significantly reduce the perceived resume time of a VM from a snapshot file.

Currently, resuming a VM from a snapshot file requires loading all RAM pages into the QEMU instance before execution begins. This extension allows the user to run the VM nearly instantly by loading only the required device states up front and loading RAM pages lazily, by trapping access to pages that have not yet been loaded.

Using the Linux userfaultfd syscall, a fault thread catches all page faults caused by the guest and loads in the pages required to keep the VM running. Concurrently, an eager background thread iteratively loads all remaining pages into RAM so the guest does not have to depend on the fault thread indefinitely.

Much of code is reused from postcopy for fault handling and precopy for reading mapped ram file. Implementation revolves around two threads named the fault thread and eager load thread. Fault thread as name suggests catches page faults by the guest and serves them using userfaultfd. Postcopy fault thread is reused but instead of requesting source for a page it loads the page directly by reading from file. In order to remove the dependency of guest on fault thread indefinitely the eager load thread loads in the entire RAM sequentially, and after iterating through the entire RAM signals fault thread to exit and calls cleanup.

In order to prevent the case of a page being loaded twice(in the case when eager load thread is loading it and fault thread also tries to serve fault on same page) a bitmap called pending_bmap is used to track pages which are pending and not being loaded by any thread. Atomic operations on this bitmap allows coordination between threads to prevent any unwanted behaviours

This patch series was tested on a single machine:
host OS : Fedora 44
host RAM : 32GB DDR5(4KB pagesize)
host CPU : Intel Ultra9 185H(22 cores, x86_64)

Guests tested:
- KVM enabled
guest OS : Fedora 44
guest RAM : 16GB(4KB pagesize)
guest CPU : 4 cores(x86_64)
- host using 2MB hugepages
guest OS : Debian 13
guest RAM : 16GB(16KB pagesize)
guest CPU : 4 cores(x86_64)

Future direction:
- Add support for multifd
- Add support for vhost-user

---
v4 -> v5
- Modify pending_bmap(Patch 6) to use variable size for better efficient atomic state management
- Added a comment(Patch 7) explaining postcopy_mapped_ram_load_page explaining why cases where guest page is larger than host page are disabled following what remote postcopy follows
- Add a new check(Patch 9) in migrate_prepare to bail out if migrate was used with fast snapshot load enabled, as suggested by Peter

Aadeshveer Singh (11):
migration: Propagate error in postcopy setup functions
migration: Extract blocktime marking helper
migration: Rename postcopy_listen_thread_bh
migration: Use file_bmap for RAMBlock during incoming file load
migration: Make qemu_get_buffer_at() thread-safe
migration: add RAMBlock field and helper for fast snapshot load
migration: add support for fault thread to load pages from disk
migration: add eager load thread and setup for fast snapshot load
migration: update capability conflict test for postcopy-ram+mapped-ram migration/tests: Add test for fast snapshot load docs/migration: Add documentation for fast snapshot load feature

 docs/devel/migration/fast-snapshot-load.rst |  82 +++++
 docs/devel/migration/features.rst           |   1 +
 include/qemu/notify.h                       |   2 +
 include/system/ramblock.h                   |   6 +
 migration/migration.c                       |  66 ++--
 migration/migration.h                       |   5 +
 migration/options.c                         |  20 +-
 migration/postcopy-ram.c                    | 333 +++++++++++++++++---
 migration/postcopy-ram.h                    |   8 +-
 migration/qemu-file.c                       |  11 +-
 migration/qemu-file.h                       |   4 +-
 migration/ram.c                             | 101 +++++-
 migration/savevm.c                          |  16 +
 migration/savevm.h                          |   2 +
 migration/trace-events                      |   2 +
 tests/qtest/migration/file-tests.c          |  24 ++
 tests/qtest/migration/misc-tests.c          |  52 ---
 util/notify.c                               |   5 +
 18 files changed, 597 insertions(+), 143 deletions(-)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst
Signature · 1 line
2.55.0