Skip to content

Postcopy implementation

Dr. David Alan Gilbert (git) qemu-devel 12 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest56 patches 35 repliesIn review
  2. v854 patches 63 repliesSuperseded
  3. v742 patches 166 repliesSuperseded
Earlier versions · 9
VersionPostedPatchesRepliesStatus
v6 posting 2/247 patches26 repliesSuperseded
v5 posting 2/245 patches135 repliesSuperseded
v6 posting 1/22 patches2 repliesApplied
v4 posting 2/247 patches156 repliesSuperseded
v5 posting 1/22 patches6 repliesSuperseded
v4 posting 1/22 patches3 repliesSuperseded
v32 patches1 replySuperseded
v22 patches2 repliesSuperseded
v12 patches3 repliesSuperseded

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/56Add postcopy documentationdocs/migration.txt+191 -00 replies
2/56Provide runtime Target page informationexec.c include/sysemu/sysemu.h+11 -00 replies
3/56Move configuration section writingmigration/savevm.c+6 -50 replies
4/56Move page_size_init earlierinclude/exec/exec-all.h include/qemu-common.h kvm-all.c +2 more+2 -31 reply
5/56qemu_ram_block_from_hostexec.c include/exec/cpu-common.h include/exec/ram_addr.h +1 more+49 -110 replies
6/56qemu_ram_block_by_nameexec.c include/exec/cpu-common.h migration/ram.c+36 -201 reply
7/56Rename mis->file to from_src_fileinclude/migration/migration.h migration/migration.c+2 -20 replies
8/56Add qemu_get_buffer_in_place to avoid copies some of the timeinclude/migration/qemu-file.h migration/qemu-file.c+39 -01 reply
9/56Add wrapper for setting blocking status on a QEMUFileinclude/migration/qemu-file.h migration/qemu-file.c+16 -00 replies
10/56Add QEMU_MADV_NOHUGEPAGEinclude/qemu/osdep.h+9 -00 replies
11/56ram_debug_dump_bitmap: Dump a migration bitmap as textinclude/migration/migration.h migration/ram.c+40 -00 replies
12/56ram_load: Factor out host_from_stream_offset call and checkmigration/ram.c+15 -261 reply
13/56migrate_init: Call from savevminclude/migration/migration.h include/qemu/typedefs.h migration/migration.c +1 more+5 -40 replies
14/56Rename save_live_complete to save_live_complete_precopyhw/ppc/spapr.c include/migration/vmstate.h include/sysemu/sysemu.h +5 more+12 -120 replies
15/56Add Linux userfaultfd.h headerlinux-headers/linux/userfaultfd.h+167 -01 reply
16/56Return path: Open a return path on QEMUFile for socketsinclude/migration/qemu-file.h migration/qemu-file-unix.c migration/qemu-file.c+79 -90 replies
17/56Return path: socket_writev_buffer: Block even on non-blocking fd'smigration/qemu-file-unix.c+37 -50 replies
18/56Migration commandsinclude/migration/migration.h include/sysemu/sysemu.h migration/savevm.c +1 more+81 -00 replies
19/56Return path: Control commandsinclude/migration/migration.h include/sysemu/sysemu.h migration/savevm.c +1 more+51 -10 replies
20/56Return path: Send responses from destination to sourceinclude/migration/migration.h migration/migration.c migration/savevm.c +1 more+66 -11 reply
21/56migration_is_setup_or_activemigration/migration.c+20 -61 reply
22/56Return path: Source handling of return pathinclude/migration/migration.h migration/migration.c trace-events+175 -20 replies
23/56Rework loadvm path for subloopsinclude/migration/migration.h migration/migration.c migration/savevm.c +1 more+86 -670 replies
24/56Add migration-capability boolean for postcopy-ram.include/migration/migration.h migration/migration.c qapi-schema.json+29 -10 replies
25/56Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.include/migration/migration.h include/sysemu/sysemu.h migration/migration.c +2 more+330 -00 replies
26/56MIG_CMD_PACKAGED: Send a packaged chunk of migration streaminclude/sysemu/sysemu.h migration/savevm.c trace-events+107 -50 replies
27/56Modify save_live_pending for postcopyinclude/migration/vmstate.h include/sysemu/sysemu.h migration/block.c +4 more+38 -140 replies
28/56postcopy: OS support testinclude/migration/postcopy-ram.h migration/Makefile.objs migration/postcopy-ram.c +1 more+182 -10 replies
29/56migrate_start_postcopy: Command to trigger transition to postcopyhmp-commands.hx hmp.c hmp.h +4 more+75 -00 replies
30/56migration_completion: Take current statemigration/migration.c+7 -41 reply
31/56MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration stateinclude/migration/migration.h migration/migration.c qapi-schema.json +1 more+51 -30 replies
32/56Avoid sending vmdescription during postcopymigration/savevm.c+2 -10 replies
33/56Add qemu_savevm_state_complete_postcopyinclude/migration/vmstate.h include/sysemu/sysemu.h migration/ram.c +1 more+50 -20 replies
34/56Postcopy: Maintain unsentmapmigration/ram.c+45 -61 reply
35/56Postcopy: Calculate discardinclude/migration/migration.h include/migration/postcopy-ram.h include/qemu/typedefs.h +5 more+354 -32 replies
36/56postcopy: Incoming initialisationinclude/migration/migration.h include/migration/postcopy-ram.h migration/postcopy-ram.c +3 more+136 -01 reply
37/56postcopy: ram_enable_notify to switch on userfaultinclude/migration/migration.h include/migration/postcopy-ram.h migration/postcopy-ram.c +1 more+86 -00 replies
38/56Postcopy: Postcopy startup in migration threadinclude/migration/migration.h migration/migration.c trace-events+174 -70 replies
39/56Postcopy: End of iterationmigration/migration.c trace-events+42 -150 replies
40/56Page request: Add MIG_RP_MSG_REQ_PAGES reverse commandinclude/migration/migration.h migration/migration.c trace-events+76 -00 replies
41/56Page request: Process incoming page requestinclude/migration/migration.h migration/migration.c migration/ram.c +1 more+138 -10 replies
42/56Page request: Consume pages off the post-copy queuemigration/ram.c trace-events+220 -311 reply
43/56postcopy_ram.c: place_page and helpersinclude/migration/migration.h include/migration/postcopy-ram.h migration/postcopy-ram.c +1 more+121 -00 replies
44/56Postcopy: Use helpers to map pages during migrationmigration/ram.c trace-events+130 -11 reply
45/56postcopy: Check order of received target pagesmigration/ram.c+11 -00 replies
46/56Don't sync dirty bitmaps in postcopymigration/ram.c+5 -20 replies
47/56Don't iterate on precopy-only devices during postcopyinclude/sysemu/sysemu.h migration/migration.c migration/savevm.c+13 -40 replies
48/56Host page!=target page: Cleanup bitmapsmigration/ram.c+172 -01 reply
49/56Round up RAMBlock sizes to host page sizesexec.c+4 -40 replies
50/56Postcopy; Handle userfault requestsinclude/migration/migration.h migration/postcopy-ram.c trace-events+158 -90 replies
51/56Start up a postcopy/listener thread ready for incoming page datainclude/migration/migration.h migration/migration.c migration/savevm.c +1 more+90 -10 replies
52/56postcopy: Wire up loadvm_postcopy_handle_ commandsmigration/savevm.c trace-events+29 -10 replies
53/56Postcopy: Mark nohugepage before discardinclude/migration/postcopy-ram.h migration/postcopy-ram.c migration/savevm.c +1 more+58 -40 replies
54/56End of migration for postcopymigration/migration.c trace-events+29 -30 replies
55/56Disable mlock around incoming postcopyinclude/sysemu/sysemu.h migration/postcopy-ram.c+25 -00 replies
56/56Inhibit ballooning during postcopyballoon.c hw/virtio/virtio-balloon.c include/sysemu/balloon.h +1 more+25 -10 replies

Reviews trailers per patch

All 56 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchAmit ShahJuan QuintelaDavid GibsonEric BlakeDr. David Alan Gilbert
SeriesR-b
1/56R-b R-b
2/56R-b R-b
3/56R-b R-b
4/56R-b
5/56R-b R-b R-b
6/56R-b
7/56R-b
8/56R-b
9/56R-b R-b
10/56R-b R-b
11/56R-b R-b
12/56R-b
13/56Reviewd-by R-b R-b
14/56R-b R-b
15/56R-b
16/56R-b R-b
17/56R-b R-b
18/56R-b R-b
19/56R-b R-b
20/56R-b R-b
21/56R-b
22/56R-b
23/56R-b R-b
24/56R-b R-b
25/56R-b R-b
26/56R-b R-b
27/56R-b R-b
28/56R-b R-b
29/56R-b R-b R-b
30/56R-b
31/56R-b R-b R-b R-b
32/56R-b R-b
33/56R-b R-b
34/56R-b
35/56R-b
36/56R-b R-b R-b
37/56R-b R-b R-b
38/56R-b R-b
39/56R-b R-b
40/56R-b R-b
41/56R-b R-b
42/56R-b
43/56R-b R-b
44/56R-b
45/56R-b
46/56R-b R-b R-b
47/56R-b
48/56R-b
49/56R-b
50/56R-b R-b
51/56R-b
52/56R-b R-b
53/56R-b
54/56R-b R-b
55/56R-b R-b R-b
56/56R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v9 thread

Cover letter [Qemu-devel] [PATCH v9 00/56] Postcopy implementation

From: "Dr. David Alan Gilbert" <[email protected]>

This is the 9th cut of my version of postcopy.

The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.

This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tag

Testing status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)

This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/

v9
Almost all of the changes are changes from review comments, and most are minor.

The following are new patches, mostly split out from earlier patches (The exception being adding userfaultfd.h header back in - which was in v5
but we took out, but is needed again due to 1842bdfd)

04/56 Move page_size_init earlier
08/56 qemu_ram_block_by_name
12/56 Factor out host_from_stream_offset call and check 15/56 Add Linux userfaultfd.h header back 21/56 migration_is_setup_or_active
30/56 migration_completion: Take current state 34/56 Maintain unsentmap

The previous patches 03,10,13,36/54 went in upstream already.

Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for migrate_cancel a second time
Rework for migration_bitmap_rcu after Denis's deadlock fix
Rework ram_load into a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_path added
split migration_is_active function into separate patch and made
migration_is_setup_or_active
move file reads into migrate_handle_advise and migrate_handle_packaged use of MIN in send-packaged-chunk
migration_thread_started -> migration_thread_running
updated qemu_get_buffer_in_place with Juan's version (and size_t'ified it) split the host_from_stream_offset change into a separate patch
'ram_load: Factor out host_from_stream_offset call and check'
split out ram_find_block_by_id into a separate patch and now
called qemu_ram_block_by_name
postcopy_discard_send_range etc now take start/length rather than start/end
(also added another trace)
split ram_save_host_page into ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passes

Dave

Dr. David Alan Gilbert (56):
Add postcopy documentation
Provide runtime Target page information
Move configuration section writing
Move page_size_init earlier
qemu_ram_block_from_host
qemu_ram_block_by_name
Rename mis->file to from_src_file
Add qemu_get_buffer_in_place to avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile Add QEMU_MADV_NOHUGEPAGE
ram_debug_dump_bitmap: Dump a migration bitmap as text
ram_load: Factor out host_from_stream_offset call and check
migrate_init: Call from savevm
Rename save_live_complete to save_live_complete_precopy Add Linux userfaultfd.h header
Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source migration_is_setup_or_active
Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modify save_live_pending for postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
migration_completion: Take current state MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Avoid sending vmdescription during postcopy
Add qemu_savevm_state_complete_postcopy
Postcopy: Maintain unsentmap
Postcopy: Calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: End of iteration
Page request: Add MIG_RP_MSG_REQ_PAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration
postcopy: Check order of received target pages
Don't sync dirty bitmaps in postcopy
Don't iterate on precopy-only devices during postcopy Host page!=target page: Cleanup bitmaps
Round up RAMBlock sizes to host page sizes
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_handle_ commands
Postcopy: Mark nohugepage before discard
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopy

 balloon.c                         |  11 +
 docs/migration.txt                | 191 ++++++++
 exec.c                            |  92 +++-
 hmp-commands.hx                   |  15 +
 hmp.c                             |   7 +
 hmp.h                             |   1 +
 hw/ppc/spapr.c                    |   2 +-
 hw/virtio/virtio-balloon.c        |   4 +-
 include/exec/cpu-common.h         |   4 +
 include/exec/exec-all.h           |   1 -
 include/exec/ram_addr.h           |   2 -
 include/migration/migration.h     | 121 ++++-
 include/migration/postcopy-ram.h  |  99 ++++
 include/migration/qemu-file.h     |  10 +
 include/migration/vmstate.h       |   8 +-
 include/qemu-common.h             |   1 +
 include/qemu/osdep.h              |   9 +
 include/qemu/typedefs.h           |   3 +
 include/sysemu/balloon.h          |   2 +
 include/sysemu/sysemu.h           |  46 +-
 kvm-all.c                         |   1 -
 linux-headers/linux/userfaultfd.h | 167 +++++++
 migration/Makefile.objs           |   2 +-
 migration/block.c                 |   9 +-
 migration/migration.c             | 723 +++++++++++++++++++++++++--
 migration/postcopy-ram.c          | 760 +++++++++++++++++++++++++++++
 migration/qemu-file-unix.c        | 111 ++++-
 migration/qemu-file.c             |  64 +++
 migration/ram.c                   | 997 +++++++++++++++++++++++++++++++++++---
 migration/savevm.c                | 823 +++++++++++++++++++++++++++----
 qapi-schema.json                  |  18 +-
 qmp-commands.hx                   |  19 +
 qtest.c                           |   1 -
 trace-events                      |  84 +++-
 vl.c                              |   1 +
 35 files changed, 4159 insertions(+), 250 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 linux-headers/linux/userfaultfd.h
 create mode 100644 migration/postcopy-ram.c
Signature · 1 line
2.5.0