Skip to content

migration: Postcopy Preemption

Peter Xu qemu-devel 9 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest14 patches 7 repliesIn review
  2. v815 patches 1 replySuperseded
  3. v714 patches 5 repliesSuperseded
Earlier versions · 6
VersionPostedPatchesRepliesStatus
v613 patches12 repliesSuperseded
v521 patches18 repliesSuperseded
v419 patches34 repliesSuperseded
v319 patches1 replySuperseded
v225 patches21 repliesSuperseded
v120 patches33 repliesSuperseded

Changes in v9 from the cover letter

- Rebase upon latest master (plus the test fix above on "tests:
migration-test: Allow test to run without uffd")
- Added missing R-bs in v7

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/14migration: Add postcopy-preempt capabilitymigration/migration.c migration/migration.h qapi/migration.json+25 -10 replies
2/14migration: Postcopy preemption preparation on channel creationmigration/migration.c migration/migration.h migration/postcopy-ram.c +7 more+218 -315 replies
3/14migration: Postcopy preemption enablementmigration/migration.c migration/migration.h migration/ram.c +1 more+253 -90 replies
4/14migration: Postcopy recover with preempt enabledmigration/migration.c migration/migration.h migration/postcopy-ram.c +4 more+119 -80 replies
5/14migration: Create the postcopy preempt channel asynchronouslymigration/migration.c migration/migration.h migration/postcopy-ram.c +1 more+68 -120 replies
6/14migration: Add property x-postcopy-preempt-break-hugemigration/migration.c migration/migration.h migration/ram.c+16 -00 replies
7/14migration: Add helpers to detect TLS capabilitymigration/channel.c migration/migration.c migration/multifd.c +2 more+17 -100 replies
8/14migration: Export tls-[creds|hostname|authz] params to cmdline toomigration/migration.c+3 -30 replies
9/14migration: Enable TLS for preempt channelmigration/postcopy-ram.c migration/trace-events+50 -80 replies
10/14migration: Respect postcopy request order in preemption modemigration/ram.c+52 -130 replies
11/14tests: Move MigrateCommon uppertests/qtest/migration-test.c+72 -720 replies
12/14tests: Add postcopy tls migration testtests/qtest/migration-test.c+51 -101 reply
13/14tests: Add postcopy tls recovery migration testtests/qtest/migration-test.c+29 -80 replies
14/14tests: Add postcopy preempt teststests/qtest/migration-test.c+56 -00 replies

Reviews trailers per patch

All 14 patches have a Reviewed-by or Acked-by.

PatchDr. David Alan GilbertDaniel P. BerrangéJuan QuintelaManish Mishra
1/14R-b R-b
2/14R-b R-b R-b
3/14R-b
4/14R-b
5/14R-b R-b
6/14R-b R-b
7/14R-b
8/14R-b
9/14R-b
10/14R-b R-b
11/14R-b
12/14R-b
13/14R-b
14/14R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v9 thread

Cover letter [PATCH v9 00/14] migration: Postcopy Preemption

Based on:
[PATCH] tests: migration-test: Allow test to run without uffd
https://lore.kernel.org/qemu-devel/[email protected]/

This is v8 of postcopy preempt series. It can also be found here:
https://github.com/xzpeter/qemu/tree/postcopy-preempt

RFC: https://lore.kernel.org/qemu-devel/[email protected]
V1:  https://lore.kernel.org/qemu-devel/[email protected]
V2:  https://lore.kernel.org/qemu-devel/[email protected]
V3:  https://lore.kernel.org/qemu-devel/[email protected]
V4:  https://lore.kernel.org/qemu-devel/[email protected]
V5:  https://lore.kernel.org/qemu-devel/[email protected]
V6:  https://lore.kernel.org/qemu-devel/[email protected]
V7:  https://lore.kernel.org/qemu-devel/[email protected]
V8:  https://lore.kernel.org/qemu-devel/[email protected]

v9:
- Rebase upon latest master (plus the test fix above on "tests:
migration-test: Allow test to run without uffd")
- Added missing R-bs in v7

Abstract
========

This series added a new migration capability called "postcopy-preempt". It can be enabled when postcopy is enabled, and it'll simply (but greatly) speed up postcopy page requests handling process.

Below are some initial postcopy page request latency measurements after the new series applied.

For each page size, I measured page request latency for three cases:

  (a) Vanilla:                the old postcopy
  (b) Preempt no-break-huge:  preempt enabled, x-postcopy-preempt-break-huge=off
  (c) Preempt full:           preempt enabled, x-postcopy-preempt-break-huge=on

(this is the default option when preempt enabled)

Here x-postcopy-preempt-break-huge parameter is just added in v2 so as to conditionally disable the behavior to break sending a precopy huge page for debugging purpose. So when it's off, postcopy will not preempt precopy sending a huge page, but still postcopy will use its own channel.

I tested it separately to give a rough idea on which part of the change helped how much of it. The overall benefit should be the comparison between case (a) and (c).

  |-----------+---------+-----------------------+--------------|
  | Page size | Vanilla | Preempt no-break-huge | Preempt full |
  |-----------+---------+-----------------------+--------------|
  | 4K        |   10.68 |               N/A [*] |         0.57 |
  | 2M        |   10.58 |                  5.49 |         5.02 |
  | 1G        | 2046.65 |               933.185 |      649.445 |
  |-----------+---------+-----------------------+--------------|

[*]: This case is N/A because 4K page does not contain huge page at all

[1] https://github.com/xzpeter/small-stuffs/blob/master/tools/huge_vm/uffd-latency.bpf

TODO List
=========

Avoid precopy write() blocks postcopy
-------------------------------------

I didn't prove this, but I always think the write() syscalls being blocked for precopy pages can affect postcopy services. If we can solve this problem then my wild guess is we can further reduce the average page latency.

Two solutions at least in mind: (1) we could have made the write side of the migration channel NON_BLOCK too, or (2) multi-threads on send side, just like multifd, but we may use lock to protect which page to send too (e.g., the core idea is we should _never_ rely anything on the main thread, multifd has that dependency on queuing pages only on main thread).

That can definitely be done and thought about later.

Multi-channel for preemption threads
------------------------------------

Currently the postcopy preempt feature use only one extra channel and one extra thread on dest (no new thread on src QEMU). It should be mostly good enough for major use cases, but when the postcopy queue is long enough (e.g. hundreds of vCPUs faulted on different pages) logically we could still observe more delays in average. Whether growing threads/channels can solve it is debatable, but sounds worthwhile a try. That's yet another thing we can think about after this patchset lands.

Logically the design provides space for that - the receiving postcopy preempt thread can understand all ram-layer migration protocol, and for multi channel and multi threads we could simply grow that into multile threads handling the same protocol (with multiple PostcopyTmpPage). The source needs more thoughts on synchronizations, though, but it shouldn't affect the whole protocol layer, so should be easy to keep compatible.

Please review, thanks.

Peter Xu (14):
migration: Add postcopy-preempt capability
migration: Postcopy preemption preparation on channel creation
migration: Postcopy preemption enablement
migration: Postcopy recover with preempt enabled
migration: Create the postcopy preempt channel asynchronously
migration: Add property x-postcopy-preempt-break-huge
migration: Add helpers to detect TLS capability
migration: Export tls-[creds|hostname|authz] params to cmdline too
migration: Enable TLS for preempt channel
migration: Respect postcopy request order in preemption mode
tests: Move MigrateCommon upper
tests: Add postcopy tls migration test
tests: Add postcopy tls recovery migration test
tests: Add postcopy preempt tests

 migration/channel.c          |   9 +-
 migration/migration.c        | 134 ++++++++++++--
 migration/migration.h        |  44 ++++-
 migration/multifd.c          |   4 +-
 migration/postcopy-ram.c     | 186 +++++++++++++++++++-
 migration/postcopy-ram.h     |  11 ++
 migration/qemu-file.c        |  27 +++
 migration/qemu-file.h        |   1 +
 migration/ram.c              | 326 +++++++++++++++++++++++++++++++++--
 migration/ram.h              |   4 +-
 migration/savevm.c           |  46 +++--
 migration/socket.c           |  22 ++-
 migration/socket.h           |   1 +
 migration/tls.c              |   9 +
 migration/tls.h              |   4 +
 migration/trace-events       |  15 +-
 qapi/migration.json          |   7 +-
 tests/qtest/migration-test.c | 286 +++++++++++++++++++++---------
 18 files changed, 990 insertions(+), 146 deletions(-)
Signature · 1 line
2.32.0