Skip to content

Ram blocks with resizeable anonymous allocations under POSIX

David Hildenbrand qemu-devel 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest15 patches 11 repliesIn review
  2. v315 patches 15 repliesSuperseded
  3. v2 posting 2/216 patches 29 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 posting 1/216 patches4 repliesSuperseded
v113 patches30 repliesSuperseded

Changes in v4 from the cover letter

- Added RBs
- "util/mmap-alloc: Factor out activating of memory to mmap_activate()" -- use "activate" instead of "populate"
- "util: vfio-helpers: Implement ram_block_resized()"
-- Also store max_size in mappings and assert against i
-- Better comment why atomic resizes are not possible
- "exec: Ram blocks with resizeable anonymous allocations under POSIX" -- Assert that RAM_RESIZEABLE_ALLOC is not set before allocating

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/15util: vfio-helpers: Fix qemu_vfio_close()util/vfio-helpers.c+2 -41 reply
2/15util: vfio-helpers: Remove Error parameter from qemu_vfio_undo_mapping()util/vfio-helpers.c+5 -61 reply
3/15util: vfio-helpers: Factor out removal from qemu_vfio_undo_mapping()util/vfio-helpers.c+27 -161 reply
4/15exec: Factor out setting ram settings (madvise ...) into qemu_ram_apply_settings()exec.c+16 -120 replies
5/15exec: Reuse qemu_ram_apply_settings() in qemu_ram_remap()exec.c+1 -20 replies
6/15exec: Drop "shared" parameter from ram_block_add()exec.c+8 -40 replies
7/15util/mmap-alloc: Factor out calculation of the pagesize for the guard pageutil/mmap-alloc.c+16 -151 reply
8/15util/mmap-alloc: Factor out reserving of a memory region to mmap_reserve()util/mmap-alloc.c+33 -250 replies
9/15util/mmap-alloc: Factor out activating of memory to mmap_activate()util/mmap-alloc.c+48 -420 replies
10/15util/mmap-alloc: Prepare for resizeable mmapsutil/mmap-alloc.c+23 -111 reply
11/15util/mmap-alloc: Implement resizeable mmapsinclude/qemu/mmap-alloc.h util/mmap-alloc.c+44 -201 reply
12/15util: vfio-helpers: Implement ram_block_resized()util/trace-events util/vfio-helpers.c+88 -141 reply
13/15util: oslib: Resizeable anonymous allocations under POSIXinclude/qemu/osdep.h util/oslib-posix.c util/oslib-win32.c +1 more+56 -51 reply
14/15numa: Introduce ram_block_notifiers_support_resize()hw/core/numa.c include/exec/ramlist.h+13 -01 reply
15/15exec: Ram blocks with resizeable anonymous allocations under POSIXexec.c hw/core/numa.c include/exec/cpu-common.h +1 more+77 -52 replies

Reviews trailers per patch

All 15 patches have a Reviewed-by or Acked-by.

PatchPeter XuRichard HendersonMurilo Opsfelder AraujoIgor KotrasinskiPhilippe Mathieu-Daudé
1/15R-b R-b
2/15A-b
3/15R-b A-b
4/15R-b R-b
5/15R-b R-b
6/15R-b R-b R-b
7/15R-b A-b
8/15R-b R-b A-b R-b
9/15R-b R-b A-b
10/15R-b R-b A-b
11/15R-b A-b
12/15R-b A-b
13/15R-b A-b
14/15R-b A-b
15/15R-b A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v4 thread

Cover letter [PATCH v4 00/15] Ram blocks with resizeable anonymous allocations under POSIX

This series is based on [1]:
"[PATCH v3 00/13] migrate/ram: Fix resizing RAM blocks while migrating"

We already allow resizable ram blocks for anonymous memory, however, they are not actually resized. All memory is mmaped() R/W, including the memory exceeding the used_length, up to the max_length.

When resizing, effectively only the boundary is moved. Implement actually resizable anonymous allocations and make use of them in resizable ram blocks when possible. Memory exceeding the used_length will be inaccessible. Especially ram block notifiers require care.

Having actually resizable anonymous allocations (via mmap-hackery) allows to reserve a big region in virtual address space and grow the accessible/usable part on demand. Even if "/proc/sys/vm/overcommit_memory" is set to "never" under Linux, huge reservations will succeed. If there is not enough memory when resizing (to populate parts of the reserved region), trying to resize will fail. Only the actually used size is reserved in the OS.

Especially, memory notifiers already handle resizing by first removing the old region, and then re-adding the resized region. prealloc is currently not possible with resizable ram blocks. mlock() should continue to work as is. Resizing is currently rare and must only happen on the start of an incoming migration, or during resets. No code path (except HAX and SEV ram block notifiers) should access memory outside of the usable range - and if we ever find one, that one has to be fixed (I did not identify any).

E.g., virtio-mem [2] wants to reserve big resizable memory regions and grow the usable part on demand. I think this change is worth sending out individually. I did excessive tests of this with virtio-mem (which makes it very easy to trigger resizes), including precopy and postcopy migration.

Accompanied by a bunch of minor fixes and cleanups.

v3 -> v4:
- Added RBs
- "util/mmap-alloc: Factor out activating of memory to mmap_activate()" -- use "activate" instead of "populate"
- "util: vfio-helpers: Implement ram_block_resized()"
-- Also store max_size in mappings and assert against i
-- Better comment why atomic resizes are not possible
- "exec: Ram blocks with resizeable anonymous allocations under POSIX" -- Assert that RAM_RESIZEABLE_ALLOC is not set before allocating

v2 -> v3:
- Rebased on current master/[1].
- "util: vfio-helpers: Factor out and fix processing of existing ram blocks"
-- moved to [1]
- "util: vfio-helpers: Remove Error parameter from qemu_vfio_undo_mapping()" -- Better parch description
- "util/mmap-alloc: Factor out calculation of pagesize to mmap_pagesize()" -- is now "util/mmap-alloc: Factor out calculation of the pagesize for the guard page"
-- Decided to keep special handling for the guard page for now -- Dropped rb's
- "util/mmap-alloc: Prepare for resizeable mmaps"
-- No asserts sizes against the real page size
- "numa: Teach ram block notifiers about resizable ram blocks" -- Split. One part is in [1], the other is now in "numa: Introduce ram_block_notifiers_support_resize()"
- "exec: Ram blocks with resizeable anonymous allocations under POSIX" -- Call qemu_ram_apply_settings() only populated parts. Call it on freshly populated parts when growing.
- Minor changes

v1 -> v2:
- Add "util: vfio-helpers: Fix qemu_vfio_close()"
- Add "util: vfio-helpers: Remove Error parameter from qemu_vfio_undo_mapping()"
- Add "util: vfio-helpers: Factor out removal from qemu_vfio_undo_mapping()"
- "util/mmap-alloc: ..."
-- Minor changes due to review feedback (e.g., assert alignment, return
bool when resizing)
- "util: vfio-helpers: Implement ram_block_resized()"
-- Reserve max_size in the IOVA address space.
-- On resize, undo old mapping and do new mapping. We can later implement
a new ioctl to resize the mapping directly.
- "numa: Teach ram block notifiers about resizable ram blocks" -- Pass size/max_size to ram block notifiers, which makes things easier an cleaner
- "exec: Ram blocks with resizable anonymous allocations under POSIX" -- Adapt to new ram block notifiers
-- Shrink after notifying. Always trigger ram block notifiers on resizes -- Add a safety net that all ram block notifiers registered at runtime
support resizes.

[1] https://lkml.kernel.org/r/[email protected]
[2] https://lore.kernel.org/kvm/[email protected]/

David Hildenbrand (15):
util: vfio-helpers: Fix qemu_vfio_close()
util: vfio-helpers: Remove Error parameter from
qemu_vfio_undo_mapping()
util: vfio-helpers: Factor out removal from qemu_vfio_undo_mapping()
exec: Factor out setting ram settings (madvise ...) into
qemu_ram_apply_settings()
exec: Reuse qemu_ram_apply_settings() in qemu_ram_remap()
exec: Drop "shared" parameter from ram_block_add() util/mmap-alloc: Factor out calculation of the pagesize for the guard
page
util/mmap-alloc: Factor out reserving of a memory region to
mmap_reserve()
util/mmap-alloc: Factor out activating of memory to mmap_activate() util/mmap-alloc: Prepare for resizeable mmaps util/mmap-alloc: Implement resizeable mmaps
util: vfio-helpers: Implement ram_block_resized()
util: oslib: Resizeable anonymous allocations under POSIX
numa: Introduce ram_block_notifiers_support_resize()
exec: Ram blocks with resizeable anonymous allocations under POSIX

 exec.c                    | 100 ++++++++++++++++++-----
 hw/core/numa.c            |  19 +++++
 include/exec/cpu-common.h |   2 +
 include/exec/memory.h     |   8 ++
 include/exec/ramlist.h    |   1 +
 include/qemu/mmap-alloc.h |  21 +++--
 include/qemu/osdep.h      |   6 +-
 util/mmap-alloc.c         | 168 ++++++++++++++++++++++++--------------
 util/oslib-posix.c        |  37 ++++++++-
 util/oslib-win32.c        |  14 ++++
 util/trace-events         |  11 ++-
 util/vfio-helpers.c       | 145 ++++++++++++++++++++++++--------
 12 files changed, 404 insertions(+), 128 deletions(-)
Signature · 1 line
2.24.1