Skip to content

RFC hostmem: NUMA-aware memory preallocation using ThreadContext

David Hildenbrand qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest7 patches 13 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7util: Cleanup and rename os_mem_prealloc()backends/hostmem.c hw/virtio/virtio-mem.c include/qemu/osdep.h +3 more+36 -230 replies
2/7util: Introduce qemu_thread_set_affinity() and qemu_thread_get_affinity()include/qemu/thread.h meson.build util/qemu-thread-posix.c +1 more+102 -00 replies
3/7util: Introduce ThreadContext user-creatable objectinclude/qemu/thread-context.h qapi/qom.json util/meson.build +2 more+355 -01 reply
4/7util: Add write-only "node-affinity" property for ThreadContextqapi/qom.json util/meson.build util/thread-context.c+91 -20 replies
5/7util: Make qemu_prealloc_mem() optionally consume a ThreadContextbackends/hostmem.c hw/virtio/virtio-mem.c include/qemu/osdep.h +2 more+22 -110 replies
6/7hostmem: Allow for specifying a ThreadContext for preallocationbackends/hostmem.c include/sysemu/hostmem.h qapi/qom.json+15 -30 replies
7/7vl: Allow ThreadContext objects to be created before the sandbox optionsoftmmu/vl.c+29 -12 replies

Reviews trailers per patch

None of the 7 patches has a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchMichal Privoznik
SeriesR-b
1/7
2/7
3/7
4/7
5/7
6/7
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH RFC 0/7] hostmem: NUMA-aware memory preallocation using ThreadContext

This is a follow-up on "util: NUMA aware memory preallocation" [1] by Michal.

Setting the CPU affinity of threads from inside QEMU usually isn't easily possible, because we don't want QEMU -- once started and running guest code -- to be able to mess up the system. QEMU disallows relevant syscalls using seccomp, such that any such invocation will fail.

Especially for memory preallocation in memory backends, the CPU affinity can significantly increase guest startup time, for example, when running large VMs backed by huge/gigantic pages, because of NUMA effects. For NUMA-aware preallocation, we have to set the CPU affinity, however:

(1) Once preallocation threads are created during preallocation, management tools cannot intercept anymore to change the affinity. These threads are created automatically on demand.
(2) QEMU cannot easily set the CPU affinity itself.
(3) The CPU affinity derived from the NUMA bindings of the memory backend might not necessarily be exactly the CPUs we actually want to use (e.g., CPU-less NUMA nodes, CPUs that are pinned/used for other VMs).

There is an easy "workaround". If we have a thread with the right CPU affinity, we can simply create new threads on demand via that prepared context. So, all we have to do is setup and create such a context ahead of time, to then configure preallocation to create new threads via that environment.

So, let's introduce a user-creatable "thread-context" object that essentially consists of a context thread used to create new threads. QEMU can either try setting the CPU affinity itself ("cpu-affinity", "node-affinity" property), or upper layers can extract the thread id ("thread-id" property) to configure it externally.

Make memory-backends consume a thread-context object
(via the "prealloc-context" property) and use it when preallocating to create new threads with the desired CPU affinity. Further, to make it easier to use, allow creation of "thread-context" objects, including setting the CPU affinity directly from QEMU, *before* enabling the sandbox option.

Quick test on a system with 2 NUMA nodes:

Without CPU affinity:

    time qemu-system-x86_64 \
        -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind \
        -nographic -monitor stdio

    real    0m5.383s
    real    0m3.499s
    real    0m5.129s
    real    0m4.232s
    real    0m5.220s
    real    0m4.288s
    real    0m3.582s
    real    0m4.305s
    real    0m5.421s
    real    0m4.502s

    -> It heavily depends on the scheduler CPU selection

With CPU affinity:

    time qemu-system-x86_64 \
        -object thread-context,id=tc1,node-affinity=0 \
        -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind,prealloc-context=tc1 \
        -sandbox enable=on,resourcecontrol=deny \
        -nographic -monitor stdio

    real    0m1.959s
    real    0m1.942s
    real    0m1.943s
    real    0m1.941s
    real    0m1.948s
    real    0m1.964s
    real    0m1.949s
    real    0m1.948s
    real    0m1.941s
    real    0m1.937s

On reasonably large VMs, the speedup can be quite significant.

While this concept is currently only used for short-lived preallocation threads, nothing major speaks against reusing the concept for other threads that are harder to identify/configure -- except that we need additional (idle) context threads that are otherwise left unused.

[1] https://lkml.kernel.org/r/ffdcd118d59b379ede2b64745144165a40f6a813.1652165704.git.mprivozn@redhat.com

Cc Michal Privoznik <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Daniel P. Berrangé" <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc Dr. David Alan Gilbert <[email protected]>
Cc Eric Blake <[email protected]>
Cc Markus Armbruster <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Stefan Weil <[email protected]>

David Hildenbrand (7):
util: Cleanup and rename os_mem_prealloc()
util: Introduce qemu_thread_set_affinity() and
qemu_thread_get_affinity()
util: Introduce ThreadContext user-creatable object
util: Add write-only "node-affinity" property for ThreadContext
util: Make qemu_prealloc_mem() optionally consume a ThreadContext
hostmem: Allow for specifying a ThreadContext for preallocation
vl: Allow ThreadContext objects to be created before the sandbox
option

 backends/hostmem.c            |  13 +-
 hw/virtio/virtio-mem.c        |   2 +-
 include/qemu/osdep.h          |  19 +-
 include/qemu/thread-context.h |  58 ++++++
 include/qemu/thread.h         |   4 +
 include/sysemu/hostmem.h      |   2 +
 meson.build                   |  16 ++
 qapi/qom.json                 |  25 +++
 softmmu/cpus.c                |   2 +-
 softmmu/vl.c                  |  30 ++-
 util/meson.build              |   1 +
 util/oslib-posix.c            |  39 ++--
 util/oslib-win32.c            |   8 +-
 util/qemu-thread-posix.c      |  70 +++++++
 util/qemu-thread-win32.c      |  12 ++
 util/thread-context.c         | 363 ++++++++++++++++++++++++++++++++++
 16 files changed, 637 insertions(+), 27 deletions(-)
 create mode 100644 include/qemu/thread-context.h
 create mode 100644 util/thread-context.c
Signature · 1 line
2.35.3