hostmem: NUMA-aware memory preallocation using ThreadContext
Subsystems (2)
v2 cover hostmem: NUMA-aware memory preallocation using ThreadContext
This is a follow-up on "util: NUMA aware memory preallocation" [1] by Michal.
Setting the CPU affinity of threads from inside QEMU usually isn't easily possible, because we don't want QEMU -- once started and running guest code -- to be able to mess up the system. QEMU disallows relevant syscalls using seccomp, such that any such invocation will fail.
Especially for memory preallocation in memory backends, the CPU affinity can significantly increase guest startup time, for example, when running large VMs backed by huge/gigantic pages, because of NUMA effects. For NUMA-aware preallocation, we have to set the CPU affinity, however:
(1) Once preallocation threads are created during preallocation, management tools cannot intercept anymore to change the affinity. These threads are created automatically on demand.
(2) QEMU cannot easily set the CPU affinity itself.
(3) The CPU affinity derived from the NUMA bindings of the memory backend might not necessarily be exactly the CPUs we actually want to use (e.g., CPU-less NUMA nodes, CPUs that are pinned/used for other VMs).
There is an easy "workaround". If we have a thread with the right CPU affinity, we can simply create new threads on demand via that prepared context. So, all we have to do is setup and create such a context ahead of time, to then configure preallocation to create new threads via that environment.
So, let's introduce a user-creatable "thread-context" object that essentially consists of a context thread used to create new threads. QEMU can either try setting the CPU affinity itself ("cpu-affinity", "node-affinity" property), or upper layers can extract the thread id ("thread-id" property) to configure it externally.
Make memory-backends consume a thread-context object
(via the "prealloc-context" property) and use it when preallocating to create new threads with the desired CPU affinity. Further, to make it easier to use, allow creation of "thread-context" objects, including setting the CPU affinity directly from QEMU, before enabling the sandbox option.
Quick test on a system with 2 NUMA nodes:
Without CPU affinity:
time qemu-system-x86_64 \
-object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind \
-nographic -monitor stdio
real 0m5.383s
real 0m3.499s
real 0m5.129s
real 0m4.232s
real 0m5.220s
real 0m4.288s
real 0m3.582s
real 0m4.305s
real 0m5.421s
real 0m4.502s
-> It heavily depends on the scheduler CPU selectionWith CPU affinity:
time qemu-system-x86_64 \
-object thread-context,id=tc1,node-affinity=0 \
-object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind,prealloc-context=tc1 \
-sandbox enable=on,resourcecontrol=deny \
-nographic -monitor stdio
real 0m1.959s
real 0m1.942s
real 0m1.943s
real 0m1.941s
real 0m1.948s
real 0m1.964s
real 0m1.949s
real 0m1.948s
real 0m1.941s
real 0m1.937sOn reasonably large VMs, the speedup can be quite significant.
While this concept is currently only used for short-lived preallocation threads, nothing major speaks against reusing the concept for other threads that are harder to identify/configure -- except that we need additional (idle) context threads that are otherwise left unused.
This series does not yet tackle concurrent preallocation of memory backends. Memory backend objects are created and memory is preallocated one memory backend at a time -- and there is currently no way to do preallocation asynchronously.
v1 -> v2:
* Fixed some minor style nits
* "util: Introduce ThreadContext user-creatable object" -> Impove documentation and patch description. [Markus]
* "util: Add write-only "node-affinity" property for ThreadContext" -> Impove documentation and patch description. [Markus]
RFC -> v1:
* "vl: Allow ThreadContext objects to be created before the sandbox option" -> Move parsing of the "name" property before object_create_pre_sandbox
* Added RB's
David Hildenbrand (7):
util: Cleanup and rename os_mem_prealloc()
util: Introduce qemu_thread_set_affinity() and
qemu_thread_get_affinity()
util: Introduce ThreadContext user-creatable object
util: Add write-only "node-affinity" property for ThreadContext
util: Make qemu_prealloc_mem() optionally consume a ThreadContext
hostmem: Allow for specifying a ThreadContext for preallocation
vl: Allow ThreadContext objects to be created before the sandbox
option
backends/hostmem.c | 13 +- hw/virtio/virtio-mem.c | 2 +- include/qemu/osdep.h | 19 +- include/qemu/thread-context.h | 57 ++++++ include/qemu/thread.h | 4 + include/sysemu/hostmem.h | 2 + meson.build | 16 ++ qapi/qom.json | 28 +++ softmmu/cpus.c | 2 +- softmmu/vl.c | 36 +++- util/meson.build | 1 + util/oslib-posix.c | 39 ++-- util/oslib-win32.c | 8 +- util/qemu-thread-posix.c | 70 +++++++ util/qemu-thread-win32.c | 12 ++ util/thread-context.c | 362 ++++++++++++++++++++++++++++++++++ 16 files changed, 641 insertions(+), 30 deletions(-) create mode 100644 include/qemu/thread-context.h create mode 100644 util/thread-context.c
Signature · 1 line
2.37.3
PATCH v2 1/7 util: Cleanup and rename os_mem_prealloc()
Let's
* give the function a "qemu_*" style name
* make sure the parameters in the implementation match the prototype
* rename smp_cpus to max_threads, which makes the semantics of that parameter clearer
... and add a function documentation.
--- backends/hostmem.c | 6 +++--- hw/virtio/virtio-mem.c | 2 +- include/qemu/osdep.h | 17 +++++++++++++++-- softmmu/cpus.c | 2 +- util/oslib-posix.c | 24 ++++++++++++------------ util/oslib-win32.c | 8 ++++---- 6 files changed, 36 insertions(+), 23 deletions(-)
backends/hostmem.c +3 -3
@@ -232,7 +232,7 @@ static void host_memory_backend_set_prealloc(Object *obj, bool value,void *ptr =memory_region_get_ram_ptr (&backend->mr);uint64_t sz =memory_region_size (&backend->mr); -os_mem_prealloc (fd, ptr, sz, backend->prealloc_threads, &local_err); +qemu_prealloc_mem (fd, ptr, sz, backend->prealloc_threads, &local_err);if (local_err) {error_propagate (errp, local_err);return ;
@@ -383,8 +383,8 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)* specified NUMA policy in place. */ if (backend->prealloc) { -os_mem_prealloc (memory_region_get_fd (&backend->mr), ptr, sz, - backend->prealloc_threads, &local_err); +qemu_prealloc_mem (memory_region_get_fd (&backend->mr), ptr, sz, + backend->prealloc_threads, &local_err);if (local_err) { goto out; }
hw/virtio/virtio-mem.c +1 -1
@@ -467,7 +467,7 @@ static int virtio_mem_set_block_state(VirtIOMEM *vmem, uint64_t start_gpa,int fd =memory_region_get_fd (&vmem->memdev->mr);Error *local_err =NULL ; -os_mem_prealloc (fd, area, size,1 , &local_err); +qemu_prealloc_mem (fd, area, size,1 , &local_err);if (local_err) {static bool warned;
include/qemu/osdep.h +15 -2
@@ -568,8 +568,21 @@ unsigned long qemu_getauxval(unsigned long type);void qemu_set_tty_echo (int fd,bool echo); -void os_mem_prealloc (int fd,char *area,size_t sz,int smp_cpus, -Error **errp); +/** +* qemu_prealloc_mem: +* @fd: the fd mapped into the area, -1 for anonymous memory +* @area: start address of the are to preallocate +* @sz: the size of the area to preallocate +* @max_threads: maximum number of threads to use +* @errp: returns an error if this function fails +* +* Preallocate memory (populate/prefault page tables writable) for the virtual +* memory area starting at @area with the size of @sz. After a successful call, +* each page in the area was faulted in writable at least once, for example, +* after allocating file blocks for mapped files. +*/ +void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, +Error **errp);/** * qemu_get_pid_name:
softmmu/cpus.c +1 -1
@@ -354,7 +354,7 @@ static void qemu_init_sigbus(void)/* * ALERT: when modifying this, take care that SIGBUS forwarding in -* os_mem_prealloc() will continue working as expected. +* qemu_prealloc_mem() will continue working as expected. */ memset (&action,0 ,sizeof (action)); action.sa_flags =SA_SIGINFO ;
util/oslib-posix.c +12 -12
@@ -329,7 +329,7 @@ static void sigbus_handler(int signal) return; }#endif /* CONFIG_LINUX */ -warn_report ("os_mem_prealloc: unrelated SIGBUS detected and ignored" ); +warn_report ("qemu_prealloc_mem: unrelated SIGBUS detected and ignored" ); }static void *do_touch_pages (void *arg)
@@ -399,13 +399,13 @@ static void *do_madv_populate_write_pages(void *arg) }static inline int get_memset_num_threads (size_t hpagesize,size_t numpages, -int smp_cpus) +int max_threads) {long host_procs =sysconf (_SC_NPROCESSORS_ONLN);int ret =1 ;if (host_procs >0 ) { - ret =MIN (MIN (host_procs,MAX_MEM_PREALLOC_THREAD_COUNT ), smp_cpus); + ret =MIN (MIN (host_procs,MAX_MEM_PREALLOC_THREAD_COUNT ), max_threads); }/* Especially with gigantic pages, don't create more threads than pages. */
@@ -418,11 +418,11 @@ static inline int get_memset_num_threads(size_t hpagesize, size_t numpages, }static int touch_all_pages (char *area,size_t hpagesize,size_t numpages, -int smp_cpus,bool use_madv_populate_write) +int max_threads,bool use_madv_populate_write) {static gsize initialized =0 ;MemsetContext context = { - .num_threads =get_memset_num_threads (hpagesize, numpages, smp_cpus), + .num_threads =get_memset_num_threads (hpagesize, numpages, max_threads), };size_t numpages_per_thread, leftover;void *(*touch_fn)(void *);
@@ -494,13 +494,13 @@ static bool madv_populate_write_possible(char *area, size_t pagesize) errno !=EINVAL ; } -void os_mem_prealloc (int fd,char *area,size_t memory,int smp_cpus, -Error **errp) +void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, +Error **errp) {static gsize initialized;int ret;size_t hpagesize =qemu_fd_getpagesize (fd); -size_t numpages =DIV_ROUND_UP (memory, hpagesize); +size_t numpages =DIV_ROUND_UP (sz, hpagesize);bool use_madv_populate_write;struct sigaction act;
@@ -530,24 +530,24 @@ void os_mem_prealloc(int fd, char *area, size_t memory, int smp_cpus,if (ret) {qemu_mutex_unlock (&sigbus_mutex);error_setg_errno (errp, errno, -"os_mem_prealloc: failed to install signal handler" ); +"qemu_prealloc_mem: failed to install signal handler" );return ; } }/* touch pages simultaneously */ - ret =touch_all_pages (area, hpagesize, numpages, smp_cpus, + ret =touch_all_pages (area, hpagesize, numpages, max_threads, use_madv_populate_write);if (ret) {error_setg_errno (errp, -ret, -"os_mem_prealloc: preallocating memory failed" ); +"qemu_prealloc_mem: preallocating memory failed" ); }if (!use_madv_populate_write) { ret =sigaction (SIGBUS , &sigbus_oldact,NULL );if (ret) {/* Terminate QEMU since it can't recover from error */ -perror ("os_mem_prealloc: failed to reinstall signal handler" ); +perror ("qemu_prealloc_mem: failed to reinstall signal handler" );exit (1 ); }qemu_mutex_unlock (&sigbus_mutex);
util/oslib-win32.c +4 -4
@@ -268,14 +268,14 @@ int getpagesize(void)return system_info.dwPageSize; } -void os_mem_prealloc (int fd,char *area,size_t memory,int smp_cpus, -Error **errp) +void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, +Error **errp) {int i;size_t pagesize =qemu_real_host_page_size (); - memory = (memory + pagesize -1 ) & -pagesize; -for (i =0 ; i < memory / pagesize; i++) { + sz = (sz + pagesize -1 ) & -pagesize; +for (i =0 ; i < sz / pagesize; i++) {memset (area + pagesize * i,0 ,1 ); } }
Signature · 1 line
2.37.3
PATCH v2 2/7 util: Introduce qemu_thread_set_affinity() and qemu_thread_get_affinity()
In reply to David HildenbrandUsually, we let upper layers handle CPU pinning, because pthread_setaffinity_np() (-> sched_setaffinity()) is blocked via seccomp when starting QEMU with
-sandbox enable=on,resourcecontrol=deny
However, we want to configure and observe the CPU affinity of threads from QEMU directly in some cases when the sandbox option is either not enabled or not active yet.
So let's add a way to configure CPU pinning via qemu_thread_set_affinity() and obtain CPU affinity via qemu_thread_get_affinity() and implement them under POSIX using pthread_setaffinity_np() + pthread_getaffinity_np().
Implementation under Windows is possible using SetProcessAffinityMask()
+ GetProcessAffinityMask(), however, that is left as future work.
--- include/qemu/thread.h | 4 +++ meson.build | 16 +++++++++ util/qemu-thread-posix.c | 70 ++++++++++++++++++++++++++++++++++++++++ util/qemu-thread-win32.c | 12 +++++++ 4 files changed, 102 insertions(+)
include/qemu/thread.h +4 -0
@@ -185,6 +185,10 @@ void qemu_event_destroy(QemuEvent *ev);void qemu_thread_create (QemuThread *thread,const char *name,void *(*start_routine)(void *),void *arg,int mode); +int qemu_thread_set_affinity (QemuThread *thread,unsigned long *host_cpus, +unsigned long nbits); +int qemu_thread_get_affinity (QemuThread *thread,unsigned long **host_cpus, +unsigned long *nbits);void *qemu_thread_join (QemuThread *thread);void qemu_thread_get_self (QemuThread *thread);bool qemu_thread_is_self (QemuThread *thread);
meson.build +16 -0
@@ -2114,7 +2114,23 @@ config_host_data.set('CONFIG_PTHREAD_CONDATTR_SETCLOCK', cc.links(gnu_source_pre pthread_condattr_setclock(&attr, CLOCK_MONOTONIC); return 0; }''', dependencies: threads)) +config_host_data.set('CONFIG_PTHREAD_AFFINITY_NP', cc.links(gnu_source_prefix + ''' + #include <pthread.h> + static void *f(void *p) { return NULL; } + int main(void) + { + int setsize = CPU_ALLOC_SIZE(64); + pthread_t thread; + cpu_set_t *cpuset; + pthread_create(&thread, 0, f, 0); + cpuset = CPU_ALLOC(64); + CPU_ZERO_S(setsize, cpuset); + pthread_setaffinity_np(thread, setsize, cpuset); + pthread_getaffinity_np(thread, setsize, cpuset); + CPU_FREE(cpuset); + return 0; + }''', dependencies: threads)) config_host_data.set('CONFIG_SIGNALFD', cc.links(gnu_source_prefix + ''' #include <sys/signalfd.h> #include <stddef.h>
util/qemu-thread-posix.c +70 -0
82 lines not on this page · raw message
util/qemu-thread-win32.c +12 -0
@@ -427,6 +427,18 @@ void qemu_thread_create(QemuThread *thread, const char *name, thread->data = data; } +int qemu_thread_set_affinity (QemuThread *thread,unsigned long *host_cpus, +unsigned long nbits) +{ +return -ENOSYS ; +} + +int qemu_thread_get_affinity (QemuThread *thread,unsigned long **host_cpus, +unsigned long *nbits) +{ +return -ENOSYS ; +} +void qemu_thread_get_self (QemuThread *thread) { thread->data = qemu_thread_data;
Signature · 1 line
2.37.3
PATCH v2 3/7 util: Introduce ThreadContext user-creatable object
In reply to David HildenbrandSetting the CPU affinity of QEMU threads is a bit problematic, because QEMU doesn't always have permissions to set the CPU affinity itself, for example, with seccomp after initialized by QEMU:
-sandbox enable=on,resourcecontrol=deny
General information about CPU affinities can be found in the man page of taskset:
CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs.
While upper layers are already aware of how to handle CPU affinities for long-lived threads like iothreads or vcpu threads, especially short-lived threads, as used for memory-backend preallocation, are more involved to handle. These threads are created on demand and upper layers are not even able to identify and configure them.
Introduce the concept of a ThreadContext, that is essentially a thread used for creating new threads. All threads created via that context thread inherit the configured CPU affinity. Consequently, it's sufficient to create a ThreadContext and configure it once, and have all threads created via that ThreadContext inherit the same CPU affinity.
The CPU affinity of a ThreadContext can be configured two ways:
(1) Obtaining the thread id via the "thread-id" property and setting the CPU affinity manually.
(2) Setting the "cpu-affinity" property and letting QEMU try set the CPU affinity itself. This will fail if QEMU doesn't have permissions to do so anymore after seccomp was initialized.
A simple QEMU example to set the CPU affinity to CPU 0,1,6,7 would be:
qemu-system-x86_64 -S \
-object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7And we can query it via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity
[
0,
1,
6,
7
]But note that due to dynamic library loading this example will not work before we actually make use of thread_context_create_thread() in QEMU code, because the type will otherwise not get registered.
A ThreadContext can be reused, simply by reconfiguring the CPU affinity.
--- include/qemu/thread-context.h | 57 +++++++ qapi/qom.json | 17 +++ util/meson.build | 1 + util/oslib-posix.c | 1 + util/thread-context.c | 278 ++++++++++++++++++++++++++++++++++ 5 files changed, 354 insertions(+) create mode 100644 include/qemu/thread-context.h create mode 100644 util/thread-context.c
include/qemu/thread-context.h new file +57 -0
@@ -0,0 +1,57 @@ +/* +* QEMU Thread Context +* +* Copyright Red Hat Inc., 2022 +* +* Authors: +* David Hildenbrand <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifndef SYSEMU_THREAD_CONTEXT_H +#define SYSEMU_THREAD_CONTEXT_H + +#include "qapi/qapi-types-machine.h" +#include "qemu/thread.h" +#include "qom/object.h" + +#define TYPE_THREAD_CONTEXT "thread-context" +OBJECT_DECLARE_TYPE (ThreadContext, ThreadContextClass, + THREAD_CONTEXT) + +struct ThreadContextClass { +ObjectClass parent_class; +}; + +struct ThreadContext { +/* private */ +Object parent; + +/* private */ +unsigned int thread_id; +QemuThread thread; + +/* Semaphore to wait for context thread action. */ +QemuSemaphore sem; +/* Semaphore to wait for action in context thread. */ +QemuSemaphore sem_thread; +/* Mutex to synchronize requests. */ +QemuMutex mutex; + +/* Commands for the thread to execute. */ +int thread_cmd; +void *thread_cmd_data; + +/* CPU affinity bitmap used for initialization. */ +unsigned long *init_cpu_bitmap; +int init_cpu_nbits; +}; + +void thread_context_create_thread (ThreadContext *tc,QemuThread *thread, +const char *name, +void *(*start_routine)(void *),void *arg, +int mode); + +#endif /* SYSEMU_THREAD_CONTEXT_H */
qapi/qom.json +17 -0
@@ -830,6 +830,21 @@ 'reduced-phys-bits': 'uint32', '*kernel-hashes': 'bool' } } +## +# @ThreadContextProperties: +# +# Properties for thread context objects. +# +# @cpu-affinity: the list of CPU numbers used as CPU affinity for all threads +# created in the thread context (default: QEMU main thread +# affinity) +# +# Since: 7.2 +## +{ 'struct': 'ThreadContextProperties', + 'data': { '*cpu-affinity': ['uint16'] } } + + ## # @ObjectType: #
@@ -882,6 +897,7 @@ { 'name': 'secret_keyring', 'if': 'CONFIG_SECRET_KEYRING' }, 'sev-guest', + 'thread-context', 's390-pv-guest', 'throttle-group', 'tls-creds-anon',
@@ -948,6 +964,7 @@ 'secret_keyring': { 'type': 'SecretKeyringProperties', 'if': 'CONFIG_SECRET_KEYRING' }, 'sev-guest': 'SevGuestProperties', + 'thread-context': 'ThreadContextProperties', 'throttle-group': 'ThrottleGroupProperties', 'tls-creds-anon': 'TlsCredsAnonProperties', 'tls-creds-psk': 'TlsCredsPskProperties',
util/meson.build +1 -0
@@ -1,4 +1,5 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) +util_ss.add(files('thread-context.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif
util/oslib-posix.c +1 -0
@@ -42,6 +42,7 @@#include "qemu/cutils.h" #include "qemu/compiler.h" #include "qemu/units.h" +#include "qemu/thread-context.h" #ifdef CONFIG_LINUX #include <sys/syscall.h>
util/thread-context.c new file +278 -0
278 lines not on this page · raw message
Signature · 1 line
2.37.3
Re: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Quoted 45 lines · David Hildenbrand
Setting the CPU affinity of QEMU threads is a bit problematic, because QEMU doesn't always have permissions to set the CPU affinity itself, for example, with seccomp after initialized by QEMU:
-sandbox enable=on,resourcecontrol=denyGeneral information about CPU affinities can be found in the man page of taskset:
CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs.While upper layers are already aware of how to handle CPU affinities for long-lived threads like iothreads or vcpu threads, especially short-lived threads, as used for memory-backend preallocation, are more involved to handle. These threads are created on demand and upper layers are not even able to identify and configure them.
Introduce the concept of a ThreadContext, that is essentially a thread used for creating new threads. All threads created via that context thread inherit the configured CPU affinity. Consequently, it's sufficient to create a ThreadContext and configure it once, and have all threads created via that ThreadContext inherit the same CPU affinity.
The CPU affinity of a ThreadContext can be configured two ways:
(1) Obtaining the thread id via the "thread-id" property and setting the CPU affinity manually.
(2) Setting the "cpu-affinity" property and letting QEMU try set the CPU affinity itself. This will fail if QEMU doesn't have permissions to do so anymore after seccomp was initialized.
A simple QEMU example to set the CPU affinity to CPU 0,1,6,7 would be:
qemu-system-x86_64 -S \ -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7And we can query it via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity [ 0, 1, 6, 7 ]
Quoted 3 lines
But note that due to dynamic library loading this example will not work before we actually make use of
thread_context_create_thread()in QEMU code, because the type will otherwise not get registered.
What do you mean exactly by "not work"? It's not "CLI option or HMP command fails":
$ upstream-qemu -S -display none -nodefaults -monitor stdio -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7
QEMU 7.1.50 monitor - type 'help' for more information
(qemu) qom-get tc1 cpu-affinity
[
0,
1,
6,
7
]
(qemu) info cpus
* CPU #0: thread_id=1670613Even though the affinities refer to nonexistent CPUs :)
Quoted 1 line
A ThreadContext can be reused, simply by reconfiguring the CPU affinity.
So, when a thread is created, its affinity comes from its thread context (if any). When I later change the context's affinity, it does *not* affect existing threads, only future ones. Correct?
Quoted 81 lines
Reviewed-by Michal Privoznik <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- include/qemu/thread-context.h | 57 +++++++ qapi/qom.json | 17 +++ util/meson.build | 1 + util/oslib-posix.c | 1 + util/thread-context.c | 278 ++++++++++++++++++++++++++++++++++ 5 files changed, 354 insertions(+) create mode 100644 include/qemu/thread-context.h create mode 100644 util/thread-context.cinclude/qemu/thread-context.h new file +57
-0@@ -0,0 +1,57 @@ +/* +* QEMU Thread Context +* +* Copyright Red Hat Inc., 2022 +* +* Authors: +* David Hildenbrand <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifndef SYSEMU_THREAD_CONTEXT_H +#define SYSEMU_THREAD_CONTEXT_H + +#include "qapi/qapi-types-machine.h" +#include "qemu/thread.h" +#include "qom/object.h" + +#define TYPE_THREAD_CONTEXT "thread-context" +OBJECT_DECLARE_TYPE (ThreadContext, ThreadContextClass, + THREAD_CONTEXT) + +struct ThreadContextClass { +ObjectClass parent_class; +}; + +struct ThreadContext { +/* private */ +Object parent; + +/* private */ +unsigned int thread_id; +QemuThread thread; + +/* Semaphore to wait for context thread action. */ +QemuSemaphore sem; +/* Semaphore to wait for action in context thread. */ +QemuSemaphore sem_thread; +/* Mutex to synchronize requests. */ +QemuMutex mutex; + +/* Commands for the thread to execute. */ +int thread_cmd; +void *thread_cmd_data; + +/* CPU affinity bitmap used for initialization. */ +unsigned long *init_cpu_bitmap; +int init_cpu_nbits; +}; + +void thread_context_create_thread (ThreadContext *tc,QemuThread *thread, +const char *name, +void *(*start_routine)(void *),void *arg, +int mode); + +#endif /* SYSEMU_THREAD_CONTEXT_H */ qapi/qom.json +0
-0@@ -830,6 +830,21 @@ 'reduced-phys-bits': 'uint32',
qapi/qom.json line 840
Quoted 10 lines
qapi/qom.json +8
-0'*kernel-hashes': 'bool' } } +## +# @ThreadContextProperties: +# +# Properties for thread context objects. +# +# @cpu-affinity: the list of CPU numbers used as CPU affinity for all threads +# created in the thread context (default: QEMU main thread +# affinity)
Another ignorant question: is the QEMU main thread affinity fixed or configurable? If configurable, how?
Quoted 26 lines
qapi/qom.json +9
-0+# +# Since: 7.2 +## +{ 'struct': 'ThreadContextProperties', + 'data': { '*cpu-affinity': ['uint16'] } } + + ## # @ObjectType: #@@ -882,6 +897,7 @@ { 'name': 'secret_keyring', 'if': 'CONFIG_SECRET_KEYRING' }, 'sev-guest', + 'thread-context', 's390-pv-guest', 'throttle-group', 'tls-creds-anon',@@ -948,6 +964,7 @@ 'secret_keyring': { 'type': 'SecretKeyringProperties', 'if': 'CONFIG_SECRET_KEYRING' }, 'sev-guest': 'SevGuestProperties', + 'thread-context': 'ThreadContextProperties', 'throttle-group': 'ThrottleGroupProperties', 'tls-creds-anon': 'TlsCredsAnonProperties', 'tls-creds-psk': 'TlsCredsPskProperties',
[...]
Re: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Quoted 7 lines
Quoted 3 lines
But note that due to dynamic library loading this example will not work before we actually make use of
thread_context_create_thread()in QEMU code, because the type will otherwise not get registered.What do you mean exactly by "not work"? It's not "CLI option or HMP command fails":
For me, if I compile patch #1-#3 only, I get:
$ ./build/qemu-system-x86_64 -S -display none -nodefaults -monitor stdio -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7 qemu-system-x86_64: invalid object type: thread-context
Reason is that, without a call to thread_context_create_thread(), we won't trigger type_init(thread_context_register_types) and consequently, the type won't be registered.
Is it really different in your environment? Maybe it depends on the QEMU config?
Quoted 12 lines
$ upstream-qemu -S -display none -nodefaults -monitor stdio -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7
QEMU 7.1.50 monitor - type 'help' for more information
(qemu) qom-get tc1 cpu-affinity
[
0,
1,
6,
7
]
(qemu) info cpus
* CPU #0: thread_id=1670613Quoted 1 line
Even though the affinities refer to nonexistent CPUs :)
CPU affinities are CPU numbers on your system (host), not QEMU vCPU numbers. I could talk about physical CPU numbers in the doc here, although I am not sure if that really helps. What about "host CPU numbers" and in patch #4 "host node numbers"?
Seems to match what we document for @MemoryBackendProperties: "@host-nodes: the list of NUMA host nodes to bind the memory to"
But unrelated to that, pthread_setaffinity_np() won't bail out on CPUs that are currently not available in the host -- because one might online/hotplug them later. It only bails out if none of the CPUs is currently available in the host:
https://man7.org/linux/man-pages/man3/pthread_setaffinity_np.3.html
EINVAL (pthread_setaffinity_np()) The affinity bit mask mask
contains no processors that are currently physically on the system and permitted to the thread according to any restrictions that may be imposed by the "cpuset" mechanism described in cpuset(7).
It will bail out on CPUs that cannot be available in the host though, because it's impossible due to the kernel config:
EINVAL (pthread_setaffinity_np()) cpuset specified a CPU that was
outside the set supported by the kernel. (The kernel configuration option CONFIG_NR_CPUS defines the range of the set supported by the kernel data type used to represent CPU sets.)
Quoted 6 lines
Quoted 2 lines
A ThreadContext can be reused, simply by reconfiguring the CPU affinity.
So, when a thread is created, its affinity comes from its thread context (if any). When I later change the context's affinity, it does *not* affect existing threads, only future ones. Correct?
Yes, that's the current state.
Quoted 84 lines
Quoted 84 lines
Reviewed-by Michal Privoznik <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- include/qemu/thread-context.h | 57 +++++++ qapi/qom.json | 17 +++ util/meson.build | 1 + util/oslib-posix.c | 1 + util/thread-context.c | 278 ++++++++++++++++++++++++++++++++++ 5 files changed, 354 insertions(+)create mode 100644
include/qemu/thread-context.hcreate mode 100644util/thread-context.cinclude/qemu/thread-context.h new file +57
-0@@ -0,0 +1,57 @@ +/* +* QEMU Thread Context +* +* Copyright Red Hat Inc., 2022 +* +* Authors: +* David Hildenbrand <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifndef SYSEMU_THREAD_CONTEXT_H +#define SYSEMU_THREAD_CONTEXT_H + +#include "qapi/qapi-types-machine.h" +#include "qemu/thread.h" +#include "qom/object.h" + +#define TYPE_THREAD_CONTEXT "thread-context" +OBJECT_DECLARE_TYPE (ThreadContext, ThreadContextClass, + THREAD_CONTEXT) + +struct ThreadContextClass { +ObjectClass parent_class; +}; + +struct ThreadContext { +/* private */ +Object parent; + +/* private */ +unsigned int thread_id; +QemuThread thread; + +/* Semaphore to wait for context thread action. */ +QemuSemaphore sem; +/* Semaphore to wait for action in context thread. */ +QemuSemaphore sem_thread; +/* Mutex to synchronize requests. */ +QemuMutex mutex; + +/* Commands for the thread to execute. */ +int thread_cmd; +void *thread_cmd_data; + +/* CPU affinity bitmap used for initialization. */ +unsigned long *init_cpu_bitmap; +int init_cpu_nbits; +}; + +void thread_context_create_thread (ThreadContext *tc,QemuThread *thread, +const char *name, +void *(*start_routine)(void *),void *arg, +int mode); + +#endif /* SYSEMU_THREAD_CONTEXT_H */ qapi/qom.json +0
-0@@ -830,6 +830,21 @@ 'reduced-phys-bits': 'uint32', '*kernel-hashes': 'bool' } }
Quoted 11 lines
qapi/qom.json line 840
Quoted 8 lines
qapi/qom.json +8
-0+## +# @ThreadContextProperties: +# +# Properties for thread context objects. +# +# @cpu-affinity: the list of CPU numbers used as CPU affinity for all threads +# created in the thread context (default: QEMU main thread +# affinity)Another ignorant question: is the QEMU main thread affinity fixed or configurable? If configurable, how?
AFAIK, it's only configurable externally, for example, via "virsh emulatorpin". There is no QEMU interface to adjust that (because it wouldn't work).
Libvirt will essentially trigger "taskset" on the emulator thread to change its CPU affinity.
Signature · 3 lines
Thanks, David / dhildenb
Markus Armbruster
Acked-byRe: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Quoted 9 lines · David Hildenbrand
Quoted 6 lines
Quoted 3 lines
But note that due to dynamic library loading this example will not work before we actually make use of
thread_context_create_thread()in QEMU code, because the type will otherwise not get registered.What do you mean exactly by "not work"? It's not "CLI option or HMP command fails":
For me, if I compile patch #1-#3 only, I get:
Quoted 8 lines
$ ./build/qemu-system-x86_64 -S -display none -nodefaults -monitor stdio -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7 qemu-system-x86_64: invalid object type: thread-contextReason is that, without a call to
thread_context_create_thread(), we won't triggertype_init(thread_context_register_types)and consequently, the type won't be registered.Is it really different in your environment? Maybe it depends on the QEMU config?
I just tested again, and get the same result as you. I figure my previous test was with the complete series.
PATCH 5 appears to make it work. Suggest to say something like "The commit after next will make this work".
Quoted 12 lines
Quoted 12 lines
$ upstream-qemu -S -display none -nodefaults -monitor stdio -object thread-context,id=tc1,cpu-affinity=0-1,cpu-affinity=6-7 QEMU 7.1.50 monitor - type 'help' for more information (qemu) qom-get tc1 cpu-affinity [ 0, 1, 6, 7 ] (qemu) info cpus * CPU #0: thread_id=1670613
Quoted 4 lines
Quoted 1 line
Even though the affinities refer to nonexistent CPUs :)
CPU affinities are CPU numbers on your system (host), not QEMU vCPU numbers. I could talk about physical CPU numbers in the doc here, although I am not sure if that really helps. What about "host CPU numbers" and in patch #4 "host node numbers"?
I think this would reduce the confusion opportunities for noobs like me.
Quoted 1 line
Seems to match what we document for @MemoryBackendProperties: "@host-nodes: the list of NUMA host nodes to bind the memory to"
Even better.
Quoted 22 lines
But unrelated to that,
pthread_setaffinity_np()won't bail out on CPUs that are currently not available in the host -- because one might online/hotplug them later. It only bails out if none of the CPUs is currently available in the host:https://man7.org/linux/man-pages/man3/pthread_setaffinity_np.3.html
EINVAL (
pthread_setaffinity_np()) The affinity bit mask mask
contains no processors that are currently physically on the system and permitted to the thread according to any restrictions that may be imposed by the "cpuset" mechanism described in cpuset(7).It will bail out on CPUs that cannot be available in the host though, because it's impossible due to the kernel config:
EINVAL (
pthread_setaffinity_np()) cpuset specified a CPU that was
outside the set supported by the kernel. (The kernel configuration optionCONFIG_NR_CPUSdefines the range of the set supported by the kernel data type used to represent CPU sets.)
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
A ThreadContext can be reused, simply by reconfiguring the CPU affinity.
So, when a thread is created, its affinity comes from its thread context (if any). When I later change the context's affinity, it does *not* affect existing threads, only future ones. Correct?
Yes, that's the current state.
Thanks!
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Reviewed-by Michal Privoznik <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
[...]
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
qapi/qom.json +2
-0@@ -830,6 +830,21 @@ 'reduced-phys-bits': 'uint32', '*kernel-hashes': 'bool' } } +## +# @ThreadContextProperties: +#Quoted 8 lines
qapi/qom.json line 840
Quoted 5 lines
qapi/qom.json +5
-0+# Properties for thread context objects. +# +# @cpu-affinity: the list of CPU numbers used as CPU affinity for all threads +# created in the thread context (default: QEMU main thread +# affinity)Another ignorant question: is the QEMU main thread affinity fixed or configurable? If configurable, how?
Quoted 4 lines
AFAIK, it's only configurable externally, for example, via "virsh emulatorpin". There is no QEMU interface to adjust that (because it wouldn't work).
Libvirt will essentially trigger "taskset" on the emulator thread to change its CPU affinity.
I see.
QAPI schema
Re: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Thanks Markus!
Quoted 5 lines
I just tested again, and get the same result as you. I figure my previous test was with the complete series.
PATCH 5 appears to make it work. Suggest to say something like "The commit after next will make this work".
I'll phrase it like " We'll wire this up next to make it work."
[...]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So, when a thread is created, its affinity comes from its thread context (if any). When I later change the context's affinity, it does *not* affect existing threads, only future ones. Correct?
Yes, that's the current state.
Thanks!
I'm adding
"Note that the CPU affinity of previously created threads will not get adjusted."
and
"In general, the interface behaves like pthread_setaffinity_np(): host CPU numbers that are currently not available are ignored; only host CPU numbers that are impossible with the current kernel will fail. If the list of host CPU numbers does not include a single CPU that is available, setting the CPU affinity will fail."
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Quoted 8 lines · David Hildenbrand
Thanks Markus!
Quoted 4 lines
I just tested again, and get the same result as you. I figure my previous test was with the complete series.
PATCH 5 appears to make it work. Suggest to say something like "The commit after next will make this work".I'll phrase it like " We'll wire this up next to make it work."
Works for me!
Quoted 12 lines
[...]
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
So, when a thread is created, its affinity comes from its thread context (if any). When I later change the context's affinity, it does *not* affect existing threads, only future ones. Correct?
Yes, that's the current state.
Thanks!
I'm adding
Quoted 7 lines
"Note that the CPU affinity of previously created threads will not get adjusted."
and
"In general, the interface behaves like
pthread_setaffinity_np(): host CPU numbers that are currently not available are ignored; only host CPU numbers that are impossible with the current kernel will fail. If the list of host CPU numbers does not include a single CPU that is available, setting the CPU affinity will fail."
This is one of the reasons why reviewing your work is such a pleasure:
not only do you answer my questions with clarity and patience, you proactively improve your patches before I can even think to ask.
Thank you!
Re: [PATCH v2 3/7] util: Introduce ThreadContext user-creatable object
Quoted 12 lines
Quoted 11 lines
Quoted 1 line
Thanks!
I'm adding
"Note that the CPU affinity of previously created threads will not get adjusted."
and
"In general, the interface behaves like
pthread_setaffinity_np(): host CPU numbers that are currently not available are ignored; only host CPU numbers that are impossible with the current kernel will fail. If the list of host CPU numbers does not include a single CPU that is available, setting the CPU affinity will fail."
Quoted 5 lines
This is one of the reasons why reviewing your work is such a pleasure:
not only do you answer my questions with clarity and patience, you proactively improve your patches before I can even think to ask.Thank you!
Thanks a lot Markus -- I have to say that your reviews are extremely helpful! You ask just the right questions that make one realize which parts of the documentation need improvement!
Signature · 3 lines
Thanks, David / dhildenb
PATCH v2 4/7 util: Add write-only "node-affinity" property for ThreadContext
In reply to David HildenbrandLet's make it easier to pin threads created via a ThreadContext to all CPUs currently belonging to a given set of NUMA nodes -- which is the common case.
"node-affinity" is simply a shortcut for setting "cpu-affinity" manually to the list of CPUs belonging to the set of nodes. This property can only be written.
A simple QEMU example to set the CPU affinity to Node 1 on a system with two NUMA nodes, 24 CPUs each:
qemu-system-x86_64 -S \
-object thread-context,id=tc1,node-affinity=1And we can query the cpu-affinity via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity
[
1,
3,
5,
7,
9,
11,
13,
15,
17,
19,
21,
23,
25,
27,
29,
31,
33,
35,
37,
39,
41,
43,
45,
47
]We cannot query the node-affinity:
(qemu) qom-get tc1 node-affinity
Error: Insufficient permission to perform this operation
But note that due to dynamic library loading this example will not work before we actually make use of thread_context_create_thread() in QEMU code, because the type will otherwise not get registered.
Note that if the CPUs for a node change due do physical CPU hotplug or hotunplug (e.g., lscpu output changes) after the ThreadContext was started, the CPU affinity will not get updated.
--- qapi/qom.json | 9 ++++- util/meson.build | 2 +- util/thread-context.c | 84 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+), 2 deletions(-)
qapi/qom.json +8 -1
@@ -839,10 +839,17 @@ # created in the thread context (default: QEMU main thread # affinity) # +# @node-affinity: the list of node numbers that will be resolved to a list +# of CPU numbers used as CPU affinity. This is a shortcut for +# specifying the list of CPU numbers belonging to the nodes +# manually by setting @cpu-affinity. (default: QEMU main thread +# affinity) +# # Since: 7.2 ## { 'struct': 'ThreadContextProperties', - 'data': { '*cpu-affinity': ['uint16'] } } + 'data': { '*cpu-affinity': ['uint16'], + '*node-affinity': ['uint16'] } } ##
util/meson.build +1 -1
@@ -1,5 +1,5 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) -util_ss.add(files('thread-context.c')) +util_ss.add(files('thread-context.c'), numa) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif
util/thread-context.c +84 -0
108 lines not on this page · raw message
Signature · 1 line
2.37.3
Re: [PATCH v2 4/7] util: Add write-only "node-affinity" property for ThreadContext
Quoted 33 lines · David Hildenbrand
Let's make it easier to pin threads created via a ThreadContext to all CPUs currently belonging to a given set of NUMA nodes -- which is the common case.
"node-affinity" is simply a shortcut for setting "cpu-affinity" manually to the list of CPUs belonging to the set of nodes. This property can only be written.
A simple QEMU example to set the CPU affinity to Node 1 on a system with two NUMA nodes, 24 CPUs each:
qemu-system-x86_64 -S \ -object thread-context,id=tc1,node-affinity=1And we can query the cpu-affinity via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity [ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33,
Quoted 8 lines
35,
37,
39,
41,
43,
45,
47
]Double-checking my understanding: on this system, the even CPUs belong to NUMA node 0, and the odd ones to node 1. Setting node-affinity=1 is therefore sugar for setting cpu-affinity to the set of even CPUs. Correct?
Quoted 3 lines
We cannot query the node-affinity:
(qemu) qom-get tc1 node-affinity
Error: Insufficient permission to perform this operation
The error message is somewhat misleading. "Insufficient permission" suggests this could work if I had more "permission". Not the case. The message comes from object_property_get(), i.e. it's not this patch's fault. I'll post a patch to improve it.
Quoted 39 lines
But note that due to dynamic library loading this example will not work before we actually make use of
thread_context_create_thread()in QEMU code, because the type will otherwise not get registered.Note that if the CPUs for a node change due do physical CPU hotplug or hotunplug (e.g., lscpu output changes) after the ThreadContext was started, the CPU affinity will not get updated.
Reviewed-by Michal Privoznik <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- qapi/qom.json | 9 ++++- util/meson.build | 2 +- util/thread-context.c | 84 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+), 2 deletions(-)qapi/qom.json +8
-1@@ -839,10 +839,17 @@ # created in the thread context (default: QEMU main thread # affinity) # +# @node-affinity: the list of node numbers that will be resolved to a list +# of CPU numbers used as CPU affinity. This is a shortcut for +# specifying the list of CPU numbers belonging to the nodes +# manually by setting @cpu-affinity. (default: QEMU main thread +# affinity) +# # Since: 7.2 ## { 'struct': 'ThreadContextProperties', - 'data': { '*cpu-affinity': ['uint16'] } } + 'data': { '*cpu-affinity': ['uint16'], + '*node-affinity': ['uint16'] } } ##
[...]
Re: [PATCH v2 4/7] util: Add write-only "node-affinity" property for ThreadContext
Quoted 44 lines · Markus Armbruster
Quoted 33 lines · David Hildenbrand
Let's make it easier to pin threads created via a ThreadContext to all CPUs currently belonging to a given set of NUMA nodes -- which is the common case.
"node-affinity" is simply a shortcut for setting "cpu-affinity" manually to the list of CPUs belonging to the set of nodes. This property can only be written.
A simple QEMU example to set the CPU affinity to Node 1 on a system with two NUMA nodes, 24 CPUs each:
qemu-system-x86_64 -S \ -object thread-context,id=tc1,node-affinity=1And we can query the cpu-affinity via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity [ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33,Quoted 8 lines
35, 37, 39, 41, 43, 45, 47 ]
Quoted 4 lines
Double-checking my understanding: on this system, the even CPUs belong to NUMA node 0, and the odd ones to node 1. Setting node-affinity=1 is therefore sugar for setting cpu-affinity to the set of even CPUs. Correct?
Yes!
# lscpu ... NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
Quoted 10 lines
Quoted 4 lines
We cannot query the node-affinity:
(qemu) qom-get tc1 node-affinity
Error: Insufficient permission to perform this operationThe error message is somewhat misleading. "Insufficient permission" suggests this could work if I had more "permission". Not the case. The message comes from
object_property_get(), i.e. it's not this patch's fault. I'll post a patch to improve it.
I agree, thanks!
Signature · 3 lines
Thanks, David / dhildenb
Markus Armbruster
Acked-byRe: [PATCH v2 4/7] util: Add write-only "node-affinity" property for ThreadContext
Quoted 50 lines · David Hildenbrand
Quoted 44 lines · Markus Armbruster
Quoted 33 lines · David Hildenbrand
Let's make it easier to pin threads created via a ThreadContext to all CPUs currently belonging to a given set of NUMA nodes -- which is the common case.
"node-affinity" is simply a shortcut for setting "cpu-affinity" manually to the list of CPUs belonging to the set of nodes. This property can only be written.
A simple QEMU example to set the CPU affinity to Node 1 on a system with two NUMA nodes, 24 CPUs each:
qemu-system-x86_64 -S \ -object thread-context,id=tc1,node-affinity=1And we can query the cpu-affinity via HMP/QMP:
(qemu) qom-get tc1 cpu-affinity [ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33,Quoted 8 lines
35, 37, 39, 41, 43, 45, 47 ]Quoted 4 lines
Double-checking my understanding: on this system, the even CPUs belong to NUMA node 0, and the odd ones to node 1. Setting node-affinity=1 is therefore sugar for setting cpu-affinity to the set of even CPUs. Correct?
Quoted 8 lines
Yes!
# lscpu ... NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
Thanks!
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
We cannot query the node-affinity:
(qemu) qom-get tc1 node-affinity
Error: Insufficient permission to perform this operationThe error message is somewhat misleading. "Insufficient permission" suggests this could work if I had more "permission". Not the case. The message comes from
object_property_get(), i.e. it's not this patch's fault. I'll post a patch to improve it.I agree, thanks!
QAPI schema
Re: [PATCH v2 4/7] util: Add write-only "node-affinity" property for ThreadContext
Quoted 1 line · Markus Armbruster
Acked-by Markus Armbruster<[email protected]>
Thanks, I'll similarly adjust to use "host nodes" and "host CPUs".
Signature · 3 lines
Thanks, David / dhildenb
PATCH v2 5/7 util: Make qemu_prealloc_mem() optionally consume a ThreadContext
In reply to David Hildenbrand... and implement it under POSIX. When a ThreadContext is provided, create new threads via the context such that these new threads obtain a porperly configured CPU affinity.
--- backends/hostmem.c | 5 +++-- hw/virtio/virtio-mem.c | 2 +- include/qemu/osdep.h | 4 +++- util/oslib-posix.c | 20 ++++++++++++++------ util/oslib-win32.c | 2 +- 5 files changed, 22 insertions(+), 11 deletions(-)
backends/hostmem.c +3 -2
@@ -232,7 +232,8 @@ static void host_memory_backend_set_prealloc(Object *obj, bool value,void *ptr =memory_region_get_ram_ptr (&backend->mr);uint64_t sz =memory_region_size (&backend->mr); -qemu_prealloc_mem (fd, ptr, sz, backend->prealloc_threads, &local_err); +qemu_prealloc_mem (fd, ptr, sz, backend->prealloc_threads,NULL , + &local_err);if (local_err) {error_propagate (errp, local_err);return ;
@@ -384,7 +385,7 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)*/ if (backend->prealloc) {qemu_prealloc_mem (memory_region_get_fd (&backend->mr), ptr, sz, - backend->prealloc_threads, &local_err); + backend->prealloc_threads,NULL , &local_err);if (local_err) { goto out; }
hw/virtio/virtio-mem.c +1 -1
@@ -467,7 +467,7 @@ static int virtio_mem_set_block_state(VirtIOMEM *vmem, uint64_t start_gpa,int fd =memory_region_get_fd (&vmem->memdev->mr);Error *local_err =NULL ; -qemu_prealloc_mem (fd, area, size,1 , &local_err); +qemu_prealloc_mem (fd, area, size,1 ,NULL , &local_err);if (local_err) {static bool warned;
include/qemu/osdep.h +3 -1
@@ -568,6 +568,8 @@ unsigned long qemu_getauxval(unsigned long type);void qemu_set_tty_echo (int fd,bool echo); +typedef struct ThreadContext ThreadContext ; +/** * qemu_prealloc_mem: * @fd: the fd mapped into the area, -1 for anonymous memory
@@ -582,7 +584,7 @@ void qemu_set_tty_echo(int fd, bool echo);* after allocating file blocks for mapped files. */ void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, -Error **errp); +ThreadContext *tc,Error **errp);/** * qemu_get_pid_name:
util/oslib-posix.c +14 -6
@@ -419,7 +419,8 @@ static inline int get_memset_num_threads(size_t hpagesize, size_t numpages, }static int touch_all_pages (char *area,size_t hpagesize,size_t numpages, -int max_threads,bool use_madv_populate_write) +int max_threads,ThreadContext *tc, +bool use_madv_populate_write) {static gsize initialized =0 ;MemsetContext context = {
@@ -458,9 +459,16 @@ static int touch_all_pages(char *area, size_t hpagesize, size_t numpages, context.threads[i].numpages = numpages_per_thread + (i < leftover); context.threads[i].hpagesize = hpagesize; context.threads[i].context = &context; -qemu_thread_create (&context.threads[i].pgthread,"touch_pages" , - touch_fn, &context.threads[i], -QEMU_THREAD_JOINABLE ); +if (tc) { +thread_context_create_thread (tc, &context.threads[i].pgthread, +"touch_pages" , + touch_fn, &context.threads[i], +QEMU_THREAD_JOINABLE ); + }else { +qemu_thread_create (&context.threads[i].pgthread,"touch_pages" , + touch_fn, &context.threads[i], +QEMU_THREAD_JOINABLE ); + } addr += context.threads[i].numpages * hpagesize; }
@@ -496,7 +504,7 @@ static bool madv_populate_write_possible(char *area, size_t pagesize) }void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, -Error **errp) +ThreadContext *tc,Error **errp) {static gsize initialized;int ret;
@@ -537,7 +545,7 @@ void qemu_prealloc_mem(int fd, char *area, size_t sz, int max_threads, }/* touch pages simultaneously */ - ret =touch_all_pages (area, hpagesize, numpages, max_threads, + ret =touch_all_pages (area, hpagesize, numpages, max_threads, tc, use_madv_populate_write);if (ret) { error_setg_errno(errp, -ret,
util/oslib-win32.c +1 -1
@@ -269,7 +269,7 @@ int getpagesize(void) }void qemu_prealloc_mem (int fd,char *area,size_t sz,int max_threads, -Error **errp) +ThreadContext *tc,Error **errp) {int i;size_t pagesize =qemu_real_host_page_size ();
Signature · 1 line
2.37.3
PATCH v2 6/7 hostmem: Allow for specifying a ThreadContext for preallocation
In reply to David HildenbrandLet's allow for specifying a thread context via the "prealloc-context" property. When set, preallcoation threads will be crated via the thread context -- inheriting the same CPU affinity as the thread context.
Pinning preallcoation threads to CPUs can heavily increase performance in NUMA setups, because, preallocation from a CPU close to the target NUMA node(s) is faster then preallocation from a CPU further remote, simply because of memory bandwidth for initializing memory with zeroes. This is especially relevant for very large VMs backed by huge/gigantic pages, whereby preallocation is mandatory.
--- backends/hostmem.c | 12 +++++++++--- include/sysemu/hostmem.h | 2 ++ qapi/qom.json | 4 ++++ 3 files changed, 15 insertions(+), 3 deletions(-)
backends/hostmem.c +9 -3
@@ -232,8 +232,8 @@ static void host_memory_backend_set_prealloc(Object *obj, bool value,void *ptr =memory_region_get_ram_ptr (&backend->mr);uint64_t sz =memory_region_size (&backend->mr); -qemu_prealloc_mem (fd, ptr, sz, backend->prealloc_threads,NULL , - &local_err); +qemu_prealloc_mem (fd, ptr, sz, backend->prealloc_threads, + backend->prealloc_context, &local_err);if (local_err) {error_propagate (errp, local_err);return ;
@@ -385,7 +385,8 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)*/ if (backend->prealloc) {qemu_prealloc_mem (memory_region_get_fd (&backend->mr), ptr, sz, - backend->prealloc_threads,NULL , &local_err); + backend->prealloc_threads, + backend->prealloc_context, &local_err);if (local_err) { goto out; }
@@ -493,6 +494,11 @@ host_memory_backend_class_init(ObjectClass *oc, void *data) NULL, NULL);object_class_property_set_description (oc,"prealloc-threads" ,"Number of CPU threads to use for prealloc" ); +object_class_property_add_link (oc,"prealloc-context" , +TYPE_THREAD_CONTEXT , offsetof(HostMemoryBackend , prealloc_context), + object_property_allow_set_link,OBJ_PROP_LINK_STRONG ); +object_class_property_set_description (oc,"prealloc-context" , +"Context to use for creating CPU threads for preallocation" ); object_class_property_add(oc, "size", "int", host_memory_backend_get_size, host_memory_backend_set_size,
include/sysemu/hostmem.h +2 -0
@@ -18,6 +18,7 @@#include "qom/object.h" #include "exec/memory.h" #include "qemu/bitmap.h" +#include "qemu/thread-context.h" #define TYPE_MEMORY_BACKEND "memory-backend" OBJECT_DECLARE_TYPE(HostMemoryBackend, HostMemoryBackendClass,
@@ -66,6 +67,7 @@ struct HostMemoryBackend {bool merge, dump, use_canonical_path;bool prealloc, is_mapped, share, reserve;uint32_t prealloc_threads; +ThreadContext *prealloc_context;DECLARE_BITMAP (host_nodes,MAX_NODES +1 );HostMemPolicy policy;
qapi/qom.json +4 -0
@@ -578,6 +578,9 @@ # # @prealloc-threads: number of CPU threads to use for prealloc (default: 1) # +# @prealloc-context: thread context to use for creation of preallocation threads +# (default: none) (since 7.2) +# # @share: if false, the memory is private to QEMU; if true, it is shared # (default: false) #
@@ -608,6 +611,7 @@ '*policy': 'HostMemPolicy', '*prealloc': 'bool', '*prealloc-threads': 'uint32', + '*prealloc-context': 'str', '*share': 'bool', '*reserve': 'bool', 'size': 'size',
Signature · 1 line
2.37.3
PATCH v2 7/7 vl: Allow ThreadContext objects to be created before the sandbox option
In reply to David HildenbrandCurrently, there is no way to configure a CPU affinity inside QEMU when the sandbox option disables it for QEMU as a whole, for example, via:
-sandbox enable=on,resourcecontrol=deny
While ThreadContext objects can be created on the QEMU commandline and the CPU affinity can be configured externally via the thread-id, this is insufficient if a ThreadContext with a certain CPU affinity is already required during QEMU startup, before we can intercept QEMU and configure the CPU affinity.
Blocking sched_setaffinity() was introduced in 24f8cdc57224 ("seccomp:
add resourcecontrol argument to command line"), "to avoid any bigger of the process". However, we only care about once QEMU is running, not when the instance starting QEMU explicitly requests a certain CPU affinity on the QEMU comandline.
Right now, for NUMA-aware preallocation of memory backends used for initial machine RAM, one has to:
1) Start QEMU with the memory-backend with "prealloc=off"
2) Pause QEMU before it starts the guest (-S)
3) Create ThreadContext, configure the CPU affinity using the thread-id
4) Configure the ThreadContext as "prealloc-context" of the memory backend
5) Trigger preallocation by setting "prealloc=on"
To simplify this handling especially for initial machine RAM, allow creation of ThreadContext objects before parsing sandbox options, such that the CPU affinity requested on the QEMU commandline alongside the sandbox option can be set. As ThreadContext objects essentially only create a persistant context thread and set the CPU affinity, this is easily possible.
With this change, we can create a ThreadContext with a CPU affinity on the QEMU commandline and use it for preallocation of memory backends glued to the machine (simplified example):
To make "-name debug-threads=on" keep working as expected for the context threads, perform earlier parsing of "-name".
qemu-system-x86_64 -m 1G \
-object thread-context,id=tc1,cpu-affinity=3-4 \
-object memory-backend-ram,id=pc.ram,size=1G,prealloc=on,prealloc-threads=2,prealloc-context=tc1 \
-machine memory-backend=pc.ram \
-S -monitor stdio -sandbox enable=on,resourcecontrol=deny
And while we can query the current CPU affinity:
(qemu) qom-get tc1 cpu-affinity
[
3,
4]
We can no longer change it from QEMU directly:
(qemu) qom-set tc1 cpu-affinity 1-2
Error: Setting CPU affinity failed: Operation not permitted
--- softmmu/vl.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)
softmmu/vl.c +32 -4
@@ -1759,6 +1759,27 @@ static void object_option_parse(const char *optarg)visit_free (v); } +/* +* Very early object creation, before the sandbox options have been activated. +*/ +static bool object_create_pre_sandbox (const char *type) +{ +/* +* Objects should in general not get initialized "too early" without +* a reason. If you add one, state the reason in a comment! +*/ + +/* +* Reason: -sandbox on,resourcecontrol=deny disallows setting CPU +* affinity of threads. +*/ +if (g_str_equal (type,"thread-context" )) { +return true; + } + +return false; +} +/* * Initial object creation happens before all other * QEMU data types are created. The majority of objects
@@ -1773,6 +1794,11 @@ static bool object_create_early(const char *type)* add one, state the reason in a comment! */ +/* Reason: already created. */ +if (object_create_pre_sandbox (type)) { +return false; + } +/* Reason: property "chardev" */ if (g_str_equal (type,"rng-egd" ) ||g_str_equal (type,"qtest" )) {
@@ -1895,7 +1921,7 @@ static void qemu_create_early_backends(void)*/ static bool object_create_late (const char *type) { -return !object_create_early (type); +return !object_create_early (type) && !object_create_pre_sandbox (type); }static void qemu_create_late_backends (void )
@@ -2351,6 +2377,11 @@ static int process_runstate_actions(void *opaque, QemuOpts *opts, Error **errp)static void qemu_process_early_options (void ) { +qemu_opts_foreach (qemu_find_opts ("name" ), + parse_name,NULL , &error_fatal); + +object_option_foreach_add (object_create_pre_sandbox); +#ifdef CONFIG_SECCOMP QemuOptsList *olist =qemu_find_opts_err ("sandbox" ,NULL );if (olist) {
@@ -2358,9 +2389,6 @@ static void qemu_process_early_options(void) }#endif -qemu_opts_foreach (qemu_find_opts ("name" ), - parse_name,NULL , &error_fatal); -if (qemu_opts_foreach (qemu_find_opts ("action" ), process_runstate_actions,NULL , &error_fatal)) {exit (1 );
Signature · 1 line
2.37.3
Quoted 86 lines · * David Hildenbrand ([email protected]) wrote:
This is a follow-up on "util: NUMA aware memory preallocation" [1] by Michal.
Setting the CPU affinity of threads from inside QEMU usually isn't easily possible, because we don't want QEMU -- once started and running guest code -- to be able to mess up the system. QEMU disallows relevant syscalls using seccomp, such that any such invocation will fail.
Especially for memory preallocation in memory backends, the CPU affinity can significantly increase guest startup time, for example, when running large VMs backed by huge/gigantic pages, because of NUMA effects. For NUMA-aware preallocation, we have to set the CPU affinity, however:
(1) Once preallocation threads are created during preallocation, management tools cannot intercept anymore to change the affinity. These threads are created automatically on demand.
(2) QEMU cannot easily set the CPU affinity itself.
(3) The CPU affinity derived from the NUMA bindings of the memory backend might not necessarily be exactly the CPUs we actually want to use (e.g., CPU-less NUMA nodes, CPUs that are pinned/used for other VMs).There is an easy "workaround". If we have a thread with the right CPU affinity, we can simply create new threads on demand via that prepared context. So, all we have to do is setup and create such a context ahead of time, to then configure preallocation to create new threads via that environment.
So, let's introduce a user-creatable "thread-context" object that essentially consists of a context thread used to create new threads. QEMU can either try setting the CPU affinity itself ("cpu-affinity", "node-affinity" property), or upper layers can extract the thread id ("thread-id" property) to configure it externally.
Make memory-backends consume a thread-context object
(via the "prealloc-context" property) and use it when preallocating to create new threads with the desired CPU affinity. Further, to make it easier to use, allow creation of "thread-context" objects, including setting the CPU affinity directly from QEMU, before enabling the sandbox option.Quick test on a system with 2 NUMA nodes:
Without CPU affinity:
time qemu-system-x86_64 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind \ -nographic -monitor stdio real 0m5.383s real 0m3.499s real 0m5.129s real 0m4.232s real 0m5.220s real 0m4.288s real 0m3.582s real 0m4.305s real 0m5.421s real 0m4.502s -> It heavily depends on the scheduler CPU selectionWith CPU affinity:
time qemu-system-x86_64 \ -object thread-context,id=tc1,node-affinity=0 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind,prealloc-context=tc1 \ -sandbox enable=on,resourcecontrol=deny \ -nographic -monitor stdio real 0m1.959s real 0m1.942s real 0m1.943s real 0m1.941s real 0m1.948s real 0m1.964s real 0m1.949s real 0m1.948s real 0m1.941s real 0m1.937sOn reasonably large VMs, the speedup can be quite significant.
While this concept is currently only used for short-lived preallocation threads, nothing major speaks against reusing the concept for other threads that are harder to identify/configure -- except that we need additional (idle) context threads that are otherwise left unused.
Quoted 4 lines
This series does not yet tackle concurrent preallocation of memory backends. Memory backend objects are created and memory is preallocated one memory backend at a time -- and there is currently no way to do preallocation asynchronously.
Since you seem to have a full set of r-b's - do you intend to merge this as-is or do the cuncurrenct preallocation first?
Dave
Quoted 60 lines
v1 -> v2:
* Fixed some minor style nits
* "util: Introduce ThreadContext user-creatable object" -> Impove documentation and patch description. [Markus]
* "util: Add write-only "node-affinity" property for ThreadContext" -> Impove documentation and patch description. [Markus]RFC -> v1:
* "vl: Allow ThreadContext objects to be created before the sandbox option" -> Move parsing of the "name" property beforeobject_create_pre_sandbox
* Added RB'sCc Michal Privoznik <[email protected]>Cc Igor Mammedov <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Paolo Bonzini <[email protected]>Cc "Daniel P. Berrangé" <[email protected]>Cc Eduardo Habkost <[email protected]>Cc Dr. David Alan Gilbert <[email protected]>Cc Eric Blake <[email protected]>Cc Markus Armbruster <[email protected]>Cc Richard Henderson <[email protected]>Cc Stefan Weil <[email protected]>David Hildenbrand (7):
util: Cleanup and renameos_mem_prealloc()
util: Introduceqemu_thread_set_affinity()and
qemu_thread_get_affinity()
util: Introduce ThreadContext user-creatable object
util: Add write-only "node-affinity" property for ThreadContext
util: Makeqemu_prealloc_mem()optionally consume a ThreadContext
hostmem: Allow for specifying a ThreadContext for preallocation
vl: Allow ThreadContext objects to be created before the sandbox
optionbackends/hostmem.c | 13 +- hw/virtio/virtio-mem.c | 2 +- include/qemu/osdep.h | 19 +- include/qemu/thread-context.h | 57 ++++++ include/qemu/thread.h | 4 + include/sysemu/hostmem.h | 2 + meson.build | 16 ++ qapi/qom.json | 28 +++ softmmu/cpus.c | 2 +- softmmu/vl.c | 36 +++- util/meson.build | 1 + util/oslib-posix.c | 39 ++-- util/oslib-win32.c | 8 +- util/qemu-thread-posix.c | 70 +++++++ util/qemu-thread-win32.c | 12 ++ util/thread-context.c | 362 ++++++++++++++++++++++++++++++++++ 16 files changed, 641 insertions(+), 30 deletions(-) create mode 100644 include/qemu/thread-context.h create mode 100644 util/thread-context.cSignature · 1 line
2.37.3
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 87 lines · Dr. David Alan Gilbert
Quoted 86 lines · * David Hildenbrand ([email protected]) wrote:
This is a follow-up on "util: NUMA aware memory preallocation" [1] by Michal.
Setting the CPU affinity of threads from inside QEMU usually isn't easily possible, because we don't want QEMU -- once started and running guest code -- to be able to mess up the system. QEMU disallows relevant syscalls using seccomp, such that any such invocation will fail.
Especially for memory preallocation in memory backends, the CPU affinity can significantly increase guest startup time, for example, when running large VMs backed by huge/gigantic pages, because of NUMA effects. For NUMA-aware preallocation, we have to set the CPU affinity, however:
(1) Once preallocation threads are created during preallocation, management tools cannot intercept anymore to change the affinity. These threads are created automatically on demand.
(2) QEMU cannot easily set the CPU affinity itself.
(3) The CPU affinity derived from the NUMA bindings of the memory backend might not necessarily be exactly the CPUs we actually want to use (e.g., CPU-less NUMA nodes, CPUs that are pinned/used for other VMs).There is an easy "workaround". If we have a thread with the right CPU affinity, we can simply create new threads on demand via that prepared context. So, all we have to do is setup and create such a context ahead of time, to then configure preallocation to create new threads via that environment.
So, let's introduce a user-creatable "thread-context" object that essentially consists of a context thread used to create new threads. QEMU can either try setting the CPU affinity itself ("cpu-affinity", "node-affinity" property), or upper layers can extract the thread id ("thread-id" property) to configure it externally.
Make memory-backends consume a thread-context object
(via the "prealloc-context" property) and use it when preallocating to create new threads with the desired CPU affinity. Further, to make it easier to use, allow creation of "thread-context" objects, including setting the CPU affinity directly from QEMU, before enabling the sandbox option.Quick test on a system with 2 NUMA nodes:
Without CPU affinity:
time qemu-system-x86_64 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind \ -nographic -monitor stdio real 0m5.383s real 0m3.499s real 0m5.129s real 0m4.232s real 0m5.220s real 0m4.288s real 0m3.582s real 0m4.305s real 0m5.421s real 0m4.502s -> It heavily depends on the scheduler CPU selectionWith CPU affinity:
time qemu-system-x86_64 \ -object thread-context,id=tc1,node-affinity=0 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind,prealloc-context=tc1 \ -sandbox enable=on,resourcecontrol=deny \ -nographic -monitor stdio real 0m1.959s real 0m1.942s real 0m1.943s real 0m1.941s real 0m1.948s real 0m1.964s real 0m1.949s real 0m1.948s real 0m1.941s real 0m1.937sOn reasonably large VMs, the speedup can be quite significant.
While this concept is currently only used for short-lived preallocation threads, nothing major speaks against reusing the concept for other threads that are harder to identify/configure -- except that we need additional (idle) context threads that are otherwise left unused.
Quoted 4 lines
Quoted 4 lines
This series does not yet tackle concurrent preallocation of memory backends. Memory backend objects are created and memory is preallocated one memory backend at a time -- and there is currently no way to do preallocation asynchronously.
Hi Dave,
Quoted 3 lines
Since you seem to have a full set of r-b's - do you intend to merge this as-is or do the cuncurrenct preallocation first?
I intent to merge this as is, as it provides a benefit as it stands and concurrent preallcoation might not require user interface changes.
I do have some ideas on how to implement concurrent preallocation, but it needs more thought (and more importantly, time).
Signature · 3 lines
Thanks, David / dhildenb
Quoted 93 lines · * David Hildenbrand ([email protected]) wrote:
Quoted 87 lines · Dr. David Alan Gilbert
Quoted 86 lines · * David Hildenbrand ([email protected]) wrote:
This is a follow-up on "util: NUMA aware memory preallocation" [1] by Michal.
Setting the CPU affinity of threads from inside QEMU usually isn't easily possible, because we don't want QEMU -- once started and running guest code -- to be able to mess up the system. QEMU disallows relevant syscalls using seccomp, such that any such invocation will fail.
Especially for memory preallocation in memory backends, the CPU affinity can significantly increase guest startup time, for example, when running large VMs backed by huge/gigantic pages, because of NUMA effects. For NUMA-aware preallocation, we have to set the CPU affinity, however:
(1) Once preallocation threads are created during preallocation, management tools cannot intercept anymore to change the affinity. These threads are created automatically on demand.
(2) QEMU cannot easily set the CPU affinity itself.
(3) The CPU affinity derived from the NUMA bindings of the memory backend might not necessarily be exactly the CPUs we actually want to use (e.g., CPU-less NUMA nodes, CPUs that are pinned/used for other VMs).There is an easy "workaround". If we have a thread with the right CPU affinity, we can simply create new threads on demand via that prepared context. So, all we have to do is setup and create such a context ahead of time, to then configure preallocation to create new threads via that environment.
So, let's introduce a user-creatable "thread-context" object that essentially consists of a context thread used to create new threads. QEMU can either try setting the CPU affinity itself ("cpu-affinity", "node-affinity" property), or upper layers can extract the thread id ("thread-id" property) to configure it externally.
Make memory-backends consume a thread-context object
(via the "prealloc-context" property) and use it when preallocating to create new threads with the desired CPU affinity. Further, to make it easier to use, allow creation of "thread-context" objects, including setting the CPU affinity directly from QEMU, before enabling the sandbox option.Quick test on a system with 2 NUMA nodes:
Without CPU affinity:
time qemu-system-x86_64 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind \ -nographic -monitor stdio real 0m5.383s real 0m3.499s real 0m5.129s real 0m4.232s real 0m5.220s real 0m4.288s real 0m3.582s real 0m4.305s real 0m5.421s real 0m4.502s -> It heavily depends on the scheduler CPU selectionWith CPU affinity:
time qemu-system-x86_64 \ -object thread-context,id=tc1,node-affinity=0 \ -object memory-backend-memfd,id=md1,hugetlb=on,hugetlbsize=2M,size=64G,prealloc-threads=12,prealloc=on,host-nodes=0,policy=bind,prealloc-context=tc1 \ -sandbox enable=on,resourcecontrol=deny \ -nographic -monitor stdio real 0m1.959s real 0m1.942s real 0m1.943s real 0m1.941s real 0m1.948s real 0m1.964s real 0m1.949s real 0m1.948s real 0m1.941s real 0m1.937sOn reasonably large VMs, the speedup can be quite significant.
While this concept is currently only used for short-lived preallocation threads, nothing major speaks against reusing the concept for other threads that are harder to identify/configure -- except that we need additional (idle) context threads that are otherwise left unused.
Quoted 4 lines
Quoted 4 lines
This series does not yet tackle concurrent preallocation of memory backends. Memory backend objects are created and memory is preallocated one memory backend at a time -- and there is currently no way to do preallocation asynchronously.
Quoted 8 lines
Hi Dave,
Quoted 3 lines
Since you seem to have a full set of r-b's - do you intend to merge this as-is or do the cuncurrenct preallocation first?
I intent to merge this as is, as it provides a benefit as it stands and concurrent preallcoation might not require user interface changes.
Yep, that's fair enough.
Quoted 2 lines
I do have some ideas on how to implement concurrent preallocation, but it needs more thought (and more importantly, time).
Yep, it would be nice for the really huge VMs.
Dave
Quoted 5 lines
Signature · 3 lines
Thanks, David / dhildenb
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK