os-posix: asynchronous teardown for shutdown on Linux
PATCH v4 1/1 os-posix: asynchronous teardown for shutdown on Linux
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates completely, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The cleanup process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown. The cleanup process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
If possible, close_range will be used in the cleanup process to close all open file descriptors. If it is not available or if it fails, /proc will be used to determine which file descriptors to close.
If the cleanup process is forcefully killed with SIGKILL before the main qemu process has terminated completely, the mechanism is defeated and the teardown will not be asynchronous.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>
--- include/qemu/async-teardown.h | 22 +++++ meson.build | 1 + os-posix.c | 6 ++ qemu-options.hx | 19 +++++ util/async-teardown.c | 155 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 6 files changed, 204 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.c
include/qemu/async-teardown.h new file +22 -0
@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif
meson.build +1 -0
@@ -1892,6 +1892,7 @@ config_host_data.set('HAVE_SYS_IOCCOM_H', cc.has_header('sys/ioccom.h')) config_host_data.set('HAVE_SYS_KCOV_H', cc.has_header('sys/kcov.h')) # has_function +config_host_data.set('CONFIG_CLOSE_RANGE', cc.has_function('close_range')) config_host_data.set('CONFIG_ACCEPT4', cc.has_function('accept4')) config_host_data.set('CONFIG_CLOCK_ADJTIME', cc.has_function('clock_adjtime')) config_host_data.set('CONFIG_DUP3', cc.has_function('dup3'))
os-posix.c +6 -0
@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /*
@@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }
qemu-options.hx +19 -0
@@ -4743,6 +4743,25 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new process called "cleanup/<QEMU_PID>" + will be created at startup sharing the address space with the main qemu + process, using clone. It will wait for the main qemu process to + terminate completely, and then exit. + This allows qemu to terminate very quickly even if the guest was + huge, leaving the teardown of the address space to the cleanup + process. Since the cleanup process shares the same cgroups as the + main qemu process, accounting is performed correctly. This only + works if the cleanup process is not forcefully killed with SIGKILL + before the main qemu process has terminated completely. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"
util/async-teardown.c new file +155 -0
155 lines not on this page · raw message
util/meson.build +1 -0
@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')
Signature · 1 line
2.37.1
Quoted 47 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates completely, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The cleanup process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown. The cleanup process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
If possible,
close_rangewill be used in the cleanup process to close all open file descriptors. If it is not available or if it fails, /proc will be used to determine which file descriptors to close.If the cleanup process is forcefully killed with SIGKILL before the main qemu process has terminated completely, the mechanism is defeated and the teardown will not be asynchronous.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>Signed-off-by Claudio Imbrenda <[email protected]>Reviewed-by Murilo Opsfelder Araujo <[email protected]>
Quoted 10 lines
--- include/qemu/async-teardown.h | 22 +++++ meson.build | 1 + os-posix.c | 6 ++ qemu-options.hx | 19 +++++ util/async-teardown.c | 155 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 6 files changed, 204 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.c
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 40 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates completely, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The cleanup process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown. The cleanup process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
If possible,
close_rangewill be used in the cleanup process to close all open file descriptors. If it is not available or if it fails, /proc will be used to determine which file descriptors to close.If the cleanup process is forcefully killed with SIGKILL before the main qemu process has terminated completely, the mechanism is defeated and the teardown will not be asynchronous.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
Quoted 8 lines
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/> </commandline>Signed-off-by Claudio Imbrenda<[email protected]>Reviewed-by Murilo Opsfelder Araujo<[email protected]>---
Nice trick indeed!
The only question I have is whether it would make sense to do this in Libvirt instead.
Having a new independent one-off option like this is not great, but I think it's fine because it's not a very reusable grouping.
Paolo
Quoted 50 lines · Paolo Bonzini
Quoted 40 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates completely, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The cleanup process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown. The cleanup process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
If possible,
close_rangewill be used in the cleanup process to close all open file descriptors. If it is not available or if it fails, /proc will be used to determine which file descriptors to close.If the cleanup process is forcefully killed with SIGKILL before the main qemu process has terminated completely, the mechanism is defeated and the teardown will not be asynchronous.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
Quoted 8 lines
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/> </commandline>Signed-off-by Claudio Imbrenda<[email protected]>Reviewed-by Murilo Opsfelder Araujo<[email protected]>---
Quoted 4 lines
Nice trick indeed!
The only question I have is whether it would make sense to do this in Libvirt instead.
Hmm, interesting idea, that hadn't occurred to me.
Current flow is
libvirtd -> fork -> fork -> execve(qemu)
\-> clone(async handler)
Due to CLONE_VM, both main QEMU PID and async handler PID are sharing the memory mappings of code being executed. Everything is fine.
If doing it in libvirt the flow would be
libvirtd -> fork -> fork -> execve(qemu)
\-> clone(async handler)
In the latter case the async handler would be running code from the libvirt binary. When the parent calls execve that will load the code from QEMU, and because of CLONE_VM, this affect the memory mappings in the async handler proces too. I think that will end up quite explody, no ?
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 10 lines · Daniel P. Berrangé
If doing it in libvirt the flow would be
libvirtd -> fork -> fork -> execve(qemu)
\-> clone(async handler)In the latter case the async handler would be running code from the libvirt binary. When the parent calls execve that will load the code from QEMU, and because of
CLONE_VM, this affect the memory mappings in the async handler process too. I think that will end up quite explody, no ?
Oh yes it will. :)
Paolo