os-posix: asynchronous teardown for shutdown on Linux
PATCH v3 1/1 os-posix: asynchronous teardown for shutdown on Linux
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>
More advanced interfaces like pidfd or close_range have intentionally been avoided in order to be more compatible with older kernels.
--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.c
include/qemu/async-teardown.h new file +22 -0
@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif
os-posix.c +6 -0
@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /*
@@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }
qemu-options.hx +17 -0
@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the main + process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"
util/async-teardown.c new file +123 -0
123 lines not on this page · raw message
util/meson.build +1 -0
@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')
Signature · 1 line
2.37.1
Hi, Claudio.
Quoted 34 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
Quoted 8 lines
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/> </commandline>More advanced interfaces like pidfd or
close_rangehave intentionally been avoided in order to be more compatible with older kernels.Signed-off-by Claudio Imbrenda <[email protected]>
I've smoke-tested this on ppc and everything looks fine. For what's worth:
Have you measured the benefits of using -async-teardown vs. not using it? If so, can you please share the details so I can give it a try on ppc, too?
The wall-clock perception is that nothing has changed, for better or worse. My tests used mid-sized VMs, like 128 vCPUs, 64GB RAM.
Cheers!
Quoted 231 lines
--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+)create mode 100644
include/qemu/async-teardown.hcreate mode 100644util/async-teardown.cinclude/qemu/async-teardown.h new file +22
-0@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif os-posix.c +6
-0@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /* @@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }qemu-options.hx +17
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the main + process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"util/async-teardown.c new file +123
-0@@ -0,0 +1,123 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <dirent.h> +#include <sys/prctl.h> +#include <signal.h> +#include <sched.h> + +#include "qemu/async-teardown.h" + +static pid_t the_ppid; + +/* +* Close all open file descriptors. +*/ +static void close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +return ; + } +/* Avoid closing the directory. */ + dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) { +close (fd); + } + } +closedir (dir); +} + +static void hup_handler (int signal) +{ +/* Check every second if this process has been reparented. */ +while (the_ppid ==getppid ()) { +/* sleep() is safe to use in a signal handler. */ +sleep (1 ); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +static int async_teardown_fn (void *arg) +{ +struct sigaction sa = { .sa_handler = hup_handler }; +sigset_t hup_signal; +char name[16 ]; + +/* Set a meaningful name for this process. */ +snprintf (name,16 ,"cleanup/%d" , the_ppid); +prctl (PR_SET_NAME , (unsigned long )name); + +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone, needed to make libvirt happy. +* Not using close_range for increased compatibility with older kernels. +*/ +close_all_open_fd (); + +/* Set up a handler for SIGHUP and unblock SIGHUP. */ +sigaction (SIGHUP , &sa,NULL ); +sigemptyset (&hup_signal); +sigaddset (&hup_signal,SIGHUP ); +sigprocmask (SIG_UNBLOCK , &hup_signal,NULL ); + +/* Ask to receive SIGHUP when the parent dies. */ +prctl (PR_SET_PDEATHSIG ,SIGHUP ); + +/* +* Sleep forever, unless the parent process has already terminated. The +* only interruption can come from the SIGHUP signal, which in normal +* operation is received when the parent process dies. +*/ +if (the_ppid ==getppid ()) { +pause (); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +/* +* Block all signals, start (clone) a new process sharing the address space +* with qemu (CLONE_VM), then restore signals. +*/ +void init_async_teardown (void ) +{ +sigset_t all_signals, old_signals; +const int stack_size =8192 ;/* Should be more than enough */ +char *stack, *stack_ptr; + + the_ppid =getpid (); + stack =malloc (stack_size); +if (!stack) { +return ; + } + stack_ptr = stack + stack_size; + +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals, &old_signals); +clone (async_teardown_fn, stack_ptr,CLONE_VM ,NULL ,NULL ,NULL ,NULL ); +sigprocmask (SIG_SETMASK , &old_signals,NULL ); +}util/meson.build +1
-0@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')
Signature · 1 line
Murilo
Quoted 1 line · On Wed, 10 Aug 2022 17:30:41 -0300 Murilo Opsfelder Araújo…
Hi, Claudio.
Hi Murilo,
[...]
Quoted 5 lines
I've smoke-tested this on ppc and everything looks fine. For what's worth:
Reviewed-by Murilo Opsfelder Araujo <[email protected]>
thanks a lot for testing this!
Quoted 7 lines
Have you measured the benefits of using -async-teardown vs. not using it? If so, can you please share the details so I can give it a try on ppc, too?
The wall-clock perception is that nothing has changed, for better or worse. My tests used mid-sized VMs, like 128 vCPUs, 64GB RAM.
The number of CPUs doesn't really have any impact. 64G of RAM is quite small to notice a sizeable difference, although you should be able to see a few seconds of speedup in the shutdown. Also, starting a guest with a lot of RAM is not enough, you have to make sure that the guest ram is actually allocated (completely fill the ram in the guest before shutting it down)
I just tested a 64G and a 256G guest on s390x. I measured the time between the last line in the console ("Reached target Power-Off.") and the moment when control comes back to the shell. The measurement was not exactly super accurate (I manually ran "date +%s" in another shell when I saw the last line in the console, and then again when I got the shell back from qemu).
The 64G guest needs a few seconds, the 256G needs almost exactly 4 times as much. With the asynchronous teardown it's almost instant in both cases (less than 1s, too fast to measure manually).
Try a multi-TB guest if you can (at the moment I can't) to see a more marked effect.
Also consider that this is for _normal_ guests. Protected guests on s390x have an even slower teardown due to the way protected virtualization is implemented in the hardware.
I hope this was helpful
Quoted 236 lines
Cheers!
Quoted 231 lines
--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+)create mode 100644
include/qemu/async-teardown.hcreate mode 100644util/async-teardown.cinclude/qemu/async-teardown.h new file +22
-0@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif os-posix.c +6
-0@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /* @@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }qemu-options.hx +17
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the main + process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"util/async-teardown.c new file +123
-0@@ -0,0 +1,123 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <dirent.h> +#include <sys/prctl.h> +#include <signal.h> +#include <sched.h> + +#include "qemu/async-teardown.h" + +static pid_t the_ppid; + +/* +* Close all open file descriptors. +*/ +static void close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +return ; + } +/* Avoid closing the directory. */ + dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) { +close (fd); + } + } +closedir (dir); +} + +static void hup_handler (int signal) +{ +/* Check every second if this process has been reparented. */ +while (the_ppid ==getppid ()) { +/* sleep() is safe to use in a signal handler. */ +sleep (1 ); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +static int async_teardown_fn (void *arg) +{ +struct sigaction sa = { .sa_handler = hup_handler }; +sigset_t hup_signal; +char name[16 ]; + +/* Set a meaningful name for this process. */ +snprintf (name,16 ,"cleanup/%d" , the_ppid); +prctl (PR_SET_NAME , (unsigned long )name); + +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone, needed to make libvirt happy. +* Not using close_range for increased compatibility with older kernels. +*/ +close_all_open_fd (); + +/* Set up a handler for SIGHUP and unblock SIGHUP. */ +sigaction (SIGHUP , &sa,NULL ); +sigemptyset (&hup_signal); +sigaddset (&hup_signal,SIGHUP ); +sigprocmask (SIG_UNBLOCK , &hup_signal,NULL ); + +/* Ask to receive SIGHUP when the parent dies. */ +prctl (PR_SET_PDEATHSIG ,SIGHUP ); + +/* +* Sleep forever, unless the parent process has already terminated. The +* only interruption can come from the SIGHUP signal, which in normal +* operation is received when the parent process dies. +*/ +if (the_ppid ==getppid ()) { +pause (); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +/* +* Block all signals, start (clone) a new process sharing the address space +* with qemu (CLONE_VM), then restore signals. +*/ +void init_async_teardown (void ) +{ +sigset_t all_signals, old_signals; +const int stack_size =8192 ;/* Should be more than enough */ +char *stack, *stack_ptr; + + the_ppid =getpid (); + stack =malloc (stack_size); +if (!stack) { +return ; + } + stack_ptr = stack + stack_size; + +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals, &old_signals); +clone (async_teardown_fn, stack_ptr,CLONE_VM ,NULL ,NULL ,NULL ,NULL ); +sigprocmask (SIG_SETMASK , &old_signals,NULL ); +}util/meson.build +1
-0@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')
Quoted 123 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>More advanced interfaces like pidfd or
close_rangehave intentionally been avoided in order to be more compatible with older kernels.Signed-off-by Claudio Imbrenda <[email protected]>--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.cinclude/qemu/async-teardown.h new file +22
-0@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif os-posix.c +6
-0@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /* @@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }qemu-options.hx +10
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the main
qemu-options.hx HXCOMM Internal use · line 4765
Quoted 10 lines
qemu-options.hx +7
-0+ process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"
It occurrs to me that we've got a general goal of getting away from adding new top level command line arguments. Most of the time there's an obvious existing place to put them, but I'm really not sure where this particular option would fit ?
it isn't tied to any aspect of the VM backend configuration nor hardware frontends.
The closest match is the lifecycle action option (-no-shutdown) which were merged into a -action arg, but even that's a bit of a stretch.
Markus/Paolo: do you have suggestions ?
Quoted 45 lines
util/async-teardown.c new file +39
-0@@ -0,0 +1,123 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <dirent.h> +#include <sys/prctl.h> +#include <signal.h> +#include <sched.h> + +#include "qemu/async-teardown.h" + +static pid_t the_ppid; + +/* +* Close all open file descriptors. +*/ +static void close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +return ; + } +/* Avoid closing the directory. */
util/async-teardown.c line 49
Quoted 10 lines
util/async-teardown.c new file +10
-0+ dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) { +close (fd); + } + } +closedir (dir); +}
I know you said you're intentionally avoided close_range, but the cost is that this makes 'proc' a mandatory dependancy.
IMHO, we should build with close_range support if it is available in the glibc headers. Only use proc as a fallback if close_range reports ENOSYS, or is compiled out.
Quoted 28 lines
util/async-teardown.c new file +28
-0+ +static void hup_handler (int signal) +{ +/* Check every second if this process has been reparented. */ +while (the_ppid ==getppid ()) { +/* sleep() is safe to use in a signal handler. */ +sleep (1 ); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +static int async_teardown_fn (void *arg) +{ +struct sigaction sa = { .sa_handler = hup_handler }; +sigset_t hup_signal; +char name[16 ]; + +/* Set a meaningful name for this process. */ +snprintf (name,16 ,"cleanup/%d" , the_ppid); +prctl (PR_SET_NAME , (unsigned long )name); + +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone, needed to make libvirt happy. +* Not using close_range for increased compatibility with older kernels. +*/
util/async-teardown.c line 87
Quoted 10 lines
util/async-teardown.c new file +10
-0+close_all_open_fd (); + +/* Set up a handler for SIGHUP and unblock SIGHUP. */ +sigaction (SIGHUP , &sa,NULL ); +sigemptyset (&hup_signal); +sigaddset (&hup_signal,SIGHUP ); +sigprocmask (SIG_UNBLOCK , &hup_signal,NULL ); + +/* Ask to receive SIGHUP when the parent dies. */ +prctl (PR_SET_PDEATHSIG ,SIGHUP );
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
Quoted 18 lines
util/async-teardown.c new file +18
-0+ +/* +* Sleep forever, unless the parent process has already terminated. The +* only interruption can come from the SIGHUP signal, which in normal +* operation is received when the parent process dies. +*/ +if (the_ppid ==getppid ()) { +pause (); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +/* +* Block all signals, start (clone) a new process sharing the address space +* with qemu (CLONE_VM), then restore signals. +*/
util/async-teardown.c line 115
Quoted 10 lines
util/async-teardown.c new file +10
-0+void init_async_teardown (void ) +{ +sigset_t all_signals, old_signals; +const int stack_size =8192 ;/* Should be more than enough */ +char *stack, *stack_ptr; + + the_ppid =getpid (); + stack =malloc (stack_size); +if (!stack) { +return ;
THis should be fatal. How about using qemu_alloc_stack instead ?
Quoted 23 lines
util/async-teardown.c new file +8
-0+ } + stack_ptr = stack + stack_size; + +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals, &old_signals); +clone (async_teardown_fn, stack_ptr,CLONE_VM ,NULL ,NULL ,NULL ,NULL ); +sigprocmask (SIG_SETMASK , &old_signals,NULL ); +}util/meson.build +1
-0@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')Signature · 1 line
2.37.1
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Am 11.08.22 um 14:27 schrieb Daniel P. Berrangé:
[...]
Quoted 32 lines
Quoted 16 lines
qemu-options.hx +10
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the mainqemu-options.hx HXCOMM Internal use · line 4765
Quoted 10 lines
qemu-options.hx +7
-0+ process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"It occurrs to me that we've got a general goal of getting away from adding new top level command line arguments. Most of the time there's an obvious existing place to put them, but I'm really not sure where this particular option would fit ?
Quoted 8 lines
it isn't tied to any aspect of the VM backend configuration nor hardware frontends.
The closest match is the lifecycle action option (-no-shutdown) which were merged into a -action arg, but even that's a bit of a stretch.
Markus/Paolo: do you have suggestions ?
Also extending this to libvirt, would it make sense to add this to the event list <on_poweroff>
<on_reboot>
<on_crash>
<on_lockfailure>
as
<on_teardown> with async/sync
This might give an indication where to put it in qemu.
Quoted 140 lines · On Thu, 11 Aug 2022 13:27:44 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 123 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>More advanced interfaces like pidfd or
close_rangehave intentionally been avoided in order to be more compatible with older kernels.Signed-off-by Claudio Imbrenda <[email protected]>--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.cinclude/qemu/async-teardown.h new file +22
-0@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif os-posix.c +6
-0@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /* @@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }qemu-options.hx +10
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the mainqemu-options.hx HXCOMM Internal use · line 4765
Quoted 10 lines
qemu-options.hx +7
-0+ process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"It occurrs to me that we've got a general goal of getting away from adding new top level command line arguments. Most of the time there's an obvious existing place to put them, but I'm really not sure where this particular option would fit ?
Quoted 7 lines
it isn't tied to any aspect of the VM backend configuration nor hardware frontends.
The closest match is the lifecycle action option (-no-shutdown) which were merged into a -action arg, but even that's a bit of a stretch.
I have no preferences as to how the parameter is passed
Quoted 53 lines
Markus/Paolo: do you have suggestions ?
Quoted 50 lines
util/async-teardown.c new file +44
-0@@ -0,0 +1,123 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <dirent.h> +#include <sys/prctl.h> +#include <signal.h> +#include <sched.h> + +#include "qemu/async-teardown.h" + +static pid_t the_ppid; + +/* +* Close all open file descriptors. +*/ +static void close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +return ; + } +/* Avoid closing the directory. */ + dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) {
Quoted 8 lines
util/async-teardown.c line 49
Quoted 5 lines
util/async-teardown.c new file +5
-0+close (fd); + } + } +closedir (dir); +}I know you said you're intentionally avoided
close_range, but the cost is that this makes 'proc' a mandatory dependancy.
Running qemu without /proc sounds a little bit weird, but ok
Quoted 4 lines
IMHO, we should build with
close_rangesupport if it is available in the glibc headers. Only use proc as a fallback ifclose_rangereports ENOSYS, or is compiled out.
sounds reasonable.
what is the best way to do it? just some #ifdefs to check if the syscall is available, or a config option to check at config time if the syscall is defined?
(and then I guess the ENOSYS check needs to be there anyway)
Quoted 35 lines
Quoted 35 lines
util/async-teardown.c new file +34
-0+ +static void hup_handler (int signal) +{ +/* Check every second if this process has been reparented. */ +while (the_ppid ==getppid ()) { +/* sleep() is safe to use in a signal handler. */ +sleep (1 ); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +static int async_teardown_fn (void *arg) +{ +struct sigaction sa = { .sa_handler = hup_handler }; +sigset_t hup_signal; +char name[16 ]; + +/* Set a meaningful name for this process. */ +snprintf (name,16 ,"cleanup/%d" , the_ppid); +prctl (PR_SET_NAME , (unsigned long )name); + +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone, needed to make libvirt happy. +* Not using close_range for increased compatibility with older kernels. +*/ +close_all_open_fd (); + +/* Set up a handler for SIGHUP and unblock SIGHUP. */ +sigaction (SIGHUP , &sa,NULL ); +sigemptyset (&hup_signal); +sigaddset (&hup_signal,SIGHUP );
Quoted 8 lines
util/async-teardown.c line 87
Quoted 4 lines
util/async-teardown.c new file +4
-0+sigprocmask (SIG_UNBLOCK , &hup_signal,NULL ); + +/* Ask to receive SIGHUP when the parent dies. */ +prctl (PR_SET_PDEATHSIG ,SIGHUP );Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Quoted 4 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Quoted 23 lines
Quoted 23 lines
util/async-teardown.c new file +22
-0+ +/* +* Sleep forever, unless the parent process has already terminated. The +* only interruption can come from the SIGHUP signal, which in normal +* operation is received when the parent process dies. +*/ +if (the_ppid ==getppid ()) { +pause (); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +/* +* Block all signals, start (clone) a new process sharing the address space +* with qemu (CLONE_VM), then restore signals. +*/ +void init_async_teardown (void ) +{ +sigset_t all_signals, old_signals; +const int stack_size =8192 ;/* Should be more than enough */
Quoted 8 lines
util/async-teardown.c line 115
Quoted 6 lines
util/async-teardown.c new file +6
-0+char *stack, *stack_ptr; + + the_ppid =getpid (); + stack =malloc (stack_size); +if (!stack) { +return ;THis should be fatal. How about using
qemu_alloc_stackinstead ?
sounds like a great idea
Quoted 27 lines
Quoted 24 lines
util/async-teardown.c new file +8
-0+ } + stack_ptr = stack + stack_size; + +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals, &old_signals); +clone (async_teardown_fn, stack_ptr,CLONE_VM ,NULL ,NULL ,NULL ,NULL ); +sigprocmask (SIG_SETMASK , &old_signals,NULL ); +}util/meson.build +1
-0@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c')) if not config_host_data.get('CONFIG_ATOMIC64') util_ss.add(files('atomic64.c')) endif +util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c')) util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c')) if config_host_data.get('CONFIG_EPOLL_CREATE1')Signature · 1 line
2.37.1With regards,
Daniel
Quoted 215 lines · Claudio Imbrenda
Quoted 140 lines · On Thu, 11 Aug 2022 13:27:44 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 123 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>More advanced interfaces like pidfd or
close_rangehave intentionally been avoided in order to be more compatible with older kernels.Signed-off-by Claudio Imbrenda <[email protected]>--- include/qemu/async-teardown.h | 22 ++++++ os-posix.c | 6 ++ qemu-options.hx | 17 +++++ util/async-teardown.c | 123 ++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 5 files changed, 169 insertions(+) create mode 100644 include/qemu/async-teardown.h create mode 100644 util/async-teardown.cinclude/qemu/async-teardown.h new file +22
-0@@ -0,0 +1,22 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_ASYNC_TEARDOWN_H +#define QEMU_ASYNC_TEARDOWN_H + +#include "config-host.h" + +#ifdef CONFIG_LINUX +void init_async_teardown (void ); +#endif + +#endif os-posix.c +6
-0@@ -39,6 +39,7 @@#ifdef CONFIG_LINUX #include <sys/prctl.h> +#include "qemu/async-teardown.h" #endif /* @@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }qemu-options.hx +10
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the mainqemu-options.hx HXCOMM Internal use · line 4765
Quoted 10 lines
qemu-options.hx +7
-0+ process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"It occurrs to me that we've got a general goal of getting away from adding new top level command line arguments. Most of the time there's an obvious existing place to put them, but I'm really not sure where this particular option would fit ?
Quoted 7 lines
it isn't tied to any aspect of the VM backend configuration nor hardware frontends.
The closest match is the lifecycle action option (-no-shutdown) which were merged into a -action arg, but even that's a bit of a stretch.
I have no preferences as to how the parameter is passed
Quoted 53 lines
Markus/Paolo: do you have suggestions ?
Quoted 50 lines
util/async-teardown.c new file +44
-0@@ -0,0 +1,123 @@ +/* +* Asynchronous teardown +* +* Copyright IBM, Corp. 2022 +* +* Authors: +* Claudio Imbrenda <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at your +* option) any later version. See the COPYING file in the top-level directory. +* +*/ +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <dirent.h> +#include <sys/prctl.h> +#include <signal.h> +#include <sched.h> + +#include "qemu/async-teardown.h" + +static pid_t the_ppid; + +/* +* Close all open file descriptors. +*/ +static void close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +return ; + } +/* Avoid closing the directory. */ + dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) {Quoted 8 lines
util/async-teardown.c line 49
Quoted 5 lines
util/async-teardown.c new file +5
-0+close (fd); + } + } +closedir (dir); +}I know you said you're intentionally avoided
close_range, but the cost is that this makes 'proc' a mandatory dependancy.
Quoted 1 line
Running qemu without /proc sounds a little bit weird, but ok
People have been doing increasingly wierd things with containers ;-)
Quoted 11 lines
Quoted 3 lines
IMHO, we should build with
close_rangesupport if it is available in the glibc headers. Only use proc as a fallback ifclose_rangereports ENOSYS, or is compiled out.sounds reasonable.
what is the best way to do it? just some #ifdefs to check if the syscall is available, or a config option to check at config time if the syscall is defined?
(and then I guess the ENOSYS check needs to be there anyway)
Probably a check for the function in meson.build, and then some #ifdefs, and a catch of ENOSYS.
Quoted 43 lines
Quoted 34 lines
Quoted 34 lines
util/async-teardown.c new file +34
-0+ +static void hup_handler (int signal) +{ +/* Check every second if this process has been reparented. */ +while (the_ppid ==getppid ()) { +/* sleep() is safe to use in a signal handler. */ +sleep (1 ); + } + +/* At this point the parent process has terminated completely. */ +_exit (0 ); +} + +static int async_teardown_fn (void *arg) +{ +struct sigaction sa = { .sa_handler = hup_handler }; +sigset_t hup_signal; +char name[16 ]; + +/* Set a meaningful name for this process. */ +snprintf (name,16 ,"cleanup/%d" , the_ppid); +prctl (PR_SET_NAME , (unsigned long )name); + +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone, needed to make libvirt happy. +* Not using close_range for increased compatibility with older kernels. +*/ +close_all_open_fd (); + +/* Set up a handler for SIGHUP and unblock SIGHUP. */ +sigaction (SIGHUP , &sa,NULL ); +sigemptyset (&hup_signal); +sigaddset (&hup_signal,SIGHUP );Quoted 8 lines
util/async-teardown.c line 87
Quoted 4 lines
util/async-teardown.c new file +4
-0+sigprocmask (SIG_UNBLOCK , &hup_signal,NULL ); + +/* Ask to receive SIGHUP when the parent dies. */ +prctl (PR_SET_PDEATHSIG ,SIGHUP );Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
Quoted 3 lines
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 8 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 8/11/22 11:02, Daniel P. Berrangé wrote:
[...]
Quoted 14 lines
Quoted 7 lines
Quoted 3 lines
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 4 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
Quoted 7 lines
Quoted 4 lines
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
It seems Libvirt sends SIGKILL if qemu process doesn't terminate within 10 seconds after Libvirt sent SIGTERM:
So I guess this patch happened to work with Libvirt because the main qemu process terminated before the timeout and before SIGKILL was delivered.
The cleanup process is trying to solve the problem where the main qemu process takes too long to terminate. However, if the cleanup process itself takes too long, SIGKILL will be sent by Libvirt anyway.
Perhaps we can describe this situation in the parameter help, e.g.: If management layer decides to send SIGKILL (e.g.: due to timeout or deliberate decision), the cleanup process can exit before the main process, deceiving its purpose.
Signature · 1 line
Murilo
Quoted 21 lines · On Thu, 11 Aug 2022 23:05:52 -0300 Murilo Opsfelder Araújo…
On 8/11/22 11:02, Daniel P. Berrangé wrote:
[...]Quoted 19 lines
Quoted 7 lines
Quoted 3 lines
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 8 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Quoted 7 lines
Quoted 2 lines
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
It seems Libvirt sends SIGKILL if qemu process doesn't terminate within 10 seconds after Libvirt sent SIGTERM:
but this is fine.
with asynchronous teardown, qemu will exit almost immediately when receiving SIGTERM, and the cleanup process will start cleaning up.
Quoted 3 lines
So I guess this patch happened to work with Libvirt because the main qemu process terminated before the timeout and before SIGKILL was delivered.
it seems so
Quoted 4 lines
The cleanup process is trying to solve the problem where the main qemu process takes too long to terminate. However, if the cleanup process itself takes too long, SIGKILL will be sent by Libvirt anyway.
but that is not a problem, the sole purpose of the cleanup process is to terminate _after_ qemu. it doesn't matter what happens after qemu has terminated. if you look at the patch, after going to great lengths to assure that qemu has terminated, all the child process does is _exit(0).
Quoted 5 lines
Perhaps we can describe this situation in the parameter help, e.g.: If management layer decides to send SIGKILL (e.g.: due to timeout or deliberate decision), the cleanup process can exit before the main process, deceiving its purpose.
if the management layer (or the user) decides to send SIGKILL immediately to the whole cgroup without sending SIGTERM first, then this whole asynchronous teardown mechanism is defeated, yes.
Quoted 30 lines · Claudio Imbrenda
Quoted 27 lines · On Thu, 11 Aug 2022 23:05:52 -0300 Murilo Opsfelder Araújo…
On 8/11/22 11:02, Daniel P. Berrangé wrote:
[...]Quoted 14 lines
Quoted 7 lines
Quoted 3 lines
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 4 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
Quoted 7 lines
Quoted 4 lines
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
It seems Libvirt sends SIGKILL if qemu process doesn't terminate within 10 seconds after Libvirt sent SIGTERM:
Quoted 6 lines
Quoted 1 line
but this is fine.
with asynchronous teardown, qemu will exit almost immediately when receiving SIGTERM, and the cleanup process will start cleaning up.
Under normal and orderly conditions, yes.
Quoted 17 lines
Quoted 2 lines
So I guess this patch happened to work with Libvirt because the main qemu process terminated before the timeout and before SIGKILL was delivered.
it seems so
Quoted 4 lines
The cleanup process is trying to solve the problem where the main qemu process takes too long to terminate. However, if the cleanup process itself takes too long, SIGKILL will be sent by Libvirt anyway.
but that is not a problem, the sole purpose of the cleanup process is to terminate _after_ qemu. it doesn't matter what happens after qemu has terminated. if you look at the patch, after going to great lengths to assure that qemu has terminated, all the child process does is
_exit(0).Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Perhaps we can describe this situation in the parameter help, e.g.: If management layer decides to send SIGKILL (e.g.: due to timeout or deliberate decision), the cleanup process can exit before the main process, deceiving its purpose.
if the management layer (or the user) decides to send SIGKILL immediately to the whole cgroup without sending SIGTERM first, then this whole asynchronous teardown mechanism is defeated, yes.
This situation is what we likely want to describe in the parameter help. I don't want to give users the false impression that this option will *always* behave the manner we expect it to work *most* of the time.
--
Murilo
Quoted 62 lines · On Fri, 12 Aug 2022 08:38:59 -0300 Murilo Opsfelder Araújo…
Quoted 30 lines · Claudio Imbrenda
Quoted 27 lines · On Thu, 11 Aug 2022 23:05:52 -0300 Murilo Opsfelder Araújo…
On 8/11/22 11:02, Daniel P. Berrangé wrote:
[...]Quoted 14 lines
Quoted 7 lines
Quoted 3 lines
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 4 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
Quoted 7 lines
Quoted 4 lines
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
It seems Libvirt sends SIGKILL if qemu process doesn't terminate within 10 seconds after Libvirt sent SIGTERM:
Quoted 6 lines
Quoted 1 line
but this is fine.
with asynchronous teardown, qemu will exit almost immediately when receiving SIGTERM, and the cleanup process will start cleaning up.
Under normal and orderly conditions, yes.
Quoted 22 lines
Quoted 2 lines
So I guess this patch happened to work with Libvirt because the main qemu process terminated before the timeout and before SIGKILL was delivered.
it seems so
Quoted 4 lines
The cleanup process is trying to solve the problem where the main qemu process takes too long to terminate. However, if the cleanup process itself takes too long, SIGKILL will be sent by Libvirt anyway.
but that is not a problem, the sole purpose of the cleanup process is to terminate _after_ qemu. it doesn't matter what happens after qemu has terminated. if you look at the patch, after going to great lengths to assure that qemu has terminated, all the child process does is
_exit(0).Quoted 5 lines
Perhaps we can describe this situation in the parameter help, e.g.: If management layer decides to send SIGKILL (e.g.: due to timeout or deliberate decision), the cleanup process can exit before the main process, deceiving its purpose.
Quoted 7 lines
Quoted 3 lines
if the management layer (or the user) decides to send SIGKILL immediately to the whole cgroup without sending SIGTERM first, then this whole asynchronous teardown mechanism is defeated, yes.
This situation is what we likely want to describe in the parameter help. I don't want to give users the false impression that this option will *always* behave the manner we expect it to work *most* of the time.
fair enough, I'll improve the documentation
Quoted 3 lines
--
Murilo
Quoted 30 lines · Claudio Imbrenda
Quoted 27 lines · On Thu, 11 Aug 2022 23:05:52 -0300 Murilo Opsfelder Araújo…
On 8/11/22 11:02, Daniel P. Berrangé wrote:
[...]Quoted 14 lines
Quoted 7 lines
Quoted 3 lines
Hmm, I was hoping you could just use SIGKILL to guarantee that this gets killed off. Is SIGKILL delivered too soon to allow for the main QEMU process to have exited quickly ?
yes, I tried. qemu has not finished exiting when the signal is delivered, the cleanup process dies before qemu, which defeats the purpose
Ok, too bad.
Quoted 4 lines
Quoted 3 lines
If so I wonder what happens when systemd just delivers SIGKILL to all processes in the cgroup - I'm not sure there's a guarantee it will SIGKILL the main qemu before it SIGKILLs this helper
Quoted 7 lines
Quoted 4 lines
I'm afraid in that case there is no guarantee.
for what it's worth, both virsh shutdown and destroy seem to do things properly.
Hmm, probably because libvirt tells QEMU to exit before systemd comes along and tells everything in the cgroup to die with SIGKILL.
It seems Libvirt sends SIGKILL if qemu process doesn't terminate within 10 seconds after Libvirt sent SIGTERM:
Quoted 6 lines
Quoted 1 line
but this is fine.
with asynchronous teardown, qemu will exit almost immediately when receiving SIGTERM, and the cleanup process will start cleaning up.
Note, when you have PCI host devices attahced it can take a very long time for QEMU to exit. For this reason, the 10 second wait before switching to SIGKILL is extended by 2 seconds for each attachec PCI hostdev.
I think the main time we will have problems is where there are storage failures that cause QEMU to get stuck in an uninterruptible sleep in kernel space. The classic example of this is an NFS server that goes away, QEMU will get stuck waiting for the NFS server to come back to life and be unkillable in this time even with SIGKILL.
That said, this call to virProcessKillPainfully shouldn't impact the cleanmup process, becaused the SIGTERM/KILL are both directed to the QEMU PID alone, not the process group.
The cleanup process should only get any signal later once libvirt has finished sending SIGTERM/KILL, it then asks systemd to cleanup the cgroups and at that time systemd can send SIGKILL to the cleanup process. So in fact I think we should be fine in all respects, except for the unkillable sleeps in kernel space.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Please excuse my late reply; I was on vacation.
Quoted 43 lines · Daniel P. Berrangé
Quoted 42 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will have the name "cleanup/<QEMU_PID>". It will wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>More advanced interfaces like pidfd or
close_rangehave intentionally been avoided in order to be more compatible with older kernels.Signed-off-by Claudio Imbrenda <[email protected]>
[...]
Quoted 34 lines
Quoted 18 lines
qemu-options.hx +10
-0@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the mainqemu-options.hx HXCOMM Internal use · line 4765
Quoted 10 lines
qemu-options.hx +7
-0+ process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"It occurrs to me that we've got a general goal of getting away from adding new top level command line arguments. Most of the time there's an obvious existing place to put them, but I'm really not sure where this particular option would fit ?
Quoted 6 lines
it isn't tied to any aspect of the VM backend configuration nor hardware frontends.
The closest match is the lifecycle action option (-no-shutdown) which were merged into a -action arg, but even that's a bit of a stretch.
If I understand the proposed new option correctly, it modifies how QEMU terminates, independent of why it terminates. Could be guest reboot with -action reboot-shutdown, monitor command quit, SIGTERM, ...
I agree putting it under -action would be a bit of a stretch, as so far -action is entirely about configuring the reaction to guest certain actions:
-action reboot=reset|shutdown
action when guest reboots [default=reset]
-action shutdown=poweroff|pause
action when guest shuts down [default=poweroff]
-action panic=pause|shutdown|exit-failure|none
action when guest panics [default=shutdown]
-action watchdog=reset|shutdown|poweroff|inject-nmi|pause|debug|none
action when watchdog fires [default=reset]
A different stretch: -daemonize, -runas, -chroot. These modify how QEMU starts. They too are "top-level".
Quoted 1 line
Markus/Paolo: do you have suggestions ?
Ramblings^WThoughts, not actionable suggestions, I'm afraid.
[...]