osdep: asynchronous teardown for shutdown on Linux
PATCH v2 1/1 osdep: asynchronous teardown for shutdown on Linux
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>
--- include/qemu/osdep.h | 2 ++ os-posix.c | 5 ++++ qemu-options.hx | 17 ++++++++++++++ util/osdep.c | 55 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 79 insertions(+)
include/qemu/osdep.h +2 -0
@@ -549,6 +549,8 @@ ssize_t qemu_write_full(int fd, const void *buf, size_t count)void qemu_set_cloexec (int fd); +void init_async_teardown (void ); +/* Return a dynamically allocated directory path that is appropriate for storing * local state. *
os-posix.c +5 -0
@@ -150,6 +150,11 @@ int os_parse_cmd_args(int index, const char *optarg)case QEMU_OPTION_daemonize: daemonize =1 ;break ; +#if defined(CONFIG_LINUX ) +case QEMU_OPTION_asyncteardown: +init_async_teardown (); +break ; +#endif default :return -1 ; }
qemu-options.hx +17 -0
@@ -4743,6 +4743,23 @@ HXCOMM Internal use DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL) DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL) +#ifdef __linux__ +DEF("async-teardown", 0, QEMU_OPTION_asyncteardown, + "-async-teardown enable asynchronous teardown\n", + QEMU_ARCH_ALL) +#endif +SRST +``-async-teardown`` + Enable asynchronous teardown. A new teardown process will be + created at startup, using clone. The teardown process will share + the address space of the main qemu process, and wait for the main + process to terminate. At that point, the teardown process will + also exit. This allows qemu to terminate quickly if the guest was + huge, leaving the teardown of the address space to the teardown + process. Since the teardown process shares the same cgroups as the + main qemu process, accounting is performed correctly. +ERST + DEF("msg", HAS_ARG, QEMU_OPTION_msg, "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n" " control error message format\n"
util/osdep.c +55 -0
67 lines not on this page · raw message
Signature · 1 line
2.37.1
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>
How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 8 lines
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until
I don't know the inner details of how libvirt works..
Quoted 2 lines
all processes in the cgroup have exited, including this cloned child process.
..but I tested it and it works
my impression is that libvirt by default is only waiting for the main qemu process.
the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The close_range() call solves the issue.
Quoted 3 lines
With regards,
Daniel
Quoted 46 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 8 lines
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until
I don't know the inner details of how libvirt works..
Quoted 7 lines
Quoted 2 lines
all processes in the cgroup have exited, including this cloned child process.
..but I tested it and it works
my impression is that libvirt by default is only waiting for the main qemu process.
If true, that would be a bug that needs fixing and should not be relied on.
Quoted 5 lines
the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The
close_range()call solves the issue.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 50 lines · Daniel P. Berrangé
Quoted 49 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 8 lines
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until
I don't know the inner details of how libvirt works..
Quoted 2 lines
all processes in the cgroup have exited, including this cloned child process.
Quoted 7 lines
Quoted 4 lines
..but I tested it and it works
my impression is that libvirt by default is only waiting for the main qemu process.
If true, that would be a bug that needs fixing and should not be relied on.
Libvirt is invoking 'TerminateMachine' DBus call on systemd-machined. That in turn iterates over every process in the cgroup and kills them off.
Docs are a little vague and I've not followed the code perfectly, but that should mean TeminateMachine doesnt return until every process in the cgroup has exited.
That said, since this is a dbus API call, libvirt will probably timeout waiting for the DBus reply after something like 30-60 seconds IIRC.
Quoted 6 lines
Quoted 6 lines
the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The
close_range()call solves the issue.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 63 lines · On Thu, 4 Aug 2022 17:58:34 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 50 lines · Daniel P. Berrangé
Quoted 49 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 8 lines
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until
I don't know the inner details of how libvirt works..
Quoted 2 lines
all processes in the cgroup have exited, including this cloned child process.
Quoted 7 lines
Quoted 4 lines
..but I tested it and it works
my impression is that libvirt by default is only waiting for the main qemu process.
If true, that would be a bug that needs fixing and should not be relied on.
Libvirt is invoking 'TerminateMachine' DBus call on systemd-machined. That in turn iterates over every process in the cgroup and kills them off.
Quoted 7 lines
Docs are a little vague and I've not followed the code perfectly, but that should mean TeminateMachine doesnt return until every process in the cgroup has exited.
That said, since this is a dbus API call, libvirt will probably timeout waiting for the DBus reply after something like 30-60 seconds IIRC.
I have not observed any delays.
could it be that DBus doesn't wait for the process to be completely dead, but only that the signal is delivered?
and which signal will DBus use?
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The
close_range()call solves the issue.With regards,
Daniel
Quoted 35 lines · Daniel P. Berrangé
Quoted 34 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition:
Quoted 7 lines
Quoted 3 lines
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.
Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of clone() except for the creation of threads.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 40 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 39 lines · Daniel P. Berrangé
Quoted 37 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>
Quoted 7 lines
Quoted 3 lines
How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.
Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of
clone()except for the creation of threads.
it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default
Quoted 3 lines
With regards,
Daniel
Quoted 47 lines · Claudio Imbrenda
Quoted 44 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 42 lines · Daniel P. Berrangé
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.
Quoted 6 lines
Quoted 3 lines
Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of
clone()except for the creation of threads.it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default
Ok, so the reason it is working is because the extra process is cloned() right in middle of processing argv. This is before the seccomp filter is applied to the process, so clone() is not blocked.
One think I note about this in practice is that (unsurprisingly) if you do a process listing, users now see 2 QEMU processes instead of one.
I wonder if we should consider overwriting argv in the child process with "[qemu async teardown]" to give users a hint as to why this duplicate process exists.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 59 lines · On Thu, 4 Aug 2022 17:41:01 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 47 lines · Claudio Imbrenda
Quoted 44 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 42 lines · Daniel P. Berrangé
Quoted 31 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
Quoted 6 lines
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.
Quoted 6 lines
Quoted 3 lines
Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of
clone()except for the creation of threads.it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default
Ok, so the reason it is working is because the extra process is
cloned()right in middle of processing argv. This is before the seccomp filter is applied to the process, soclone()is not blocked.
Quoted 7 lines
One think I note about this in practice is that (unsurprisingly) if you do a process listing, users now see 2 QEMU processes instead of one.
I wonder if we should consider overwriting argv in the child process with "[qemu async teardown]" to give users a hint as to why this duplicate process exists.
sounds like a good idea
Quoted 3 lines
With regards,
Daniel
Quoted 70 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>Signed-off-by Claudio Imbrenda <[email protected]>--- include/qemu/osdep.h | 2 ++ os-posix.c | 5 ++++ qemu-options.hx | 17 ++++++++++++++ util/osdep.c | 55 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 79 insertions(+)util/osdep.c +9
-0@@ -23,6 +23,15 @@*/ #include "qemu/osdep.h" #include "qapi/error.h" + +#ifdef CONFIG_LINUX +#include <sys/types.h> +#include <sys/select.h> +#include <sys/unistd.h> +#include <sys/syscall.h> +#include <signal.h> +#endif +#include "qemu/cutils.h" #include "qemu/sockets.h" #include "qemu/error-report.h" @@ -512,6 +521,52 @@ const char *qemu_hw_version(void)return hw_version; }
util/osdep.c const char *qemu_hw_version(void) · line 532
Quoted 10 lines
util/osdep.c +9
-0+#ifdef __linux__ +static int async_teardown_fn (void *arg) +{ +sigset_t all_signals; +fd_set r, w, e; +int fd; + +/* open a pidfd descriptor for the parent qemu process */ + fd =syscall (__NR_pidfd_open,getppid (),0 );
We ought to open this FD in the parent process to avoid a race where the parent crashes immediately after clone() and gets reparented to 'init' before this child process calls pidfd_open, otherwise it'll sit around waiting for init to exit.
util/osdep.c const char *qemu_hw_version(void) · line 545
Quoted 13 lines
util/osdep.c +13
-0+/* if something went wrong, or if the file descriptor is too big */ +if ((fd <0 ) || (fd >=FD_SETSIZE )) { +_exit (1 ); + } +/* zero all fd sets */ +FD_ZERO (&r); +FD_ZERO (&w); +FD_ZERO (&e); +/* set the fd for the pidfd in the "read" set */ +FD_SET (fd, &r); +/* block all signals */ +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals,NULL );
Technnically this is racy as there's still a window in which the child is running when signals are not blocked.
util/osdep.c const char *qemu_hw_version(void) · line 547
Quoted 2 lines
util/osdep.c +2
-0+/* wait for the pid to disappear -> fd will appear as ready for read */ + (void )select (fd +1 , &r, &w, &e,NULL );
While using pidfd can work, a stronger protection would be to use
prctl(PR_SET_PDEATHSIG, SIGKILL, 0, 0, 0);
this guarantees that the kernel will deliver SIGKILL to this process immediately when the parent QEMU exits.
We should probably do both in fact.
util/osdep.c const char *qemu_hw_version(void) · line 554
Quoted 7 lines
util/osdep.c +7
-0+ +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone. This is needed to make libvirt +* happy. +*/ +close_range (0 , ~0U ,0 );
Shouldn't we close all the FDs immediately when this process is created, rather than only at the end when we're exiting. I don't see there's any need to keep them open. Doing it immediately would be better when using prctl(PR_SET_PDEATHSIG)
util/osdep.c const char *qemu_hw_version(void) · line 562
Quoted 8 lines
util/osdep.c +8
-0+_exit (0 ); +} + +void init_async_teardown (void ) +{ +const int size =8192 ;/* should be more than enough */ +char *stack =malloc (size); +
You need to block all signals here.
util/osdep.c const char *qemu_hw_version(void) · line 564
Quoted 2 lines
util/osdep.c +2
-0+/* start a new process sharing the address space with qemu */ +clone (async_teardown_fn, stack + size,CLONE_VM ,NULL ,NULL ,NULL ,NULL );
And unblock signals again here.
That way the "everything blocked" mask is inherited by the child from the very first moment of its existance.
Quoted 11 lines
util/osdep.c +5
-0+} +#else /* __linux__ */ +void init_async_teardown (void ) {} +#endif +#ifdef _WIN32static void socket_cleanup (void ) {Signature · 1 line
2.37.1
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 72 lines · On Thu, 4 Aug 2022 17:56:34 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 71 lines · Claudio Imbrenda
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).
Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.
If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.
The new process will then simpy wait until qemu terminates, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.
The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.
This feature can already be used with libvirt by adding the following to the XML domain definition:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>Signed-off-by Claudio Imbrenda <[email protected]>--- include/qemu/osdep.h | 2 ++ os-posix.c | 5 ++++ qemu-options.hx | 17 ++++++++++++++ util/osdep.c | 55 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 79 insertions(+)util/osdep.c +9
-0@@ -23,6 +23,15 @@*/ #include "qemu/osdep.h" #include "qapi/error.h" + +#ifdef CONFIG_LINUX +#include <sys/types.h> +#include <sys/select.h> +#include <sys/unistd.h> +#include <sys/syscall.h> +#include <signal.h> +#endif +#include "qemu/cutils.h" #include "qemu/sockets.h" #include "qemu/error-report.h" @@ -512,6 +521,52 @@ const char *qemu_hw_version(void)return hw_version; }
Quoted 14 lines
util/osdep.c const char *qemu_hw_version(void) · line 532
Quoted 9 lines
util/osdep.c +9
-0+#ifdef __linux__ +static int async_teardown_fn (void *arg) +{ +sigset_t all_signals; +fd_set r, w, e; +int fd; + +/* open a pidfd descriptor for the parent qemu process */ + fd =syscall (__NR_pidfd_open,getppid (),0 );We ought to open this FD in the parent process to avoid a race where the parent crashes immediately after
clone()and gets reparented to 'init' before this child process callspidfd_open, otherwise it'll sit around waiting for init to exit.
sounds good
Quoted 15 lines
Quoted 4 lines
util/osdep.c +3
-0+/* if something went wrong, or if the file descriptor is too big */ +if ((fd <0 ) || (fd >=FD_SETSIZE )) { +_exit (1 );util/osdep.c const char *qemu_hw_version(void) · line 545
Quoted 10 lines
util/osdep.c +10
-0+ } +/* zero all fd sets */ +FD_ZERO (&r); +FD_ZERO (&w); +FD_ZERO (&e); +/* set the fd for the pidfd in the "read" set */ +FD_SET (fd, &r); +/* block all signals */ +sigfillset (&all_signals); +sigprocmask (SIG_BLOCK , &all_signals,NULL );
Quoted 2 lines
Technnically this is racy as there's still a window in which the child is running when signals are not blocked.
true, I will apply the workaround you suggest below
Quoted 4 lines
util/osdep.c const char *qemu_hw_version(void) · line 547
Quoted 3 lines
util/osdep.c +2
-0+/* wait for the pid to disappear -> fd will appear as ready for read */ + (void )select (fd +1 , &r, &w, &e,NULL );
Quoted 8 lines
While using pidfd can work, a stronger protection would be to use
prctl(PR_SET_PDEATHSIG, SIGKILL, 0, 0, 0);this guarantees that the kernel will deliver SIGKILL to this process immediately when the parent QEMU exits.
We should probably do both in fact.
makes sense
Quoted 10 lines
util/osdep.c const char *qemu_hw_version(void) · line 554
Quoted 8 lines
util/osdep.c +7
-0+ +/* +* Close all file descriptors that might have been inherited from the +* main qemu process when doing clone. This is needed to make libvirt +* happy. +*/ +close_range (0 , ~0U ,0 );Shouldn't we close all the FDs immediately when this process is
that's exactly what I did in the beginning, but some things broke... but I just tried again now and it seems to work, so probably the breakage was due to something else
Quoted 3 lines
created, rather than only at the end when we're exiting. I don't see there's any need to keep them open. Doing it immediately would be better when using
prctl(PR_SET_PDEATHSIG)
yes
Quoted 12 lines
util/osdep.c const char *qemu_hw_version(void) · line 562
Quoted 9 lines
util/osdep.c +8
-0+_exit (0 ); +} + +void init_async_teardown (void ) +{ +const int size =8192 ;/* should be more than enough */ +char *stack =malloc (size); +You need to block all signals here.
Quoted 7 lines
util/osdep.c const char *qemu_hw_version(void) · line 564
Quoted 2 lines
util/osdep.c +2
-0+/* start a new process sharing the address space with qemu */ +clone (async_teardown_fn, stack + size,CLONE_VM ,NULL ,NULL ,NULL ,NULL );And unblock signals again here.
That way the "everything blocked" mask is inherited by the child from the very first moment of its existance.
sounds like a very good idea
Quoted 15 lines
Quoted 12 lines
util/osdep.c +5
-0+} +#else /* __linux__ */ +void init_async_teardown (void ) {} +#endif +#ifdef _WIN32static void socket_cleanup (void ) {Signature · 1 line
2.37.1With regards,
Daniel