Skip to contentSkip to conversation

osdep: asynchronous teardown for shutdown on Linux

PATCH Superseded Claudio Imbrenda 2 participants 12 messages Aug 3, 2022 – Aug 5, 2022

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 osdep: asynchronous teardown for shutdown on Linux

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

---
 include/qemu/osdep.h |  2 ++
 os-posix.c           |  5 ++++
 qemu-options.hx      | 17 ++++++++++++++
 util/osdep.c         | 55 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 79 insertions(+)
include/qemu/osdep.h +2 -0
index b1c161c035..3154759d79 100644
@@ -549,6 +549,8 @@ ssize_t qemu_write_full(int fd, const void *buf, size_t count)
 
 void qemu_set_cloexec(int fd);
 
+void init_async_teardown(void);
+
 /* Return a dynamically allocated directory path that is appropriate for storing
  * local state.
  *
os-posix.c +5 -0
index 321fc4bd13..dd3e42b4c4 100644
@@ -150,6 +150,11 @@ int os_parse_cmd_args(int index, const char *optarg)
     case QEMU_OPTION_daemonize:
         daemonize = 1;
         break;
+#if defined(CONFIG_LINUX)
+    case QEMU_OPTION_asyncteardown:
+        init_async_teardown();
+        break;
+#endif
     default:
         return -1;
     }
qemu-options.hx +17 -0
index 3f23a42fa8..d434353159 100644
@@ -4743,6 +4743,23 @@ HXCOMM Internal use
 DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL)
 DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL)
 
+#ifdef __linux__
+DEF("async-teardown", 0, QEMU_OPTION_asyncteardown,
+    "-async-teardown enable asynchronous teardown\n",
+    QEMU_ARCH_ALL)
+#endif
+SRST
+``-async-teardown``
+    Enable asynchronous teardown. A new teardown process will be
+    created at startup, using clone. The teardown process will share
+    the address space of the main qemu process, and wait for the main
+    process to terminate. At that point, the teardown process will
+    also exit. This allows qemu to terminate quickly if the guest was
+    huge, leaving the teardown of the address space to the teardown
+    process. Since the teardown process shares the same cgroups as the
+    main qemu process, accounting is performed correctly.
+ERST
+
 DEF("msg", HAS_ARG, QEMU_OPTION_msg,
     "-msg [timestamp[=on|off]][,guest-name=[on|off]]\n"
     "                control error message format\n"
util/osdep.c +55 -0

67 lines not on this page · raw message

Signature · 1 line
2.37.1
#1 lore raw
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 8 lines
Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until

I don't know the inner details of how libvirt works..

Quoted 2 lines

all processes in the cgroup have exited, including this cloned child process.

..but I tested it and it works

my impression is that libvirt by default is only waiting for the main qemu process.

the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The close_range() call solves the issue.

Quoted 3 lines

With regards,
Daniel

#3 lore raw
Quoted 46 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 8 lines
Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until

I don't know the inner details of how libvirt works..

Quoted 7 lines
Quoted 2 lines

all processes in the cgroup have exited, including this cloned child process.

..but I tested it and it works

my impression is that libvirt by default is only waiting for the main qemu process.

If true, that would be a bug that needs fixing and should not be relied on.

Quoted 5 lines

the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The close_range() call solves the issue.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 50 lines · Daniel P. Berrangé
Quoted 49 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 8 lines
Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until

I don't know the inner details of how libvirt works..

Quoted 2 lines

all processes in the cgroup have exited, including this cloned child process.

Quoted 7 lines
Quoted 4 lines

..but I tested it and it works

my impression is that libvirt by default is only waiting for the main qemu process.

If true, that would be a bug that needs fixing and should not be relied on.

Libvirt is invoking 'TerminateMachine' DBus call on systemd-machined. That in turn iterates over every process in the cgroup and kills them off.

Docs are a little vague and I've not followed the code perfectly, but that should mean TeminateMachine doesnt return until every process in the cgroup has exited.

That said, since this is a dbus API call, libvirt will probably timeout waiting for the DBus reply after something like 30-60 seconds IIRC.

Quoted 6 lines
Quoted 6 lines

the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The close_range() call solves the issue.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw
Quoted 63 lines · On Thu, 4 Aug 2022 17:58:34 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 50 lines · Daniel P. Berrangé
Quoted 49 lines · Claudio Imbrenda
Quoted 32 lines · On Wed, 3 Aug 2022 18:34:45 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 8 lines
Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until

I don't know the inner details of how libvirt works..

Quoted 2 lines

all processes in the cgroup have exited, including this cloned child process.

Quoted 7 lines
Quoted 4 lines

..but I tested it and it works

my impression is that libvirt by default is only waiting for the main qemu process.

If true, that would be a bug that needs fixing and should not be relied on.

Libvirt is invoking 'TerminateMachine' DBus call on systemd-machined. That in turn iterates over every process in the cgroup and kills them off.

Quoted 7 lines

Docs are a little vague and I've not followed the code perfectly, but that should mean TeminateMachine doesnt return until every process in the cgroup has exited.

That said, since this is a dbus API call, libvirt will probably timeout waiting for the DBus reply after something like 30-60 seconds IIRC.

I have not observed any delays.

could it be that DBus doesn't wait for the process to be completely dead, but only that the signal is delivered?

and which signal will DBus use?

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines

the only issue I have found is the log file, which stays open as long as some file descriptors (which the cloned process inherits from the main qemu process) stay open. A new VM cannot be started if its log file is still open by the logger process. The close_range() call solves the issue.

With regards,
Daniel

#6 lore raw
In reply to Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé
Quoted 34 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

This feature can already be used with libvirt by adding the following to the XML domain definition:

Quoted 7 lines
Quoted 3 lines

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.

Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of clone() except for the creation of threads.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw
Quoted 40 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 39 lines · Daniel P. Berrangé
Quoted 37 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

Quoted 7 lines
Quoted 3 lines

How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.

Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of clone() except for the creation of threads.

it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default

Quoted 3 lines

With regards,
Daniel

#8 lore raw
Quoted 47 lines · Claudio Imbrenda
Quoted 44 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 42 lines · Daniel P. Berrangé
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.

Quoted 6 lines
Quoted 3 lines

Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of clone() except for the creation of threads.

it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default

Ok, so the reason it is working is because the extra process is cloned() right in middle of processing argv. This is before the seccomp filter is applied to the process, so clone() is not blocked.

One think I note about this in practice is that (unsurprisingly) if you do a process listing, users now see 2 QEMU processes instead of one.

I wonder if we should consider overwriting argv in the child process with "[qemu async teardown]" to give users a hint as to why this duplicate process exists.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw
Quoted 59 lines · On Thu, 4 Aug 2022 17:41:01 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 47 lines · Claudio Imbrenda
Quoted 44 lines · On Thu, 4 Aug 2022 09:29:39 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 42 lines · Daniel P. Berrangé
Quoted 31 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

Quoted 6 lines

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

How does this work in practice ? Libvirt should be blocking until all processes in the cgroup have exited, including this cloned child process.

Quoted 6 lines
Quoted 3 lines

Also, have you disabled use of seccomp with QEMU when testing this, as the seccomp filter that libivrt enables is supposed to block any use of clone() except for the creation of threads.

it was just a vanilla libvirt 8.0.0 as found on ubuntu 22.04; I have no idea how it is configured by default

Ok, so the reason it is working is because the extra process is cloned() right in middle of processing argv. This is before the seccomp filter is applied to the process, so clone() is not blocked.

Quoted 7 lines

One think I note about this in practice is that (unsurprisingly) if you do a process listing, users now see 2 QEMU processes instead of one.

I wonder if we should consider overwriting argv in the child process with "[qemu async teardown]" to give users a hint as to why this duplicate process exists.

sounds like a good idea

Quoted 3 lines

With regards,
Daniel

#10 lore raw
In reply to Claudio Imbrenda
Quoted 70 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

---
 include/qemu/osdep.h |  2 ++
 os-posix.c           |  5 ++++
 qemu-options.hx      | 17 ++++++++++++++
 util/osdep.c         | 55 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 79 insertions(+)
util/osdep.c +9 -0
index 60fcbbaebe..bb0baf97a0 100644
@@ -23,6 +23,15 @@
  */
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+
+#ifdef CONFIG_LINUX
+#include <sys/types.h>
+#include <sys/select.h>
+#include <sys/unistd.h>
+#include <sys/syscall.h>
+#include <signal.h>
+#endif
+
 #include "qemu/cutils.h"
 #include "qemu/sockets.h"
 #include "qemu/error-report.h"
@@ -512,6 +521,52 @@ const char *qemu_hw_version(void)
     return hw_version;
 }

util/osdep.c const char *qemu_hw_version(void) · line 532

Quoted 10 lines
util/osdep.c +9 -0
 
+#ifdef __linux__
+static int async_teardown_fn(void *arg)
+{
+    sigset_t all_signals;
+    fd_set r, w, e;
+    int fd;
+
+    /* open a pidfd descriptor for the parent qemu process */
+    fd = syscall(__NR_pidfd_open, getppid(), 0);

We ought to open this FD in the parent process to avoid a race where the parent crashes immediately after clone() and gets reparented to 'init' before this child process calls pidfd_open, otherwise it'll sit around waiting for init to exit.

util/osdep.c const char *qemu_hw_version(void) · line 545

Quoted 13 lines
util/osdep.c +13 -0
+    /* if something went wrong, or if the file descriptor is too big */
+    if ((fd < 0) || (fd >= FD_SETSIZE)) {
+        _exit(1);
+    }
+    /* zero all fd sets */
+    FD_ZERO(&r);
+    FD_ZERO(&w);
+    FD_ZERO(&e);
+    /* set the fd for the pidfd in the "read" set */
+    FD_SET(fd, &r);
+    /* block all signals */
+    sigfillset(&all_signals);
+    sigprocmask(SIG_BLOCK, &all_signals, NULL);

Technnically this is racy as there's still a window in which the child is running when signals are not blocked.

util/osdep.c const char *qemu_hw_version(void) · line 547

Quoted 2 lines
util/osdep.c +2 -0
+    /* wait for the pid to disappear -> fd will appear as ready for read */
+    (void) select(fd + 1, &r, &w, &e, NULL);

While using pidfd can work, a stronger protection would be to use

   prctl(PR_SET_PDEATHSIG, SIGKILL, 0, 0, 0);

this guarantees that the kernel will deliver SIGKILL to this process immediately when the parent QEMU exits.

We should probably do both in fact.

util/osdep.c const char *qemu_hw_version(void) · line 554

Quoted 7 lines
util/osdep.c +7 -0
+
+    /*
+     * Close all file descriptors that might have been inherited from the
+     * main qemu process when doing clone. This is needed to make libvirt
+     * happy.
+     */
+    close_range(0, ~0U, 0);

Shouldn't we close all the FDs immediately when this process is created, rather than only at the end when we're exiting. I don't see there's any need to keep them open. Doing it immediately would be better when using prctl(PR_SET_PDEATHSIG)

util/osdep.c const char *qemu_hw_version(void) · line 562

Quoted 8 lines
util/osdep.c +8 -0
+    _exit(0);
+}
+
+void init_async_teardown(void)
+{
+    const int size = 8192; /* should be more than enough */
+    char *stack = malloc(size);
+

You need to block all signals here.

util/osdep.c const char *qemu_hw_version(void) · line 564

Quoted 2 lines
util/osdep.c +2 -0
+    /* start a new process sharing the address space with qemu */
+    clone(async_teardown_fn, stack + size, CLONE_VM, NULL, NULL, NULL, NULL);

And unblock signals again here.

That way the "everything blocked" mask is inherited by the child from the very first moment of its existance.

Quoted 11 lines
util/osdep.c +5 -0
+}
+#else /* __linux__ */
+void init_async_teardown(void) {}
+#endif
+
 #ifdef _WIN32
 static void socket_cleanup(void)
 {
Signature · 1 line
2.37.1

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw
Quoted 72 lines · On Thu, 4 Aug 2022 17:56:34 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 71 lines · Claudio Imbrenda

This patch adds support for asynchronously tearing down a VM on Linux.

When qemu terminates, either naturally or because of a fatal signal, the VM is torn down. If the VM is huge, it can take a considerable amount of time for it to be cleaned up. In case of a protected VM, it might take even longer than a non-protected VM (this is the case on s390x, for example).

Some users might want to shut down a VM and restart it immediately, without having to wait. This is especially true if management infrastructure like libvirt is used.

This patch implements a simple trick on Linux to allow qemu to return immediately, with the teardown of the VM being performed asynchronously.

If the new commandline option -async-teardown is used, a new process is spawned from qemu at startup, using the clone syscall, in such way that it will share its address space with qemu.

The new process will then simpy wait until qemu terminates, and then it will exit itself.

This allows qemu to terminate quickly, without having to wait for the whole address space to be torn down. The teardown process will exit after qemu, so it will be the last user of the address space, and therefore it will take care of the actual teardown.

The teardown process will share the same cgroups as qemu, so both memory usage and cpu time will be accounted properly.

This feature can already be used with libvirt by adding the following to the XML domain definition:

<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0"> <arg value='-async-teardown'/>
</commandline>

---
 include/qemu/osdep.h |  2 ++
 os-posix.c           |  5 ++++
 qemu-options.hx      | 17 ++++++++++++++
 util/osdep.c         | 55 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 79 insertions(+)  
util/osdep.c +9 -0
index 60fcbbaebe..bb0baf97a0 100644
@@ -23,6 +23,15 @@
  */
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+
+#ifdef CONFIG_LINUX
+#include <sys/types.h>
+#include <sys/select.h>
+#include <sys/unistd.h>
+#include <sys/syscall.h>
+#include <signal.h>
+#endif
+
 #include "qemu/cutils.h"
 #include "qemu/sockets.h"
 #include "qemu/error-report.h"
@@ -512,6 +521,52 @@ const char *qemu_hw_version(void)
     return hw_version;
 }
 
Quoted 14 lines

util/osdep.c const char *qemu_hw_version(void) · line 532

Quoted 9 lines
util/osdep.c +9 -0
+#ifdef __linux__
+static int async_teardown_fn(void *arg)
+{
+    sigset_t all_signals;
+    fd_set r, w, e;
+    int fd;
+
+    /* open a pidfd descriptor for the parent qemu process */
+    fd = syscall(__NR_pidfd_open, getppid(), 0);  

We ought to open this FD in the parent process to avoid a race where the parent crashes immediately after clone() and gets reparented to 'init' before this child process calls pidfd_open, otherwise it'll sit around waiting for init to exit.

sounds good

Quoted 15 lines
Quoted 4 lines
util/osdep.c +3 -0
+    /* if something went wrong, or if the file descriptor is too big */
+    if ((fd < 0) || (fd >= FD_SETSIZE)) {
+        _exit(1);

util/osdep.c const char *qemu_hw_version(void) · line 545

Quoted 10 lines
util/osdep.c +10 -0
+    }
+    /* zero all fd sets */
+    FD_ZERO(&r);
+    FD_ZERO(&w);
+    FD_ZERO(&e);
+    /* set the fd for the pidfd in the "read" set */
+    FD_SET(fd, &r);
+    /* block all signals */
+    sigfillset(&all_signals);
+    sigprocmask(SIG_BLOCK, &all_signals, NULL);  
Quoted 2 lines

Technnically this is racy as there's still a window in which the child is running when signals are not blocked.

true, I will apply the workaround you suggest below

Quoted 4 lines

util/osdep.c const char *qemu_hw_version(void) · line 547

Quoted 3 lines
util/osdep.c +2 -0
+    /* wait for the pid to disappear -> fd will appear as ready for read */
+    (void) select(fd + 1, &r, &w, &e, NULL);  
Quoted 8 lines

While using pidfd can work, a stronger protection would be to use

   prctl(PR_SET_PDEATHSIG, SIGKILL, 0, 0, 0);

this guarantees that the kernel will deliver SIGKILL to this process immediately when the parent QEMU exits.

We should probably do both in fact.

makes sense

Quoted 10 lines

util/osdep.c const char *qemu_hw_version(void) · line 554

Quoted 8 lines
util/osdep.c +7 -0
+
+    /*
+     * Close all file descriptors that might have been inherited from the
+     * main qemu process when doing clone. This is needed to make libvirt
+     * happy.
+     */
+    close_range(0, ~0U, 0);  

Shouldn't we close all the FDs immediately when this process is

that's exactly what I did in the beginning, but some things broke... but I just tried again now and it seems to work, so probably the breakage was due to something else

Quoted 3 lines

created, rather than only at the end when we're exiting. I don't see there's any need to keep them open. Doing it immediately would be better when using prctl(PR_SET_PDEATHSIG)

yes

Quoted 12 lines

util/osdep.c const char *qemu_hw_version(void) · line 562

Quoted 9 lines
util/osdep.c +8 -0
+    _exit(0);
+}
+
+void init_async_teardown(void)
+{
+    const int size = 8192; /* should be more than enough */
+    char *stack = malloc(size);
+  

You need to block all signals here.

Quoted 7 lines

util/osdep.c const char *qemu_hw_version(void) · line 564

Quoted 2 lines
util/osdep.c +2 -0
+    /* start a new process sharing the address space with qemu */
+    clone(async_teardown_fn, stack + size, CLONE_VM, NULL, NULL, NULL, NULL);  

And unblock signals again here.

That way the "everything blocked" mask is inherited by the child from the very first moment of its existance.

sounds like a very good idea

Quoted 15 lines
Quoted 12 lines
util/osdep.c +5 -0
+}
+#else /* __linux__ */
+void init_async_teardown(void) {}
+#endif
+
 #ifdef _WIN32
 static void socket_cleanup(void)
 {
Signature · 1 line
2.37.1

With regards,
Daniel