Skip to contentSkip to conversation

python/machine.py: refactor shutdown

PATCH In review John Snow 3 participants 45 messages Jul 10, 2020 – Jul 14, 2020

lore
Share & tools
lore
author
#10 lore raw

PATCH v5 4/12 python/machine.py: Perform early cleanup for wait() calls, too

In reply to John Snow

This is primarily for consistency, and is a step towards wait() and shutdown() sharing the same implementation so that the two cleanup paths cannot diverge.

---
 python/qemu/machine.py | 1 +
 1 file changed, 1 insertion(+)
python/qemu/machine.py +1 -0
index 4280aab380..cac466fbe6 100644
@@ -369,6 +369,7 @@ def wait(self):
         """
         Wait for the VM to power off
         """
+        self._early_cleanup()
         self._popen.wait()
         self._post_shutdown()
 
Signature · 1 line
2.21.3
author
#4 lore raw

PATCH v5 2/12 python/machine.py: Close QMP socket in cleanup

In reply to John Snow

It's not important to do this before waiting for the process to exit, so it can be done during generic post-shutdown cleanup.

---
 python/qemu/machine.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
python/qemu/machine.py +4 -3
index f7e68191c2..938c891b1d 100644
@@ -283,6 +283,10 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        if self._qmp:
+            self._qmp.close()
+            self._qmp = None
+
         self._load_io_log()
 
         if self._qemu_log_file is not None:
@@ -355,8 +359,6 @@ def wait(self):
         Wait for the VM to power off
         """
         self._popen.wait()
-        if self._qmp:
-            self._qmp.close()
         self._post_shutdown()
 
     def shutdown(self, has_quit=False, hard=False):
@@ -377,7 +379,6 @@ def shutdown(self, has_quit=False, hard=False):
                 try:
                     if not has_quit:
                         self._qmp.cmd('quit')
-                    self._qmp.close()
                     self._popen.wait(timeout=3)
                 except:
                     self._popen.kill()
Signature · 1 line
2.21.3

John Snow

author
#0 lore raw

v5 cover python/machine.py: refactor shutdown

v5: More or less rewritten.

This series is motivated by a desire to move python/qemu onto a strict mypy/pylint regime to help prevent regressions in the python codebase.

1. Remove the "bare except" pattern in the existing shutdown code, which can mask problems and make debugging difficult.

2. Ensure that post-shutdown cleanup is always performed, even when graceful termination fails.

3. Unify cleanup paths such that no matter how the VM is terminated, the same functions and steps are always taken to reset the object state.

4. Rewrite shutdown() such that any error encountered when attempting a graceful shutdown will be raised as an AbnormalShutdown exception. The pythonic idiom is to allow the caller to decide if this is a problem or not.

Previous versions of this series did not engage the fourth goal, and ran into race conditions. When I was trying to allow shutdown to succeed if QEMU was already closed, it became impossible to tell in which cases QEMU not being present was "OK" and in which cases it was evidence of a problem.

This refactoring is even more explicit. If a graceful shutdown is requested and cannot be performed, an exception /will/ be raised.

In cases where the test writer expects QEMU to already have exited, vm.wait() should be used in preference to vm.shutdown(). In cases where a graceful shutdown is not interesting or necessary to the test, vm.kill() should be used.

John Snow (12):
python/machine.py: consolidate _post_shutdown() python/machine.py: Close QMP socket in cleanup python/machine.py: Add _early_cleanup hook python/machine.py: Perform early cleanup for wait() calls, too python/machine.py: Prohibit multiple shutdown() calls python/machine.py: Add a configurable timeout to shutdown() python/machine.py: Make wait() call shutdown() tests/acceptance: wait() instead of shutdown() where appropriate tests/acceptance: Don't test reboot on cubieboard python/machine.py: split shutdown into hard and soft flavors python/machine.py: re-add sigkill warning suppression python/machine.py: change default wait timeout to 3 seconds

 python/qemu/machine.py                   | 166 ++++++++++++++++++-----
 tests/acceptance/boot_linux_console.py   |  14 +-
 tests/acceptance/linux_ssh_mips_malta.py |   2 +
 3 files changed, 141 insertions(+), 41 deletions(-)
Signature · 1 line
2.21.3
author
#7 lore raw

PATCH v5 3/12 python/machine.py: Add _early_cleanup hook

Some parts of cleanup need to occur prior to shutdown, otherwise shutdown might break. Move this into a suitably named method/callback.

---
 python/qemu/machine.py | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
python/qemu/machine.py +15 -9
index 938c891b1d..4280aab380 100644
@@ -354,16 +354,9 @@ def _launch(self):
                                        close_fds=False)
         self._post_launch()
 
-    def wait(self):
+    def _early_cleanup(self) -> None:
         """
-        Wait for the VM to power off
-        """
-        self._popen.wait()
-        self._post_shutdown()
-
-    def shutdown(self, has_quit=False, hard=False):
-        """
-        Terminate the VM and clean up
+        Perform any cleanup that needs to happen before the VM exits.
         """
         # If we keep the console socket open, we may deadlock waiting
         # for QEMU to exit, while QEMU is waiting for the socket to
@@ -372,6 +365,19 @@ def shutdown(self, has_quit=False, hard=False):
             self._console_socket.close()
             self._console_socket = None
 
+    def wait(self):
+        """
+        Wait for the VM to power off
+        """
+        self._popen.wait()
+        self._post_shutdown()
+
+    def shutdown(self, has_quit=False, hard=False):
+        """
+        Terminate the VM and clean up
+        """
+        self._early_cleanup()
+
         if self.is_running():
             if hard:
                 self._popen.kill()
Signature · 1 line
2.21.3
author
#13 lore raw

PATCH v5 5/12 python/machine.py: Prohibit multiple shutdown() calls

In reply to John Snow

If the VM is not launched, don't try to shut it down. As a change, _post_shutdown now unconditionally also calls _early_cleanup in order to offer comprehensive object cleanup in failure cases.

As a courtesy, treat it as a NOP instead of rejecting it as an error. This is slightly nicer for acceptance tests where vm.shutdown() is issued unconditionally in tearDown callbacks.

---
 python/qemu/machine.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
python/qemu/machine.py +13 -1
index cac466fbe6..e66a7d66dd 100644
@@ -283,6 +283,13 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        """
+        Called to cleanup the VM instance after the process has exited.
+        May also be called after a failed launch.
+        """
+        # Comprehensive reset for the failed launch case:
+        self._early_cleanup()
+
         if self._qmp:
             self._qmp.close()
             self._qmp = None
@@ -328,7 +335,7 @@ def launch(self):
             self._launch()
             self._launched = True
         except:
-            self.shutdown()
+            self._post_shutdown()
 
             LOG.debug('Error launching VM')
             if self._qemu_full_args:
@@ -357,6 +364,8 @@ def _launch(self):
     def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
+
+        Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
         # for QEMU to exit, while QEMU is waiting for the socket to
@@ -377,6 +386,9 @@ def shutdown(self, has_quit=False, hard=False):
         """
         Terminate the VM and clean up
         """
+        if not self._launched:
+            return
+
         self._early_cleanup()
 
         if self.is_running():
Signature · 1 line
2.21.3
author
#18 lore raw

PATCH v5 6/12 python/machine.py: Add a configurable timeout to shutdown()

In reply to John Snow

Three seconds is hardcoded. Use it as a default parameter instead, and use that value for both waits that may occur in the function.

---
 python/qemu/machine.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
python/qemu/machine.py +5 -3
index e66a7d66dd..120e0df3ee 100644
@@ -382,7 +382,9 @@ def wait(self):
         self._popen.wait()
         self._post_shutdown()
 
-    def shutdown(self, has_quit=False, hard=False):
+    def shutdown(self, has_quit: bool = False,
+                 hard: bool = False,
+                 timeout: Optional[int] = 3) -> None:
         """
         Terminate the VM and clean up
         """
@@ -398,10 +400,10 @@ def shutdown(self, has_quit=False, hard=False):
                 try:
                     if not has_quit:
                         self._qmp.cmd('quit')
-                    self._popen.wait(timeout=3)
+                    self._popen.wait(timeout=timeout)
                 except:
                     self._popen.kill()
-            self._popen.wait()
+            self._popen.wait(timeout=timeout)
 
         self._post_shutdown()
 
Signature · 1 line
2.21.3
author
#21 lore raw

PATCH v5 7/12 python/machine.py: Make wait() call shutdown()

In reply to John Snow

At this point, shutdown(has_quit=True) and wait() do essentially the same thing; they perform cleanup without actually instructing QEMU to quit.

Define one in terms of the other.

---
 python/qemu/machine.py | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
python/qemu/machine.py +9 -8
index 120e0df3ee..aaa173f046 100644
@@ -374,14 +374,6 @@ def _early_cleanup(self) -> None:
             self._console_socket.close()
             self._console_socket = None
 
-    def wait(self):
-        """
-        Wait for the VM to power off
-        """
-        self._early_cleanup()
-        self._popen.wait()
-        self._post_shutdown()
-
     def shutdown(self, has_quit: bool = False,
                  hard: bool = False,
                  timeout: Optional[int] = 3) -> None:
@@ -410,6 +402,15 @@ def shutdown(self, has_quit: bool = False,
     def kill(self):
         self.shutdown(hard=True)
 
+    def wait(self, timeout: Optional[int] = None) -> None:
+        """
+        Wait for the VM to power off and perform post-shutdown cleanup.
+
+        :param timeout: Optional timeout in seconds.
+                        Default None, an infinite wait.
+        """
+        self.shutdown(has_quit=True, timeout=timeout)
+
     def set_qmp_monitor(self, enabled=True):
         """
         Set the QMP monitor.
Signature · 1 line
2.21.3
author
#24 lore raw

PATCH v5 8/12 tests/acceptance: wait() instead of shutdown() where appropriate

In reply to John Snow

When issuing 'reboot' to a VM with the no-reboot option, that VM will exit. When then issuing a shutdown command, the cleanup may race.

Add calls to vm.wait() which will gracefully mark the VM as having exited. Subsequent vm.shutdown() calls in generic tearDown code will not race when called after completion of the call.

---
 tests/acceptance/boot_linux_console.py   | 10 ++++++++++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 2 files changed, 12 insertions(+)
tests/acceptance/boot_linux_console.py +10 -0
index 3d02519660..5867ef760c 100644
@@ -191,6 +191,8 @@ def test_mips_malta_cpio(self):
                                                 'Debian')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # Wait for VM to shut down gracefully
+        self.vm.wait()
 
     @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_mips64el_malta_5KEc_cpio(self):
@@ -231,6 +233,8 @@ def test_mips64el_malta_5KEc_cpio(self):
                                                 '3.19.3.mtoman.20150408')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # Wait for VM to shut down gracefully
+        self.vm.wait()
 
     def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
         kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
@@ -506,6 +510,7 @@ def test_arm_cubieboard_initrd(self):
                                                 'system-control@1c00000')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
 
     def test_arm_cubieboard_sata(self):
         """
@@ -550,6 +555,7 @@ def test_arm_cubieboard_sata(self):
                                                 'sda')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
 
     def test_arm_orangepi(self):
         """
@@ -615,6 +621,8 @@ def test_arm_orangepi_initrd(self):
                                                 'system-control@1c00000')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # Wait for VM to shut down gracefully
+        self.vm.wait()
 
     def test_arm_orangepi_sd(self):
         """
@@ -662,6 +670,8 @@ def test_arm_orangepi_sd(self):
             '3 packets transmitted, 3 packets received, 0% packet loss')
         exec_command_and_wait_for_pattern(self, 'reboot',
                                                 'reboot: Restarting system')
+        # Wait for VM to shut down gracefully
+        self.vm.wait()
 
     @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
     @skipUnless(P7ZIP_AVAILABLE, '7z not installed')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index 90d7f2f167..25c5c5f741 100644
@@ -212,6 +212,8 @@ def check_mips_malta(self, uname_m, endianess):
 
         self.run_common_commands(wordsize)
         self.shutdown_via_ssh()
+        # Wait for VM to shut down gracefully
+        self.vm.wait()
 
     def test_mips_malta32eb_kernel3_2_0(self):
         """
Signature · 1 line
2.21.3
author
#1 lore raw

PATCH v5 1/12 python/machine.py: consolidate _post_shutdown()

In reply to John Snow

Move more cleanup actions into _post_shutdown. As a change, if QEMU should so happen to be terminated during a call to wait(), that event will now be logged.

This is not likely to occur during normative use.

---
 python/qemu/machine.py | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
python/qemu/machine.py +13 -14
index 041c615052..f7e68191c2 100644
@@ -283,6 +283,8 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        self._load_io_log()
+
         if self._qemu_log_file is not None:
             self._qemu_log_file.close()
             self._qemu_log_file = None
@@ -296,6 +298,17 @@ def _post_shutdown(self):
         while len(self._remove_files) > 0:
             self._remove_if_exists(self._remove_files.pop())
 
+        exitcode = self.exitcode()
+        if exitcode is not None and exitcode < 0:
+            msg = 'qemu received signal %i; command: "%s"'
+            if self._qemu_full_args:
+                command = ' '.join(self._qemu_full_args)
+            else:
+                command = ''
+            LOG.warning(msg, -int(exitcode), command)
+
+        self._launched = False
+
     def launch(self):
         """
         Launch the VM and make sure we cleanup and expose the
@@ -344,7 +357,6 @@ def wait(self):
         self._popen.wait()
         if self._qmp:
             self._qmp.close()
-        self._load_io_log()
         self._post_shutdown()
 
     def shutdown(self, has_quit=False, hard=False):
@@ -371,21 +383,8 @@ def shutdown(self, has_quit=False, hard=False):
                     self._popen.kill()
             self._popen.wait()
 
-        self._load_io_log()
         self._post_shutdown()
 
-        exitcode = self.exitcode()
-        if exitcode is not None and exitcode < 0 and \
-                not (exitcode == -9 and hard):
-            msg = 'qemu received signal %i: %s'
-            if self._qemu_full_args:
-                command = ' '.join(self._qemu_full_args)
-            else:
-                command = ''
-            LOG.warning(msg, -int(exitcode), command)
-
-        self._launched = False
-
     def kill(self):
         self.shutdown(hard=True)
 
Signature · 1 line
2.21.3
author
#32 lore raw

PATCH v5 10/12 python/machine.py: split shutdown into hard and soft flavors

In reply to John Snow

This is done primarily to avoid the 'bare except' pattern, which suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens, _post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful shutdown and is unable to, it will now always raise an exception to indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of exiting, wait() can be used instead of shutdown() to clean up resources instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown at all, kill() can be used instead.

Handling shutdown in this way makes it much more explicit which type of shutdown we want and allows the library to report problems with this process.

---
 python/qemu/machine.py | 95 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 80 insertions(+), 15 deletions(-)
python/qemu/machine.py +80 -15
index aaa173f046..b24ce8a268 100644
@@ -48,6 +48,12 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 
+class AbnormalShutdown(QEMUMachineError):
+    """
+    Exception raised when a graceful shutdown was requested, but not performed.
+    """
+
+
 class MonitorResponseError(qmp.QMPError):
     """
     Represents erroneous QMP monitor reply
@@ -365,6 +371,7 @@ def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
 
+        May be invoked by both soft and hard shutdown in failover scenarios.
         Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
@@ -374,32 +381,90 @@ def _early_cleanup(self) -> None:
             self._console_socket.close()
             self._console_socket = None
 
+    def _hard_shutdown(self) -> None:
+        """
+        Perform early cleanup, kill the VM, and wait for it to terminate.
+
+        :raise subprocess.Timeout: When timeout is exceeds 60 seconds
+            waiting for the QEMU process to terminate.
+        """
+        self._early_cleanup()
+        self._popen.kill()
+        self._popen.wait(timeout=60)
+
+    def _soft_shutdown(self, has_quit: bool = False,
+                       timeout: Optional[int] = 3) -> None:
+        """
+        Perform early cleanup, attempt to gracefully shut down the VM, and wait
+        for it to terminate.
+
+        :param has_quit: When True, don't attempt to issue 'quit' QMP command
+        :param timeout: Optional timeout in seconds for graceful shutdown.
+                        Default 3 seconds, A value of None is an infinite wait.
+
+        :raise ConnectionReset: On QMP communication errors
+        :raise subprocess.TimeoutExpired: When timeout is exceeded waiting for
+            the QEMU process to terminate.
+        """
+        self._early_cleanup()
+
+        if self._qmp is not None:
+            if not has_quit:
+                # Might raise ConnectionReset
+                self._qmp.cmd('quit')
+
+        # May raise subprocess.TimeoutExpired
+        self._popen.wait(timeout=timeout)
+
+    def _do_shutdown(self, has_quit: bool = False,
+                     timeout: Optional[int] = 3) -> None:
+        """
+        Attempt to shutdown the VM gracefully; fallback to a hard shutdown.
+
+        :param has_quit: When True, don't attempt to issue 'quit' QMP command
+        :param timeout: Optional timeout in seconds for graceful shutdown.
+                        Default 3 seconds, A value of None is an infinite wait.
+
+        :raise AbnormalShutdown: When the VM could not be shut down gracefully.
+            The inner exception will likely be ConnectionReset or
+            subprocess.TimeoutExpired. In rare cases, non-graceful termination
+            may result in its own exceptions, likely subprocess.TimeoutExpired.
+        """
+        try:
+            self._soft_shutdown(has_quit, timeout)
+        except Exception as exc:
+            self._hard_shutdown()
+            raise AbnormalShutdown("Could not perform graceful shutdown") \
+                from exc
+
     def shutdown(self, has_quit: bool = False,
                  hard: bool = False,
                  timeout: Optional[int] = 3) -> None:
         """
-        Terminate the VM and clean up
+        Terminate the VM (gracefully if possible) and perform cleanup.
+        Cleanup will always be performed.
+
+        :param has_quit: When true, do not attempt to issue 'quit' QMP command.
+        :param hard: When true, do not attempt graceful shutdown, and
+                     suppress the SIGKILL warning log message.
+        :param timeout: Optional timeout in seconds for graceful shutdown.
+                        Default 3 seconds, A value of None is an infinite wait.
         """
         if not self._launched:
             return
 
-        self._early_cleanup()
-
-        if self.is_running():
+        try:
             if hard:
-                self._popen.kill()
-            elif self._qmp:
-                try:
-                    if not has_quit:
-                        self._qmp.cmd('quit')
-                    self._popen.wait(timeout=timeout)
-                except:
-                    self._popen.kill()
-            self._popen.wait(timeout=timeout)
-
-        self._post_shutdown()
+                self._hard_shutdown()
+            else:
+                self._do_shutdown(has_quit, timeout=timeout)
+        finally:
+            self._post_shutdown()
 
     def kill(self):
+        """
+        Terminate the VM forcefully, wait for it to exit, and perform cleanup.
+        """
         self.shutdown(hard=True)
 
     def wait(self, timeout: Optional[int] = None) -> None:
Signature · 1 line
2.21.3
author
#27 lore raw

PATCH v5 9/12 tests/acceptance: Don't test reboot on cubieboard

In reply to John Snow

cubieboard does not have a functioning reboot, it halts and QEMU does not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant of race conditions on shutdown; tests should consciously decide to WAIT or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would be to WAIT for the VM to exit. However, since that's broken, we should SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown, however, if anyone fixes cubieboard reboot in the future, this test will develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is still running when the test concludes, where the test teardown will do the right thing.

---
 tests/acceptance/boot_linux_console.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
tests/acceptance/boot_linux_console.py +2 -6
index 5867ef760c..8b8b828bc5 100644
@@ -508,9 +508,7 @@ def test_arm_cubieboard_initrd(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
                                                 'system-control@1c00000')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_cubieboard_sata(self):
         """
@@ -553,9 +551,7 @@ def test_arm_cubieboard_sata(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/partitions',
                                                 'sda')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_orangepi(self):
         """
Signature · 1 line
2.21.3
author
#37 lore raw

PATCH v5 11/12 python/machine.py: re-add sigkill warning suppression

In reply to John Snow

If the user kills QEMU on purpose, we don't need to warn them about that having happened: they know already.

---
 python/qemu/machine.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
python/qemu/machine.py +6 -1
index b24ce8a268..02d66e3cff 100644
@@ -22,6 +22,7 @@
 import os
 import subprocess
 import shutil
+import signal
 import socket
 import tempfile
 from typing import Optional, Type
@@ -128,6 +129,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         self._console_address = None
         self._console_socket = None
         self._remove_files = []
+        self._user_killed = False
 
     def __enter__(self):
         return self
@@ -316,7 +318,8 @@ def _post_shutdown(self):
             self._remove_if_exists(self._remove_files.pop())
 
         exitcode = self.exitcode()
-        if exitcode is not None and exitcode < 0:
+        if (exitcode is not None and exitcode < 0
+                and not (self._user_killed and exitcode == -signal.SIGKILL)):
             msg = 'qemu received signal %i; command: "%s"'
             if self._qemu_full_args:
                 command = ' '.join(self._qemu_full_args)
@@ -324,6 +327,7 @@ def _post_shutdown(self):
                 command = ''
             LOG.warning(msg, -int(exitcode), command)
 
+        self._user_killed = False
         self._launched = False
 
     def launch(self):
@@ -455,6 +459,7 @@ def shutdown(self, has_quit: bool = False,
 
         try:
             if hard:
+                self._user_killed = True
                 self._hard_shutdown()
             else:
                 self._do_shutdown(has_quit, timeout=timeout)
Signature · 1 line
2.21.3
author
#40 lore raw

PATCH v5 12/12 python/machine.py: change default wait timeout to 3 seconds

In reply to John Snow

Machine.wait() does not appear to be used except in the acceptance tests, and an infinite timeout by default in a test suite is not the most helpful.

Change it to 3 seconds, like the default shutdown timeout.

---
 python/qemu/machine.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
python/qemu/machine.py +2 -2
index 02d66e3cff..d08a8e4a6e 100644
@@ -472,12 +472,12 @@ def kill(self):
         """
         self.shutdown(hard=True)
 
-    def wait(self, timeout: Optional[int] = None) -> None:
+    def wait(self, timeout: Optional[int] = 3) -> None:
         """
         Wait for the VM to power off and perform post-shutdown cleanup.
 
         :param timeout: Optional timeout in seconds.
-                        Default None, an infinite wait.
+                        Default 3 seconds, A value of None is an infinite wait.
         """
         self.shutdown(has_quit=True, timeout=timeout)
 
Signature · 1 line
2.21.3
Reviewed-by on v5 2/12
#5 lore raw

Re: [PATCH v5 02/12] python/machine.py: Close QMP socket in cleanup

In reply to John Snow
Quoted 8 lines · John Snow

It's not important to do this before waiting for the process to exit, so it can be done during generic post-shutdown cleanup.

---
 python/qemu/machine.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Reviewed-by on v5 5/12
#14 lore raw

Re: [PATCH v5 05/12] python/machine.py: Prohibit multiple shutdown() calls

In reply to John Snow
Quoted 4 lines · John Snow

If the VM is not launched, don't try to shut it down. As a change, _post_shutdown now unconditionally also calls _early_cleanup in order to offer comprehensive object cleanup in failure cases.

Quoted 9 lines

As a courtesy, treat it as a NOP instead of rejecting it as an error. This is slightly nicer for acceptance tests where vm.shutdown() is issued unconditionally in tearDown callbacks.

---
 python/qemu/machine.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
Reviewed-by on v5 6/12
#19 lore raw

Re: [PATCH v5 06/12] python/machine.py: Add a configurable timeout to shutdown()

In reply to John Snow
Quoted 8 lines · John Snow

Three seconds is hardcoded. Use it as a default parameter instead, and use that value for both waits that may occur in the function.

---
 python/qemu/machine.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
Reviewed-by on v5 7/12
#22 lore raw

Re: [PATCH v5 07/12] python/machine.py: Make wait() call shutdown()

In reply to John Snow
Quoted 11 lines · John Snow

At this point, shutdown(has_quit=True) and wait() do essentially the same thing; they perform cleanup without actually instructing QEMU to quit.

Define one in terms of the other.

---
 python/qemu/machine.py | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Reviewed-by on v5 11/12
#38 lore raw

Re: [PATCH v5 11/12] python/machine.py: re-add sigkill warning suppression

In reply to John Snow
Quoted 8 lines · John Snow

If the user kills QEMU on purpose, we don't need to warn them about that having happened: they know already.

---
 python/qemu/machine.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

Good idea :)

Reviewed-by on v5 12/12
#41 lore raw

Re: [PATCH v5 12/12] python/machine.py: change default wait timeout to 3 seconds

In reply to John Snow
Quoted 20 lines · John Snow

Machine.wait() does not appear to be used except in the acceptance tests, and an infinite timeout by default in a test suite is not the most helpful.

Change it to 3 seconds, like the default shutdown timeout.

---
 python/qemu/machine.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
python/qemu/machine.py +0 -1
index 02d66e3cff..d08a8e4a6e 100644
@@ -472,12 +472,12 @@ def kill(self):
         """
         self.shutdown(hard=True)
 
-    def wait(self, timeout: Optional[int] = None) -> None:

python/qemu/machine.py def kill(self): · line 483

Quoted 10 lines
python/qemu/machine.py +2 -1
+    def wait(self, timeout: Optional[int] = 3) -> None:
         """
         Wait for the VM to power off and perform post-shutdown cleanup.
 
         :param timeout: Optional timeout in seconds.
-                        Default None, an infinite wait.
+                        Default 3 seconds, A value of None is an infinite wait.
         """
         self.shutdown(has_quit=True, timeout=timeout)
 
Reviewed-by on v5 10/12
#33 lore raw

Re: [PATCH v5 10/12] python/machine.py: split shutdown into hard and soft flavors

In reply to John Snow
Quoted 25 lines · John Snow

This is done primarily to avoid the 'bare except' pattern, which suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens, _post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful shutdown and is unable to, it will now always raise an exception to indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of exiting, wait() can be used instead of shutdown() to clean up resources instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown at all, kill() can be used instead.

Quoted 9 lines

Handling shutdown in this way makes it much more explicit which type of shutdown we want and allows the library to report problems with this process.

---
 python/qemu/machine.py | 95 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 80 insertions(+), 15 deletions(-)

:))

Reviewed-by on v5 9/12
#28 lore raw

Re: [PATCH v5 09/12] tests/acceptance: Don't test reboot on cubieboard

In reply to John Snow
Quoted 41 lines · John Snow

cubieboard does not have a functioning reboot, it halts and QEMU does not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant of race conditions on shutdown; tests should consciously decide to WAIT or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would be to WAIT for the VM to exit. However, since that's broken, we should SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown, however, if anyone fixes cubieboard reboot in the future, this test will develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is still running when the test concludes, where the test teardown will do the right thing.

---
 tests/acceptance/boot_linux_console.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
tests/acceptance/boot_linux_console.py +1 -3
index 5867ef760c..8b8b828bc5 100644
@@ -508,9 +508,7 @@ def test_arm_cubieboard_initrd(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
                                                 'system-control@1c00000')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_cubieboard_sata(self):
         """

tests/acceptance/boot_linux_console.py def test_arm_cubieboard_sata(self): · line 557

Quoted 10 lines
tests/acceptance/boot_linux_console.py +1 -3
@@ -553,9 +551,7 @@ def test_arm_cubieboard_sata(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/partitions',
                                                 'sda')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_orangepi(self):
         """

Note, if I do the pull request, I might reorder this one before the previous one "tests/acceptance: wait() instead of shutdown() where appropriate".

Reviewed-by on v5 8/12
#25 lore raw

Re: [PATCH v5 08/12] tests/acceptance: wait() instead of shutdown() where appropriate

In reply to John Snow
Quoted 7 lines · John Snow

When issuing 'reboot' to a VM with the no-reboot option, that VM will exit. When then issuing a shutdown command, the cleanup may race.

Add calls to vm.wait() which will gracefully mark the VM as having exited. Subsequent vm.shutdown() calls in generic tearDown code will not race when called after completion of the call.

Quoted 6 lines
---
 tests/acceptance/boot_linux_console.py   | 10 ++++++++++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 2 files changed, 12 insertions(+)
Reviewed-by Tested-by on v5 1/12
#2 lore raw

Re: [PATCH v5 01/12] python/machine.py: consolidate _post_shutdown()

In reply to John Snow
Quoted 43 lines · John Snow

Move more cleanup actions into _post_shutdown. As a change, if QEMU should so happen to be terminated during a call to wait(), that event will now be logged.

This is not likely to occur during normative use.

---
 python/qemu/machine.py | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
python/qemu/machine.py +13 -0
index 041c615052..f7e68191c2 100644
@@ -283,6 +283,8 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        self._load_io_log()
+
         if self._qemu_log_file is not None:
             self._qemu_log_file.close()
             self._qemu_log_file = None
@@ -296,6 +298,17 @@ def _post_shutdown(self):
         while len(self._remove_files) > 0:
             self._remove_if_exists(self._remove_files.pop())
 
+        exitcode = self.exitcode()
+        if exitcode is not None and exitcode < 0:
+            msg = 'qemu received signal %i; command: "%s"'
+            if self._qemu_full_args:
+                command = ' '.join(self._qemu_full_args)
+            else:
+                command = ''
+            LOG.warning(msg, -int(exitcode), command)
+
+        self._launched = False
+
     def launch(self):
         """
         Launch the VM and make sure we cleanup and expose the

python/qemu/machine.py def wait(self): · line 347

Quoted 4 lines
python/qemu/machine.py +0 -1
@@ -344,7 +357,6 @@ def wait(self):
         self._popen.wait()
         if self._qmp:
             self._qmp.close()
-        self._load_io_log()

Nice consolidation of responsibilities!

author
#29 lore raw

Re: [PATCH v5 09/12] tests/acceptance: Don't test reboot on cubieboard

In reply to Philippe Mathieu-Daudé
Quoted 55 lines · Philippe Mathieu-Daudé
Quoted 41 lines · John Snow

cubieboard does not have a functioning reboot, it halts and QEMU does not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant of race conditions on shutdown; tests should consciously decide to WAIT or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would be to WAIT for the VM to exit. However, since that's broken, we should SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown, however, if anyone fixes cubieboard reboot in the future, this test will develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is still running when the test concludes, where the test teardown will do the right thing.

---
 tests/acceptance/boot_linux_console.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
tests/acceptance/boot_linux_console.py +1 -3
index 5867ef760c..8b8b828bc5 100644
@@ -508,9 +508,7 @@ def test_arm_cubieboard_initrd(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
                                                 'system-control@1c00000')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_cubieboard_sata(self):
         """

tests/acceptance/boot_linux_console.py def test_arm_cubieboard_sata(self): · line 557

Quoted 10 lines
tests/acceptance/boot_linux_console.py +1 -3
@@ -553,9 +551,7 @@ def test_arm_cubieboard_sata(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/partitions',
                                                 'sda')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_orangepi(self):
         """
Quoted 4 lines

Note, if I do the pull request, I might reorder this one before the previous one "tests/acceptance: wait() instead of shutdown() where appropriate".

you could -- in practice it didn't seem to matter. I tested both with and without this patch.

I was just trying to isolate each intentional semantic change as its own commit so it could be observed/understood/debated.

--js

#30 lore raw

Re: [PATCH v5 09/12] tests/acceptance: Don't test reboot on cubieboard

Quoted 63 lines · John Snow
Quoted 55 lines · Philippe Mathieu-Daudé
Quoted 41 lines · John Snow

cubieboard does not have a functioning reboot, it halts and QEMU does not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant of race conditions on shutdown; tests should consciously decide to WAIT or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would be to WAIT for the VM to exit. However, since that's broken, we should SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown, however, if anyone fixes cubieboard reboot in the future, this test will develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is still running when the test concludes, where the test teardown will do the right thing.

---
 tests/acceptance/boot_linux_console.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
tests/acceptance/boot_linux_console.py +1 -3
index 5867ef760c..8b8b828bc5 100644
@@ -508,9 +508,7 @@ def test_arm_cubieboard_initrd(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
                                                 'system-control@1c00000')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_cubieboard_sata(self):
         """

tests/acceptance/boot_linux_console.py def test_arm_cubieboard_sata(self): · line 557

Quoted 10 lines
tests/acceptance/boot_linux_console.py +1 -3
@@ -553,9 +551,7 @@ def test_arm_cubieboard_sata(self):
                                                 'Allwinner sun4i/sun5i')
         exec_command_and_wait_for_pattern(self, 'cat /proc/partitions',
                                                 'sda')
-        exec_command_and_wait_for_pattern(self, 'reboot',
-                                                'reboot: Restarting system')
-        # NB: Do not issue vm.wait() here, cubieboard's reboot does not exit!
+        # cubieboard's reboot is not functioning; omit reboot test.
 
     def test_arm_orangepi(self):
         """
Quoted 4 lines

Note, if I do the pull request, I might reorder this one before the previous one "tests/acceptance: wait() instead of shutdown() where appropriate".

Quoted 5 lines

you could -- in practice it didn't seem to matter. I tested both with and without this patch.

I was just trying to isolate each intentional semantic change as its own commit so it could be observed/understood/debated.

As both patches are correct, there is no need to debate IMO :) I'm fine either way. The simpler the easier, and here the simpler is to take your series as it.

Reviewed-by Tested-by on v5 2/12
#6 lore raw

Re: [PATCH v5 02/12] python/machine.py: Close QMP socket in cleanup

In reply to John Snow
Quoted 43 lines · John Snow

It's not important to do this before waiting for the process to exit, so it can be done during generic post-shutdown cleanup.

---
 python/qemu/machine.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
python/qemu/machine.py +4 -3
index f7e68191c2..938c891b1d 100644
@@ -283,6 +283,10 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        if self._qmp:
+            self._qmp.close()
+            self._qmp = None
+
         self._load_io_log()
 
         if self._qemu_log_file is not None:
@@ -355,8 +359,6 @@ def wait(self):
         Wait for the VM to power off
         """
         self._popen.wait()
-        if self._qmp:
-            self._qmp.close()
         self._post_shutdown()
 
     def shutdown(self, has_quit=False, hard=False):
@@ -377,7 +379,6 @@ def shutdown(self, has_quit=False, hard=False):
                 try:
                     if not has_quit:
                         self._qmp.cmd('quit')
-                    self._qmp.close()
                     self._popen.wait(timeout=3)
                 except:
                     self._popen.kill()
Signature · 1 line
2.21.3
Reviewed-by on v5 3/12
#8 lore raw

Re: [PATCH v5 03/12] python/machine.py: Add _early_cleanup hook

In reply to John Snow
Quoted 7 lines · John Snow

Some parts of cleanup need to occur prior to shutdown, otherwise shutdown might break. Move this into a suitably named method/callback.

---
 python/qemu/machine.py | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
Reviewed-by on v5 1/12
#3 lore raw

Re: [PATCH v5 01/12] python/machine.py: consolidate _post_shutdown()

In reply to John Snow
Quoted 10 lines · John Snow

Move more cleanup actions into _post_shutdown. As a change, if QEMU should so happen to be terminated during a call to wait(), that event will now be logged.

This is not likely to occur during normative use.

---
 python/qemu/machine.py | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
Reviewed-by on v5 4/12
#11 lore raw

Re: [PATCH v5 04/12] python/machine.py: Perform early cleanup for wait() calls, too

In reply to John Snow
Quoted 15 lines · John Snow

This is primarily for consistency, and is a step towards wait() and shutdown() sharing the same implementation so that the two cleanup paths cannot diverge.

---
 python/qemu/machine.py | 1 +
 1 file changed, 1 insertion(+)
python/qemu/machine.py +0 -0
index 4280aab380..cac466fbe6 100644

python/qemu/machine.py def wait(self): · line 375

Quoted 7 lines
python/qemu/machine.py +1 -0
@@ -369,6 +369,7 @@ def wait(self):
         """
         Wait for the VM to power off
         """
+        self._early_cleanup()
         self._popen.wait()
         self._post_shutdown()
 
Reviewed-by Tested-by on v5 3/12
#9 lore raw

Re: [PATCH v5 03/12] python/machine.py: Add _early_cleanup hook

In reply to John Snow
Quoted 13 lines · John Snow

Some parts of cleanup need to occur prior to shutdown, otherwise shutdown might break. Move this into a suitably named method/callback.

---
 python/qemu/machine.py | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
python/qemu/machine.py +0 -0
index 938c891b1d..4280aab380 100644

python/qemu/machine.py def _launch(self): · line 357

Quoted 5 lines
python/qemu/machine.py +1 -1
@@ -354,16 +354,9 @@ def _launch(self):
                                        close_fds=False)
         self._post_launch()
 
-    def wait(self):
+    def _early_cleanup(self) -> None:

Inaugurating type hints "around here", huh? :)

WRT method name, up to the end of this series, this method is only about closing the console socket, so *maybe* just name it accordingly and move on to an abstract method name when/if the need arises?

Either way,

Reviewed-by Tested-by on v5 4/12
#12 lore raw

Re: [PATCH v5 04/12] python/machine.py: Perform early cleanup for wait() calls, too

In reply to John Snow
Quoted 9 lines · John Snow

This is primarily for consistency, and is a step towards wait() and shutdown() sharing the same implementation so that the two cleanup paths cannot diverge.

---
 python/qemu/machine.py | 1 +
 1 file changed, 1 insertion(+)
#15 lore raw

Re: [PATCH v5 05/12] python/machine.py: Prohibit multiple shutdown() calls

In reply to John Snow
Quoted 18 lines · John Snow

If the VM is not launched, don't try to shut it down. As a change, _post_shutdown now unconditionally also calls _early_cleanup in order to offer comprehensive object cleanup in failure cases.

As a courtesy, treat it as a NOP instead of rejecting it as an error. This is slightly nicer for acceptance tests where vm.shutdown() is issued unconditionally in tearDown callbacks.

---
 python/qemu/machine.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
python/qemu/machine.py +0 -0
index cac466fbe6..e66a7d66dd 100644

python/qemu/machine.py def _post_launch(self): · line 292

Quoted 10 lines
python/qemu/machine.py +7 -0
@@ -283,6 +283,13 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        """
+        Called to cleanup the VM instance after the process has exited.
+        May also be called after a failed launch.
+        """
+        # Comprehensive reset for the failed launch case:
+        self._early_cleanup()
+

I'm not following why this is needed here. AFAICT, the closing of the console socket file was introduced when the QEMU process was alive, and doing I/O on the serial device attached to the console file (and was only necessary because of that). In those scenarios, a race between the time of sending the QMP command to quit, and getting a response from QMP could occur.

But here, IIUC, there's no expectations for the QEMU process to be alive. To the best of my understanding and testing, this line did not contribute to the robustness of the shutdown behavior.

Finally, given that currently, only the closing of the console socket is done within _early_cleanup(), and that is conditional, this also does no harm AFAICT. If more early cleanups operations were needed, then I would feel less bothered about calling it here.

Quoted 22 lines
python/qemu/machine.py +3 -1
         if self._qmp:
             self._qmp.close()
             self._qmp = None
@@ -328,7 +335,7 @@ def launch(self):
             self._launch()
             self._launched = True
         except:
-            self.shutdown()
+            self._post_shutdown()
 
             LOG.debug('Error launching VM')
             if self._qemu_full_args:
@@ -357,6 +364,8 @@ def _launch(self):
     def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
+
+        Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
         # for QEMU to exit, while QEMU is waiting for the socket to

python/qemu/machine.py def shutdown(self, has_quit=False, hard=False): · line 391

Quoted 6 lines
python/qemu/machine.py +3 -0
@@ -377,6 +386,9 @@ def shutdown(self, has_quit=False, hard=False):
         """
         Terminate the VM and clean up
         """
+        if not self._launched:
+            return
+

Because of the additional logic, it'd be a good opportunity to make the docstring more accurate. This method may now *not* do *any* of if termination and cleaning (while previously it would attempt those anyhow).

- Cleber.

Reviewed-by on v5 6/12
#20 lore raw

Re: [PATCH v5 06/12] python/machine.py: Add a configurable timeout to shutdown()

In reply to John Snow
Quoted 7 lines · John Snow

Three seconds is hardcoded. Use it as a default parameter instead, and use that value for both waits that may occur in the function.

---
 python/qemu/machine.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
Reviewed-by Tested-by on v5 7/12
#23 lore raw

Re: [PATCH v5 07/12] python/machine.py: Make wait() call shutdown()

In reply to John Snow
Quoted 11 lines · John Snow

At this point, shutdown(has_quit=True) and wait() do essentially the same thing; they perform cleanup without actually instructing QEMU to quit.

Define one in terms of the other.

---
 python/qemu/machine.py | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Reviewed-by Tested-by on v5 8/12
#26 lore raw

Re: [PATCH v5 08/12] tests/acceptance: wait() instead of shutdown() where appropriate

In reply to John Snow
Quoted 7 lines · John Snow

When issuing 'reboot' to a VM with the no-reboot option, that VM will exit. When then issuing a shutdown command, the cleanup may race.

Add calls to vm.wait() which will gracefully mark the VM as having exited. Subsequent vm.shutdown() calls in generic tearDown code will not race when called after completion of the call.

Quoted 6 lines
---
 tests/acceptance/boot_linux_console.py   | 10 ++++++++++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 2 files changed, 12 insertions(+)
Reviewed-by Tested-by on v5 9/12
#31 lore raw

Re: [PATCH v5 09/12] tests/acceptance: Don't test reboot on cubieboard

In reply to John Snow
Quoted 15 lines · John Snow

cubieboard does not have a functioning reboot, it halts and QEMU does not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant of race conditions on shutdown; tests should consciously decide to WAIT or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would be to WAIT for the VM to exit. However, since that's broken, we should SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown, however, if anyone fixes cubieboard reboot in the future, this test will develop a new race condition that might be hard to debug.

Quoted 9 lines

Therefore: remove the reboot test and make it obvious that the VM is still running when the test concludes, where the test teardown will do the right thing.

---
 tests/acceptance/boot_linux_console.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
Reviewed-by Tested-by on v5 10/12
#34 lore raw

Re: [PATCH v5 10/12] python/machine.py: split shutdown into hard and soft flavors

In reply to John Snow
Quoted 61 lines · John Snow

This is done primarily to avoid the 'bare except' pattern, which suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens, _post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful shutdown and is unable to, it will now always raise an exception to indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of exiting, wait() can be used instead of shutdown() to clean up resources instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown at all, kill() can be used instead.

Handling shutdown in this way makes it much more explicit which type of shutdown we want and allows the library to report problems with this process.

---
 python/qemu/machine.py | 95 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 80 insertions(+), 15 deletions(-)
python/qemu/machine.py +7 -0
index aaa173f046..b24ce8a268 100644
@@ -48,6 +48,12 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 
+class AbnormalShutdown(QEMUMachineError):
+    """
+    Exception raised when a graceful shutdown was requested, but not performed.
+    """
+
+
 class MonitorResponseError(qmp.QMPError):
     """
     Represents erroneous QMP monitor reply
@@ -365,6 +371,7 @@ def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
 
+        May be invoked by both soft and hard shutdown in failover scenarios.
         Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
@@ -374,32 +381,90 @@ def _early_cleanup(self) -> None:
             self._console_socket.close()

python/qemu/machine.py def _early_cleanup(self) -> None: · line 391

Quoted 10 lines
python/qemu/machine.py +8 -0
             self._console_socket = None
 
+    def _hard_shutdown(self) -> None:
+        """
+        Perform early cleanup, kill the VM, and wait for it to terminate.
+
+        :raise subprocess.Timeout: When timeout is exceeds 60 seconds
+            waiting for the QEMU process to terminate.
+        """
+        self._early_cleanup()

Like I commented on patch 5, I don't think the *current* type of cleanup done is needed on a scenario like this...

python/qemu/machine.py def _early_cleanup(self) -> None: · line 392

Quoted 1 line
python/qemu/machine.py +1 -0
+        self._popen.kill()

... as I don't remember QEMU's SIGKILL handler to be susceptible to the race condition that motivated the closing of the console file in the first place. But, I also can not prove it's not susceptible at this time.

Note: I have some old patches that added tests for QEMUMachine itself. I intend to respin them on top of your work, so we may have a clearer understanding of the QEMU behaviors we need to handle. So, feel free to take the prudent route here, and keep the early cleanup.

Reviewed-by Tested-by on v5 12/12
#42 lore raw

Re: [PATCH v5 12/12] python/machine.py: change default wait timeout to 3 seconds

In reply to John Snow
Quoted 10 lines · John Snow

Machine.wait() does not appear to be used except in the acceptance tests, and an infinite timeout by default in a test suite is not the most helpful.

Change it to 3 seconds, like the default shutdown timeout.

---
 python/qemu/machine.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Well, for the acceptance tests, there's usually a test wide timeout, but this is indeed a good idea!

author
#16 lore raw

Re: [PATCH v5 05/12] python/machine.py: Prohibit multiple shutdown() calls

In reply to Cleber Rosa
Quoted 37 lines · Cleber Rosa
Quoted 18 lines · John Snow

If the VM is not launched, don't try to shut it down. As a change, _post_shutdown now unconditionally also calls _early_cleanup in order to offer comprehensive object cleanup in failure cases.

As a courtesy, treat it as a NOP instead of rejecting it as an error. This is slightly nicer for acceptance tests where vm.shutdown() is issued unconditionally in tearDown callbacks.

---
 python/qemu/machine.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
python/qemu/machine.py +0 -0
index cac466fbe6..e66a7d66dd 100644

python/qemu/machine.py def _post_launch(self): · line 292

Quoted 10 lines
python/qemu/machine.py +7 -0
@@ -283,6 +283,13 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        """
+        Called to cleanup the VM instance after the process has exited.
+        May also be called after a failed launch.
+        """
+        # Comprehensive reset for the failed launch case:
+        self._early_cleanup()
+

I'm not following why this is needed here. AFAICT, the closing of the console socket file was introduced when the QEMU process was alive, and doing I/O on the serial device attached to the console file (and was only necessary because of that). In those scenarios, a race between the time of sending the QMP command to quit, and getting a response from QMP could occur.

Quoted 9 lines

But here, IIUC, there's no expectations for the QEMU process to be alive. To the best of my understanding and testing, this line did not contribute to the robustness of the shutdown behavior.

Finally, given that currently, only the closing of the console socket is done within _early_cleanup(), and that is conditional, this also does no harm AFAICT. If more early cleanups operations were needed, then I would feel less bothered about calling it here.

Sure, I can tie it up to be a little less abstract, but I'll go over some of my reasons below to see if any of them resonate for you.

What I wanted was three things:

(1) A single call that comprehensively cleaned up the VM object back to a known state, no matter what state it was in.

This is the answer to your first paragraph: I wanted a function to always thoroughly reset an object back to base state. In the current code, we won't have a console socket object here yet.

I felt this was good as it illustrates to other contributors what the comprehensive steps to reset the object are.

(It might be considered bad in that it does perform potentially unnecessary cleanup at times; but as a matter of taste I prefer this to conditional inconsistency.)

(2) A guarantee that no matter how a VM object was terminated (either via context handler, shutdown(), or wait()) that the exact same cleanup steps would be performed for consistency

This is why wait() now points to shutdown(), and why _post_shutdown is called outside of either of the shutdown handlers. No matter what, _post_shutdown is getting called and it is going to clean everything.

(3) Avoiding special-casing certain cleanup actions open-coded in the shutdown handlers.

It's #3 that wound up with me creating the "_early_cleanup" hook. I created it because I had the idea to create "Console" and "QMP Socket" as inheritable mixins to help keep the core class simpler.

In these cases, the mixins need a hook for "early cleanup."

No, I didn't finish that work and it's not ready. We could remove it for now, but that's why this is here and why it's named like it is.

Long term, I think Machine Mixins are the only viable way to add more features. The upstream version of the code now has more than 10 arguments and 20+ fields. It's getting unwieldy.

(....all that said, we can still do it later. If you prefer, for now, I can at least rename the function "_close_socket" to make it less abstract.

(I think it should still be called in _post_shutdown, though.))

Quoted 25 lines
Quoted 25 lines
python/qemu/machine.py +3 -1
         if self._qmp:
             self._qmp.close()
             self._qmp = None
@@ -328,7 +335,7 @@ def launch(self):
             self._launch()
             self._launched = True
         except:
-            self.shutdown()
+            self._post_shutdown()
 
             LOG.debug('Error launching VM')
             if self._qemu_full_args:
@@ -357,6 +364,8 @@ def _launch(self):
     def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
+
+        Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
         # for QEMU to exit, while QEMU is waiting for the socket to
@@ -377,6 +386,9 @@ def shutdown(self, has_quit=False, hard=False):
         """
         Terminate the VM and clean up
         """
Quoted 9 lines

python/qemu/machine.py def shutdown(self, has_quit=False, hard=False): · line 391

Quoted 3 lines
python/qemu/machine.py +3 -0
+        if not self._launched:
+            return
+

Because of the additional logic, it'd be a good opportunity to make the docstring more accurate. This method may now *not* do *any* of if termination and cleaning (while previously it would attempt those anyhow).

Kinda-sorta. The old version used to have "if self.running", which could race.

I'll amend the docstring(s) as needed to make it clear that it's a NOP if the machine has been cleaned up already.

Quoted 2 lines

- Cleber.

author
#35 lore raw

Re: [PATCH v5 10/12] python/machine.py: split shutdown into hard and soft flavors

In reply to Cleber Rosa
Quoted 76 lines · Cleber Rosa
Quoted 61 lines · John Snow

This is done primarily to avoid the 'bare except' pattern, which suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens, _post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful shutdown and is unable to, it will now always raise an exception to indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of exiting, wait() can be used instead of shutdown() to clean up resources instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown at all, kill() can be used instead.

Handling shutdown in this way makes it much more explicit which type of shutdown we want and allows the library to report problems with this process.

---
 python/qemu/machine.py | 95 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 80 insertions(+), 15 deletions(-)
python/qemu/machine.py +7 -0
index aaa173f046..b24ce8a268 100644
@@ -48,6 +48,12 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 
+class AbnormalShutdown(QEMUMachineError):
+    """
+    Exception raised when a graceful shutdown was requested, but not performed.
+    """
+
+
 class MonitorResponseError(qmp.QMPError):
     """
     Represents erroneous QMP monitor reply
@@ -365,6 +371,7 @@ def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
 
+        May be invoked by both soft and hard shutdown in failover scenarios.
         Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
@@ -374,32 +381,90 @@ def _early_cleanup(self) -> None:
             self._console_socket.close()

python/qemu/machine.py def _early_cleanup(self) -> None: · line 391

Quoted 10 lines
python/qemu/machine.py +8 -0
             self._console_socket = None
 
+    def _hard_shutdown(self) -> None:
+        """
+        Perform early cleanup, kill the VM, and wait for it to terminate.
+
+        :raise subprocess.Timeout: When timeout is exceeds 60 seconds
+            waiting for the QEMU process to terminate.
+        """
+        self._early_cleanup()

Like I commented on patch 5, I don't think the *current* type of cleanup done is needed on a scenario like this...

Quoted 7 lines

python/qemu/machine.py def _early_cleanup(self) -> None: · line 392

Quoted 1 line
python/qemu/machine.py +1 -0
+        self._popen.kill()

... as I don't remember QEMU's SIGKILL handler to be susceptible to the race condition that motivated the closing of the console file in the first place. But, I also can not prove it's not susceptible at this time.

It probably isn't. It was for consistency in when and where that hook is called, again. It does happen to be "pointless" here.

Quoted 5 lines

Note: I have some old patches that added tests for QEMUMachine itself. I intend to respin them on top of your work, so we may have a clearer understanding of the QEMU behaviors we need to handle. So, feel free to take the prudent route here, and keep the early cleanup.

Oh, adding formal tests to this folder would be incredible; especially if we wanted to package it on PyPI. Basically a necessity.

Quoted 3 lines
author
#43 lore raw

Re: [PATCH v5 12/12] python/machine.py: change default wait timeout to 3 seconds

In reply to Cleber Rosa
Quoted 12 lines · Cleber Rosa
Quoted 10 lines · John Snow

Machine.wait() does not appear to be used except in the acceptance tests, and an infinite timeout by default in a test suite is not the most helpful.

Change it to 3 seconds, like the default shutdown timeout.

---
 python/qemu/machine.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 6 lines

Well, for the acceptance tests, there's usually a test wide timeout, but this is indeed a good idea!

Yes, there's a bigger timeout for acceptance tests, but iotests doesn't have the same just yet.

In general, it helps for most of the python library methods to time out by default to prevent hangs in the various test suites.

So, anticipating that iotest callers will probably want to use wait() sooner or later, I just went ahead and made the change primarily for consistency again.

--js

author
#17 lore raw

Re: [PATCH v5 05/12] python/machine.py: Prohibit multiple shutdown() calls

In reply to Cleber Rosa
Quoted 71 lines · Cleber Rosa
Quoted 18 lines · John Snow

If the VM is not launched, don't try to shut it down. As a change, _post_shutdown now unconditionally also calls _early_cleanup in order to offer comprehensive object cleanup in failure cases.

As a courtesy, treat it as a NOP instead of rejecting it as an error. This is slightly nicer for acceptance tests where vm.shutdown() is issued unconditionally in tearDown callbacks.

---
 python/qemu/machine.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
python/qemu/machine.py +0 -0
index cac466fbe6..e66a7d66dd 100644

python/qemu/machine.py def _post_launch(self): · line 292

Quoted 10 lines
python/qemu/machine.py +7 -0
@@ -283,6 +283,13 @@ def _post_launch(self):
             self._qmp.accept()
 
     def _post_shutdown(self):
+        """
+        Called to cleanup the VM instance after the process has exited.
+        May also be called after a failed launch.
+        """
+        # Comprehensive reset for the failed launch case:
+        self._early_cleanup()
+

I'm not following why this is needed here. AFAICT, the closing of the console socket file was introduced when the QEMU process was alive, and doing I/O on the serial device attached to the console file (and was only necessary because of that). In those scenarios, a race between the time of sending the QMP command to quit, and getting a response from QMP could occur.

But here, IIUC, there's no expectations for the QEMU process to be alive. To the best of my understanding and testing, this line did not contribute to the robustness of the shutdown behavior.

Finally, given that currently, only the closing of the console socket is done within _early_cleanup(), and that is conditional, this also does no harm AFAICT. If more early cleanups operations were needed, then I would feel less bothered about calling it here.

Quoted 25 lines
python/qemu/machine.py +3 -1
         if self._qmp:
             self._qmp.close()
             self._qmp = None
@@ -328,7 +335,7 @@ def launch(self):
             self._launch()
             self._launched = True
         except:
-            self.shutdown()
+            self._post_shutdown()
 
             LOG.debug('Error launching VM')
             if self._qemu_full_args:
@@ -357,6 +364,8 @@ def _launch(self):
     def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
+
+        Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
         # for QEMU to exit, while QEMU is waiting for the socket to
@@ -377,6 +386,9 @@ def shutdown(self, has_quit=False, hard=False):
         """
         Terminate the VM and clean up
         """
Quoted 9 lines

python/qemu/machine.py def shutdown(self, has_quit=False, hard=False): · line 391

Quoted 3 lines
python/qemu/machine.py +3 -0
+        if not self._launched:
+            return
+

Because of the additional logic, it'd be a good opportunity to make the docstring more accurate. This method may now *not* do *any* of if termination and cleaning (while previously it would attempt those anyhow).

The docstring gets hit with a polish beam later in the series, but still neglects that it might do nothing.

I squashed in a change to patch #10 and pushed to my gitlab to add this clarification to the docstring.

https://gitlab.com/jsnow/qemu/-/tree/python-package-shutdown

https://gitlab.com/jsnow/qemu/-/commit/b3f14deb730fda9bb2a28a9366a8096d3617f4f4

#44 lore raw
In reply to John Snow
Quoted 33 lines · John Snow

v5: More or less rewritten.

This series is motivated by a desire to move python/qemu onto a strict mypy/pylint regime to help prevent regressions in the python codebase.

1. Remove the "bare except" pattern in the existing shutdown code, which can mask problems and make debugging difficult.

2. Ensure that post-shutdown cleanup is always performed, even when graceful termination fails.

3. Unify cleanup paths such that no matter how the VM is terminated, the same functions and steps are always taken to reset the object state.

4. Rewrite shutdown() such that any error encountered when attempting a graceful shutdown will be raised as an AbnormalShutdown exception. The pythonic idiom is to allow the caller to decide if this is a problem or not.

Previous versions of this series did not engage the fourth goal, and ran into race conditions. When I was trying to allow shutdown to succeed if QEMU was already closed, it became impossible to tell in which cases QEMU not being present was "OK" and in which cases it was evidence of a problem.

This refactoring is even more explicit. If a graceful shutdown is requested and cannot be performed, an exception /will/ be raised.

In cases where the test writer expects QEMU to already have exited, vm.wait() should be used in preference to vm.shutdown(). In cases where a graceful shutdown is not interesting or necessary to the test, vm.kill() should be used.

Quoted 13 lines

John Snow (12):
python/machine.py: consolidate _post_shutdown() python/machine.py: Close QMP socket in cleanup python/machine.py: Add _early_cleanup hook python/machine.py: Perform early cleanup for wait() calls, too python/machine.py: Prohibit multiple shutdown() calls python/machine.py: Add a configurable timeout to shutdown() python/machine.py: Make wait() call shutdown() tests/acceptance: wait() instead of shutdown() where appropriate tests/acceptance: Don't test reboot on cubieboard python/machine.py: split shutdown into hard and soft flavors python/machine.py: re-add sigkill warning suppression python/machine.py: change default wait timeout to 3 seconds

Series (finally) queued on python-next, thanks.

#36 lore raw

Re: [PATCH v5 10/12] python/machine.py: split shutdown into hard and soft flavors

In reply to John Snow
Quoted 90 lines · John Snow
Quoted 76 lines · Cleber Rosa
Quoted 61 lines · John Snow

This is done primarily to avoid the 'bare except' pattern, which suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens, _post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful shutdown and is unable to, it will now always raise an exception to indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of exiting, wait() can be used instead of shutdown() to clean up resources instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown at all, kill() can be used instead.

Handling shutdown in this way makes it much more explicit which type of shutdown we want and allows the library to report problems with this process.

---
 python/qemu/machine.py | 95 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 80 insertions(+), 15 deletions(-)
python/qemu/machine.py +7 -0
index aaa173f046..b24ce8a268 100644
@@ -48,6 +48,12 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 
+class AbnormalShutdown(QEMUMachineError):
+    """
+    Exception raised when a graceful shutdown was requested, but not performed.
+    """
+
+
 class MonitorResponseError(qmp.QMPError):
     """
     Represents erroneous QMP monitor reply
@@ -365,6 +371,7 @@ def _early_cleanup(self) -> None:
         """
         Perform any cleanup that needs to happen before the VM exits.
 
+        May be invoked by both soft and hard shutdown in failover scenarios.
         Called additionally by _post_shutdown for comprehensive cleanup.
         """
         # If we keep the console socket open, we may deadlock waiting
@@ -374,32 +381,90 @@ def _early_cleanup(self) -> None:
             self._console_socket.close()

python/qemu/machine.py def _early_cleanup(self) -> None: · line 391

Quoted 10 lines
python/qemu/machine.py +8 -0
             self._console_socket = None
 
+    def _hard_shutdown(self) -> None:
+        """
+        Perform early cleanup, kill the VM, and wait for it to terminate.
+
+        :raise subprocess.Timeout: When timeout is exceeds 60 seconds
+            waiting for the QEMU process to terminate.
+        """
+        self._early_cleanup()

Like I commented on patch 5, I don't think the *current* type of cleanup done is needed on a scenario like this...

Quoted 7 lines

python/qemu/machine.py def _early_cleanup(self) -> None: · line 392

Quoted 1 line
python/qemu/machine.py +1 -0
+        self._popen.kill()

... as I don't remember QEMU's SIGKILL handler to be susceptible to the race condition that motivated the closing of the console file in the first place. But, I also can not prove it's not susceptible at this time.

It probably isn't. It was for consistency in when and where that hook is called, again. It does happen to be "pointless" here.

Quoted 13 lines
Quoted 5 lines

Note: I have some old patches that added tests for QEMUMachine itself. I intend to respin them on top of your work, so we may have a clearer understanding of the QEMU behaviors we need to handle. So, feel free to take the prudent route here, and keep the early cleanup.

Oh, adding formal tests to this folder would be incredible; especially if we wanted to package it on PyPI. Basically a necessity.

Quoted 3 lines

Queuing with this hunk from jsnow:

-- >8 --
@@ -455,6 +444,9 @@ def shutdown(self, has_quit: bool = False,
Terminate the VM (gracefully if possible) and perform cleanup. Cleanup will always be performed.

+ If the VM has not yet been launched, or shutdown(), wait(), or
kill()
+ have already been called, this method does nothing.
+
:param has_quit: When true, do not attempt to issue 'quit' QMP
command.
:param hard: When true, do not attempt graceful shutdown, and
suppress the SIGKILL warning log message.
---