python: misc machine.py improvements
v2 cover python: misc machine.py improvements
Improve machine.py logging and fix a shutdown bug that we *probably* weren't actually hitting anywhere.
Changes for console input logging are on the way, but separate because it will touch avocado.
More comprehensive fixes for multiple socket polling are also on the way, but decidedly separate. This is the simple stuff that might make it easier to diagnose failures in the meantime.
V2:
- Adjusted messages just a pinch. (Can't leave well enough alone.)
- Added exc_info=True for a log message in the shutdown failover case. (danpb)
John Snow (2):
python/machine: Add debug logging to key state changes python/machine: Handle termination cases without QMP
python/qemu/machine/machine.py | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
Signature · 1 line
2.37.3
PATCH v2 1/2 python/machine: Add debug logging to key state changes
When key decisions are made about the lifetime of the VM process being managed, there's no log entry. Juxtaposed with the very verbose runstate change logging of the QMP module, machine seems a bit too introverted now.
Season the machine.py module with logging statements to taste to help make a tastier soup.
--- python/qemu/machine/machine.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
python/qemu/machine/machine.py +25 -0
@@ -373,6 +373,7 @@ def _post_shutdown(self) -> None: Called to cleanup the VM instance after the process has exited. May also be called after a failed launch. """ + LOG.debug("Cleaning up after VM process") try: self._close_qmp_connection() except Exception as err: # pylint: disable=broad-except
@@ -497,6 +498,7 @@ def _early_cleanup(self) -> None: # for QEMU to exit, while QEMU is waiting for the socket to # become writable. if self._console_socket is not None: + LOG.debug("Closing console socket") self._console_socket.close() self._console_socket = None
@@ -507,6 +509,7 @@ def _hard_shutdown(self) -> None: :raise subprocess.Timeout: When timeout is exceeds 60 seconds waiting for the QEMU process to terminate. """ + LOG.debug("Performing hard shutdown") self._early_cleanup() self._subp.kill() self._subp.wait(timeout=60)
@@ -523,8 +526,18 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None: :raise subprocess.TimeoutExpired: When timeout is exceeded waiting for the QEMU process to terminate. """ + LOG.debug("Attempting graceful termination") + self._early_cleanup() + if self._quit_issued: + LOG.debug( + "Anticipating QEMU termination due to prior 'quit' command, " + "or explicit call to wait()" + ) + else: + LOG.debug("Politely asking QEMU to terminate") + if self._qmp_connection: try: if not self._quit_issued:
@@ -536,6 +549,10 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None: self._close_qmp_connection() # May raise subprocess.TimeoutExpired + LOG.debug( + "Waiting (timeout=%s) for QEMU process (pid=%s) to terminate", + timeout, self._subp.pid + ) self._subp.wait(timeout=timeout) def _do_shutdown(self, timeout: Optional[int]) -> None:
@@ -553,6 +570,10 @@ def _do_shutdown(self, timeout: Optional[int]) -> None: try: self._soft_shutdown(timeout) except Exception as exc: + if isinstance(exc, subprocess.TimeoutExpired): + LOG.debug("Timed out waiting for QEMU process to exit") + LOG.debug("Graceful shutdown failed", exc_info=True) + LOG.debug("Falling back to hard shutdown") self._hard_shutdown() raise AbnormalShutdown("Could not perform graceful shutdown") \ from exc
@@ -575,6 +596,10 @@ def shutdown(self, if not self._launched: return + LOG.debug("Shutting down VM appliance; timeout=%s", timeout) + if hard: + LOG.debug("Caller requests immediate termination of QEMU process.") + try: if hard: self._user_killed = True
Signature · 1 line
2.37.3
PATCH v2 2/2 python/machine: Handle termination cases without QMP
In reply to John SnowIf we request a shutdown of a VM without a QMP console, we'll just hang waiting. Not ideal.
Add in code that attempts graceful termination in these cases. Tested lightly; it appears to work and I doubt we rely on this case anywhere, but it's a corner you're allowed to wedge yourself in, so it should be handled.
--- python/qemu/machine/machine.py | 6 ++++++ 1 file changed, 6 insertions(+)
python/qemu/machine/machine.py +6 -0
@@ -547,6 +547,12 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None: finally: # Regardless, we want to quiesce the connection. self._close_qmp_connection() + elif not self._quit_issued: + LOG.debug( + "Not anticipating QEMU quit and no QMP connection present, " + "issuing SIGTERM" + ) + self._subp.terminate() # May raise subprocess.TimeoutExpired LOG.debug(
Signature · 1 line
2.37.3