python/machine.py: refactor shutdown
v3 cover python/machine.py: refactor shutdown
v3:
- Split _post_shutdown refactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)
NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32] python//machine.py: remove bare except [PATCH 2/4] python/machine.py: remove bare except
It's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.py
John Snow (3):
python/machine.py: consolidate _post_shutdown() python/machine.py: refactor shutdown python/machine.py: re-add sigkill warning suppression
python/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)
Signature · 1 line
2.21.3
PATCH v3 1/3 python/machine.py: consolidate _post_shutdown()
Move more cleanup actions into _post_shutdown. As a change, if QEMU should so happen to be terminated during a call to wait(), that event will now be logged.
This is not likely to occur during normative use.
--- python/qemu/machine.py | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
python/qemu/machine.py +13 -14
@@ -283,6 +283,8 @@ def _post_launch(self): self._qmp.accept() def _post_shutdown(self): + self._load_io_log() + if self._qemu_log_file is not None: self._qemu_log_file.close() self._qemu_log_file = None
@@ -296,6 +298,17 @@ def _post_shutdown(self): while len(self._remove_files) > 0: self._remove_if_exists(self._remove_files.pop()) + exitcode = self.exitcode() + if exitcode is not None and exitcode < 0: + msg = 'qemu received signal %i; command: "%s"' + if self._qemu_full_args: + command = ' '.join(self._qemu_full_args) + else: + command = '' + LOG.warning(msg, -int(exitcode), command) + + self._launched = False + def launch(self): """ Launch the VM and make sure we cleanup and expose the
@@ -344,7 +357,6 @@ def wait(self): self._popen.wait() if self._qmp: self._qmp.close() - self._load_io_log() self._post_shutdown() def shutdown(self, has_quit=False, hard=False):
@@ -371,21 +383,8 @@ def shutdown(self, has_quit=False, hard=False): self._popen.kill() self._popen.wait() - self._load_io_log() self._post_shutdown() - exitcode = self.exitcode() - if exitcode is not None and exitcode < 0 and \ - not (exitcode == -9 and hard): - msg = 'qemu received signal %i: %s' - if self._qemu_full_args: - command = ' '.join(self._qemu_full_args) - else: - command = '' - LOG.warning(msg, -int(exitcode), command) - - self._launched = False - def kill(self): self.shutdown(hard=True)
Signature · 1 line
2.21.3
PATCH v3 2/3 python/machine.py: refactor shutdown
In reply to John SnowThis is done primarily to avoid the 'bare except' pattern, which suppresses ALL exceptions and not just ones that we are anticipating to see.
Replace this with a pattern that isolates the different kind of shutdown paradigms and a new fallback shutdown handler that gracefully attempts one before the other.
Ensure that the main shutdown() function ALWAYS calls the post_shutdown logic, no matter what kind of error we encountered. Subprocess wait timeouts are considered expected, everything else is unexpected.
In cases where we encounter an expected error in the graceful shutdown timeout, we will not re-raise an exception above shutdown(). Otherwise, after post_shutdown cleanup, we will.
I anticipate that this WILL lead to additional bug reports filed against this module, but that is unfortunately somewhat the point: This code shouldn't be hiding failures that exist elsewhere within the python code.
--- python/qemu/machine.py | 68 ++++++++++++++++++++++++++++++++---------- 1 file changed, 53 insertions(+), 15 deletions(-)
python/qemu/machine.py +53 -15
@@ -359,9 +359,51 @@ def wait(self): self._qmp.close() self._post_shutdown() - def shutdown(self, has_quit=False, hard=False): + def _hard_shutdown(self) -> None: """ - Terminate the VM and clean up + Kill the VM if it is running. + """ + if not self.is_running(): + return + + self._popen.kill() + self._popen.wait(timeout=60) + + def _soft_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None: + """ + Attempt to shutdown the VM gracefully if it is running. + + :param has_quit: When True, don't attempt to issue 'quit' QMP command + :param timeout: Timeout for graceful shutdown. Default 3 seconds. + """ + if not self.is_running(): + return + + if self._qmp is not None: + if not has_quit: + self._qmp.cmd('quit') + self._qmp.close() + + self._popen.wait(timeout=timeout) + + def _do_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None: + """ + Attempt to shutdown the VM gracefully; fallback to a hard shutdown. + + :param has_quit: When True, don't attempt to issue 'quit' QMP command + :param timeout: Timeout for graceful shutdown. Default 3 seconds. + """ + try: + self._soft_shutdown(has_quit, timeout) + except subprocess.TimeoutExpired: + self._hard_shutdown() + except: + self._hard_shutdown() + raise + + def shutdown(self, has_quit: bool = False, hard: bool = False) -> None: + """ + Terminate the VM (gracefully if possible) and perform cleanup. """ # If we keep the console socket open, we may deadlock waiting # for QEMU to exit, while QEMU is waiting for the socket to
@@ -370,22 +412,18 @@ def shutdown(self, has_quit=False, hard=False): self._console_socket.close() self._console_socket = None - if self.is_running(): + try: if hard: - self._popen.kill() - elif self._qmp: - try: - if not has_quit: - self._qmp.cmd('quit') - self._qmp.close() - self._popen.wait(timeout=3) - except: - self._popen.kill() - self._popen.wait() - - self._post_shutdown() + self._hard_shutdown() + else: + self._do_shutdown(has_quit) + finally: + self._post_shutdown() def kill(self): + """ + Terminate the VM forcefully and perform cleanup. + """ self.shutdown(hard=True) def set_qmp_monitor(self, enabled=True):
Signature · 1 line
2.21.3
PATCH v3 3/3 python/machine.py: re-add sigkill warning suppression
In reply to John SnowIf the user kills QEMU on purpose, we don't need to warn them about that having happened: they know already.
--- python/qemu/machine.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
python/qemu/machine.py +7 -2
@@ -22,6 +22,7 @@ import os import subprocess import shutil +import signal import socket import tempfile from typing import Optional, Type
@@ -122,6 +123,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._console_address = None self._console_socket = None self._remove_files = [] + self._killed = False def __enter__(self): return self
@@ -282,7 +284,7 @@ def _post_launch(self): if self._qmp: self._qmp.accept() - def _post_shutdown(self): + def _post_shutdown(self) -> None: self._load_io_log() if self._qemu_log_file is not None:
@@ -299,7 +301,8 @@ def _post_shutdown(self): self._remove_if_exists(self._remove_files.pop()) exitcode = self.exitcode() - if exitcode is not None and exitcode < 0: + if (exitcode is not None and exitcode < 0 + and not (self._killed and exitcode == -signal.SIGKILL)): msg = 'qemu received signal %i; command: "%s"' if self._qemu_full_args: command = ' '.join(self._qemu_full_args)
@@ -307,6 +310,7 @@ def _post_shutdown(self): command = '' LOG.warning(msg, -int(exitcode), command) + self._killed = False self._launched = False def launch(self):
@@ -414,6 +418,7 @@ def shutdown(self, has_quit: bool = False, hard: bool = False) -> None: try: if hard: + self._killed = True self._hard_shutdown() else: self._do_shutdown(has_quit)
Signature · 1 line
2.21.3
Am 04.06.2020 um 21:52 hat John Snow geschrieben:
Quoted 2 lines
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
Feels much easier to read now!
Quoted 5 lines
- Re-add sigkill warning squelch (now 3/3)
NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
As I found in the v2 review that we don't even have a single user of hard=True, I don't think it matters. A separate patch makes things easier to understand, so I would leave it as you have it in this series.
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.py
Quoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)
I'm now seeing this error:
21:31:58 DEBUG| / # reboot
21:32:01 DEBUG| / # reboot: Restarting system
21:32:01 DEBUG| >>> {'execute': 'quit'}
21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel /tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta -initrd /tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio -append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR| vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR| self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR| self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR| self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| return self.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, in cmd_obj
21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe
21:32:01 ERROR|
21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging /var/log/syslog (lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |
https://travis-ci.org/github/philmd/qemu/jobs/696142277#L5329
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |
Gotcha.
The problem here is that test_mips_malta_cpio in boot_linux_console.py does this:
self.vm.add_args('-kernel', kernel_path,
'-initrd', initrd_path,
'-append', kernel_command_line,
'-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')
and (in avocado_qemu/) __init__.py does this:
def tearDown(self):
for vm in self._vms.values():
vm.shutdown()
What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None and self._popen.poll() is None
But, well, race conditions.
When we make it here:
if self._qmp is not None:
if not has_quit:
self._qmp.cmd('quit')
self._qmp.close()
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Ultimately:
- Avocado should not try to shut down QEMU twice, but
- machine.py shouldn't enable the race condition either.
for my part, how about this:
python/qemu/machine.py +0 -0
@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,
timeout: int = 3) -> None:
if self._qmp is not None:
if not has_quit:
- self._qmp.cmd('quit')
+ try:
+ self._qmp.cmd('quit')
+ except (BrokenPipeError, ConnectionResetError):
+ # QMP went away just before or just after sending
'quit'
+ if not self.is_running():
+ # "Mission Accomplished"
+ pass
+ raise
self._qmp.close()
self._popen.wait(timeout=timeout)Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()
Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 25 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py
Quoted 4 lines
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raise
Looks OK to me, Cleber/Eduardo can you Ack?
Quoted 4 lines
self._qmp.close()
self._popen.wait(timeout=timeout)Eduardo Habkost
Reviewed-byQuoted 168 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 17 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
Quoted 14 lines
python/qemu/machine.py
Quoted 12 lines
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
Looks good to me.
Quoted 6 lines
Quoted 5 lines
self._qmp.close() self._popen.wait(timeout=timeout)
Signature · 1 line
Eduardo
Quoted 184 lines · Eduardo Habkost
Quoted 168 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 17 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
Quoted 14 lines
python/qemu/machine.py
Quoted 12 lines
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
Quoted 12 lines
Looks good to me.
Reviewed-by Eduardo Habkost <[email protected]>Quoted 6 lines
Quoted 5 lines
self._qmp.close() self._popen.wait(timeout=timeout)
I'll respin with the fix squashed.
Thanks!
--js
Quoted 193 lines · John Snow
Quoted 190 lines · Eduardo Habkost
Quoted 168 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 17 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
Quoted 14 lines
python/qemu/machine.py
Quoted 12 lines
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
Looks good to me.
Reviewed-by Eduardo Habkost <[email protected]>Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
self._qmp.close() self._popen.wait(timeout=timeout)I'll respin with the fix squashed.
Not needed, it is here already:
https://gitlab.com/philmd/qemu/-/commit/d95ff1cf5d47f96d
I was waiting the build to success to respond to this thread.
Quoted 5 lines
Thanks!
--js
Quoted 168 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 17 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
Quoted 15 lines
python/qemu/machine.py
Quoted 12 lines
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
John's description seems spot on, and so does the solution. I was unable, though, to reproduce it given that the window is indeed tiny.
Phil,
In addition to this, wouldn't it make sense to:
tests/acceptance/boot_linux_console.py +4 -4
@@ -180,8 +180,8 @@ class BootLinuxConsole(LinuxKernelTest): + 'rdinit=/sbin/init noreboot') self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, - '-append', kernel_command_line, - '-no-reboot') + '-append', kernel_command_line) + self.vm.launch() self.wait_for_console_pattern('Boot successful.')
@@ -189,8 +189,8 @@ class BootLinuxConsole(LinuxKernelTest): 'BogoMIPS') exec_command_and_wait_for_pattern(self, 'uname -a', 'Debian') - exec_command_and_wait_for_pattern(self, 'reboot', - 'reboot: Restarting system') + exec_command_and_wait_for_pattern(self, 'halt', + 'reboot: System halted') @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code') def test_mips64el_malta_5KEc_cpio(self):
Do you know of any reason to "fake halt" (reboot) instead of actually halting from the guests?
Quoted 5 lines
Quoted 4 lines
self._qmp.close() self._popen.wait(timeout=timeout)
Either way (^ with this extra block):
Quoted 197 lines · Cleber Rosa
Quoted 168 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 17 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
Quoted 15 lines
python/qemu/machine.py
Quoted 12 lines
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
John's description seems spot on, and so does the solution. I was unable, though, to reproduce it given that the window is indeed tiny.
Phil,
In addition to this, wouldn't it make sense to:
tests/acceptance/boot_linux_console.py +0
-0
tests/acceptance/boot_linux_console.py class BootLinuxConsole(LinuxKernelTest): · line 183
Quoted 6 lines
tests/acceptance/boot_linux_console.py +1
-2@@ -180,8 +180,8 @@ class BootLinuxConsole(LinuxKernelTest): + 'rdinit=/sbin/init noreboot') self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, - '-append', kernel_command_line, - '-no-reboot') + '-append', kernel_command_line)
Very few machines implement the hardware shutdown (sending the SHUTDOWN_CAUSE_GUEST_SHUTDOWN event). AFAIK The SHUTDOWN_CAUSE_GUEST_RESET is a bit more present. When the hardware doesn't model this, the guest loops infinitely. Some physical machines are not supposed to power off, until cutting its power. So firmware also loops without trying to power things off.
'-no-reboot' means "if the guest tries to reboot, consider the emulation done".
Quoted 16 lines
tests/acceptance/boot_linux_console.py +3
-2+ self.vm.launch() self.wait_for_console_pattern('Boot successful.')@@ -189,8 +189,8 @@ class BootLinuxConsole(LinuxKernelTest): 'BogoMIPS') exec_command_and_wait_for_pattern(self, 'uname -a', 'Debian') - exec_command_and_wait_for_pattern(self, 'reboot', - 'reboot: Restarting system') + exec_command_and_wait_for_pattern(self, 'halt', + 'reboot: System halted') @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code') def test_mips64el_malta_5KEc_cpio(self):
Quoted 2 lines
Do you know of any reason to "fake halt" (reboot) instead of actually halting from the guests?
This might be my embedded system reflex (explained earlier, some hardware can not power itself down). I will give your diff a try.
Quoted 11 lines
Quoted 6 lines
Quoted 5 lines
self._qmp.close() self._popen.wait(timeout=timeout)Either way (^ with this extra block):
Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>
Thanks!
Quoted 186 lines · Philippe Mathieu-Daudé
Quoted 185 lines · Cleber Rosa
Quoted 183 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 25 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py
Quoted 4 lines
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
Quoted 2 lines
Quoted 2 lines
John's description seems spot on, and so does the solution. I was unable, though, to reproduce it given that the window is indeed tiny.
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445
Quoted 64 lines
Quoted 10 lines
Phil,
In addition to this, wouldn't it make sense to:
tests/acceptance/boot_linux_console.py +0
-0tests/acceptance/boot_linux_console.py class BootLinuxConsole(LinuxKernelTest): · line 183
Quoted 6 lines
tests/acceptance/boot_linux_console.py +1
-2@@ -180,8 +180,8 @@ class BootLinuxConsole(LinuxKernelTest): + 'rdinit=/sbin/init noreboot') self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, - '-append', kernel_command_line, - '-no-reboot') + '-append', kernel_command_line)Very few machines implement the hardware shutdown (sending the
SHUTDOWN_CAUSE_GUEST_SHUTDOWNevent). AFAIK TheSHUTDOWN_CAUSE_GUEST_RESETis a bit more present. When the hardware doesn't model this, the guest loops infinitely. Some physical machines are not supposed to power off, until cutting its power. So firmware also loops without trying to power things off.'-no-reboot' means "if the guest tries to reboot, consider the emulation done".
Quoted 16 lines
tests/acceptance/boot_linux_console.py +3
-2+ self.vm.launch() self.wait_for_console_pattern('Boot successful.')@@ -189,8 +189,8 @@ class BootLinuxConsole(LinuxKernelTest): 'BogoMIPS') exec_command_and_wait_for_pattern(self, 'uname -a', 'Debian') - exec_command_and_wait_for_pattern(self, 'reboot', - 'reboot: Restarting system') + exec_command_and_wait_for_pattern(self, 'halt', + 'reboot: System halted') @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code') def test_mips64el_malta_5KEc_cpio(self):Quoted 2 lines
Do you know of any reason to "fake halt" (reboot) instead of actually halting from the guests?
This might be my embedded system reflex (explained earlier, some hardware can not power itself down). I will give your diff a try.
Quoted 11 lines
Quoted 6 lines
Quoted 5 lines
self._qmp.close() self._popen.wait(timeout=timeout)Either way (^ with this extra block):
Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>Thanks!
Quoted 2 lines · Philippe Mathieu-Daudé
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445
I tried running from master with just these patches applied. So far, the only failures I am seeing on gitlab/travis are job timeouts.
Are you doing something special to avoid timeout failures?
--js
Quoted 10 lines · John Snow
Quoted 2 lines · Philippe Mathieu-Daudé
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445I tried running from master with just these patches applied. So far, the only failures I am seeing on gitlab/travis are job timeouts.
Are you doing something special to avoid timeout failures?
Unfortunately no :S Cleber tried too and couldn't reproduce, I'm the only one lucky enough to win the race condition...
Retrying the acceptance tests on Travis-CI is painful, yes, I know...
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · John Snow
Quoted 2 lines · Philippe Mathieu-Daudé
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445I tried running from master with just these patches applied. So far, the only failures I am seeing on gitlab/travis are job timeouts.
Quoted 8 lines
Quoted 1 line
Are you doing something special to avoid timeout failures?
Unfortunately no :S Cleber tried too and couldn't reproduce, I'm the only one lucky enough to win the race condition...
Retrying the acceptance tests on Travis-CI is painful, yes, I know...
You didn't address the timeouts I asked about. Are you not seeing job timeouts? How is the CI passing at all, for anyone?
We seem to have bigger problems here.
+Alex
Quoted 16 lines · John Snow
Quoted 15 lines · Philippe Mathieu-Daudé
Quoted 10 lines · John Snow
Quoted 2 lines · Philippe Mathieu-Daudé
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445I tried running from master with just these patches applied. So far, the only failures I am seeing on gitlab/travis are job timeouts.
Are you doing something special to avoid timeout failures?
Unfortunately no :S Cleber tried too and couldn't reproduce, I'm the only one lucky enough to win the race condition...
Quoted 8 lines
Quoted 3 lines
Retrying the acceptance tests on Travis-CI is painful, yes, I know...
You didn't address the timeouts I asked about. Are you not seeing job timeouts? How is the CI passing at all, for anyone?
We seem to have bigger problems here.
OK, your problem/question was not obvious.
What helped me to is to know you setup'd your Travis-CI account today.
At some point the jobs were taking too long, by caching the ccache directory between runs, we could get quicker builds.
My guess is as your setup is new, your cache is empty, and you are reaching the 50min timeout...
If this is the case, I'm not sure how to resolve this. Retry and retry again?
Quoted 187 lines · Philippe Mathieu-Daudé
Quoted 186 lines · Philippe Mathieu-Daudé
Quoted 185 lines · Cleber Rosa
Quoted 183 lines · Philippe Mathieu-Daudé
Quoted 141 lines · John Snow
Quoted 91 lines · Philippe Mathieu-Daudé
Hi John,
Quoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.pyQuoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)I'm now seeing this error:
21:31:58 DEBUG| / # reboot 21:32:01 DEBUG| / # reboot: Restarting system 21:32:01 DEBUG| >>> {'execute': 'quit'} 21:32:01 WARNI| qemu received signal 9; command:"mips-softmmu/qemu-system-mips -display none -vga none -chardev
socket,id=mon,path=/tmp/tmp679upvrk/qemu-10292-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmp679upvrk/qemu-10292-console.sock,server,nowait
-serial chardev:console -kernel/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta-initrd/tmp/avocado_b3aaagr9/avocado_job_5bj0xe1h/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio-append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"
21:32:01 ERROR|
21:32:01 ERROR| Reproduced traceback from:/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886
21:32:01 ERROR| Traceback (most recent call last):
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py",
line 195, in tearDown
21:32:01 ERROR|vm.shutdown()
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 449, in
shutdown
21:32:01 ERROR|self._do_shutdown(has_quit)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 426, in_do_shutdown
21:32:01 ERROR|self._soft_shutdown(has_quit, timeout)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/machine.py", line 413, in_soft_shutdown
21:32:01 ERROR|self._qmp.cmd('quit')
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
21:32:01 ERROR| returnself.cmd_obj(qmp_cmd)
21:32:01 ERROR| File
"/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, incmd_obj21:32:01 ERROR| self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) 21:32:01 ERROR| BrokenPipeError: [Errno 32] Broken pipe 21:32:01 ERROR| 21:32:01 DEBUG| DATA (filename=output.expected) => NOT FOUND (datasources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data
sources: variant, test, file)
21:32:01 DEBUG| Not logging/var/log/syslog(lack of permissions) 21:32:01 ERROR| ERROR 12-tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips_malta_cpio -> TestSetupFail: [Errno 32] Broken pipe
21:32:01 INFO |Gotcha.
The problem here is that
test_mips_malta_cpioinboot_linux_console.pydoes this:self.vm.add_args('-kernel', kernel_path, '-initrd', initrd_path, '-append', kernel_command_line, '-no-reboot')and then:
exec_command_and_wait_for_pattern(self, 'reboot',
'reboot: Restarting system')and (in avocado_qemu/)
__init__.pydoes this:def tearDown(self):
for vm inself._vms.values():
vm.shutdown()What's happening here is that we are instructing QEMU to *close* when the guest reboots instead of allowing it to reboot. Then, we are issuing a reboot command to the guest, which will effectively terminate QEMU as well. Finally, we are trying to send a shutdown command to QEMU, but QEMU has already gone.
Now, in the shutdown code, we do make an attempt to catch this:
def
is_running(self):
"""Returns true if the VM is running."""
return self._popen is not None andself._popen.poll()is NoneBut, well, race conditions.
When we make it here:
if self._qmp is not None:
if nothas_quit:
self._qmp.cmd('quit')
self._qmp.close()Quoted 6 lines
We believe we are running and we believe we have an open QMP socket. Attempting to engage the socket by sending 'quit' causes the error.
It's a tight window: if quit happens earlier, we send the command successfully and everything's OK. If quit happens later, we realize QEMU isn't running and proceed to cleanup.
Nice debugging :)
Quoted 25 lines
Ultimately:
- Avocado should not try to shut down QEMU twice, but
-machine.pyshouldn't enable the race condition either.for my part, how about this:
python/qemu/machine.py +0
-0@@ -385,7 +385,14 @@ def _soft_shutdown(self, has_quit: bool = False,timeout: int = 3) -> None:
if self._qmp is not None:
if nothas_quit:python/qemu/machine.py +4
-1- self._qmp.cmd('quit') + try: + self._qmp.cmd('quit') + except (BrokenPipeError, ConnectionResetError): + # QMP went away just before or just after sending'quit'
python/qemu/machine.py
Quoted 4 lines
python/qemu/machine.py +4
-0+ if not self.is_running(): + # "Mission Accomplished" + pass + raiseLooks OK to me, Cleber/Eduardo can you Ack?
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
John's description seems spot on, and so does the solution. I was unable, though, to reproduce it given that the window is indeed tiny.
Still doesn't work...
https://travis-ci.org/github/philmd/qemu/jobs/700309163#L5445
I have overlooked a very fundamental mistake!
'pass' does not terminate execution of an exception handler...
It needs to be:
except (BrokenPipeError, ConnectionResetError):
if self.is_running():
raise
Will re-spin, sorry about that.
--js
Philippe Mathieu-Daudé
AppliedQuoted 23 lines · John Snow
v3:
- Split_post_shutdownrefactor into own patch (now 1/3)
- Re-add sigkill warning squelch (now 3/3)NOTE: I re-added the squelch in its own patch for review purposes, but for the purposes of avoiding temporary breakage, a maintainer may wish to squash patches 2 and 3 if they are accepted.
v2: Philippe took patches 1, 3 and 4.
This is a re-write of what was:
[PATCH RFC 03/32]python//machine.py: remove bare except [PATCH 2/4]python/machine.py: remove bare exceptIt's a bit heavier handed, but it should address some of kwolf's feedback from the RFC version.
Applies straight to origin/master, ought to pass pylint and flake8:
Quoted 3 lines
cd ~/src/qemu/python/qemu
pylint *.py
flake8 *.py
Quoted 8 lines
John Snow (3):
python/machine.py: consolidate_post_shutdown()python/machine.py: refactor shutdownpython/machine.py: re-add sigkill warning suppressionpython/qemu/machine.py | 100 +++++++++++++++++++++++++++++------------ 1 file changed, 71 insertions(+), 29 deletions(-)
I had prepared:
Thanks, applied to my python-next tree (including your diff in python/qemu/machine.py):
https://gitlab.com/philmd/qemu/commits/python-next
But I just noticed Cleber review comment asking for more testing, so I'm keeping this applied, but will delay the pull request since I have to re-run the testing.
Phil.