Skip to contentSkip to conversation

python: add mypy support to python/qemu

PATCH Superseded John Snow 2 participants 20 messages Jun 26, 2020 – Jul 1, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

v4 cover python: add mypy support to python/qemu

Based-on: [email protected]

This series modifies the python/qemu library to comply with mypy --strict. This requires my "refactor shutdown" patch as a pre-requisite.

v4:
001/16:[----] [--] 'python/qmp.py: Define common types' 002/16:[----] [--] 'iotests.py: use qemu.qmp type aliases' 003/16:[----] [--] 'python/qmp.py: re-absorb MonitorResponseError' 004/16:[----] [--] 'python/qmp.py: Do not return None from cmd_obj' 005/16:[----] [--] 'python/qmp.py: add casts to JSON deserialization' 006/16:[----] [--] 'python/qmp.py: add QMPProtocolError' 007/16:[----] [--] 'python/machine.py: Fix monitor address typing' 008/16:[----] [--] 'python/machine.py: reorder __init__' 009/16:[----] [--] 'python/machine.py: Don't modify state in _base_args()' 010/16:[----] [--] 'python/machine.py: Handle None events in events_wait' 011/16:[----] [--] 'python/machine.py: use qmp.command' 012/16:[0010] [FC] 'python/machine.py: Add _qmp access shim' 013/16:[----] [-C] 'python/machine.py: fix _popen access' 014/16:[----] [--] 'python/qemu: make 'args' style arguments immutable' 015/16:[----] [--] 'iotests.py: Adjust HMP kwargs typing' 016/16:[----] [-C] 'python/qemu: Add mypy type annotations'

- Rebased on "refactor shutdown" v4
- Fixed _qmp access for scripts that disable QMP

v3:
005: Removed a cast, per Kevin Wolf's tip
010: Renamed with correct function name;
Rewrote docstring and added comments
016: Use SocketAddrT instead of Union[Tuple[str,str],str]

"v2":
- This version supports iotests 297
- Many patches merged by Phil are removed
- Replaces iotests.py type aliases with centralized ones (See patch 2)
- Imports etc are reworked to use the non-installable package layout instead. (Mostly important for patch 3)

Testing this out:
- You'll need Python3.6+
- I encourage you to use a virtual environment!
- You don't necessarily need these exact versions, but I didn't test the lower bounds, use older versions at your peril:
- pylint==2.5.0
- mypy=0.770
- flake8=3.7.8

Quoted 5 lines

cd ~/src/qemu/python/
flake8 qemu
mypy --strict qemu
cd qemu
pylint *.py

These should all 100% pass.

---

Open RFC: What's the right way to hook this into make check to prevent regressions?

John Snow (16):
python/qmp.py: Define common types iotests.py: use qemu.qmp type aliases python/qmp.py: re-absorb MonitorResponseError python/qmp.py: Do not return None from cmd_obj python/qmp.py: add casts to JSON deserialization python/qmp.py: add QMPProtocolError python/machine.py: Fix monitor address typing python/machine.py: reorder __init__ python/machine.py: Don't modify state in _base_args() python/machine.py: Handle None events in events_wait python/machine.py: use qmp.command python/machine.py: Add _qmp access shim python/machine.py: fix _popen access python/qemu: make 'args' style arguments immutable iotests.py: Adjust HMP kwargs typing python/qemu: Add mypy type annotations

 python/qemu/accel.py          |   8 +-
 python/qemu/machine.py        | 294 ++++++++++++++++++++--------------
 python/qemu/qmp.py            | 111 +++++++++----
 python/qemu/qtest.py          |  53 +++---
 scripts/render_block_graph.py |   7 +-
 tests/qemu-iotests/iotests.py |  11 +-
 6 files changed, 300 insertions(+), 184 deletions(-)
Signature · 1 line
2.21.3
author
#1 lore raw

PATCH v4 1/16 python/qmp.py: Define common types

Define some common types that we'll need to annotate a lot of other functions going forward.

---
 python/qemu/qmp.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
python/qemu/qmp.py +18 -0
index e64b6b5faa..8388c7b603 100644
@@ -12,13 +12,31 @@
 import socket
 import logging
 from typing import (
+    Any,
+    Dict,
     Optional,
     TextIO,
     Type,
+    Tuple,
+    Union,
 )
 from types import TracebackType
 
 
+# QMPMessage is a QMP Message of any kind.
+# e.g. {'yee': 'haw'}
+#
+# QMPReturnValue is the inner value of return values only.
+# {'return': {}} is the QMPMessage,
+# {} is the QMPReturnValue.
+QMPMessage = Dict[str, Any]
+QMPReturnValue = Dict[str, Any]
+
+InternetAddrT = Tuple[str, str]
+UnixAddrT = str
+SocketAddrT = Union[InternetAddrT, UnixAddrT]
+
+
 class QMPError(Exception):
     """
     QMP base exception
Signature · 1 line
2.21.3
author
#2 lore raw

PATCH v4 2/16 iotests.py: use qemu.qmp type aliases

In reply to John Snow

iotests.py should use the type definitions from qmp.py instead of its own.

---
 tests/qemu-iotests/iotests.py | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
tests/qemu-iotests/iotests.py +3 -6
index 5ea4c4df8b..a8576f811b 100644
@@ -35,13 +35,10 @@
 # pylint: disable=import-error, wrong-import-position
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import qtest
+from qemu.qmp import QMPMessage
 
 assert sys.version_info >= (3, 6)
 
-# Type Aliases
-QMPResponse = Dict[str, Any]
-
-
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
 logger.addHandler(logging.NullHandler())
@@ -554,7 +551,7 @@ def add_incoming(self, addr):
         self._args.append(addr)
         return self
 
-    def hmp(self, command_line: str, use_log: bool = False) -> QMPResponse:
+    def hmp(self, command_line: str, use_log: bool = False) -> QMPMessage:
         cmd = 'human-monitor-command'
         kwargs = {'command-line': command_line}
         if use_log:
@@ -575,7 +572,7 @@ def resume_drive(self, drive: str) -> None:
         self.hmp(f'qemu-io {drive} "remove_break bp_{drive}"')
 
     def hmp_qemu_io(self, drive: str, cmd: str,
-                    use_log: bool = False) -> QMPResponse:
+                    use_log: bool = False) -> QMPMessage:
         """Write to a given drive using an HMP command"""
         return self.hmp(f'qemu-io {drive} "{cmd}"', use_log=use_log)
 
Signature · 1 line
2.21.3
author
#3 lore raw

PATCH v4 3/16 python/qmp.py: re-absorb MonitorResponseError

In reply to John Snow

When I initially split this out, I considered this more of a machine error than a QMP protocol error, but I think that's misguided.

Move this back to qmp.py and name it QMPResponseError. Convert qmp.command() to use this exception type.

---
 python/qemu/machine.py        | 15 +--------------
 python/qemu/qmp.py            | 17 +++++++++++++++--
 scripts/render_block_graph.py |  7 +++++--
 3 files changed, 21 insertions(+), 18 deletions(-)
python/qemu/machine.py +1 -14
index 8c9050af5d..a5a579095b 100644
@@ -49,19 +49,6 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 
-class MonitorResponseError(qmp.QMPError):
-    """
-    Represents erroneous QMP monitor reply
-    """
-    def __init__(self, reply):
-        try:
-            desc = reply["error"]["desc"]
-        except KeyError:
-            desc = reply
-        super().__init__(desc)
-        self.reply = reply
-
-
 class QEMUMachine:
     """
     A QEMU VM
@@ -477,7 +464,7 @@ def command(self, cmd, conv_keys=True, **args):
         if reply is None:
             raise qmp.QMPError("Monitor is closed")
         if "error" in reply:
-            raise MonitorResponseError(reply)
+            raise qmp.QMPResponseError(reply)
         return reply["return"]
 
     def get_qmp_event(self, wait=False):
python/qemu/qmp.py +15 -2
index 8388c7b603..aa8a666b8a 100644
@@ -61,6 +61,19 @@ class QMPTimeoutError(QMPError):
     """
 
 
+class QMPResponseError(QMPError):
+    """
+    Represents erroneous QMP monitor reply
+    """
+    def __init__(self, reply: QMPMessage):
+        try:
+            desc = reply['error']['desc']
+        except KeyError:
+            desc = reply
+        super().__init__(desc)
+        self.reply = reply
+
+
 class QEMUMonitorProtocol:
     """
     Provide an API to connect to QEMU via QEMU Monitor Protocol (QMP) and then
@@ -251,8 +264,8 @@ def command(self, cmd, **kwds):
         Build and send a QMP command to the monitor, report errors if any
         """
         ret = self.cmd(cmd, kwds)
-        if "error" in ret:
-            raise Exception(ret['error']['desc'])
+        if 'error' in ret:
+            raise QMPResponseError(ret)
         return ret['return']
 
     def pull_event(self, wait=False):
scripts/render_block_graph.py +5 -2
index 409b4321f2..da6acf050d 100755
@@ -25,7 +25,10 @@
 from graphviz import Digraph
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
-from qemu.machine import MonitorResponseError
+from qemu.qmp import (
+    QEMUMonitorProtocol,
+    QMPResponseError,
+)
 
 
 def perm(arr):
@@ -102,7 +105,7 @@ def command(self, cmd):
         reply = json.loads(subprocess.check_output(ar))
 
         if 'error' in reply:
-            raise MonitorResponseError(reply)
+            raise QMPResponseError(reply)
 
         return reply['return']
 
Signature · 1 line
2.21.3
author
#4 lore raw

PATCH v4 4/16 python/qmp.py: Do not return None from cmd_obj

In reply to John Snow

This makes typing the qmp library difficult, as it necessitates wrapping Optional[] around the type for every return type up the stack. At some point, it becomes difficult to discern or remember why it's None instead of the expected object.

Use the python exception system to tell us exactly why we didn't get an object. Remove this special-cased return.

---
 python/qemu/qmp.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
python/qemu/qmp.py +5 -9
index aa8a666b8a..ef3c919b76 100644
@@ -225,22 +225,18 @@ def accept(self, timeout=15.0):
         self.__sockfile = self.__sock.makefile(mode='r')
         return self.__negotiate_capabilities()
 
-    def cmd_obj(self, qmp_cmd):
+    def cmd_obj(self, qmp_cmd: QMPMessage) -> QMPMessage:
         """
         Send a QMP command to the QMP Monitor.
 
         @param qmp_cmd: QMP command to be sent as a Python dict
-        @return QMP response as a Python dict or None if the connection has
-                been closed
+        @return QMP response as a Python dict
         """
         self.logger.debug(">>> %s", qmp_cmd)
-        try:
-            self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
-        except OSError as err:
-            if err.errno == errno.EPIPE:
-                return None
-            raise err
+        self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
         resp = self.__json_read()
+        if resp is None:
+            raise QMPConnectError("Unexpected empty reply from server")
         self.logger.debug("<<< %s", resp)
         return resp
 
Signature · 1 line
2.21.3
author
#5 lore raw

PATCH v4 5/16 python/qmp.py: add casts to JSON deserialization

In reply to John Snow

mypy and python type hints are not powerful enough to properly describe JSON messages in Python 3.6. The best we can do, generally, is describe them as Dict[str, Any].

Add casts to coerce this type for static analysis; but do NOT enforce this type at runtime in any way.

Note: Python 3.8 adds a TypedDict construct which allows for the description of more arbitrary Dictionary shapes. There is a third-party module, "Pydantic", which is compatible with 3.6 that can be used instead of the JSON library that parses JSON messages to fully-typed Python objects, and may be preferable in some cases.

(That is well beyond the scope of this commit or series.)

---
 python/qemu/qmp.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
python/qemu/qmp.py +6 -2
index ef3c919b76..1ae36050a4 100644
@@ -13,6 +13,7 @@
 import logging
 from typing import (
     Any,
+    cast,
     Dict,
     Optional,
     TextIO,
@@ -130,7 +131,10 @@ def __json_read(self, only_event=False):
             data = self.__sockfile.readline()
             if not data:
                 return None
-            resp = json.loads(data)
+            # By definition, any JSON received from QMP is a QMPMessage,
+            # and we are asserting only at static analysis time that it
+            # has a particular shape.
+            resp: QMPMessage = json.loads(data)
             if 'event' in resp:
                 self.logger.debug("<<< %s", resp)
                 self.__events.append(resp)
@@ -262,7 +266,7 @@ def command(self, cmd, **kwds):
         ret = self.cmd(cmd, kwds)
         if 'error' in ret:
             raise QMPResponseError(ret)
-        return ret['return']
+        return cast(QMPReturnValue, ret['return'])
 
     def pull_event(self, wait=False):
         """
Signature · 1 line
2.21.3
author
#6 lore raw

PATCH v4 6/16 python/qmp.py: add QMPProtocolError

In reply to John Snow

In the case that we receive a reply but are unable to understand it, use this exception name to indicate that case.

---
 python/qemu/qmp.py | 10 ++++++++++
 1 file changed, 10 insertions(+)
python/qemu/qmp.py +10 -0
index 1ae36050a4..7935dababb 100644
@@ -62,6 +62,12 @@ class QMPTimeoutError(QMPError):
     """
 
 
+class QMPProtocolError(QMPError):
+    """
+    QMP protocol error; unexpected response
+    """
+
+
 class QMPResponseError(QMPError):
     """
     Represents erroneous QMP monitor reply
@@ -266,6 +272,10 @@ def command(self, cmd, **kwds):
         ret = self.cmd(cmd, kwds)
         if 'error' in ret:
             raise QMPResponseError(ret)
+        if 'return' not in ret:
+            raise QMPProtocolError(
+                "'return' key not found in QMP response '{}'".format(str(ret))
+            )
         return cast(QMPReturnValue, ret['return'])
 
     def pull_event(self, wait=False):
Signature · 1 line
2.21.3
author
#7 lore raw

PATCH v4 7/16 python/machine.py: Fix monitor address typing

In reply to John Snow

Prior to this, it's difficult for mypy to intuit what the concrete type of the monitor address is; it has difficulty inferring the type across two variables.

Create _monitor_address as a property that always returns a valid address to simplify static type analysis.

To preserve our ability to clean up, use a simple boolean to indicate whether or not we should try to clean up the sock file after execution.

---
 python/qemu/machine.py | 45 +++++++++++++++++++++++++++---------------
 1 file changed, 29 insertions(+), 16 deletions(-)
python/qemu/machine.py +29 -16
index a5a579095b..6d38147cd5 100644
@@ -25,10 +25,14 @@
 import signal
 import socket
 import tempfile
-from typing import Optional, Type
+from typing import (
+    Optional,
+    Type,
+)
 from types import TracebackType
 
 from . import qmp
+from .qmp import SocketAddrT
 
 LOG = logging.getLogger(__name__)
 
@@ -62,7 +66,8 @@ class QEMUMachine:
     """
 
     def __init__(self, binary, args=None, wrapper=None, name=None,
-                 test_dir="/var/tmp", monitor_address=None,
+                 test_dir="/var/tmp",
+                 monitor_address: Optional[SocketAddrT] = None,
                  socket_scm_helper=None, sock_dir=None):
         '''
         Initialize a QEMUMachine
@@ -85,8 +90,14 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         if sock_dir is None:
             sock_dir = test_dir
         self._name = name
-        self._monitor_address = monitor_address
-        self._vm_monitor = None
+        if monitor_address is not None:
+            self._monitor_address = monitor_address
+            self._remove_monitor_sockfile = False
+        else:
+            self._monitor_address = os.path.join(
+                sock_dir, f"{name}-monitor.sock"
+            )
+            self._remove_monitor_sockfile = True
         self._qemu_log_path = None
         self._qemu_log_file = None
         self._popen = None
@@ -225,15 +236,17 @@ def _load_io_log(self):
 
     def _base_args(self):
         args = ['-display', 'none', '-vga', 'none']
+
         if self._qmp_set:
             if isinstance(self._monitor_address, tuple):
-                moncdev = "socket,id=mon,host=%s,port=%s" % (
-                    self._monitor_address[0],
-                    self._monitor_address[1])
+                moncdev = "socket,id=mon,host={},port={}".format(
+                    *self._monitor_address
+                )
             else:
-                moncdev = 'socket,id=mon,path=%s' % self._vm_monitor
+                moncdev = f"socket,id=mon,path={self._monitor_address}"
             args.extend(['-chardev', moncdev, '-mon',
                          'chardev=mon,mode=control'])
+
         if self._machine is not None:
             args.extend(['-machine', self._machine])
         for _ in range(self._console_index):
@@ -258,14 +271,14 @@ def _pre_launch(self):
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
         if self._qmp_set:
-            if self._monitor_address is not None:
-                self._vm_monitor = self._monitor_address
-            else:
-                self._vm_monitor = os.path.join(self._sock_dir,
-                                                self._name + "-monitor.sock")
-                self._remove_files.append(self._vm_monitor)
-            self._qmp = qmp.QEMUMonitorProtocol(self._vm_monitor, server=True,
-                                                nickname=self._name)
+            if self._remove_monitor_sockfile:
+                assert isinstance(self._monitor_address, str)
+                self._remove_files.append(self._monitor_address)
+            self._qmp = qmp.QEMUMonitorProtocol(
+                self._monitor_address,
+                server=True,
+                nickname=self._name
+            )
 
     def _post_launch(self):
         if self._qmp:
Signature · 1 line
2.21.3
author
#8 lore raw

PATCH v4 8/16 python/machine.py: reorder __init__

In reply to John Snow

Put the init arg handling all at the top, and mostly in order (deviating when one is dependent on another), and put what is effectively runtime state declaration at the bottom.

---
 python/qemu/machine.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)
python/qemu/machine.py +17 -12
index 6d38147cd5..36812bd2be 100644
@@ -81,38 +81,43 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         @param socket_scm_helper: helper program, required for send_fd_scm()
         @note: Qemu process is not started until launch() is used.
         '''
+        # Direct user configuration
+
+        self._binary = binary
+
         if args is None:
             args = []
+        # Copy mutable input: we will be modifying our copy
+        self._args = list(args)
+
         if wrapper is None:
             wrapper = []
-        if name is None:
-            name = "qemu-%d" % os.getpid()
-        if sock_dir is None:
-            sock_dir = test_dir
-        self._name = name
+        self._wrapper = wrapper
+
+        self._name = name or "qemu-%d" % os.getpid()
+        self._test_dir = test_dir
+        self._sock_dir = sock_dir or self._test_dir
+        self._socket_scm_helper = socket_scm_helper
+
         if monitor_address is not None:
             self._monitor_address = monitor_address
             self._remove_monitor_sockfile = False
         else:
             self._monitor_address = os.path.join(
-                sock_dir, f"{name}-monitor.sock"
+                self._sock_dir, f"{self._name}-monitor.sock"
             )
             self._remove_monitor_sockfile = True
+
+        # Runstate
         self._qemu_log_path = None
         self._qemu_log_file = None
         self._popen = None
-        self._binary = binary
-        self._args = list(args)     # Force copy args in case we modify them
-        self._wrapper = wrapper
         self._events = []
         self._iolog = None
-        self._socket_scm_helper = socket_scm_helper
         self._qmp_set = True   # Enable QMP monitor by default.
         self._qmp = None
         self._qemu_full_args = None
-        self._test_dir = test_dir
         self._temp_dir = None
-        self._sock_dir = sock_dir
         self._launched = False
         self._machine = None
         self._console_index = 0
Signature · 1 line
2.21.3
author
#9 lore raw

PATCH v4 9/16 python/machine.py: Don't modify state in _base_args()

In reply to John Snow

Don't append to the _remove_files list during _base_args; instead do so during _launch. Rework _base_args as a @property to help facilitate this impression.

This has the additional benefit of making the type of _console_address easier to analyze statically.

---
 python/qemu/machine.py | 16 ++++++++++------
 python/qemu/qtest.py   | 11 ++++++++---
 2 files changed, 18 insertions(+), 9 deletions(-)
python/qemu/machine.py +10 -6
index 36812bd2be..8eb10a4bc1 100644
@@ -26,6 +26,7 @@
 import socket
 import tempfile
 from typing import (
+    List,
     Optional,
     Type,
 )
@@ -123,7 +124,9 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         self._console_index = 0
         self._console_set = False
         self._console_device_type = None
-        self._console_address = None
+        self._console_address = os.path.join(
+            self._sock_dir, f"{self._name}-console.sock"
+        )
         self._console_socket = None
         self._remove_files = []
         self._killed = False
@@ -239,7 +242,8 @@ def _load_io_log(self):
             with open(self._qemu_log_path, "r") as iolog:
                 self._iolog = iolog.read()
 
-    def _base_args(self):
+    @property
+    def _base_args(self) -> List[str]:
         args = ['-display', 'none', '-vga', 'none']
 
         if self._qmp_set:
@@ -257,9 +261,6 @@ def _base_args(self):
         for _ in range(self._console_index):
             args.extend(['-serial', 'null'])
         if self._console_set:
-            self._console_address = os.path.join(self._sock_dir,
-                                                 self._name + "-console.sock")
-            self._remove_files.append(self._console_address)
             chardev = ('socket,id=console,path=%s,server,nowait' %
                        self._console_address)
             args.extend(['-chardev', chardev])
@@ -275,6 +276,9 @@ def _pre_launch(self):
         self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log")
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
+        if self._console_set:
+            self._remove_files.append(self._console_address)
+
         if self._qmp_set:
             if self._remove_monitor_sockfile:
                 assert isinstance(self._monitor_address, str)
@@ -349,7 +353,7 @@ def _launch(self):
         devnull = open(os.path.devnull, 'rb')
         self._pre_launch()
         self._qemu_full_args = (self._wrapper + [self._binary] +
-                                self._base_args() + self._args)
+                                self._base_args + self._args)
         LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
         self._popen = subprocess.Popen(self._qemu_full_args,
                                        stdin=devnull,
python/qemu/qtest.py +8 -3
index 888c8bd2f6..05c63a1d58 100644
@@ -19,7 +19,11 @@
 
 import socket
 import os
-from typing import Optional, TextIO
+from typing import (
+    List,
+    Optional,
+    TextIO,
+)
 
 from .machine import QEMUMachine
 
@@ -111,8 +115,9 @@ def __init__(self, binary, args=None, name=None, test_dir="/var/tmp",
         self._qtest = None
         self._qtest_path = os.path.join(sock_dir, name + "-qtest.sock")
 
-    def _base_args(self):
-        args = super()._base_args()
+    @property
+    def _base_args(self) -> List[str]:
+        args = super()._base_args
         args.extend(['-qtest', 'unix:path=' + self._qtest_path,
                      '-accel', 'qtest'])
         return args
Signature · 1 line
2.21.3
author
#10 lore raw

PATCH v4 10/16 python/machine.py: Handle None events in events_wait

In reply to John Snow

If the timeout is 0, we can get None back. Handle this explicitly.

---
 python/qemu/machine.py | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
python/qemu/machine.py +20 -7
index 8eb10a4bc1..f65bd18c1a 100644
@@ -33,7 +33,7 @@
 from types import TracebackType
 
 from . import qmp
-from .qmp import SocketAddrT
+from .qmp import SocketAddrT, QMPMessage
 
 LOG = logging.getLogger(__name__)
 
@@ -552,13 +552,20 @@ def event_wait(self, name, timeout=60.0, match=None):
 
     def events_wait(self, events, timeout=60.0):
         """
-        events_wait waits for and returns a named event
-        from QMP with a timeout.
+        events_wait waits for and returns a single named event from QMP.
+        In the case of multiple qualifying events, this function returns the
+        first one.
 
-        events: a sequence of (name, match_criteria) tuples.
-                The match criteria are optional and may be None.
-                See event_match for details.
-        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        :param events: A sequence of (name, match_criteria) tuples.
+                       The match criteria are optional and may be None.
+                       See event_match for details.
+        :param timeout: Optional timeout, in seconds.
+                        See QEMUMonitorProtocol.pull_event.
+
+        :raise QMPTimeoutError: If timeout was non-zero and no matching events
+                                were found.
+        :return: A QMP event matching the filter criteria.
+                 If timeout was 0 and no event matched, None.
         """
         def _match(event):
             for name, match in events:
@@ -566,6 +573,8 @@ def _match(event):
                     return True
             return False
 
+        event: Optional[QMPMessage]
+
         # Search cached events
         for event in self._events:
             if _match(event):
@@ -575,6 +584,10 @@ def _match(event):
         # Poll for new events
         while True:
             event = self._qmp.pull_event(wait=timeout)
+            if event is None:
+                # NB: None is only returned when timeout is false-ish.
+                # Timeouts raise QMPTimeoutError instead!
+                break
             if _match(event):
                 return event
             self._events.append(event)
Signature · 1 line
2.21.3
author
#11 lore raw

PATCH v4 11/16 python/machine.py: use qmp.command

In reply to John Snow

machine.py and qmp.py both do the same thing here; refactor machine.py to use qmp.py's functionality more directly.

---
 python/qemu/machine.py | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)
python/qemu/machine.py +15 -11
index f65bd18c1a..9bb5d26407 100644
@@ -26,6 +26,8 @@
 import socket
 import tempfile
 from typing import (
+    Any,
+    Dict,
     List,
     Optional,
     Type,
@@ -463,17 +465,23 @@ def set_qmp_monitor(self, enabled=True):
             self._qmp_set = False
             self._qmp = None
 
-    def qmp(self, cmd, conv_keys=True, **args):
-        """
-        Invoke a QMP command and return the response dict
-        """
+    @classmethod
+    def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:
         qmp_args = dict()
         for key, value in args.items():
-            if conv_keys:
+            if _conv_keys:
                 qmp_args[key.replace('_', '-')] = value
             else:
                 qmp_args[key] = value
+        return qmp_args
 
+    def qmp(self, cmd: str,
+            conv_keys: bool = True,
+            **args: Any) -> QMPMessage:
+        """
+        Invoke a QMP command and return the response dict
+        """
+        qmp_args = self._qmp_args(conv_keys, **args)
         return self._qmp.cmd(cmd, args=qmp_args)
 
     def command(self, cmd, conv_keys=True, **args):
@@ -482,12 +490,8 @@ def command(self, cmd, conv_keys=True, **args):
         On success return the response dict.
         On failure raise an exception.
         """
-        reply = self.qmp(cmd, conv_keys, **args)
-        if reply is None:
-            raise qmp.QMPError("Monitor is closed")
-        if "error" in reply:
-            raise qmp.QMPResponseError(reply)
-        return reply["return"]
+        qmp_args = self._qmp_args(conv_keys, **args)
+        return self._qmp.command(cmd, **qmp_args)
 
     def get_qmp_event(self, wait=False):
         """
Signature · 1 line
2.21.3
author
#12 lore raw

PATCH v4 12/16 python/machine.py: Add _qmp access shim

In reply to John Snow

Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.

---
 python/qemu/machine.py | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
python/qemu/machine.py +11 -9
index 9bb5d26407..cb3db1f66a 100644
@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         self._events = []
         self._iolog = None
         self._qmp_set = True   # Enable QMP monitor by default.
-        self._qmp = None
+        self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None
         self._qemu_full_args = None
         self._temp_dir = None
         self._launched = False
@@ -285,14 +285,14 @@ def _pre_launch(self):
             if self._remove_monitor_sockfile:
                 assert isinstance(self._monitor_address, str)
                 self._remove_files.append(self._monitor_address)
-            self._qmp = qmp.QEMUMonitorProtocol(
+            self._qmp_connection = qmp.QEMUMonitorProtocol(
                 self._monitor_address,
                 server=True,
                 nickname=self._name
             )
 
     def _post_launch(self):
-        if self._qmp:
+        if self._qmp_connection:
             self._qmp.accept()
 
     def _post_shutdown(self) -> None:
@@ -370,7 +370,7 @@ def wait(self):
         Wait for the VM to power off
         """
         self._popen.wait()
-        if self._qmp:
+        if self._qmp_connection:
             self._qmp.close()
         self._post_shutdown()
 
@@ -459,11 +459,13 @@ def set_qmp_monitor(self, enabled=True):
                         line. Default is True.
         @note: call this function before launch().
         """
-        if enabled:
-            self._qmp_set = True
-        else:
-            self._qmp_set = False
-            self._qmp = None
+        self._qmp_set = enabled
+
+    @property
+    def _qmp(self) -> qmp.QEMUMonitorProtocol:
+        if self._qmp_connection is None:
+            raise QEMUMachineError("Attempt to access QMP with no connection")
+        return self._qmp_connection
 
     @classmethod
     def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:
Signature · 1 line
2.21.3
author
#13 lore raw

PATCH v4 13/16 python/machine.py: fix _popen access

In reply to John Snow

As always, Optional[T] causes problems with unchecked access. Add a helper that asserts the pipe is present before we attempt to talk with it.

---
 python/qemu/machine.py | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
python/qemu/machine.py +12 -6
index cb3db1f66a..c189fa4e28 100644
@@ -114,7 +114,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         # Runstate
         self._qemu_log_path = None
         self._qemu_log_file = None
-        self._popen = None
+        self._popen: Optional['subprocess.Popen[bytes]'] = None
         self._events = []
         self._iolog = None
         self._qmp_set = True   # Enable QMP monitor by default.
@@ -227,6 +227,12 @@ def is_running(self):
         """Returns true if the VM is running."""
         return self._popen is not None and self._popen.poll() is None
 
+    @property
+    def _subp(self) -> 'subprocess.Popen[bytes]':
+        if self._popen is None:
+            raise QEMUMachineError('Subprocess pipe not present')
+        return self._popen
+
     def exitcode(self):
         """Returns the exit code if possible, or None."""
         if self._popen is None:
@@ -237,7 +243,7 @@ def get_pid(self):
         """Returns the PID of the running process, or None."""
         if not self.is_running():
             return None
-        return self._popen.pid
+        return self._subp.pid
 
     def _load_io_log(self):
         if self._qemu_log_path is not None:
@@ -369,7 +375,7 @@ def wait(self):
         """
         Wait for the VM to power off
         """
-        self._popen.wait()
+        self._subp.wait()
         if self._qmp_connection:
             self._qmp.close()
         self._post_shutdown()
@@ -381,8 +387,8 @@ def _hard_shutdown(self) -> None:
         if not self.is_running():
             return
 
-        self._popen.kill()
-        self._popen.wait(timeout=60)
+        self._subp.kill()
+        self._subp.wait(timeout=60)
 
     def _soft_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None:
         """
@@ -407,7 +413,7 @@ def _soft_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None:
                         raise
             self._qmp.close()
 
-        self._popen.wait(timeout=timeout)
+        self._subp.wait(timeout=timeout)
 
     def _do_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None:
         """
Signature · 1 line
2.21.3
author
#14 lore raw

PATCH v4 14/16 python/qemu: make 'args' style arguments immutable

In reply to John Snow

These arguments don't need to be mutable and aren't really used as such. Clarify their types as immutable and adjust code to match where necessary.

In general, It's probably best not to accept a user-defined mutable object and store it as internal object state unless there's a strong justification for doing so. Instead, try to use generic types as input with empty tuples as the default, and coerce to list where necessary.

---
 python/qemu/machine.py | 30 +++++++++++++++++-------------
 python/qemu/qtest.py   | 16 ++++++++++++----
 2 files changed, 29 insertions(+), 17 deletions(-)
python/qemu/machine.py +17 -13
index c189fa4e28..15e854e04f 100644
@@ -18,6 +18,7 @@
 #
 
 import errno
+from itertools import chain
 import logging
 import os
 import subprocess
@@ -30,6 +31,8 @@
     Dict,
     List,
     Optional,
+    Sequence,
+    Tuple,
     Type,
 )
 from types import TracebackType
@@ -68,8 +71,12 @@ class QEMUMachine:
         # vm is guaranteed to be shut down here
     """
 
-    def __init__(self, binary, args=None, wrapper=None, name=None,
-                 test_dir="/var/tmp",
+    def __init__(self,
+                 binary: str,
+                 args: Sequence[str] = (),
+                 wrapper: Sequence[str] = (),
+                 name: Optional[str] = None,
+                 test_dir: str = "/var/tmp",
                  monitor_address: Optional[SocketAddrT] = None,
                  socket_scm_helper=None, sock_dir=None):
         '''
@@ -87,14 +94,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         # Direct user configuration
 
         self._binary = binary
-
-        if args is None:
-            args = []
-        # Copy mutable input: we will be modifying our copy
         self._args = list(args)
-
-        if wrapper is None:
-            wrapper = []
         self._wrapper = wrapper
 
         self._name = name or "qemu-%d" % os.getpid()
@@ -119,7 +119,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None,
         self._iolog = None
         self._qmp_set = True   # Enable QMP monitor by default.
         self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None
-        self._qemu_full_args = None
+        self._qemu_full_args: Tuple[str, ...] = ()
         self._temp_dir = None
         self._launched = False
         self._machine = None
@@ -340,7 +340,7 @@ def launch(self):
             raise QEMUMachineError('VM already launched')
 
         self._iolog = None
-        self._qemu_full_args = None
+        self._qemu_full_args = ()
         try:
             self._launch()
             self._launched = True
@@ -360,8 +360,12 @@ def _launch(self):
         """
         devnull = open(os.path.devnull, 'rb')
         self._pre_launch()
-        self._qemu_full_args = (self._wrapper + [self._binary] +
-                                self._base_args + self._args)
+        self._qemu_full_args = tuple(
+            chain(self._wrapper,
+                  [self._binary],
+                  self._base_args,
+                  self._args)
+        )
         LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
         self._popen = subprocess.Popen(self._qemu_full_args,
                                        stdin=devnull,
python/qemu/qtest.py +12 -4
index 05c63a1d58..ae4661d4d3 100644
@@ -22,6 +22,7 @@
 from typing import (
     List,
     Optional,
+    Sequence,
     TextIO,
 )
 
@@ -103,8 +104,13 @@ class QEMUQtestMachine(QEMUMachine):
     A QEMU VM, with a qtest socket available.
     """
 
-    def __init__(self, binary, args=None, name=None, test_dir="/var/tmp",
-                 socket_scm_helper=None, sock_dir=None):
+    def __init__(self,
+                 binary: str,
+                 args: Sequence[str] = (),
+                 name: Optional[str] = None,
+                 test_dir: str = "/var/tmp",
+                 socket_scm_helper: Optional[str] = None,
+                 sock_dir: Optional[str] = None):
         if name is None:
             name = "qemu-%d" % os.getpid()
         if sock_dir is None:
@@ -118,8 +124,10 @@ def __init__(self, binary, args=None, name=None, test_dir="/var/tmp",
     @property
     def _base_args(self) -> List[str]:
         args = super()._base_args
-        args.extend(['-qtest', 'unix:path=' + self._qtest_path,
-                     '-accel', 'qtest'])
+        args.extend([
+            '-qtest', f"unix:path={self._qtest_path}",
+            '-accel', 'qtest'
+        ])
         return args
 
     def _pre_launch(self):
Signature · 1 line
2.21.3
author
#15 lore raw

PATCH v4 15/16 iotests.py: Adjust HMP kwargs typing

In reply to John Snow

mypy wants to ensure there's consistency between the kwargs arguments types and any unspecified keyword arguments. In this case, conv_keys is a bool, but the remaining keys are Any type. Mypy (correctly) infers the **kwargs type to be **Dict[str, str], which is not compatible with
conv_keys: bool.

Because QMP typing is a little fraught right now anyway, re-type kwargs to Dict[str, Any] which has the benefit of silencing this check right now.

A future re-design might type these more aggressively, but this will give us a baseline to work from with minimal disruption.

(Thanks Kevin Wolf for the debugging assist here)

---
 tests/qemu-iotests/iotests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
index a8576f811b..9040a671f9 100644
@@ -553,7 +553,7 @@ def add_incoming(self, addr):
 
     def hmp(self, command_line: str, use_log: bool = False) -> QMPMessage:
         cmd = 'human-monitor-command'
-        kwargs = {'command-line': command_line}
+        kwargs: Dict[str, Any] = {'command-line': command_line}
         if use_log:
             return self.qmp_log(cmd, **kwargs)
         else:
Signature · 1 line
2.21.3
author
#16 lore raw

PATCH v4 16/16 python/qemu: Add mypy type annotations

In reply to John Snow

These should all be purely annotations with no changes in behavior at all. You need to be in the python folder, but you should be able to confirm that these annotations are correct (or at least self-consistent) by running mypy --strict qemu.

---
 python/qemu/accel.py   |  8 ++--
 python/qemu/machine.py | 94 ++++++++++++++++++++++++------------------
 python/qemu/qmp.py     | 44 +++++++++++---------
 python/qemu/qtest.py   | 26 +++++++-----
 4 files changed, 98 insertions(+), 74 deletions(-)
python/qemu/accel.py +5 -3
index 7fabe62920..4325114e51 100644
@@ -17,6 +17,7 @@
 import logging
 import os
 import subprocess
+from typing import List, Optional
 
 LOG = logging.getLogger(__name__)
 
@@ -29,7 +30,7 @@
 }
 
 
-def list_accel(qemu_bin):
+def list_accel(qemu_bin: str) -> List[str]:
     """
     List accelerators enabled in the QEMU binary.
 
@@ -49,7 +50,8 @@ def list_accel(qemu_bin):
     return [acc.strip() for acc in out.splitlines()[1:]]
 
 
-def kvm_available(target_arch=None, qemu_bin=None):
+def kvm_available(target_arch: Optional[str] = None,
+                  qemu_bin: Optional[str] = None) -> bool:
     """
     Check if KVM is available using the following heuristic:
       - Kernel module is present in the host;
@@ -72,7 +74,7 @@ def kvm_available(target_arch=None, qemu_bin=None):
     return True
 
 
-def tcg_available(qemu_bin):
+def tcg_available(qemu_bin: str) -> bool:
     """
     Check if TCG is available.
 
python/qemu/machine.py +53 -41

295 lines not on this page · raw message

python/qemu/qmp.py +25 -19

163 lines not on this page · raw message

python/qemu/qtest.py +15 -11
index ae4661d4d3..49c04bc98d 100644
@@ -27,6 +27,7 @@
 )
 
 from .machine import QEMUMachine
+from .qmp import SocketAddrT
 
 
 class QEMUQtestProtocol:
@@ -43,7 +44,8 @@ class QEMUQtestProtocol:
        No conection is estabalished by __init__(), this is done
        by the connect() or accept() methods.
     """
-    def __init__(self, address, server=False):
+    def __init__(self, address: SocketAddrT,
+                 server: bool = False):
         self._address = address
         self._sock = self._get_sock()
         self._sockfile: Optional[TextIO] = None
@@ -51,14 +53,14 @@ def __init__(self, address, server=False):
             self._sock.bind(self._address)
             self._sock.listen(1)
 
-    def _get_sock(self):
+    def _get_sock(self) -> socket.socket:
         if isinstance(self._address, tuple):
             family = socket.AF_INET
         else:
             family = socket.AF_UNIX
         return socket.socket(family, socket.SOCK_STREAM)
 
-    def connect(self):
+    def connect(self) -> None:
         """
         Connect to the qtest socket.
 
@@ -67,7 +69,7 @@ def connect(self):
         self._sock.connect(self._address)
         self._sockfile = self._sock.makefile(mode='r')
 
-    def accept(self):
+    def accept(self) -> None:
         """
         Await connection from QEMU.
 
@@ -76,7 +78,7 @@ def accept(self):
         self._sock, _ = self._sock.accept()
         self._sockfile = self._sock.makefile(mode='r')
 
-    def cmd(self, qtest_cmd):
+    def cmd(self, qtest_cmd: str) -> str:
         """
         Send a qtest command on the wire.
 
@@ -87,14 +89,16 @@ def cmd(self, qtest_cmd):
         resp = self._sockfile.readline()
         return resp
 
-    def close(self):
-        """Close this socket."""
+    def close(self) -> None:
+        """
+        Close this socket.
+        """
         self._sock.close()
         if self._sockfile:
             self._sockfile.close()
             self._sockfile = None
 
-    def settimeout(self, timeout):
+    def settimeout(self, timeout: Optional[float]) -> None:
         """Set a timeout, in seconds."""
         self._sock.settimeout(timeout)
 
@@ -118,7 +122,7 @@ def __init__(self,
         super().__init__(binary, args, name=name, test_dir=test_dir,
                          socket_scm_helper=socket_scm_helper,
                          sock_dir=sock_dir)
-        self._qtest = None
+        self._qtest: Optional[QEMUQtestProtocol] = None
         self._qtest_path = os.path.join(sock_dir, name + "-qtest.sock")
 
     @property
@@ -130,7 +134,7 @@ def _base_args(self) -> List[str]:
         ])
         return args
 
-    def _pre_launch(self):
+    def _pre_launch(self) -> None:
         super()._pre_launch()
         self._qtest = QEMUQtestProtocol(self._qtest_path, server=True)
 
@@ -139,7 +143,7 @@ def _post_launch(self) -> None:
         super()._post_launch()
         self._qtest.accept()
 
-    def _post_shutdown(self):
+    def _post_shutdown(self) -> None:
         super()._post_shutdown()
         self._remove_if_exists(self._qtest_path)
 
Signature · 1 line
2.21.3
#17 lore raw
In reply to John Snow

+Cleber/Wainer.

Quoted 23 lines · John Snow

Based-on: [email protected]

This series modifies the python/qemu library to comply with mypy --strict. This requires my "refactor shutdown" patch as a pre-requisite.

v4:
001/16:[----] [--] 'python/qmp.py: Define common types' 002/16:[----] [--] 'iotests.py: use qemu.qmp type aliases' 003/16:[----] [--] 'python/qmp.py: re-absorb MonitorResponseError' 004/16:[----] [--] 'python/qmp.py: Do not return None from cmd_obj' 005/16:[----] [--] 'python/qmp.py: add casts to JSON deserialization' 006/16:[----] [--] 'python/qmp.py: add QMPProtocolError' 007/16:[----] [--] 'python/machine.py: Fix monitor address typing' 008/16:[----] [--] 'python/machine.py: reorder __init__' 009/16:[----] [--] 'python/machine.py: Don't modify state in _base_args()' 010/16:[----] [--] 'python/machine.py: Handle None events in events_wait' 011/16:[----] [--] 'python/machine.py: use qmp.command' 012/16:[0010] [FC] 'python/machine.py: Add _qmp access shim' 013/16:[----] [-C] 'python/machine.py: fix _popen access' 014/16:[----] [--] 'python/qemu: make 'args' style arguments immutable' 015/16:[----] [--] 'iotests.py: Adjust HMP kwargs typing' 016/16:[----] [-C] 'python/qemu: Add mypy type annotations'

Quoted 2 lines

- Rebased on "refactor shutdown" v4
- Fixed _qmp access for scripts that disable QMP

See:

https://travis-ci.org/github/philmd/qemu/jobs/702507625#L5439

/ # uname -a
Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
/ # reboot
/ # reboot: Restarting system

Quoted 1 line
Quoted 1 line
Quoted 1 line

{'execute': 'quit'}

qemu received signal 9; command: "mips-softmmu/qemu-system-mips -display
none -vga none -chardev
socket,id=mon,path=/tmp/tmpcojsc5u3/qemu-14540-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmpcojsc5u3/qemu-14540-console.sock,server,nowait
-serial chardev:console -kernel /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta -initrd /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio -append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"

Reproduced traceback from:
/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886 Traceback (most recent call last):
File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py", line 195, in tearDown
vm.shutdown()
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
457, in shutdown
self._do_shutdown(has_quit)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
434, in _do_shutdown
self._soft_shutdown(has_quit, timeout)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
414, in _soft_shutdown

    self._qmp.cmd('quit')

File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
return self.cmd_obj(qmp_cmd)
File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, in
cmd_obj

    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))

BrokenPipeError: [Errno 32] Broken pipe

author
#18 lore raw
Quoted 66 lines · Philippe Mathieu-Daudé

+Cleber/Wainer.

Quoted 23 lines · John Snow

Based-on: [email protected]

This series modifies the python/qemu library to comply with mypy --strict. This requires my "refactor shutdown" patch as a pre-requisite.

v4:
001/16:[----] [--] 'python/qmp.py: Define common types' 002/16:[----] [--] 'iotests.py: use qemu.qmp type aliases' 003/16:[----] [--] 'python/qmp.py: re-absorb MonitorResponseError' 004/16:[----] [--] 'python/qmp.py: Do not return None from cmd_obj' 005/16:[----] [--] 'python/qmp.py: add casts to JSON deserialization' 006/16:[----] [--] 'python/qmp.py: add QMPProtocolError' 007/16:[----] [--] 'python/machine.py: Fix monitor address typing' 008/16:[----] [--] 'python/machine.py: reorder __init__' 009/16:[----] [--] 'python/machine.py: Don't modify state in _base_args()' 010/16:[----] [--] 'python/machine.py: Handle None events in events_wait' 011/16:[----] [--] 'python/machine.py: use qmp.command' 012/16:[0010] [FC] 'python/machine.py: Add _qmp access shim' 013/16:[----] [-C] 'python/machine.py: fix _popen access' 014/16:[----] [--] 'python/qemu: make 'args' style arguments immutable' 015/16:[----] [--] 'iotests.py: Adjust HMP kwargs typing' 016/16:[----] [-C] 'python/qemu: Add mypy type annotations'

Quoted 2 lines

- Rebased on "refactor shutdown" v4
- Fixed _qmp access for scripts that disable QMP

See:

https://travis-ci.org/github/philmd/qemu/jobs/702507625#L5439

/ # uname -a
Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
/ # reboot
/ # reboot: Restarting system

Quoted 1 line
Quoted 1 line
Quoted 1 line

{'execute': 'quit'}

qemu received signal 9; command: "mips-softmmu/qemu-system-mips -display
none -vga none -chardev
socket,id=mon,path=/tmp/tmpcojsc5u3/qemu-14540-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmpcojsc5u3/qemu-14540-console.sock,server,nowait
-serial chardev:console -kernel /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta -initrd /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio -append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"

Reproduced traceback from:
/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886 Traceback (most recent call last):
File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py", line 195, in tearDown
vm.shutdown()
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
457, in shutdown
self._do_shutdown(has_quit)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
434, in _do_shutdown
self._soft_shutdown(has_quit, timeout)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
414, in _soft_shutdown

Quoted 8 lines
    self._qmp.cmd('quit')

File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
return self.cmd_obj(qmp_cmd)
File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, in
cmd_obj

    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))

BrokenPipeError: [Errno 32] Broken pipe

Might be racing between QMP going away and Python not being aware that the process has closed yet. It's the only explanation I have left.

I wish I could reproduce this, though. When I submit jobs to Travis I am not seeing these failures.

I'll see what I can do, but at this point I'll not re-send the patch series until I can conclusively fix your build testing.

--js

author
#19 lore raw
Quoted 81 lines · John Snow
Quoted 66 lines · Philippe Mathieu-Daudé

+Cleber/Wainer.

Quoted 23 lines · John Snow

Based-on: [email protected]

This series modifies the python/qemu library to comply with mypy --strict. This requires my "refactor shutdown" patch as a pre-requisite.

v4:
001/16:[----] [--] 'python/qmp.py: Define common types' 002/16:[----] [--] 'iotests.py: use qemu.qmp type aliases' 003/16:[----] [--] 'python/qmp.py: re-absorb MonitorResponseError' 004/16:[----] [--] 'python/qmp.py: Do not return None from cmd_obj' 005/16:[----] [--] 'python/qmp.py: add casts to JSON deserialization' 006/16:[----] [--] 'python/qmp.py: add QMPProtocolError' 007/16:[----] [--] 'python/machine.py: Fix monitor address typing' 008/16:[----] [--] 'python/machine.py: reorder __init__' 009/16:[----] [--] 'python/machine.py: Don't modify state in _base_args()' 010/16:[----] [--] 'python/machine.py: Handle None events in events_wait' 011/16:[----] [--] 'python/machine.py: use qmp.command' 012/16:[0010] [FC] 'python/machine.py: Add _qmp access shim' 013/16:[----] [-C] 'python/machine.py: fix _popen access' 014/16:[----] [--] 'python/qemu: make 'args' style arguments immutable' 015/16:[----] [--] 'iotests.py: Adjust HMP kwargs typing' 016/16:[----] [-C] 'python/qemu: Add mypy type annotations'

Quoted 2 lines

- Rebased on "refactor shutdown" v4
- Fixed _qmp access for scripts that disable QMP

See:

https://travis-ci.org/github/philmd/qemu/jobs/702507625#L5439

/ # uname -a
Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
/ # reboot
/ # reboot: Restarting system

Quoted 1 line
Quoted 1 line
Quoted 1 line

{'execute': 'quit'}

qemu received signal 9; command: "mips-softmmu/qemu-system-mips -display
none -vga none -chardev
socket,id=mon,path=/tmp/tmpcojsc5u3/qemu-14540-monitor.sock -mon
chardev=mon,mode=control -machine malta -chardev
socket,id=console,path=/tmp/tmpcojsc5u3/qemu-14540-console.sock,server,nowait
-serial chardev:console -kernel /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpio/boot/vmlinux-4.5.0-2-4kc-malta -initrd /tmp/avocado_xgmck2k5/avocado_job_83wkzs2f/12-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_mips_malta_cpiorootfs.cpio -append printk.time=0 console=ttyS0 console=tty rdinit=/sbin/init
noreboot -no-reboot"

Reproduced traceback from:
/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/test.py:886 Traceback (most recent call last):
File
"/home/travis/build/philmd/qemu/build/tests/acceptance/avocado_qemu/__init__.py", line 195, in tearDown
vm.shutdown()
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
457, in shutdown
self._do_shutdown(has_quit)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
434, in _do_shutdown
self._soft_shutdown(has_quit, timeout)
File "/home/travis/build/philmd/qemu/python/qemu/machine.py", line
414, in _soft_shutdown

Quoted 8 lines
    self._qmp.cmd('quit')

File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 271, in cmd
return self.cmd_obj(qmp_cmd)
File "/home/travis/build/philmd/qemu/python/qemu/qmp.py", line 249, in
cmd_obj

    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))

BrokenPipeError: [Errno 32] Broken pipe

Might be racing between QMP going away and Python not being aware that the process has closed yet. It's the only explanation I have left.

Quoted 8 lines

I wish I could reproduce this, though. When I submit jobs to Travis I am not seeing these failures.

I'll see what I can do, but at this point I'll not re-send the patch series until I can conclusively fix your build testing.

--js

OK, this is a very big mea culpa. There are two problems.

1. I should catch ConnectionReset and not ConnectionResetError; one is a catch-all for socket problems and the other is a very specific errno that does not include BrokenPipeError.

2. Even if I do, it can still race, actually. QEMU might be in the middle of shutting down, but has already lost the control channel.

Solving the second problem as the interface is currently designed is hard. You can wait, but then if the wait failed, you need to re-raise the control channel error instead of the wait failure. IOW, you need to wait in order to learn if the control channel error is "important" or not.

So, the architecture of this is starting to look wrong; _soft_shutdown should keep clarity of purpose. Either it was able to to a nice, soft shutdown -- or it wasn't.

I'm starting to think that the real problem is that machine.py shouldn't try to hide connection errors on shutdown at all if we expected to be able to issue a quit command.

Changing my mind rapidly that the right thing to do is to actually just fix the test to understand that it should not try to issue a shutdown command, but instead issue a wait command; and removing the race condition from machine.py by simply reporting *all* shutdown errors.

(If callers expect shutdown errors, they can -- and should -- catch those exceptions as desired!)

--js