python: add mypy support to python/qemu
v3 cover python: add mypy support to python/qemu
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
This requires my "refactor shutdown" patch as a pre-requisite.
v3:
001/16:[----] [--] 'python/qmp.py: Define common types' 002/16:[----] [--] 'iotests.py: use qemu.qmp type aliases' 003/16:[----] [--] 'python/qmp.py: re-absorb MonitorResponseError' 004/16:[----] [--] 'python/qmp.py: Do not return None from cmd_obj' 005/16:[0002] [FC] 'python/qmp.py: add casts to JSON deserialization' 006/16:[----] [--] 'python/qmp.py: add QMPProtocolError' 007/16:[----] [-C] 'python/machine.py: Fix monitor address typing' 008/16:[----] [--] 'python/machine.py: reorder __init__' 009/16:[----] [-C] 'python/machine.py: Don't modify state in _base_args()' 010/16:[down] 'python/machine.py: Handle None events in events_wait' 011/16:[----] [--] 'python/machine.py: use qmp.command' 012/16:[----] [--] 'python/machine.py: Add _qmp access shim' 013/16:[----] [--] 'python/machine.py: fix _popen access' 014/16:[----] [--] 'python/qemu: make 'args' style arguments immutable' 015/16:[----] [--] 'iotests.py: Adjust HMP kwargs typing' 016/16:[0007] [FC] 'python/qemu: Add mypy type annotations'
005: Removed a cast, per Kevin Wolf's tip
010: Renamed with correct function name;
Rewrote docstring and added comments
016: Use SocketAddrT instead of Union[Tuple[str,str],str]
"v2":
- This version supports iotests 297
- Many patches merged by Phil are removed
- Replaces iotests.py type aliases with centralized ones (See patch 2)
- Imports etc are reworked to use the non-installable package layout instead. (Mostly important for patch 3)
Testing this out:
- You'll need Python3.6+
- I encourage you to use a virtual environment!
- You don't necessarily need these exact versions, but I didn't test the lower bounds, use older versions at your peril:
- pylint==2.5.0
- mypy=0.770
- flake8=3.7.8
Quoted 5 lines
cd ~/src/qemu/python/
flake8 qemu
mypy --strict qemu
cd qemu
pylint *.py
These should all 100% pass.
---
Open RFC: What's the right way to hook this into make check to prevent regressions?
John Snow (16):
python/qmp.py: Define common types iotests.py: use qemu.qmp type aliases python/qmp.py: re-absorb MonitorResponseError python/qmp.py: Do not return None from cmd_obj python/qmp.py: add casts to JSON deserialization python/qmp.py: add QMPProtocolError python/machine.py: Fix monitor address typing python/machine.py: reorder __init__ python/machine.py: Don't modify state in _base_args() python/machine.py: Handle None events in events_wait python/machine.py: use qmp.command python/machine.py: Add _qmp access shim python/machine.py: fix _popen access python/qemu: make 'args' style arguments immutable iotests.py: Adjust HMP kwargs typing python/qemu: Add mypy type annotations
python/qemu/accel.py | 8 +- python/qemu/machine.py | 286 ++++++++++++++++++++-------------- python/qemu/qmp.py | 111 +++++++++---- python/qemu/qtest.py | 53 ++++--- scripts/render_block_graph.py | 7 +- tests/qemu-iotests/iotests.py | 11 +- 6 files changed, 298 insertions(+), 178 deletions(-)
Signature · 1 line
2.21.3
PATCH v3 1/16 python/qmp.py: Define common types
Define some common types that we'll need to annotate a lot of other functions going forward.
--- python/qemu/qmp.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
python/qemu/qmp.py +18 -0
@@ -12,13 +12,31 @@ import socket import logging from typing import ( + Any, + Dict, Optional, TextIO, Type, + Tuple, + Union, ) from types import TracebackType +# QMPMessage is a QMP Message of any kind. +# e.g. {'yee': 'haw'} +# +# QMPReturnValue is the inner value of return values only. +# {'return': {}} is the QMPMessage, +# {} is the QMPReturnValue. +QMPMessage = Dict[str, Any] +QMPReturnValue = Dict[str, Any] + +InternetAddrT = Tuple[str, str] +UnixAddrT = str +SocketAddrT = Union[InternetAddrT, UnixAddrT] + + class QMPError(Exception): """ QMP base exception
Signature · 1 line
2.21.3
PATCH v3 2/16 iotests.py: use qemu.qmp type aliases
In reply to John Snowiotests.py should use the type definitions from qmp.py instead of its own.
--- tests/qemu-iotests/iotests.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
tests/qemu-iotests/iotests.py +3 -6
@@ -35,13 +35,10 @@ # pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest +from qemu.qmp import QMPMessage assert sys.version_info >= (3, 6) -# Type Aliases -QMPResponse = Dict[str, Any] - - # Use this logger for logging messages directly from the iotests module logger = logging.getLogger('qemu.iotests') logger.addHandler(logging.NullHandler())
@@ -554,7 +551,7 @@ def add_incoming(self, addr): self._args.append(addr) return self - def hmp(self, command_line: str, use_log: bool = False) -> QMPResponse: + def hmp(self, command_line: str, use_log: bool = False) -> QMPMessage: cmd = 'human-monitor-command' kwargs = {'command-line': command_line} if use_log:
@@ -575,7 +572,7 @@ def resume_drive(self, drive: str) -> None: self.hmp(f'qemu-io {drive} "remove_break bp_{drive}"') def hmp_qemu_io(self, drive: str, cmd: str, - use_log: bool = False) -> QMPResponse: + use_log: bool = False) -> QMPMessage: """Write to a given drive using an HMP command""" return self.hmp(f'qemu-io {drive} "{cmd}"', use_log=use_log)
Signature · 1 line
2.21.3
PATCH v3 3/16 python/qmp.py: re-absorb MonitorResponseError
In reply to John SnowWhen I initially split this out, I considered this more of a machine error than a QMP protocol error, but I think that's misguided.
Move this back to qmp.py and name it QMPResponseError. Convert qmp.command() to use this exception type.
--- python/qemu/machine.py | 15 +-------------- python/qemu/qmp.py | 17 +++++++++++++++-- scripts/render_block_graph.py | 7 +++++-- 3 files changed, 21 insertions(+), 18 deletions(-)
python/qemu/machine.py +1 -14
@@ -49,19 +49,6 @@ class QEMUMachineAddDeviceError(QEMUMachineError): """ -class MonitorResponseError(qmp.QMPError): - """ - Represents erroneous QMP monitor reply - """ - def __init__(self, reply): - try: - desc = reply["error"]["desc"] - except KeyError: - desc = reply - super().__init__(desc) - self.reply = reply - - class QEMUMachine: """ A QEMU VM
@@ -469,7 +456,7 @@ def command(self, cmd, conv_keys=True, **args): if reply is None: raise qmp.QMPError("Monitor is closed") if "error" in reply: - raise MonitorResponseError(reply) + raise qmp.QMPResponseError(reply) return reply["return"] def get_qmp_event(self, wait=False):
python/qemu/qmp.py +15 -2
@@ -61,6 +61,19 @@ class QMPTimeoutError(QMPError): """ +class QMPResponseError(QMPError): + """ + Represents erroneous QMP monitor reply + """ + def __init__(self, reply: QMPMessage): + try: + desc = reply['error']['desc'] + except KeyError: + desc = reply + super().__init__(desc) + self.reply = reply + + class QEMUMonitorProtocol: """ Provide an API to connect to QEMU via QEMU Monitor Protocol (QMP) and then
@@ -251,8 +264,8 @@ def command(self, cmd, **kwds): Build and send a QMP command to the monitor, report errors if any """ ret = self.cmd(cmd, kwds) - if "error" in ret: - raise Exception(ret['error']['desc']) + if 'error' in ret: + raise QMPResponseError(ret) return ret['return'] def pull_event(self, wait=False):
scripts/render_block_graph.py +5 -2
@@ -25,7 +25,10 @@ from graphviz import Digraph sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python')) -from qemu.machine import MonitorResponseError +from qemu.qmp import ( + QEMUMonitorProtocol, + QMPResponseError, +) def perm(arr):
@@ -102,7 +105,7 @@ def command(self, cmd): reply = json.loads(subprocess.check_output(ar)) if 'error' in reply: - raise MonitorResponseError(reply) + raise QMPResponseError(reply) return reply['return']
Signature · 1 line
2.21.3
PATCH v3 4/16 python/qmp.py: Do not return None from cmd_obj
In reply to John SnowThis makes typing the qmp library difficult, as it necessitates wrapping Optional[] around the type for every return type up the stack. At some point, it becomes difficult to discern or remember why it's None instead of the expected object.
Use the python exception system to tell us exactly why we didn't get an object. Remove this special-cased return.
--- python/qemu/qmp.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
python/qemu/qmp.py +5 -9
@@ -225,22 +225,18 @@ def accept(self, timeout=15.0): self.__sockfile = self.__sock.makefile(mode='r') return self.__negotiate_capabilities() - def cmd_obj(self, qmp_cmd): + def cmd_obj(self, qmp_cmd: QMPMessage) -> QMPMessage: """ Send a QMP command to the QMP Monitor. @param qmp_cmd: QMP command to be sent as a Python dict - @return QMP response as a Python dict or None if the connection has - been closed + @return QMP response as a Python dict """ self.logger.debug(">>> %s", qmp_cmd) - try: - self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) - except OSError as err: - if err.errno == errno.EPIPE: - return None - raise err + self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8')) resp = self.__json_read() + if resp is None: + raise QMPConnectError("Unexpected empty reply from server") self.logger.debug("<<< %s", resp) return resp
Signature · 1 line
2.21.3
PATCH v3 5/16 python/qmp.py: add casts to JSON deserialization
In reply to John Snowmypy and python type hints are not powerful enough to properly describe JSON messages in Python 3.6. The best we can do, generally, is describe them as Dict[str, Any].
Add casts to coerce this type for static analysis; but do NOT enforce this type at runtime in any way.
Note: Python 3.8 adds a TypedDict construct which allows for the description of more arbitrary Dictionary shapes. There is a third-party module, "Pydantic", which is compatible with 3.6 that can be used instead of the JSON library that parses JSON messages to fully-typed Python objects, and may be preferable in some cases.
(That is well beyond the scope of this commit or series.)
--- python/qemu/qmp.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
python/qemu/qmp.py +6 -2
@@ -13,6 +13,7 @@ import logging from typing import ( Any, + cast, Dict, Optional, TextIO,
@@ -130,7 +131,10 @@ def __json_read(self, only_event=False): data = self.__sockfile.readline() if not data: return None - resp = json.loads(data) + # By definition, any JSON received from QMP is a QMPMessage, + # and we are asserting only at static analysis time that it + # has a particular shape. + resp: QMPMessage = json.loads(data) if 'event' in resp: self.logger.debug("<<< %s", resp) self.__events.append(resp)
@@ -262,7 +266,7 @@ def command(self, cmd, **kwds): ret = self.cmd(cmd, kwds) if 'error' in ret: raise QMPResponseError(ret) - return ret['return'] + return cast(QMPReturnValue, ret['return']) def pull_event(self, wait=False): """
Signature · 1 line
2.21.3
PATCH v3 6/16 python/qmp.py: add QMPProtocolError
In reply to John SnowIn the case that we receive a reply but are unable to understand it, use this exception name to indicate that case.
--- python/qemu/qmp.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
python/qemu/qmp.py +10 -0
@@ -62,6 +62,12 @@ class QMPTimeoutError(QMPError): """ +class QMPProtocolError(QMPError): + """ + QMP protocol error; unexpected response + """ + + class QMPResponseError(QMPError): """ Represents erroneous QMP monitor reply
@@ -266,6 +272,10 @@ def command(self, cmd, **kwds): ret = self.cmd(cmd, kwds) if 'error' in ret: raise QMPResponseError(ret) + if 'return' not in ret: + raise QMPProtocolError( + "'return' key not found in QMP response '{}'".format(str(ret)) + ) return cast(QMPReturnValue, ret['return']) def pull_event(self, wait=False):
Signature · 1 line
2.21.3
PATCH v3 7/16 python/machine.py: Fix monitor address typing
In reply to John SnowPrior to this, it's difficult for mypy to intuit what the concrete type of the monitor address is; it has difficulty inferring the type across two variables.
Create _monitor_address as a property that always returns a valid address to simply static type analysis.
To preserve our ability to clean up, use a simple boolean to indicate whether or not we should try to clean up the sock file after execution.
--- python/qemu/machine.py | 45 +++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 16 deletions(-)
python/qemu/machine.py +29 -16
@@ -25,10 +25,14 @@ import signal import socket import tempfile -from typing import Optional, Type +from typing import ( + Optional, + Type, +) from types import TracebackType from . import qmp +from .qmp import SocketAddrT LOG = logging.getLogger(__name__)
@@ -62,7 +66,8 @@ class QEMUMachine: """ def __init__(self, binary, args=None, wrapper=None, name=None, - test_dir="/var/tmp", monitor_address=None, + test_dir="/var/tmp", + monitor_address: Optional[SocketAddrT] = None, socket_scm_helper=None, sock_dir=None): ''' Initialize a QEMUMachine
@@ -85,8 +90,14 @@ def __init__(self, binary, args=None, wrapper=None, name=None, if sock_dir is None: sock_dir = test_dir self._name = name - self._monitor_address = monitor_address - self._vm_monitor = None + if monitor_address is not None: + self._monitor_address = monitor_address + self._remove_monitor_sockfile = False + else: + self._monitor_address = os.path.join( + sock_dir, f"{name}-monitor.sock" + ) + self._remove_monitor_sockfile = True self._qemu_log_path = None self._qemu_log_file = None self._popen = None
@@ -225,15 +236,17 @@ def _load_io_log(self): def _base_args(self): args = ['-display', 'none', '-vga', 'none'] + if self._qmp_set: if isinstance(self._monitor_address, tuple): - moncdev = "socket,id=mon,host=%s,port=%s" % ( - self._monitor_address[0], - self._monitor_address[1]) + moncdev = "socket,id=mon,host={},port={}".format( + *self._monitor_address + ) else: - moncdev = 'socket,id=mon,path=%s' % self._vm_monitor + moncdev = f"socket,id=mon,path={self._monitor_address}" args.extend(['-chardev', moncdev, '-mon', 'chardev=mon,mode=control']) + if self._machine is not None: args.extend(['-machine', self._machine]) for _ in range(self._console_index):
@@ -258,14 +271,14 @@ def _pre_launch(self): self._qemu_log_file = open(self._qemu_log_path, 'wb') if self._qmp_set: - if self._monitor_address is not None: - self._vm_monitor = self._monitor_address - else: - self._vm_monitor = os.path.join(self._sock_dir, - self._name + "-monitor.sock") - self._remove_files.append(self._vm_monitor) - self._qmp = qmp.QEMUMonitorProtocol(self._vm_monitor, server=True, - nickname=self._name) + if self._remove_monitor_sockfile: + assert isinstance(self._monitor_address, str) + self._remove_files.append(self._monitor_address) + self._qmp = qmp.QEMUMonitorProtocol( + self._monitor_address, + server=True, + nickname=self._name + ) def _post_launch(self): if self._qmp:
Signature · 1 line
2.21.3
PATCH v3 8/16 python/machine.py: reorder __init__
In reply to John SnowPut the init arg handling all at the top, and mostly in order (deviating when one is dependent on another), and put what is effectively runtime state declaration at the bottom.
--- python/qemu/machine.py | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
python/qemu/machine.py +17 -12
@@ -81,38 +81,43 @@ def __init__(self, binary, args=None, wrapper=None, name=None, @param socket_scm_helper: helper program, required for send_fd_scm() @note: Qemu process is not started until launch() is used. ''' + # Direct user configuration + + self._binary = binary + if args is None: args = [] + # Copy mutable input: we will be modifying our copy + self._args = list(args) + if wrapper is None: wrapper = [] - if name is None: - name = "qemu-%d" % os.getpid() - if sock_dir is None: - sock_dir = test_dir - self._name = name + self._wrapper = wrapper + + self._name = name or "qemu-%d" % os.getpid() + self._test_dir = test_dir + self._sock_dir = sock_dir or self._test_dir + self._socket_scm_helper = socket_scm_helper + if monitor_address is not None: self._monitor_address = monitor_address self._remove_monitor_sockfile = False else: self._monitor_address = os.path.join( - sock_dir, f"{name}-monitor.sock" + self._sock_dir, f"{self._name}-monitor.sock" ) self._remove_monitor_sockfile = True + + # Runstate self._qemu_log_path = None self._qemu_log_file = None self._popen = None - self._binary = binary - self._args = list(args) # Force copy args in case we modify them - self._wrapper = wrapper self._events = [] self._iolog = None - self._socket_scm_helper = socket_scm_helper self._qmp_set = True # Enable QMP monitor by default. self._qmp = None self._qemu_full_args = None - self._test_dir = test_dir self._temp_dir = None - self._sock_dir = sock_dir self._launched = False self._machine = None self._console_index = 0
Signature · 1 line
2.21.3
PATCH v3 9/16 python/machine.py: Don't modify state in _base_args()
In reply to John SnowDon't append to the _remove_files list during _base_args; instead do so during _launch. Rework _base_args as a @property to help facilitate this impression.
This has the additional benefit of making the type of _console_address easier to analyze statically.
--- python/qemu/machine.py | 16 ++++++++++------ python/qemu/qtest.py | 11 ++++++++--- 2 files changed, 18 insertions(+), 9 deletions(-)
python/qemu/machine.py +10 -6
@@ -26,6 +26,7 @@ import socket import tempfile from typing import ( + List, Optional, Type, )
@@ -123,7 +124,9 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._console_index = 0 self._console_set = False self._console_device_type = None - self._console_address = None + self._console_address = os.path.join( + self._sock_dir, f"{self._name}-console.sock" + ) self._console_socket = None self._remove_files = [] self._killed = False
@@ -239,7 +242,8 @@ def _load_io_log(self): with open(self._qemu_log_path, "r") as iolog: self._iolog = iolog.read() - def _base_args(self): + @property + def _base_args(self) -> List[str]: args = ['-display', 'none', '-vga', 'none'] if self._qmp_set:
@@ -257,9 +261,6 @@ def _base_args(self): for _ in range(self._console_index): args.extend(['-serial', 'null']) if self._console_set: - self._console_address = os.path.join(self._sock_dir, - self._name + "-console.sock") - self._remove_files.append(self._console_address) chardev = ('socket,id=console,path=%s,server,nowait' % self._console_address) args.extend(['-chardev', chardev])
@@ -275,6 +276,9 @@ def _pre_launch(self): self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log") self._qemu_log_file = open(self._qemu_log_path, 'wb') + if self._console_set: + self._remove_files.append(self._console_address) + if self._qmp_set: if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str)
@@ -349,7 +353,7 @@ def _launch(self): devnull = open(os.path.devnull, 'rb') self._pre_launch() self._qemu_full_args = (self._wrapper + [self._binary] + - self._base_args() + self._args) + self._base_args + self._args) LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args)) self._popen = subprocess.Popen(self._qemu_full_args, stdin=devnull,
python/qemu/qtest.py +8 -3
@@ -19,7 +19,11 @@ import socket import os -from typing import Optional, TextIO +from typing import ( + List, + Optional, + TextIO, +) from .machine import QEMUMachine
@@ -111,8 +115,9 @@ def __init__(self, binary, args=None, name=None, test_dir="/var/tmp", self._qtest = None self._qtest_path = os.path.join(sock_dir, name + "-qtest.sock") - def _base_args(self): - args = super()._base_args() + @property + def _base_args(self) -> List[str]: + args = super()._base_args args.extend(['-qtest', 'unix:path=' + self._qtest_path, '-accel', 'qtest']) return args
Signature · 1 line
2.21.3
PATCH v3 10/16 python/machine.py: Handle None events in events_wait
In reply to John SnowIf the timeout is 0, we can get None back. Handle this explicitly.
--- python/qemu/machine.py | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
python/qemu/machine.py +20 -7
@@ -33,7 +33,7 @@ from types import TracebackType from . import qmp -from .qmp import SocketAddrT +from .qmp import SocketAddrT, QMPMessage LOG = logging.getLogger(__name__)
@@ -544,13 +544,20 @@ def event_wait(self, name, timeout=60.0, match=None): def events_wait(self, events, timeout=60.0): """ - events_wait waits for and returns a named event - from QMP with a timeout. + events_wait waits for and returns a single named event from QMP. + In the case of multiple qualifying events, this function returns the + first one. - events: a sequence of (name, match_criteria) tuples. - The match criteria are optional and may be None. - See event_match for details. - timeout: QEMUMonitorProtocol.pull_event timeout parameter. + :param events: A sequence of (name, match_criteria) tuples. + The match criteria are optional and may be None. + See event_match for details. + :param timeout: Optional timeout, in seconds. + See QEMUMonitorProtocol.pull_event. + + :raise QMPTimeoutError: If timeout was non-zero and no matching events + were found. + :return: A QMP event matching the filter criteria. + If timeout was 0 and no event matched, None. """ def _match(event): for name, match in events:
@@ -558,6 +565,8 @@ def _match(event): return True return False + event: Optional[QMPMessage] + # Search cached events for event in self._events: if _match(event):
@@ -567,6 +576,10 @@ def _match(event): # Poll for new events while True: event = self._qmp.pull_event(wait=timeout) + if event is None: + # NB: None is only returned when timeout is false-ish. + # Timeouts raise QMPTimeoutError instead! + break if _match(event): return event self._events.append(event)
Signature · 1 line
2.21.3
PATCH v3 11/16 python/machine.py: use qmp.command
In reply to John Snowmachine.py and qmp.py both do the same thing here; refactor machine.py to use qmp.py's functionality more directly.
--- python/qemu/machine.py | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)
python/qemu/machine.py +15 -11
@@ -26,6 +26,8 @@ import socket import tempfile from typing import ( + Any, + Dict, List, Optional, Type,
@@ -455,17 +457,23 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = False self._qmp = None - def qmp(self, cmd, conv_keys=True, **args): - """ - Invoke a QMP command and return the response dict - """ + @classmethod + def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]: qmp_args = dict() for key, value in args.items(): - if conv_keys: + if _conv_keys: qmp_args[key.replace('_', '-')] = value else: qmp_args[key] = value + return qmp_args + def qmp(self, cmd: str, + conv_keys: bool = True, + **args: Any) -> QMPMessage: + """ + Invoke a QMP command and return the response dict + """ + qmp_args = self._qmp_args(conv_keys, **args) return self._qmp.cmd(cmd, args=qmp_args) def command(self, cmd, conv_keys=True, **args):
@@ -474,12 +482,8 @@ def command(self, cmd, conv_keys=True, **args): On success return the response dict. On failure raise an exception. """ - reply = self.qmp(cmd, conv_keys, **args) - if reply is None: - raise qmp.QMPError("Monitor is closed") - if "error" in reply: - raise qmp.QMPResponseError(reply) - return reply["return"] + qmp_args = self._qmp_args(conv_keys, **args) + return self._qmp.command(cmd, **qmp_args) def get_qmp_event(self, wait=False): """
Signature · 1 line
2.21.3
PATCH v3 12/16 python/machine.py: Add _qmp access shim
In reply to John SnowLike many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
python/qemu/machine.py +9 -3
@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False
@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name
@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connection @classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:
Signature · 1 line
2.21.3
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
Quoted 37 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +2
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 10 lines
python/qemu/machine.py +7
-0+ self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connection @classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:
This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
Quoted 46 lines · Philippe Mathieu-Daudé
Quoted 44 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +9
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connection
Quoted 8 lines
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 3 lines
python/qemu/machine.py +0
-0@classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)
Fixed with:
-- >8 --
python/qemu/machine.py +1 -1
@@ -480,7 +480,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None: @property def _qmp(self) -> qmp.QEMUMonitorProtocol: - if self._qmp_connection is None: + if self._qmp_set and self._qmp_connection is None: raise QEMUMachineError("Attempt to access QMP with no connection") return self._qmp_connection
---
Does that sound reasonable to you?
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
Am 20.06.2020 um 10:20 hat Philippe Mathieu-Daudé geschrieben:
Quoted 75 lines
Quoted 46 lines · Philippe Mathieu-Daudé
Quoted 44 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +9
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connectionQuoted 8 lines
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 3 lines
python/qemu/machine.py +0
-0@classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)Fixed with:
-- >8 --
python/qemu/machine.py +1
-2@@ -480,7 +480,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None: @property def _qmp(self) -> qmp.QEMUMonitorProtocol: - if self._qmp_connection is None: + if self._qmp_set and self._qmp_connection is None: raise QEMUMachineError("Attempt to access QMP with no connection") return self._qmp_connection ---
Quoted 1 line
Does that sound reasonable to you?
Wouldn't that make the return type Optional[qmp.QEMUMonitorProtocol]? Maybe this is what we want, but then we don't need the shim that this patch adds but can just declare the variable this way.
And why does the feeling code even try to acess _qmp when _qmp_set is False? Shouldn't it first check whether it's even valid?
Or maybe going a step back, why do we even have a separate _qmp_set instead of only using None for _qmp?
Kevin
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
Quoted 82 lines · Kevin Wolf
Am 20.06.2020 um 10:20 hat Philippe Mathieu-Daudé geschrieben:
Quoted 75 lines
Quoted 46 lines · Philippe Mathieu-Daudé
Quoted 44 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +9
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connectionQuoted 8 lines
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 3 lines
python/qemu/machine.py +0
-0@classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)Fixed with:
-- >8 --
python/qemu/machine.py +1
-2@@ -480,7 +480,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None: @property def _qmp(self) -> qmp.QEMUMonitorProtocol: - if self._qmp_connection is None: + if self._qmp_set and self._qmp_connection is None: raise QEMUMachineError("Attempt to access QMP with no connection") return self._qmp_connection ---Quoted 1 line
Does that sound reasonable to you?
Wouldn't that make the return type Optional[qmp.QEMUMonitorProtocol]? Maybe this is what we want, but then we don't need the shim that this patch adds but can just declare the variable this way.
Quoted 5 lines
And why does the feeling code even try to acess
_qmpwhen_qmp_setis False? Shouldn't it first check whether it's even valid?Or maybe going a step back, why do we even have a separate
_qmp_setinstead of only using None for_qmp?
Better indeed.
John, at this point from a maintenance perspective it is easier if you respin the series (and please, run the Travis-CI tests).
Regards,
Phil.
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
Quoted 75 lines · Philippe Mathieu-Daudé
Quoted 74 lines · Kevin Wolf
Am 20.06.2020 um 10:20 hat Philippe Mathieu-Daudé geschrieben:
Quoted 73 lines
Quoted 46 lines · Philippe Mathieu-Daudé
Quoted 44 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +9
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connectionQuoted 8 lines
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 3 lines
python/qemu/machine.py +0
-0@classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)Fixed with:
-- >8 --
python/qemu/machine.py +1
-1@@ -480,7 +480,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None: @property def _qmp(self) -> qmp.QEMUMonitorProtocol: - if self._qmp_connection is None: + if self._qmp_set and self._qmp_connection is None: raise QEMUMachineError("Attempt to access QMP with no connection") return self._qmp_connection
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
---
Does that sound reasonable to you?
Not entirely; the idea is to protect access to the socket in all cases where the socket was not created, for any and all reasons.
Quoted 4 lines
Quoted 4 lines
Wouldn't that make the return type Optional[qmp.QEMUMonitorProtocol]? Maybe this is what we want, but then we don't need the shim that this patch adds but can just declare the variable this way.
The _qmp_connection variable is already Optional[], so you could indeed remove the shim. The shim is here to peel away the Optional[] so you don't need the "if qmp" checks everywhere.
Quoted 3 lines
Quoted 3 lines
And why does the feeling code even try to acess
_qmpwhen_qmp_setis False? Shouldn't it first check whether it's even valid?
Yup. The code in lifetime management functions should check for the socket. Code in qmp-centric commands should assume it's there with the shim.
Quoted 2 lines
Quoted 2 lines
Or maybe going a step back, why do we even have a separate
_qmp_setinstead of only using None for_qmp?
Because the socket isn't initialized until after launch, so "_qmp_set" is a request/instruction and not reflective of current state.
_qmp_set is the flag for a desire to create QMP or not, _qmp_connection is the actual connection.
set | conn | T F | OK (we are pre-launch) T T | OK (we are running) F F | OK (we set no QMP before launch) F T | Set QMP false after launch
The last one doesn't make a lot of sense. Worse, it just None's the socket variable so we can't even close it properly.
That's sorta busted, so I am patching it to just simply set the variable.
Quoted 10 lines
Better indeed.
John, at this point from a maintenance perspective it is easier if you respin the series (and please, run the Travis-CI tests).
Regards,
Phil.
OK, I am resending. I didn't make any big changes, but I thought about it.
(Read as: I tried, didn't like how it looked, and came back to this patchset which is fairly close as-is.)
--js
Re: [PATCH v3 12/16] python/machine.py: Add _qmp access shim
In reply to Kevin WolfQuoted 82 lines · Kevin Wolf
Am 20.06.2020 um 10:20 hat Philippe Mathieu-Daudé geschrieben:
Quoted 75 lines
Quoted 46 lines · Philippe Mathieu-Daudé
Quoted 44 lines · John Snow
Like many other Optional[] types, it's not always a given that this object will be set. Wrap it in a type-shim that raises a meaningful error and will always return a concrete type.
Signed-off-by John Snow <[email protected]>--- python/qemu/machine.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)python/qemu/machine.py +9
-3@@ -118,7 +118,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default. - self._qmp = None + self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None self._qemu_full_args = None self._temp_dir = None self._launched = False@@ -285,7 +285,7 @@ def _pre_launch(self): if self._remove_monitor_sockfile: assert isinstance(self._monitor_address, str) self._remove_files.append(self._monitor_address) - self._qmp = qmp.QEMUMonitorProtocol( + self._qmp_connection = qmp.QEMUMonitorProtocol( self._monitor_address, server=True, nickname=self._name@@ -455,7 +455,13 @@ def set_qmp_monitor(self, enabled=True): self._qmp_set = True else: self._qmp_set = False - self._qmp = None + self._qmp_connection = None + + @property + def _qmp(self) -> qmp.QEMUMonitorProtocol: + if self._qmp_connection is None: + raise QEMUMachineError("Attempt to access QMP with no connection") + return self._qmp_connectionQuoted 8 lines
python/qemu/machine.py def set_qmp_monitor(self, enabled=True): · line 467
Quoted 3 lines
python/qemu/machine.py +0
-0@classmethod def _qmp_args(cls, _conv_keys: bool = True, **args: Any) -> Dict[str, Any]:This patch breaks the EmptyCPUModel test:
(043/101) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
ERROR: Attempt to access QMP with no connection (0.03 s)Fixed with:
-- >8 --
python/qemu/machine.py +1
-2@@ -480,7 +480,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None: @property def _qmp(self) -> qmp.QEMUMonitorProtocol: - if self._qmp_connection is None: + if self._qmp_set and self._qmp_connection is None: raise QEMUMachineError("Attempt to access QMP with no connection") return self._qmp_connection ---Quoted 1 line
Does that sound reasonable to you?
Wouldn't that make the return type Optional[qmp.QEMUMonitorProtocol]? Maybe this is what we want, but then we don't need the shim that this patch adds but can just declare the variable this way.
Quoted 8 lines
And why does the feeling code even try to acess
_qmpwhen_qmp_setis False? Shouldn't it first check whether it's even valid?Or maybe going a step back, why do we even have a separate
_qmp_setinstead of only using None for_qmp?Kevin
Using Optional[qmp.QEMUMonitorProtocol] does work; it does require a lot more "if not None" checks littered everywhere.
From memory, it was just difficult to type a qmp(...) passthrough command that integrated the "if self._qmp_socket" check -- the faster hack was the shim around accessing the socket itself.
IIRC, mypy had difficulty reasoning about constraints otherwise -- but I'll look into it again. At this point it warrants stronger effort.
--js
PATCH v3 13/16 python/machine.py: fix _popen access
In reply to John SnowAs always, Optional[T] causes problems with unchecked access. Add a helper that asserts the pipe is present before we attempt to talk with it.
--- python/qemu/machine.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
python/qemu/machine.py +12 -6
@@ -114,7 +114,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, # Runstate self._qemu_log_path = None self._qemu_log_file = None - self._popen = None + self._popen: Optional['subprocess.Popen[bytes]'] = None self._events = [] self._iolog = None self._qmp_set = True # Enable QMP monitor by default.
@@ -227,6 +227,12 @@ def is_running(self): """Returns true if the VM is running.""" return self._popen is not None and self._popen.poll() is None + @property + def _subp(self) -> 'subprocess.Popen[bytes]': + if self._popen is None: + raise QEMUMachineError('Subprocess pipe not present') + return self._popen + def exitcode(self): """Returns the exit code if possible, or None.""" if self._popen is None:
@@ -237,7 +243,7 @@ def get_pid(self): """Returns the PID of the running process, or None.""" if not self.is_running(): return None - return self._popen.pid + return self._subp.pid def _load_io_log(self): if self._qemu_log_path is not None:
@@ -369,7 +375,7 @@ def wait(self): """ Wait for the VM to power off """ - self._popen.wait() + self._subp.wait() if self._qmp: self._qmp.close() self._post_shutdown()
@@ -381,8 +387,8 @@ def _hard_shutdown(self) -> None: if not self.is_running(): return - self._popen.kill() - self._popen.wait(timeout=60) + self._subp.kill() + self._subp.wait(timeout=60) def _soft_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None: """
@@ -399,7 +405,7 @@ def _soft_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None: self._qmp.cmd('quit') self._qmp.close() - self._popen.wait(timeout=timeout) + self._subp.wait(timeout=timeout) def _do_shutdown(self, has_quit: bool = False, timeout: int = 3) -> None: """
Signature · 1 line
2.21.3
PATCH v3 14/16 python/qemu: make 'args' style arguments immutable
In reply to John SnowThese arguments don't need to be mutable and aren't really used as such. Clarify their types as immutable and adjust code to match where necessary.
In general, It's probably best not to accept a user-defined mutable object and store it as internal object state unless there's a strong justification for doing so. Instead, try to use generic types as input with empty tuples as the default, and coerce to list where necessary.
--- python/qemu/machine.py | 30 +++++++++++++++++------------- python/qemu/qtest.py | 16 ++++++++++++---- 2 files changed, 29 insertions(+), 17 deletions(-)
python/qemu/machine.py +17 -13
@@ -18,6 +18,7 @@ # import errno +from itertools import chain import logging import os import subprocess
@@ -30,6 +31,8 @@ Dict, List, Optional, + Sequence, + Tuple, Type, ) from types import TracebackType
@@ -68,8 +71,12 @@ class QEMUMachine: # vm is guaranteed to be shut down here """ - def __init__(self, binary, args=None, wrapper=None, name=None, - test_dir="/var/tmp", + def __init__(self, + binary: str, + args: Sequence[str] = (), + wrapper: Sequence[str] = (), + name: Optional[str] = None, + test_dir: str = "/var/tmp", monitor_address: Optional[SocketAddrT] = None, socket_scm_helper=None, sock_dir=None): '''
@@ -87,14 +94,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, # Direct user configuration self._binary = binary - - if args is None: - args = [] - # Copy mutable input: we will be modifying our copy self._args = list(args) - - if wrapper is None: - wrapper = [] self._wrapper = wrapper self._name = name or "qemu-%d" % os.getpid()
@@ -119,7 +119,7 @@ def __init__(self, binary, args=None, wrapper=None, name=None, self._iolog = None self._qmp_set = True # Enable QMP monitor by default. self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None - self._qemu_full_args = None + self._qemu_full_args: Tuple[str, ...] = () self._temp_dir = None self._launched = False self._machine = None
@@ -340,7 +340,7 @@ def launch(self): raise QEMUMachineError('VM already launched') self._iolog = None - self._qemu_full_args = None + self._qemu_full_args = () try: self._launch() self._launched = True
@@ -360,8 +360,12 @@ def _launch(self): """ devnull = open(os.path.devnull, 'rb') self._pre_launch() - self._qemu_full_args = (self._wrapper + [self._binary] + - self._base_args + self._args) + self._qemu_full_args = tuple( + chain(self._wrapper, + [self._binary], + self._base_args, + self._args) + ) LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args)) self._popen = subprocess.Popen(self._qemu_full_args, stdin=devnull,
python/qemu/qtest.py +12 -4
@@ -22,6 +22,7 @@ from typing import ( List, Optional, + Sequence, TextIO, )
@@ -103,8 +104,13 @@ class QEMUQtestMachine(QEMUMachine): A QEMU VM, with a qtest socket available. """ - def __init__(self, binary, args=None, name=None, test_dir="/var/tmp", - socket_scm_helper=None, sock_dir=None): + def __init__(self, + binary: str, + args: Sequence[str] = (), + name: Optional[str] = None, + test_dir: str = "/var/tmp", + socket_scm_helper: Optional[str] = None, + sock_dir: Optional[str] = None): if name is None: name = "qemu-%d" % os.getpid() if sock_dir is None:
@@ -118,8 +124,10 @@ def __init__(self, binary, args=None, name=None, test_dir="/var/tmp", @property def _base_args(self) -> List[str]: args = super()._base_args - args.extend(['-qtest', 'unix:path=' + self._qtest_path, - '-accel', 'qtest']) + args.extend([ + '-qtest', f"unix:path={self._qtest_path}", + '-accel', 'qtest' + ]) return args def _pre_launch(self):
Signature · 1 line
2.21.3
PATCH v3 15/16 iotests.py: Adjust HMP kwargs typing
In reply to John Snowmypy wants to ensure there's consistency between the kwargs arguments types and any unspecified keyword arguments. In this case, conv_keys is a bool, but the remaining keys are Any type. Mypy (correctly) infers the **kwargs type to be **Dict[str, str], which is not compatible withconv_keys: bool.
Because QMP typing is a little fraught right now anyway, re-type kwargs to Dict[str, Any] which has the benefit of silencing this check right now.
A future re-design might type these more aggressively, but this will give us a baseline to work from with minimal disruption.
(Thanks Kevin Wolf for the debugging assist here)
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -553,7 +553,7 @@ def add_incoming(self, addr): def hmp(self, command_line: str, use_log: bool = False) -> QMPMessage: cmd = 'human-monitor-command' - kwargs = {'command-line': command_line} + kwargs: Dict[str, Any] = {'command-line': command_line} if use_log: return self.qmp_log(cmd, **kwargs) else:
Signature · 1 line
2.21.3
PATCH v3 16/16 python/qemu: Add mypy type annotations
In reply to John SnowThese should all be purely annotations with no changes in behavior at all. You need to be in the python folder, but you should be able to confirm that these annotations are correct (or at least self-consistent) by running mypy --strict qemu.
--- python/qemu/accel.py | 8 ++-- python/qemu/machine.py | 94 ++++++++++++++++++++++++------------------ python/qemu/qmp.py | 44 +++++++++++--------- python/qemu/qtest.py | 26 +++++++----- 4 files changed, 98 insertions(+), 74 deletions(-)
python/qemu/accel.py +5 -3
@@ -17,6 +17,7 @@ import logging import os import subprocess +from typing import List, Optional LOG = logging.getLogger(__name__)
@@ -29,7 +30,7 @@ } -def list_accel(qemu_bin): +def list_accel(qemu_bin: str) -> List[str]: """ List accelerators enabled in the QEMU binary.
@@ -49,7 +50,8 @@ def list_accel(qemu_bin): return [acc.strip() for acc in out.splitlines()[1:]] -def kvm_available(target_arch=None, qemu_bin=None): +def kvm_available(target_arch: Optional[str] = None, + qemu_bin: Optional[str] = None) -> bool: """ Check if KVM is available using the following heuristic: - Kernel module is present in the host;
@@ -72,7 +74,7 @@ def kvm_available(target_arch=None, qemu_bin=None): return True -def tcg_available(qemu_bin): +def tcg_available(qemu_bin: str) -> bool: """ Check if TCG is available.
python/qemu/machine.py +53 -41
295 lines not on this page · raw message
python/qemu/qmp.py +25 -19
163 lines not on this page · raw message
python/qemu/qtest.py +15 -11
@@ -27,6 +27,7 @@ ) from .machine import QEMUMachine +from .qmp import SocketAddrT class QEMUQtestProtocol:
@@ -43,7 +44,8 @@ class QEMUQtestProtocol: No conection is estabalished by __init__(), this is done by the connect() or accept() methods. """ - def __init__(self, address, server=False): + def __init__(self, address: SocketAddrT, + server: bool = False): self._address = address self._sock = self._get_sock() self._sockfile: Optional[TextIO] = None
@@ -51,14 +53,14 @@ def __init__(self, address, server=False): self._sock.bind(self._address) self._sock.listen(1) - def _get_sock(self): + def _get_sock(self) -> socket.socket: if isinstance(self._address, tuple): family = socket.AF_INET else: family = socket.AF_UNIX return socket.socket(family, socket.SOCK_STREAM) - def connect(self): + def connect(self) -> None: """ Connect to the qtest socket.
@@ -67,7 +69,7 @@ def connect(self): self._sock.connect(self._address) self._sockfile = self._sock.makefile(mode='r') - def accept(self): + def accept(self) -> None: """ Await connection from QEMU.
@@ -76,7 +78,7 @@ def accept(self): self._sock, _ = self._sock.accept() self._sockfile = self._sock.makefile(mode='r') - def cmd(self, qtest_cmd): + def cmd(self, qtest_cmd: str) -> str: """ Send a qtest command on the wire.
@@ -87,14 +89,16 @@ def cmd(self, qtest_cmd): resp = self._sockfile.readline() return resp - def close(self): - """Close this socket.""" + def close(self) -> None: + """ + Close this socket. + """ self._sock.close() if self._sockfile: self._sockfile.close() self._sockfile = None - def settimeout(self, timeout): + def settimeout(self, timeout: Optional[float]) -> None: """Set a timeout, in seconds.""" self._sock.settimeout(timeout)
@@ -118,7 +122,7 @@ def __init__(self, super().__init__(binary, args, name=name, test_dir=test_dir, socket_scm_helper=socket_scm_helper, sock_dir=sock_dir) - self._qtest = None + self._qtest: Optional[QEMUQtestProtocol] = None self._qtest_path = os.path.join(sock_dir, name + "-qtest.sock") @property
@@ -130,7 +134,7 @@ def _base_args(self) -> List[str]: ]) return args - def _pre_launch(self): + def _pre_launch(self) -> None: super()._pre_launch() self._qtest = QEMUQtestProtocol(self._qtest_path, server=True)
@@ -139,7 +143,7 @@ def _post_launch(self) -> None: super()._post_launch() self._qtest.accept() - def _post_shutdown(self): + def _post_shutdown(self) -> None: super()._post_shutdown() self._remove_if_exists(self._qtest_path)
Signature · 1 line
2.21.3
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 7 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Quoted 7 lines · Kevin Wolf
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>
I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I stg push; stg edit and copy-paste the R-B into it.
It's easier when I'm just pulling other people's patches from the ML, the patches tool handles it for me. It's updating my own patches that's a drag and prone to error.
--js
Am 08.06.2020 um 17:19 hat John Snow geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 7 lines
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Quoted 8 lines
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I
stg push; stg editand copy-paste the R-B into it.It's easier when I'm just pulling other people's patches from the ML, the patches tool handles it for me. It's updating my own patches that's a drag and prone to error.
It's a manual process for me, too. Just that I don't use stgit, but 'git rebase -i'.
Kevin
Quoted 26 lines · Kevin Wolf
Am 08.06.2020 um 17:19 hat John Snow geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 7 lines
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I
stg push; stg editand copy-paste the R-B into it.
wget https://patchew.org/QEMU/${MSG_ID}/mbox
git am mbox
Where ${MSG_ID} is the Message-Id of the series cover letter.
Quoted 10 lines
Quoted 4 lines
It's easier when I'm just pulling other people's patches from the ML, the patches tool handles it for me. It's updating my own patches that's a drag and prone to error.
It's a manual process for me, too. Just that I don't use stgit, but 'git rebase -i'.
Kevin
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 26 lines · Kevin Wolf
Am 08.06.2020 um 17:19 hat John Snow geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 7 lines
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I
stg push; stg editand copy-paste the R-B into it.
Quoted 4 lines
wget https://patchew.org/QEMU/${MSG_ID}/mbox
git am mboxWhere ${MSG_ID} is the Message-Id of the series cover letter.
Patchew's awesomeness is still under-appreciated.
applying mailing list review tags (was: Re: [PATCH v3 00/16] python: add mypy support to python/qemu)
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 26 lines · Kevin Wolf
Am 08.06.2020 um 17:19 hat John Snow geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 7 lines
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I
stg push; stg editand copy-paste the R-B into it.
Quoted 7 lines
Quoted 4 lines
wget https://patchew.org/QEMU/${MSG_ID}/mbox
git am mboxWhere ${MSG_ID} is the Message-Id of the series cover letter.
Patchew's awesomeness is still under-appreciated.
Not for lack of appreciating patchew, but the problem with this workflow is if I have already made modifications to my patches locally, I can't use this to apply tags from upstream.
It looks like I will continue to do this manually for the time being; but scripting the ability to "merge tags" from the list would be a cool trick.
I'm not sure how to do it with git, though. Let's say I've got 16 patches and I've made modifications to some, but not all; so I have a branch with 16 patches ahead of origin/master.
Does anyone have any cool tricks for being able to script:
1. Correlating a mailing list patch from e.g. patchew to a commit in my history, even if it's changed a little bit?
(git-backport-diff uses patch names, that might be sufficient... Could use that as a starting point, at least.)
2. Obtaining the commit message of that patch?git show -s --format=%B $SHA ought to do it...
3. Editing that commit message? This I'm not sure about. I'd need to understand the tags on the upstream and downstream versions, merge them, and then re-write the message. Some magic with git rebase -i ?
--js
Re: applying mailing list review tags (was: Re: [PATCH v3 00/16] python: add mypy support to python/qemu)
Quoted 36 lines · John Snow
Quoted 33 lines · Markus Armbruster
Quoted 31 lines · Philippe Mathieu-Daudé
Quoted 26 lines · Kevin Wolf
Am 08.06.2020 um 17:19 hat John Snow geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 7 lines
Am 04.06.2020 um 22:22 hat John Snow geschrieben:
Quoted 6 lines
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
Quoted 6 lines
Quoted 1 line
This requires my "refactor shutdown" patch as a pre-requisite.
You didn't like my previous R-b? Here's a new one. :-)
Reviewed-by Kevin Wolf <[email protected]>I felt like I should address the feedback, and though I could have applied the R-B to patches I didn't change, it was ... faster to just re-send it.
Serious question: How do you apply people's R-Bs to your patches? At the moment, it's pretty manually intensive for me. I use stgit and I pop all of the patches off (stg pop -n 100), and then one-at-a-time I
stg push; stg editand copy-paste the R-B into it.wget https://patchew.org/QEMU/${MSG_ID}/mbox
git am mbox
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Where ${MSG_ID} is the Message-Id of the series cover letter.
Patchew's awesomeness is still under-appreciated.
Not for lack of appreciating patchew, but the problem with this workflow is if I have already made modifications to my patches locally, I can't use this to apply tags from upstream.
Does that mean you want to respin this series?
Else you can consider it applied on python-next.
Quoted 26 lines
It looks like I will continue to do this manually for the time being; but scripting the ability to "merge tags" from the list would be a cool trick.
I'm not sure how to do it with git, though. Let's say I've got 16 patches and I've made modifications to some, but not all; so I have a branch with 16 patches ahead of origin/master.
Does anyone have any cool tricks for being able to script:
1. Correlating a mailing list patch from e.g. patchew to a commit in my history, even if it's changed a little bit?
(git-backport-diff uses patch names, that might be sufficient... Could use that as a starting point, at least.)
2. Obtaining the commit message of that patch?
git show -s --format=%B $SHAought to do it...3. Editing that commit message? This I'm not sure about. I'd need to understand the tags on the upstream and downstream versions, merge them, and then re-write the message. Some magic with
git rebase -i?--js
Quoted 2 lines · Philippe Mathieu-Daudé
Does that mean you want to respin this series?
Else you can consider it applied on python-next.
No no - you're all set. Just taking the opportunity to discuss tooling that I find cumbersome at the moment.
Thank you Philippe :)
--js
Re: applying mailing list review tags (was: Re: [PATCH v3 00/16] python: add mypy support to python/qemu)
In reply to John SnowQuoted 2 lines · John Snow
1. Correlating a mailing list patch from e.g. patchew to a commit in my history, even if it's changed a little bit?
Use "git am --message-id"?
Quoted 9 lines
(git-backport-diff uses patch names, that might be sufficient... Could use that as a starting point, at least.)
2. Obtaining the commit message of that patch?
git show -s --format=%B $SHAought to do it...3. Editing that commit message? This I'm not sure about. I'd need to understand the tags on the upstream and downstream versions, merge them, and then re-write the message. Some magic with
git rebase -i?
"patchew apply" actually uses "git filter-branch --msg-filter" to add the tags after a successful "git am", so you can do something similar yourself. (Actually I have pending patches to patchew that switch it to server-side application of tags using the "mbox" URL that Philippe mentioned earlier, but they've been pending for quite some time now).
To get the upstream tags you can use the Patchew REST API:
$ [email protected] $ curl -L https://patchew.org/api/v1/projects/by-name/QEMU/messages/$MSGID/ | jq -r '.tags[]' Reported-by: Philippe Mathieu-Daudé <[email protected]> Reviewed-by: Richard Henderson <[email protected]>
So you'd have to take a commit, look for a Message-Id header, fetch the tags from above mentioned Patchew API URL and add the tags to the commit message.
The commit message can be either emitted to stdout (and the script used with "git filter-branch)" or, for the faint of heart, the script could do a "git commit --amend" and you can use "git rebase -i --exec" to execute the script on all commits in a range.
This script is for the latter option:
#! /bin/bash BODY=$(git show -s --format=%B) MSGID=$(git interpret-trailers --parse <<< $BODY | sed -n 's/^Message-Id: <\(.*\)>/\1/ip') USER="$(git config user.name) <$(git config user.email)>" BODY=$(curl -L https://patchew.org/api/v1/projects/by-name/QEMU/messages/$MSGID/ | \ jq -r '.tags[]' | ( \ args=() while read x; do args+=(--trailer "$x") done git interpret-trailers \ --if-exists doNothing "${args[@]}" \ --if-exists replace --if-missing doNothing --trailer "Signed-off-by: $USER" <<< $BODY )) git commit --amend -m"$BODY"
The script will also move your Signed-off-by line at the end of the commit message, this might be a problem if there is more than one such line and you want to keep them all.
Paolo
Philippe Mathieu-Daudé
AppliedQuoted 6 lines · John Snow
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
[...]
Quoted 19 lines
John Snow (16):
python/qmp.py: Define common typesiotests.py: use qemu.qmp type aliasespython/qmp.py: re-absorb MonitorResponseErrorpython/qmp.py: Do not return None fromcmd_objpython/qmp.py: add casts to JSON deserializationpython/qmp.py: add QMPProtocolErrorpython/machine.py: Fix monitor address typingpython/machine.py: reorder__init__python/machine.py: Don't modify state in_base_args()python/machine.py: Handle None events inevents_waitpython/machine.py: use qmp.commandpython/machine.py: Add_qmpaccess shimpython/machine.py: fix_popenaccess python/qemu: make 'args' style arguments immutableiotests.py: Adjust HMP kwargs typing python/qemu: Add mypy type annotations
Quoted 8 lines
python/qemu/accel.py | 8 +- python/qemu/machine.py | 286 ++++++++++++++++++++-------------- python/qemu/qmp.py | 111 +++++++++---- python/qemu/qtest.py | 53 ++++--- scripts/render_block_graph.py | 7 +- tests/qemu-iotests/iotests.py | 11 +- 6 files changed, 298 insertions(+), 178 deletions(-)
Thanks, applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next
Quoted 36 lines · Philippe Mathieu-Daudé
Quoted 6 lines · John Snow
Based-on: [email protected]
This series is extracted from my larger series that attempted to bundle our python module as an installable module. These fixes don't require that, so they are being sent first as I think there's less up for debate in here.
[...]
Quoted 19 lines
John Snow (16):
python/qmp.py: Define common typesiotests.py: use qemu.qmp type aliasespython/qmp.py: re-absorb MonitorResponseErrorpython/qmp.py: Do not return None fromcmd_objpython/qmp.py: add casts to JSON deserializationpython/qmp.py: add QMPProtocolErrorpython/machine.py: Fix monitor address typingpython/machine.py: reorder__init__python/machine.py: Don't modify state in_base_args()python/machine.py: Handle None events inevents_waitpython/machine.py: use qmp.commandpython/machine.py: Add_qmpaccess shimpython/machine.py: fix_popenaccess python/qemu: make 'args' style arguments immutableiotests.py: Adjust HMP kwargs typing python/qemu: Add mypy type annotationsQuoted 8 lines
python/qemu/accel.py | 8 +- python/qemu/machine.py | 286 ++++++++++++++++++++-------------- python/qemu/qmp.py | 111 +++++++++---- python/qemu/qtest.py | 53 ++++--- scripts/render_block_graph.py | 7 +- tests/qemu-iotests/iotests.py | 11 +- 6 files changed, 298 insertions(+), 178 deletions(-)
Quoted 3 lines
Thanks, applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next
Awesome, thanks!