Skip to contentSkip to conversation

Python: move /scripts/qmp/qom* to /python/qemu/qmp/qom*

PATCH In review John Snow 2 participants 25 messages Jun 3, 2021 – Jun 9, 2021

lore
Share & tools
lore

John Snow

author
#0 lore raw

v3 cover Python: move /scripts/qmp/qom* to /python/qemu/qmp/qom*

GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-qom
CI: https://gitlab.com/jsnow/qemu/-/pipelines/313932818

Hello!
------

This series applies the usual linting cleanups to modernize the qom tools and then integrates them into the python packaging hierarchy.

This will help prevent further bitrot of these tools.

I don't expect or need a detailed review of the QOM tools themselves -- these tools are not used during build OR testing, and some are fairly bitrotted in places.

However, some details of how the python packaging system is being utilized here may attract your attention and could be worth a look. (Patches 5-6 and 16-19 are the interesting ones.)

Since these scripts aren't critical, I'm OK with sending a fairly hasty PR to merge these sooner rather than later.

Overview:
---------

Patch 1: Update Pipfile.lock (See the commit as for why ...)

Patches 2-3: Correct some existing typing issues in qemu.qmp

Patch 4: Combine qom-set, qom-get, (etc) into one, newly written script that makes all of the command invocations, help text, etc. consistent. (I ask that review for this patch should be limited to critical
mistakes: I have no interest in developing the QOM tools further.)

Patches 5-6: Integrate the qom tools into the python package.

Patches 7-15: Delinting of the qom_fuse script. Similarly, I am not terribly interested in further improvements here, personally.

Patches 16-19: Integrating qom-fuse into the Python packaging directory; additional care is taken to ensure that "optional" dependencies like fusepy are handled well.

Changelog
---------

V3:
- Technically, I sent two versions of this before, a long time ago. This has been cleaned up and based on the latest origin/master.

John Snow (19):
python/pipenv: Update Pipfile.lock
python/qmp: Fix type of SocketAddrT
python/qmp: add parse_address classmethod
python/qmp: Add qom script rewrites
python/qmp: add qom script entry points
scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/ scripts/qom-fuse: apply isort rules
scripts/qom-fuse: apply flake8 rules
python: Add 'fh' to known-good variable names scripts/qom-fuse: Apply pylint rules
scripts/qom-fuse: Add docstrings
scripts/qom-fuse: Convert to QOMCommand
scripts/qom-fuse: use QOMCommand.qom_list() scripts/qom-fuse: ensure QOMFuse.read always returns bytes scripts/qom-fuse: add static type hints
python: add optional FUSE dependencies
scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py python/qmp: add fuse command to 'qom' tools

 python/Pipfile.lock           |  97 +++++++++++-
 python/qemu/qmp/__init__.py   |  28 +++-
 python/qemu/qmp/qom.py        | 272 ++++++++++++++++++++++++++++++++++
 python/qemu/qmp/qom_common.py | 178 ++++++++++++++++++++++
 python/qemu/qmp/qom_fuse.py   | 206 +++++++++++++++++++++++++
 python/setup.cfg              |  33 ++++-
 scripts/qmp/qmp-shell         |  21 +--
 scripts/qmp/qom-fuse          | 144 +-----------------
 scripts/qmp/qom-get           |  66 +--------
 scripts/qmp/qom-list          |  63 +-------
 scripts/qmp/qom-set           |  63 +-------
 scripts/qmp/qom-tree          |  74 +--------
 12 files changed, 828 insertions(+), 417 deletions(-)
 create mode 100644 python/qemu/qmp/qom.py
 create mode 100644 python/qemu/qmp/qom_common.py
 create mode 100644 python/qemu/qmp/qom_fuse.py
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v3 1/19 python/pipenv: Update Pipfile.lock

In a previous commit, I added tox to the development requirements of the Python library. I never bothered to add them to the Pipfile, because they aren't needed there. Here, I sync it anyway in its own commit so that when we add new packages later that the diffstats will not confusingly appear to pull in lots of extra packages.

Ideally I could tell Pipenv simply not to install these, but it doesn't seem to support that, exactly. The alternative is removing Tox from the development requires, which I'd rather not do.

The other alternative is re-specifying all of the dependencies of setup.cfg in the Pipfile, which I'd also rather not do.

Picking what feels least-worst here.

---
 python/Pipfile.lock | 91 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 88 insertions(+), 3 deletions(-)
python/Pipfile.lock +88 -3
index 6e344f5fadf..f2a3f91d0fa 100644
@@ -22,6 +22,13 @@
         }
     },
     "develop": {
+        "appdirs": {
+            "hashes": [
+                "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41",
+                "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"
+            ],
+            "version": "==1.4.4"
+        },
         "astroid": {
             "hashes": [
                 "sha256:4db03ab5fc3340cf619dbc25e42c2cc3755154ce6009469766d7143d1fc2ee4e",
@@ -38,6 +45,20 @@
             "markers": "python_version >= '3.6'",
             "version": "==88.1"
         },
+        "distlib": {
+            "hashes": [
+                "sha256:106fef6dc37dd8c0e2c0a60d3fca3e77460a48907f335fa28420463a6f799736",
+                "sha256:23e223426b28491b1ced97dc3bbe183027419dfc7982b4fa2f05d5f3ff10711c"
+            ],
+            "version": "==0.3.2"
+        },
+        "filelock": {
+            "hashes": [
+                "sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59",
+                "sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836"
+            ],
+            "version": "==3.0.12"
+        },
         "flake8": {
             "hashes": [
                 "sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b",
@@ -54,6 +75,14 @@
             "markers": "python_version < '3.8'",
             "version": "==4.0.1"
         },
+        "importlib-resources": {
+            "hashes": [
+                "sha256:54161657e8ffc76596c4ede7080ca68cb02962a2e074a2586b695a93a925d36e",
+                "sha256:e962bff7440364183203d179d7ae9ad90cb1f2b74dcb84300e88ecc42dca3351"
+            ],
+            "markers": "python_version < '3.7'",
+            "version": "==5.1.4"
+        },
         "isort": {
             "hashes": [
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
@@ -132,6 +161,30 @@
             ],
             "version": "==0.4.3"
         },
+        "packaging": {
+            "hashes": [
+                "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5",
+                "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==20.9"
+        },
+        "pluggy": {
+            "hashes": [
+                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
+                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==0.13.1"
+        },
+        "py": {
+            "hashes": [
+                "sha256:21b81bda15b66ef5e1a777a21c4dcd9c20ad3efd0b3f817e7a809035269e1bd3",
+                "sha256:3b80836aa6d1feeaa108e046da6423ab8f6ceda6468545ae8d02d9d58d18818a"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==1.10.0"
+        },
         "pycodestyle": {
             "hashes": [
                 "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068",
@@ -156,18 +209,42 @@
             "markers": "python_version ~= '3.6'",
             "version": "==2.8.2"
         },
+        "pyparsing": {
+            "hashes": [
+                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
+                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
+            ],
+            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==2.4.7"
+        },
         "qemu": {
             "editable": true,
             "path": "."
         },
+        "six": {
+            "hashes": [
+                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
+                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==1.16.0"
+        },
         "toml": {
             "hashes": [
                 "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
                 "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
             ],
-            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2'",
+            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==0.10.2"
         },
+        "tox": {
+            "hashes": [
+                "sha256:307a81ddb82bd463971a273f33e9533a24ed22185f27db8ce3386bff27d324e3",
+                "sha256:b0b5818049a1c1997599d42012a637a33f24c62ab8187223fdd318fa8522637b"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
+            "version": "==3.23.1"
+        },
         "typed-ast": {
             "hashes": [
                 "sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace",
@@ -201,7 +278,7 @@
                 "sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f",
                 "sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"
             ],
-            "markers": "implementation_name == 'cpython' and python_version < '3.8'",
+            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
             "version": "==1.4.3"
         },
         "typing-extensions": {
@@ -213,6 +290,14 @@
             "markers": "python_version < '3.8'",
             "version": "==3.10.0.0"
         },
+        "virtualenv": {
+            "hashes": [
+                "sha256:14fdf849f80dbb29a4eb6caa9875d476ee2a5cf76a5f5415fa2f1606010ab467",
+                "sha256:2b0126166ea7c9c3661f5b8e06773d28f83322de7a3ff7d06f0aed18c9de6a76"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==20.4.7"
+        },
         "wrapt": {
             "hashes": [
                 "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
@@ -224,7 +309,7 @@
                 "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76",
                 "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"
             ],
-            "markers": "python_version >= '3.6'",
+            "markers": "python_version < '3.10'",
             "version": "==3.4.1"
         }
     }
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH v3 2/19 python/qmp: Fix type of SocketAddrT

In reply to John Snow

In porting the qom tools, qmp-shell, etc; it becomes evident that this type is wrong.

This is an integer, not a string. We didn't catch this before because none of QEMUMonitorProtocol's *users* happen to be checked, and the internal logic of this class is otherwise self-consistent. Additionally, mypy was not introspecting into the socket() interface to realize we were passing a bad type for AF_INET. Fixed now.

---
 python/qemu/qmp/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
python/qemu/qmp/__init__.py +1 -1
index 9606248a3d2..5fb970f8a80 100644
@@ -44,7 +44,7 @@
 QMPMessage = Dict[str, Any]
 QMPReturnValue = Dict[str, Any]
 
-InternetAddrT = Tuple[str, str]
+InternetAddrT = Tuple[str, int]
 UnixAddrT = str
 SocketAddrT = Union[InternetAddrT, UnixAddrT]
 
Signature · 1 line
2.31.1
Reviewed-by on v3 2/19
#3 lore raw

Re: [PATCH v3 02/19] python/qmp: Fix type of SocketAddrT

Quoted 9 lines · John Snow

In porting the qom tools, qmp-shell, etc; it becomes evident that this type is wrong.

This is an integer, not a string. We didn't catch this before because none of QEMUMonitorProtocol's *users* happen to be checked, and the internal logic of this class is otherwise self-consistent. Additionally, mypy was not introspecting into the socket() interface to realize we were passing a bad type for AF_INET. Fixed now.

Quoted 4 lines
---
 python/qemu/qmp/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
author
#4 lore raw

PATCH v3 3/19 python/qmp: add parse_address classmethod

In reply to John Snow

This takes the place of qmp-shell's __get_address function. It also allows other utilities to share the same parser and syntax for specifying QMP locations.

---
 python/qemu/qmp/__init__.py | 26 ++++++++++++++++++++++++++
 scripts/qmp/qmp-shell       | 21 ++-------------------
 2 files changed, 28 insertions(+), 19 deletions(-)
python/qemu/qmp/__init__.py +26 -0
index 5fb970f8a80..822c793c320 100644
@@ -92,6 +92,12 @@ def __init__(self, reply: QMPMessage):
         self.reply = reply
 
 
+class QMPBadPortError(QMPError):
+    """
+    Unable to parse socket address: Port was non-numerical.
+    """
+
+
 class QEMUMonitorProtocol:
     """
     Provide an API to connect to QEMU via QEMU Monitor Protocol (QMP) and then
@@ -219,6 +225,26 @@ def __exit__(self,
         # Implement context manager exit function.
         self.close()
 
+    @classmethod
+    def parse_address(cls, address: str) -> SocketAddrT:
+        """
+        Parse a string into a QMP address.
+
+        Figure out if the argument is in the port:host form.
+        If it's not, it's probably a file path.
+        """
+        components = address.split(':')
+        if len(components) == 2:
+            try:
+                port = int(components[1])
+            except ValueError:
+                msg = f"Bad port: '{components[1]}' in '{address}'."
+                raise QMPBadPortError(msg) from None
+            return (components[0], port)
+
+        # Treat as filepath.
+        return address
+
     def connect(self, negotiate: bool = True) -> Optional[QMPMessage]:
         """
         Connect to the QMP Monitor and perform capabilities negotiation.
scripts/qmp/qmp-shell +2 -19
index b4d06096abd..d5ae8a9b212 100755
@@ -89,8 +89,6 @@ class QMPCompleter(list):
 class QMPShellError(Exception):
     pass
 
-class QMPShellBadPort(QMPShellError):
-    pass
 
 class FuzzyJSON(ast.NodeTransformer):
     '''This extension of ast.NodeTransformer filters literal "true/false/null"
@@ -109,7 +107,7 @@ class FuzzyJSON(ast.NodeTransformer):
 #       _execute_cmd()). Let's design a better one.
 class QMPShell(qmp.QEMUMonitorProtocol):
     def __init__(self, address, pretty=False):
-        super(QMPShell, self).__init__(self.__get_address(address))
+        super(QMPShell, self).__init__(self.parse_address(address))
         self._greeting = None
         self._completer = None
         self._pretty = pretty
@@ -118,21 +116,6 @@ class QMPShell(qmp.QEMUMonitorProtocol):
         self._histfile = os.path.join(os.path.expanduser('~'),
                                       '.qmp-shell_history')
 
-    def __get_address(self, arg):
-        """
-        Figure out if the argument is in the port:host form, if it's not it's
-        probably a file path.
-        """
-        addr = arg.split(':')
-        if len(addr) == 2:
-            try:
-                port = int(addr[1])
-            except ValueError:
-                raise QMPShellBadPort
-            return ( addr[0], port )
-        # socket path
-        return arg
-
     def _fill_completion(self):
         cmds = self.cmd('query-commands')
         if 'error' in cmds:
@@ -437,7 +420,7 @@ def main():
 
         if qemu is None:
             fail_cmdline()
-    except QMPShellBadPort:
+    except qmp.QMPBadPortError:
         die('bad port number in command-line')
 
     try:
Signature · 1 line
2.31.1
author
#5 lore raw

PATCH v3 4/19 python/qmp: Add qom script rewrites

In reply to John Snow

Inspired by qom-set, qom-get, qom-tree and qom-list; combine all four of those scripts into a single script.

A later addition of qom-fuse as an 'extension' necessitates that some common features are split out and shared between them.

---
 python/qemu/qmp/qom.py        | 262 ++++++++++++++++++++++++++++++++++
 python/qemu/qmp/qom_common.py | 178 +++++++++++++++++++++++
 2 files changed, 440 insertions(+)
 create mode 100644 python/qemu/qmp/qom.py
 create mode 100644 python/qemu/qmp/qom_common.py
python/qemu/qmp/qom.py new file +262 -0
new file mode 100644
index 00000000000..7fe1448b5d9
@@ -0,0 +1,262 @@
+"""
+QEMU Object Model testing tools.
+
+usage: qom [-h] {set,get,list,tree} ...
+
+Query and manipulate QOM data
+
+optional arguments:
+  -h, --help           show this help message and exit
+
+QOM commands:
+  {set,get,list,tree}
+    set                Set a QOM property value
+    get                Get a QOM property value
+    list               List QOM properties at a given path
+    tree               Show QOM tree from a given path
+"""
+##
+# Copyright John Snow 2020, for Red Hat, Inc.
+# Copyright IBM, Corp. 2011
+#
+# Authors:
+#  John Snow <[email protected]>
+#  Anthony Liguori <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# Based on ./scripts/qmp/qom-[set|get|tree|list]
+##
+
+import argparse
+
+from . import QMPResponseError
+from .qom_common import QOMCommand
+
+
+class QOMSet(QOMCommand):
+    """
+    QOM Command - Set a property to a given value.
+
+    usage: qom-set [-h] [--socket SOCKET] <path>.<property> <value>
+
+    Set a QOM property value
+
+    positional arguments:
+      <path>.<property>     QOM path and property, separated by a period '.'
+      <value>               new QOM property value
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      --socket SOCKET, -s SOCKET
+                            QMP socket path or address (addr:port). May also be
+                            set via QMP_SOCKET environment variable.
+    """
+    name = 'set'
+    help = 'Set a QOM property value'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        cls.add_path_prop_arg(parser)
+        parser.add_argument(
+            'value',
+            metavar='<value>',
+            action='store',
+            help='new QOM property value'
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path, self.prop = args.path_prop.rsplit('.', 1)
+        self.value = args.value
+
+    def run(self) -> int:
+        rsp = self.qmp.command(
+            'qom-set',
+            path=self.path,
+            property=self.prop,
+            value=self.value
+        )
+        print(rsp)
+        return 0
+
+
+class QOMGet(QOMCommand):
+    """
+    QOM Command - Get a property's current value.
+
+    usage: qom-get [-h] [--socket SOCKET] <path>.<property>
+
+    Get a QOM property value
+
+    positional arguments:
+      <path>.<property>     QOM path and property, separated by a period '.'
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      --socket SOCKET, -s SOCKET
+                            QMP socket path or address (addr:port). May also be
+                            set via QMP_SOCKET environment variable.
+    """
+    name = 'get'
+    help = 'Get a QOM property value'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        cls.add_path_prop_arg(parser)
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        try:
+            tmp = args.path_prop.rsplit('.', 1)
+        except ValueError as err:
+            raise ValueError('Invalid format for <path>.<property>') from err
+        self.path = tmp[0]
+        self.prop = tmp[1]
+
+    def run(self) -> int:
+        rsp = self.qmp.command(
+            'qom-get',
+            path=self.path,
+            property=self.prop
+        )
+        if isinstance(rsp, dict):
+            for key, value in rsp.items():
+                print(f"{key}: {value}")
+        else:
+            print(rsp)
+        return 0
+
+
+class QOMList(QOMCommand):
+    """
+    QOM Command - List the properties at a given path.
+
+    usage: qom-list [-h] [--socket SOCKET] <path>
+
+    List QOM properties at a given path
+
+    positional arguments:
+      <path>                QOM path
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      --socket SOCKET, -s SOCKET
+                            QMP socket path or address (addr:port). May also be
+                            set via QMP_SOCKET environment variable.
+    """
+    name = 'list'
+    help = 'List QOM properties at a given path'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'path',
+            metavar='<path>',
+            action='store',
+            help='QOM path',
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path = args.path
+
+    def run(self) -> int:
+        rsp = self.qom_list(self.path)
+        for item in rsp:
+            if item.child:
+                print(f"{item.name}/")
+            elif item.link:
+                print(f"@{item.name}/")
+            else:
+                print(item.name)
+        return 0
+
+
+class QOMTree(QOMCommand):
+    """
+    QOM Command - Show the full tree below a given path.
+
+    usage: qom-tree [-h] [--socket SOCKET] [<path>]
+
+    Show QOM tree from a given path
+
+    positional arguments:
+      <path>                QOM path
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      --socket SOCKET, -s SOCKET
+                            QMP socket path or address (addr:port). May also be
+                            set via QMP_SOCKET environment variable.
+    """
+    name = 'tree'
+    help = 'Show QOM tree from a given path'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'path',
+            metavar='<path>',
+            action='store',
+            help='QOM path',
+            nargs='?',
+            default='/'
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path = args.path
+
+    def _list_node(self, path: str) -> None:
+        print(path)
+        items = self.qom_list(path)
+        for item in items:
+            if item.child:
+                continue
+            try:
+                rsp = self.qmp.command('qom-get', path=path,
+                                       property=item.name)
+                print(f"  {item.name}: {rsp} ({item.type})")
+            except QMPResponseError as err:
+                print(f"  {item.name}: <EXCEPTION: {err!s}> ({item.type})")
+        print('')
+        for item in items:
+            if not item.child:
+                continue
+            if path == '/':
+                path = ''
+            self._list_node(f"{path}/{item.name}")
+
+    def run(self) -> int:
+        self._list_node(self.path)
+        return 0
+
+
+def main() -> int:
+    """QOM script main entry point."""
+    parser = argparse.ArgumentParser(
+        description='Query and manipulate QOM data'
+    )
+    subparsers = parser.add_subparsers(
+        title='QOM commands',
+        dest='command'
+    )
+
+    for command in QOMCommand.__subclasses__():
+        command.register(subparsers)
+
+    args = parser.parse_args()
+
+    if args.command is None:
+        parser.error('Command not specified.')
+        return 1
+
+    cmd_class = args.cmd_class
+    assert isinstance(cmd_class, type(QOMCommand))
+    return cmd_class.command_runner(args)
python/qemu/qmp/qom_common.py new file +178 -0

178 lines not on this page · raw message

Signature · 1 line
2.31.1
author
#6 lore raw

PATCH v3 5/19 python/qmp: add qom script entry points

In reply to John Snow

Add the 'qom', 'qom-set', 'qom-get', 'qom-list', and 'qom-tree' scripts to the qemu.qmp package. When you install this package, these scripts will become available on your command line.

(e.g. when inside of a venv, cd python && pip install . will add 'qom', 'qom-set', etc to your $PATH.)

---
 python/setup.cfg | 8 ++++++++
 1 file changed, 8 insertions(+)
python/setup.cfg +8 -0
index 0fcdec6f322..a19029d5388 100644
@@ -37,6 +37,14 @@ devel =
     pylint >= 2.8.0
     tox >= 3.18.0
 
+[options.entry_points]
+console_scripts =
+    qom = qemu.qmp.qom:main
+    qom-set = qemu.qmp.qom:QOMSet.entry_point
+    qom-get = qemu.qmp.qom:QOMGet.entry_point
+    qom-list = qemu.qmp.qom:QOMList.entry_point
+    qom-tree = qemu.qmp.qom:QOMTree.entry_point
+
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 exclude = __pycache__,
Signature · 1 line
2.31.1
author
#7 lore raw

PATCH v3 6/19 scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/

In reply to John Snow

Redirect to the new qom scripts. These forwarders can be deleted eventually when there has been more time for the dust on the Python packaging to settle and people understand how to find these commands.

Note: You can run these by setting $PYTHONPATH in your shell and then running "python3 -m qemu.qmp.qom", or you can install the qemu namespace package and use the "qom" or "qom-set" scripts.

I've written how to install the package elsewhere, but for the sake of git-blame, cd to ./python, and then do:

- pip3 install [--user] [-e] .

--user will install to your local user install (will not work inside of
a venv), omitting this flag installs to your system-wide packages (outside of a venv) or to your current virtual environment (inside the venv).

When installing to a venv or to your system-wide packages, "qom" should be in your $PATH already. If you do a user install, you may need to add ~/.local/bin to your $PATH if you haven't already.

-e installs in editable mode: the installed package is effectively just
a symlink to this folder; so changes to your git working tree are reflected in the installed package.

Note: installing these packages to an environment outside a venv can be
dangerous: Many QEMU scripts will begin to prefer the installed version instead of the version directly in the tree. Use with caution. editable mode is recommended when working outside of a venv.

---
 scripts/qmp/qom-get  | 66 +++------------------------------------
 scripts/qmp/qom-list | 63 +++----------------------------------
 scripts/qmp/qom-set  | 63 +++----------------------------------
 scripts/qmp/qom-tree | 74 +++-----------------------------------------
 4 files changed, 16 insertions(+), 250 deletions(-)
scripts/qmp/qom-get +4 -62
index 666df718320..e4f3e0c0138 100755
@@ -1,69 +1,11 @@
 #!/usr/bin/env python3
-##
-# QEMU Object Model test tools
-#
-# Copyright IBM, Corp. 2011
-#
-# Authors:
-#  Anthony Liguori   <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or later.  See
-# the COPYING file in the top-level directory.
-##
 
-import sys
 import os
+import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.qmp.qom import QOMGet
 
-cmd, args = sys.argv[0], sys.argv[1:]
-socket_path = None
-path = None
-prop = None
 
-def usage():
-    return '''environment variables:
-    QMP_SOCKET=<path | addr:port>
-usage:
-    %s [-h] [-s <QMP socket path | addr:port>] <path>.<property>
-''' % cmd
-
-def usage_error(error_msg = "unspecified error"):
-    sys.stderr.write('%s\nERROR: %s\n' % (usage(), error_msg))
-    exit(1)
-
-if len(args) > 0:
-    if args[0] == "-h":
-        print(usage())
-        exit(0);
-    elif args[0] == "-s":
-        try:
-            socket_path = args[1]
-        except:
-            usage_error("missing argument: QMP socket path or address");
-        args = args[2:]
-
-if not socket_path:
-    if 'QMP_SOCKET' in os.environ:
-        socket_path = os.environ['QMP_SOCKET']
-    else:
-        usage_error("no QMP socket path or address given");
-
-if len(args) > 0:
-    try:
-        path, prop = args[0].rsplit('.', 1)
-    except:
-        usage_error("invalid format for path/property/value")
-else:
-    usage_error("not enough arguments")
-
-srv = QEMUMonitorProtocol(socket_path)
-srv.connect()
-
-rsp = srv.command('qom-get', path=path, property=prop)
-if type(rsp) == dict:
-    for i in rsp.keys():
-        print('%s: %s' % (i, rsp[i]))
-else:
-    print(rsp)
+if __name__ == '__main__':
+    sys.exit(QOMGet.entry_point())
scripts/qmp/qom-list +4 -59

70 lines not on this page · raw message

scripts/qmp/qom-set +4 -59

70 lines not on this page · raw message

scripts/qmp/qom-tree +4 -70

81 lines not on this page · raw message

Signature · 1 line
2.31.1
author
#8 lore raw

PATCH v3 7/19 scripts/qom-fuse: apply isort rules

In reply to John Snow

Hint: you can use symlinks to create qom_fuse.py in python/qemu/qmp/ and point to scripts/qom-fuse to apply the standard linting rules to this script.

---
 scripts/qmp/qom-fuse | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
scripts/qmp/qom-fuse +9 -3
index 7c7cff8edfb..62deb9adb12 100755
@@ -13,14 +13,20 @@
 # the COPYING file in the top-level directory.
 ##
 
-import fuse, stat
-from fuse import FUSE, FuseOSError, Operations
-import os, posix, sys
 from errno import *
+import os
+import posix
+import stat
+import sys
+
+import fuse
+from fuse import FUSE, FuseOSError, Operations
+
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu.qmp import QEMUMonitorProtocol
 
+
 fuse.fuse_python_api = (0, 2)
 
 class QOMFS(Operations):
Signature · 1 line
2.31.1
author
#9 lore raw

PATCH v3 8/19 scripts/qom-fuse: apply flake8 rules

In reply to John Snow

flake8 still has one warning because of the sys.path hack, but that will be going away by the end of this patch series.

---
 scripts/qmp/qom-fuse | 81 +++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 38 deletions(-)
scripts/qmp/qom-fuse +43 -38
index 62deb9adb12..ca30e928679 100755
@@ -9,13 +9,12 @@
 #  Anthony Liguori   <[email protected]>
 #  Markus Armbruster <[email protected]>
 #
-# This work is licensed under the terms of the GNU GPL, version 2 or later.  See
-# the COPYING file in the top-level directory.
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
 ##
 
-from errno import *
+from errno import ENOENT, EPERM
 import os
-import posix
 import stat
 import sys
 
@@ -29,6 +28,7 @@ from qemu.qmp import QEMUMonitorProtocol
 
 fuse.fuse_python_api = (0, 2)
 
+
 class QOMFS(Operations):
     def __init__(self, qmp):
         self.qmp = qmp
@@ -45,7 +45,7 @@ class QOMFS(Operations):
 
     def is_object(self, path):
         try:
-            items = self.qmp.command('qom-list', path=path)
+            self.qmp.command('qom-list', path=path)
             return True
         except:
             return False
@@ -85,7 +85,7 @@ class QOMFS(Operations):
             path = '/'
         try:
             data = self.qmp.command('qom-get', path=path, property=prop)
-            data += '\n' # make values shell friendly
+            data += '\n'  # make values shell friendly
         except:
             raise FuseOSError(EPERM)
 
@@ -104,38 +104,44 @@ class QOMFS(Operations):
 
     def getattr(self, path, fh=None):
         if self.is_link(path):
-            value = { 'st_mode': 0o755 | stat.S_IFLNK,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 2,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o755 | stat.S_IFLNK,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 2,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         elif self.is_object(path):
-            value = { 'st_mode': 0o755 | stat.S_IFDIR,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 2,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o755 | stat.S_IFDIR,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 2,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         elif self.is_property(path):
-            value = { 'st_mode': 0o644 | stat.S_IFREG,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 1,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o644 | stat.S_IFREG,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 1,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         else:
             raise FuseOSError(ENOENT)
         return value
@@ -146,8 +152,7 @@ class QOMFS(Operations):
         for item in self.qmp.command('qom-list', path=path):
             yield str(item['name'])
 
+
 if __name__ == '__main__':
-    import os
-
     fuse = FUSE(QOMFS(QEMUMonitorProtocol(os.environ['QMP_SOCKET'])),
                 sys.argv[1], foreground=True)
Signature · 1 line
2.31.1
Reviewed-by on v3 8/19
#10 lore raw

Re: [PATCH v3 08/19] scripts/qom-fuse: apply flake8 rules

Quoted 7 lines · John Snow

flake8 still has one warning because of the sys.path hack, but that will be going away by the end of this patch series.

---
 scripts/qmp/qom-fuse | 81 +++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 38 deletions(-)
author
#11 lore raw

PATCH v3 9/19 python: Add 'fh' to known-good variable names

In reply to John Snow

fd and fh are fine: we often use these for "file descriptor" or "file handle" accordingly. It is rarely the case that you need to enforce a more semantically meaningful name beyond "This is the file we are using right now."

While we're here: add comments for all of the non-standard pylint names. (And the underscore.)

---
 python/setup.cfg | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
python/setup.cfg +4 -3
index a19029d5388..c9b9445af98 100644
@@ -78,9 +78,10 @@ good-names=i,
            k,
            ex,
            Run,
-           _,
-           fd,
-           c,
+           _,   # By convention: Unused variable
+           fh,  # fh = open(...)
+           fd,  # fd = os.open(...)
+           c,   # for c in string: ...
 
 [pylint.similarities]
 # Ignore imports when computing similarities.
Signature · 1 line
2.31.1
author
#12 lore raw

PATCH v3 10/19 scripts/qom-fuse: Apply pylint rules

In reply to John Snow

- Catch specific exceptions from QMP
- Reraise errors with explicit context
- method parameters should match parent's names

---
 scripts/qmp/qom-fuse | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
scripts/qmp/qom-fuse +8 -8
index ca30e928679..805e99c8ecd 100755
@@ -23,7 +23,7 @@ from fuse import FUSE, FuseOSError, Operations
 
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.qmp import QEMUMonitorProtocol, QMPResponseError
 
 
 fuse.fuse_python_api = (0, 2)
@@ -47,7 +47,7 @@ class QOMFS(Operations):
         try:
             self.qmp.command('qom-list', path=path)
             return True
-        except:
+        except QMPResponseError:
             return False
 
     def is_property(self, path):
@@ -59,7 +59,7 @@ class QOMFS(Operations):
                 if item['name'] == prop:
                     return True
             return False
-        except:
+        except QMPResponseError:
             return False
 
     def is_link(self, path):
@@ -73,10 +73,10 @@ class QOMFS(Operations):
                         return True
                     return False
             return False
-        except:
+        except QMPResponseError:
             return False
 
-    def read(self, path, length, offset, fh):
+    def read(self, path, size, offset, fh):
         if not self.is_property(path):
             return -ENOENT
 
@@ -86,13 +86,13 @@ class QOMFS(Operations):
         try:
             data = self.qmp.command('qom-get', path=path, property=prop)
             data += '\n'  # make values shell friendly
-        except:
-            raise FuseOSError(EPERM)
+        except QMPResponseError as err:
+            raise FuseOSError(EPERM) from err
 
         if offset > len(data):
             return ''
 
-        return bytes(data[offset:][:length], encoding='utf-8')
+        return bytes(data[offset:][:size], encoding='utf-8')
 
     def readlink(self, path):
         if not self.is_link(path):
Signature · 1 line
2.31.1
author
#13 lore raw

PATCH v3 11/19 scripts/qom-fuse: Add docstrings

In reply to John Snow

The methods inherited from fuse don't need docstrings; that's up to fusepy to handle.

---
 scripts/qmp/qom-fuse | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
scripts/qmp/qom-fuse +19 -2
index 805e99c8ecd..1fb3008a167 100755
@@ -1,7 +1,19 @@
 #!/usr/bin/env python3
+"""
+QEMU Object Model FUSE filesystem tool
+
+This script offers a simple FUSE filesystem within which the QOM tree
+may be browsed, queried and edited using traditional shell tooling.
+
+This script requires the 'fusepy' python package.
+
+ENV:
+    QMP_SOCKET: Path to the QMP server socket
+
+Usage:
+    qom-fuse /mount/to/here
+"""
 ##
-# QEMU Object Model test tools
-#
 # Copyright IBM, Corp. 2012
 # Copyright (C) 2020 Red Hat, Inc.
 #
@@ -30,6 +42,7 @@ fuse.fuse_python_api = (0, 2)
 
 
 class QOMFS(Operations):
+    """QOMFS implements fuse.Operations to provide a QOM filesystem."""
     def __init__(self, qmp):
         self.qmp = qmp
         self.qmp.connect()
@@ -37,6 +50,7 @@ class QOMFS(Operations):
         self.ino_count = 1
 
     def get_ino(self, path):
+        """Get an inode number for a given QOM path."""
         if path in self.ino_map:
             return self.ino_map[path]
         self.ino_map[path] = self.ino_count
@@ -44,6 +58,7 @@ class QOMFS(Operations):
         return self.ino_map[path]
 
     def is_object(self, path):
+        """Is the given QOM path an object?"""
         try:
             self.qmp.command('qom-list', path=path)
             return True
@@ -51,6 +66,7 @@ class QOMFS(Operations):
             return False
 
     def is_property(self, path):
+        """Is the given QOM path a property?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
@@ -63,6 +79,7 @@ class QOMFS(Operations):
             return False
 
     def is_link(self, path):
+        """Is the given QOM path a link?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
Signature · 1 line
2.31.1
author
#14 lore raw

PATCH v3 12/19 scripts/qom-fuse: Convert to QOMCommand

In reply to John Snow

Move qom-fuse onto the QOMCommand base established in python/qemu/qmp/qom_common.py. The interface doesn't change incompatibly, "qom-fuse mountpoint" still works as an invocation, and QMP_SOCKET is still used as the environment variable.

---
 scripts/qmp/qom-fuse | 59 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 46 insertions(+), 13 deletions(-)
scripts/qmp/qom-fuse +46 -13
index 1fb3008a167..1676fb78d99 100755
@@ -7,11 +7,19 @@ may be browsed, queried and edited using traditional shell tooling.
 
 This script requires the 'fusepy' python package.
 
-ENV:
-    QMP_SOCKET: Path to the QMP server socket
 
-Usage:
-    qom-fuse /mount/to/here
+usage: qom-fuse [-h] [--socket SOCKET] <mount>
+
+Mount a QOM tree as a FUSE filesystem
+
+positional arguments:
+  <mount>               Mount point
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --socket SOCKET, -s SOCKET
+                        QMP socket path or address (addr:port). May also be
+                        set via QMP_SOCKET environment variable.
 """
 ##
 # Copyright IBM, Corp. 2012
@@ -25,30 +33,56 @@ Usage:
 # See the COPYING file in the top-level directory.
 ##
 
+import argparse
 from errno import ENOENT, EPERM
 import os
 import stat
 import sys
+from typing import Dict
 
 import fuse
 from fuse import FUSE, FuseOSError, Operations
 
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol, QMPResponseError
+from qemu.qmp import QMPResponseError
+from qemu.qmp.qom_common import QOMCommand
 
 
 fuse.fuse_python_api = (0, 2)
 
 
-class QOMFS(Operations):
-    """QOMFS implements fuse.Operations to provide a QOM filesystem."""
-    def __init__(self, qmp):
-        self.qmp = qmp
-        self.qmp.connect()
-        self.ino_map = {}
+class QOMFuse(QOMCommand, Operations):
+    """
+    QOMFuse implements both fuse.Operations and QOMCommand.
+
+    Operations implements the FS, and QOMCommand implements the CLI command.
+    """
+    name = 'fuse'
+    help = 'Mount a QOM tree as a FUSE filesystem'
+    fuse: FUSE
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'mount',
+            metavar='<mount>',
+            action='store',
+            help="Mount point",
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.mount = args.mount
+        self.ino_map: Dict[str, int] = {}
         self.ino_count = 1
 
+    def run(self) -> int:
+        print(f"Mounting QOMFS to '{self.mount}'", file=sys.stderr)
+        self.fuse = FUSE(self, self.mount, foreground=True)
+        return 0
+
     def get_ino(self, path):
         """Get an inode number for a given QOM path."""
         if path in self.ino_map:
@@ -171,5 +205,4 @@ class QOMFS(Operations):
 
 
 if __name__ == '__main__':
-    fuse = FUSE(QOMFS(QEMUMonitorProtocol(os.environ['QMP_SOCKET'])),
-                sys.argv[1], foreground=True)
+    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.31.1
author
#15 lore raw

PATCH v3 13/19 scripts/qom-fuse: use QOMCommand.qom_list()

In reply to John Snow

the qom_list method provides a type-safe object that's easier to type check, so switch to using it.

---
 scripts/qmp/qom-fuse | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
scripts/qmp/qom-fuse +8 -10
index 1676fb78d99..703a97e75ff 100755
@@ -94,7 +94,7 @@ class QOMFuse(QOMCommand, Operations):
     def is_object(self, path):
         """Is the given QOM path an object?"""
         try:
-            self.qmp.command('qom-list', path=path)
+            self.qom_list(path)
             return True
         except QMPResponseError:
             return False
@@ -105,8 +105,8 @@ class QOMFuse(QOMCommand, Operations):
         if path == '':
             path = '/'
         try:
-            for item in self.qmp.command('qom-list', path=path):
-                if item['name'] == prop:
+            for item in self.qom_list(path):
+                if item.name == prop:
                     return True
             return False
         except QMPResponseError:
@@ -118,11 +118,9 @@ class QOMFuse(QOMCommand, Operations):
         if path == '':
             path = '/'
         try:
-            for item in self.qmp.command('qom-list', path=path):
-                if item['name'] == prop:
-                    if item['type'].startswith('link<'):
-                        return True
-                    return False
+            for item in self.qom_list(path):
+                if item.name == prop and item.link:
+                    return True
             return False
         except QMPResponseError:
             return False
@@ -200,8 +198,8 @@ class QOMFuse(QOMCommand, Operations):
     def readdir(self, path, fh):
         yield '.'
         yield '..'
-        for item in self.qmp.command('qom-list', path=path):
-            yield str(item['name'])
+        for item in self.qom_list(path):
+            yield item.name
 
 
 if __name__ == '__main__':
Signature · 1 line
2.31.1
author
#17 lore raw

PATCH v3 14/19 scripts/qom-fuse: ensure QOMFuse.read always returns bytes

In reply to John Snow

- Use FuseOSError to signal ENOENT instead of returning it
- Wrap qom-get in str(), as we don't always know its type
- The empty return should be b'', not ''.

---
 scripts/qmp/qom-fuse | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/qmp/qom-fuse +3 -3
index 703a97e75ff..0d11f731526 100755
@@ -127,19 +127,19 @@ class QOMFuse(QOMCommand, Operations):
 
     def read(self, path, size, offset, fh):
         if not self.is_property(path):
-            return -ENOENT
+            raise FuseOSError(ENOENT)
 
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
         try:
-            data = self.qmp.command('qom-get', path=path, property=prop)
+            data = str(self.qmp.command('qom-get', path=path, property=prop))
             data += '\n'  # make values shell friendly
         except QMPResponseError as err:
             raise FuseOSError(EPERM) from err
 
         if offset > len(data):
-            return ''
+            return b''
 
         return bytes(data[offset:][:size], encoding='utf-8')
 
Signature · 1 line
2.31.1
author
#18 lore raw

PATCH v3 15/19 scripts/qom-fuse: add static type hints

In reply to John Snow

Because fusepy does not have type hints, add some targeted warning suppressions.

Namely, we need to allow subclassing something of an unknown type (in qom_fuse.py), and we need to allow missing imports (recorded against fuse itself) because mypy will be unable to import fusepy (even when installed) as it has no types nor type stubs available.

Note: Until now, it was possible to run invocations like 'mypy qemu/' from ./python and have that work. However, these targeted suppressions require that you run 'mypy -p qemu/' instead. The correct, canonical invocation is recorded in ./python/tests/mypy.sh and all of the various CI invocations always use this correct form.

---
 python/setup.cfg     |  8 ++++++++
 scripts/qmp/qom-fuse | 26 +++++++++++++++++---------
 2 files changed, 25 insertions(+), 9 deletions(-)
python/setup.cfg +8 -0
index c9b9445af98..ba8d29fd62d 100644
@@ -57,6 +57,14 @@ python_version = 3.6
 warn_unused_configs = True
 namespace_packages = True
 
+[mypy-qemu.qmp.qom_fuse]
+# fusepy has no type stubs:
+allow_subclassing_any = True
+
+[mypy-fuse]
+# fusepy has no type stubs:
+ignore_missing_imports = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
scripts/qmp/qom-fuse +17 -9
index 0d11f731526..a5a7a304a3e 100755
@@ -38,7 +38,14 @@ from errno import ENOENT, EPERM
 import os
 import stat
 import sys
-from typing import Dict
+from typing import (
+    IO,
+    Dict,
+    Iterator,
+    Mapping,
+    Optional,
+    Union,
+)
 
 import fuse
 from fuse import FUSE, FuseOSError, Operations
@@ -83,7 +90,7 @@ class QOMFuse(QOMCommand, Operations):
         self.fuse = FUSE(self, self.mount, foreground=True)
         return 0
 
-    def get_ino(self, path):
+    def get_ino(self, path: str) -> int:
         """Get an inode number for a given QOM path."""
         if path in self.ino_map:
             return self.ino_map[path]
@@ -91,7 +98,7 @@ class QOMFuse(QOMCommand, Operations):
         self.ino_count += 1
         return self.ino_map[path]
 
-    def is_object(self, path):
+    def is_object(self, path: str) -> bool:
         """Is the given QOM path an object?"""
         try:
             self.qom_list(path)
@@ -99,7 +106,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def is_property(self, path):
+    def is_property(self, path: str) -> bool:
         """Is the given QOM path a property?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
@@ -112,7 +119,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def is_link(self, path):
+    def is_link(self, path: str) -> bool:
         """Is the given QOM path a link?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
@@ -125,7 +132,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def read(self, path, size, offset, fh):
+    def read(self, path: str, size: int, offset: int, fh: IO[bytes]) -> bytes:
         if not self.is_property(path):
             raise FuseOSError(ENOENT)
 
@@ -143,7 +150,7 @@ class QOMFuse(QOMCommand, Operations):
 
         return bytes(data[offset:][:size], encoding='utf-8')
 
-    def readlink(self, path):
+    def readlink(self, path: str) -> Union[bool, str]:
         if not self.is_link(path):
             return False
         path, prop = path.rsplit('/', 1)
@@ -151,7 +158,8 @@ class QOMFuse(QOMCommand, Operations):
         return prefix + str(self.qmp.command('qom-get', path=path,
                                              property=prop))
 
-    def getattr(self, path, fh=None):
+    def getattr(self, path: str,
+                fh: Optional[IO[bytes]] = None) -> Mapping[str, object]:
         if self.is_link(path):
             value = {
                 'st_mode': 0o755 | stat.S_IFLNK,
@@ -195,7 +203,7 @@ class QOMFuse(QOMCommand, Operations):
             raise FuseOSError(ENOENT)
         return value
 
-    def readdir(self, path, fh):
+    def readdir(self, path: str, fh: IO[bytes]) -> Iterator[str]:
         yield '.'
         yield '..'
         for item in self.qom_list(path):
Signature · 1 line
2.31.1
author
#19 lore raw

PATCH v3 16/19 python: add optional FUSE dependencies

In reply to John Snow

In preparation for moving qom-fuse over to the python package, we need some new dependencies to support it.

Add an optional 'fusepy' dependency that users of the package can opt into with e.g. "pip install qemu[fuse]" which installs the requirements necessary to obtain the additional functionality.

Add the same fusepy dependency to the 'devel' extras group -- unfortunately I do not see a way for optional groups to imply other optional groups at present, so the dependency is repeated. The development group needs to include the full set of dependencies for the purpose of static analysis of all features offered by this library.

Lastly, add the [fuse] extras group to tox's configuration as a workaround so that if a stale tox environment is found when running make check-tox, tox will know to rebuild its environments.

---
 python/Pipfile.lock | 6 ++++++
 python/setup.cfg    | 9 ++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
python/Pipfile.lock +6 -0
index f2a3f91d0fa..5bb3f1b6351 100644
@@ -67,6 +67,12 @@
             "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
             "version": "==3.9.2"
         },
+        "fusepy": {
+            "hashes": [
+                "sha256:72ff783ec2f43de3ab394e3f7457605bf04c8cf288a2f4068b4cde141d4ee6bd"
+            ],
+            "version": "==3.0.1"
+        },
         "importlib-metadata": {
             "hashes": [
                 "sha256:8c501196e49fb9df5df43833bdb1e4328f64847763ec8a50703148b73784d581",
python/setup.cfg +8 -1
index ba8d29fd62d..aca6f311853 100644
@@ -32,11 +32,16 @@ packages =
 devel =
     avocado-framework >= 87.0
     flake8 >= 3.6.0
+    fusepy >= 2.0.4
     isort >= 5.1.2
     mypy >= 0.770
     pylint >= 2.8.0
     tox >= 3.18.0
 
+# Provides qom-fuse functionality
+fuse =
+    fusepy >= 2.0.4
+
 [options.entry_points]
 console_scripts =
     qom = qemu.qmp.qom:main
@@ -114,6 +119,8 @@ envlist = py36, py37, py38, py39, py310
 
 [testenv]
 allowlist_externals = make
-deps = .[devel]
+deps =
+    .[devel]
+    .[fuse]  # Workaround to trigger tox venv rebuild
 commands =
     make check
Signature · 1 line
2.31.1
author
#20 lore raw

PATCH v3 17/19 scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py

In reply to John Snow

Move qom-fuse over to the python package now that it passes the linter. Update the import paradigms so that it continues to pass in the context of the Python package.

---
 scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
 rename scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py (95%)
 mode change 100755 => 100644
python/qemu/qmp/qom_fuse.py renamed from scripts/qmp/qom-fuse +2 -10
old mode 100755
new mode 100644
similarity index 95%
rename from scripts/qmp/qom-fuse
rename to python/qemu/qmp/qom_fuse.py
index a5a7a304a3e..43f4671fdb1
@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """
 QEMU Object Model FUSE filesystem tool
 
@@ -35,7 +34,6 @@
 
 import argparse
 from errno import ENOENT, EPERM
-import os
 import stat
 import sys
 from typing import (
@@ -50,10 +48,8 @@
 import fuse
 from fuse import FUSE, FuseOSError, Operations
 
-
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QMPResponseError
-from qemu.qmp.qom_common import QOMCommand
+from . import QMPResponseError
+from .qom_common import QOMCommand
 
 
 fuse.fuse_python_api = (0, 2)
@@ -208,7 +204,3 @@ def readdir(self, path: str, fh: IO[bytes]) -> Iterator[str]:
         yield '..'
         for item in self.qom_list(path):
             yield item.name
-
-
-if __name__ == '__main__':
-    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.31.1
author
#21 lore raw

PATCH v3 18/19 scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py

In reply to John Snow

By leaving the script absent for a commit, git-blame travels to the new file instead of staying on the shim.

---
 scripts/qmp/qom-fuse | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 scripts/qmp/qom-fuse
scripts/qmp/qom-fuse new file +11 -0
new file mode 100755
index 00000000000..a58c8ef9793
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp.qom_fuse import QOMFuse
+
+
+if __name__ == '__main__':
+    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.31.1
author
#22 lore raw

PATCH v3 19/19 python/qmp: add fuse command to 'qom' tools

In reply to John Snow

The 'fuse' command will be unavailable if 'fusepy' is not installed. It will simply not load and subsequently be unavailable as a subcommand.

---
 python/qemu/qmp/qom.py | 14 ++++++++++++--
 python/setup.cfg       |  1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
python/qemu/qmp/qom.py +12 -2
index 7fe1448b5d9..7ec7843d57a 100644
@@ -1,7 +1,7 @@
 """
 QEMU Object Model testing tools.
 
-usage: qom [-h] {set,get,list,tree} ...
+usage: qom [-h] {set,get,list,tree,fuse} ...
 
 Query and manipulate QOM data
 
@@ -9,11 +9,12 @@
   -h, --help           show this help message and exit
 
 QOM commands:
-  {set,get,list,tree}
+  {set,get,list,tree,fuse}
     set                Set a QOM property value
     get                Get a QOM property value
     list               List QOM properties at a given path
     tree               Show QOM tree from a given path
+    fuse               Mount a QOM tree as a FUSE filesystem
 """
 ##
 # Copyright John Snow 2020, for Red Hat, Inc.
@@ -35,6 +36,15 @@
 from .qom_common import QOMCommand
 
 
+try:
+    from .qom_fuse import QOMFuse
+except ModuleNotFoundError as err:
+    if err.name != 'fuse':
+        raise
+else:
+    assert issubclass(QOMFuse, QOMCommand)
+
+
 class QOMSet(QOMCommand):
     """
     QOM Command - Set a property to a given value.
python/setup.cfg +1 -0
index aca6f311853..6b6be8b03c6 100644
@@ -49,6 +49,7 @@ console_scripts =
     qom-get = qemu.qmp.qom:QOMGet.entry_point
     qom-list = qemu.qmp.qom:QOMList.entry_point
     qom-tree = qemu.qmp.qom:QOMTree.entry_point
+    qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse]
 
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
Signature · 1 line
2.31.1
Reviewed-by on v3 19/19
#23 lore raw

Re: [PATCH v3 19/19] python/qmp: add fuse command to 'qom' tools

Quoted 8 lines · John Snow

The 'fuse' command will be unavailable if 'fusepy' is not installed. It will simply not load and subsequently be unavailable as a subcommand.

---
 python/qemu/qmp/qom.py | 14 ++++++++++++--
 python/setup.cfg       |  1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
author
#24 lore raw
In reply to John Snow
Quoted 85 lines · John Snow

GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-qom
CI: https://gitlab.com/jsnow/qemu/-/pipelines/313932818

Hello!
------

This series applies the usual linting cleanups to modernize the qom tools and then integrates them into the python packaging hierarchy.

This will help prevent further bitrot of these tools.

I don't expect or need a detailed review of the QOM tools themselves -- these tools are not used during build OR testing, and some are fairly bitrotted in places.

However, some details of how the python packaging system is being utilized here may attract your attention and could be worth a look. (Patches 5-6 and 16-19 are the interesting ones.)

Since these scripts aren't critical, I'm OK with sending a fairly hasty PR to merge these sooner rather than later.

Overview:
---------

Patch 1: Update Pipfile.lock (See the commit as for why ...)

Patches 2-3: Correct some existing typing issues in qemu.qmp

Patch 4: Combine qom-set, qom-get, (etc) into one, newly written script that makes all of the command invocations, help text, etc. consistent. (I ask that review for this patch should be limited to critical
mistakes: I have no interest in developing the QOM tools further.)

Patches 5-6: Integrate the qom tools into the python package.

Patches 7-15: Delinting of the qom_fuse script. Similarly, I am not terribly interested in further improvements here, personally.

Patches 16-19: Integrating qom-fuse into the Python packaging directory; additional care is taken to ensure that "optional" dependencies like fusepy are handled well.

Changelog
---------

V3:
- Technically, I sent two versions of this before, a long time ago. This has been cleaned up and based on the latest origin/master.

John Snow (19):
python/pipenv: Update Pipfile.lock
python/qmp: Fix type of SocketAddrT
python/qmp: add parse_address classmethod
python/qmp: Add qom script rewrites
python/qmp: add qom script entry points
scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/ scripts/qom-fuse: apply isort rules
scripts/qom-fuse: apply flake8 rules
python: Add 'fh' to known-good variable names scripts/qom-fuse: Apply pylint rules
scripts/qom-fuse: Add docstrings
scripts/qom-fuse: Convert to QOMCommand
scripts/qom-fuse: use QOMCommand.qom_list() scripts/qom-fuse: ensure QOMFuse.read always returns bytes scripts/qom-fuse: add static type hints
python: add optional FUSE dependencies
scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py python/qmp: add fuse command to 'qom' tools

  python/Pipfile.lock           |  97 +++++++++++-
  python/qemu/qmp/__init__.py   |  28 +++-
  python/qemu/qmp/qom.py        | 272 ++++++++++++++++++++++++++++++++++
  python/qemu/qmp/qom_common.py | 178 ++++++++++++++++++++++
  python/qemu/qmp/qom_fuse.py   | 206 +++++++++++++++++++++++++
  python/setup.cfg              |  33 ++++-
  scripts/qmp/qmp-shell         |  21 +--
  scripts/qmp/qom-fuse          | 144 +-----------------
  scripts/qmp/qom-get           |  66 +--------
  scripts/qmp/qom-list          |  63 +-------
  scripts/qmp/qom-set           |  63 +-------
  scripts/qmp/qom-tree          |  74 +--------
  12 files changed, 828 insertions(+), 417 deletions(-)
Quoted 4 lines

create mode 100644 python/qemu/qmp/qom.py create mode 100644 python/qemu/qmp/qom_common.py create mode 100644 python/qemu/qmp/qom_fuse.py

Provisionally staged to my Python branch:
https://gitlab.com/jsnow/qemu/-/commits/python/

(Barring objections, I intend to send a PR for all the ./scripts/qmp/ cleanups at once, next Friday.)

--js