Skip to contentSkip to conversation

python: absorb scripts/qmp/qom-* tooling

PATCH Superseded John Snow 2 participants 24 messages Oct 21, 2020 – Nov 4, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover python: absorb scripts/qmp/qom-* tooling

Based-on: <[email protected]>
[PATCH v3 00/15] python: create installable package

This is a bit of a demonstration of the direction I want to take our python tooling, and how it *might* work.

By moving items from ./scripts/*.py over to ./python/qemu/* somewhere, they can be checked with the same isort/flake8/pylint/mypy tooling as everything else. This will help prevent regressions.

I would like to, over time, move all applicable python scripts from ./scripts to ./python. That will be a long, gradual stream of changes, but the more we do it, the better off we'll be for these tools.

Reviewer notes:

- I just rewrote qom-xxx entirely, though it is based on the original scripts. Doing it brick by brick was too slow.

- I added a symlink to the qom-fuse file under the python/ tree so I could check it with the usual linters. This causes some future knowledge to bleed through in a few places; notably I update the python setup.cfg several times in the middle of the series where it doesn't seem like that should have an effect.

- qom-fuse disappears from the tree for a single commit, but that preserves git-blame history. Best I could do.

John Snow (15):
python/qmp: Add qom script rewrites
python/qmp: add qom script entry points
scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/ scripts/qom-fuse: apply isort rules
scripts/qom-fuse: apply flake8 rules
python: Add 'fh' to known-good variable names scripts/qom-fuse: Apply pylint rules
scripts/qom-fuse: Add docstrings
scripts/qom-fuse: Convert to QOMCommand
scripts/qom-fuse: use QOMCommand.qom_list() scripts/qom-fuse: ensure QOMFuse.read always returns bytes scripts/qom-fuse: add static type hints
scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py
python: add fuse command to 'qom' tools

 python/qemu/qmp/qom.py        | 217 ++++++++++++++++++++++++++++++++++
 python/qemu/qmp/qom_common.py | 153 ++++++++++++++++++++++++
 python/qemu/qmp/qom_fuse.py   | 207 ++++++++++++++++++++++++++++++++
 python/setup.cfg              |  24 +++-
 scripts/qmp/qom-fuse          | 144 +---------------------
 scripts/qmp/qom-get           |  66 +----------
 scripts/qmp/qom-list          |  63 +---------
 scripts/qmp/qom-set           |  63 +---------
 scripts/qmp/qom-tree          |  74 +-----------
 9 files changed, 618 insertions(+), 393 deletions(-)
 create mode 100644 python/qemu/qmp/qom.py
 create mode 100644 python/qemu/qmp/qom_common.py
 create mode 100644 python/qemu/qmp/qom_fuse.py
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH 1/15 python/qmp: Add qom script rewrites

Inspired by qom-set, qom-get, qom-tree and qom-list; combine all four of those scripts into one script.

A later addition of qom-fuse necessitates that some common features are split out and shared between them.

---
 python/qemu/qmp/qom.py        | 207 ++++++++++++++++++++++++++++++++++
 python/qemu/qmp/qom_common.py | 153 +++++++++++++++++++++++++
 2 files changed, 360 insertions(+)
 create mode 100644 python/qemu/qmp/qom.py
 create mode 100644 python/qemu/qmp/qom_common.py
python/qemu/qmp/qom.py new file +207 -0
new file mode 100644
index 0000000000..912d1809e6
@@ -0,0 +1,207 @@
+"""
+QEMU Object Model testing tools.
+
+usage: qom.py [-h] {set,get,list,tree} ...
+
+Query and manipulate QOM data
+
+optional arguments:
+  -h, --help           show this help message and exit
+
+QOM commands:
+  {set,get,list,tree,fuse}
+    set                Set a QOM property value
+    get                Get a QOM property value
+    list               List QOM properties at a given path
+    tree               Show QOM tree from a given path
+"""
+##
+# Copyright John Snow 2020, for Red Hat, Inc.
+# Copyright IBM, Corp. 2011
+#
+# Authors:
+#  John Snow <[email protected]>
+#  Anthony Liguori <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# Based on ./scripts/qmp/qom-[set|get|tree|list]
+##
+
+import argparse
+import sys
+
+from . import QMPResponseError
+from .qom_common import QOMCommand
+
+
+class QOMSet(QOMCommand):
+    """QOM Command - Set a property to a given value."""
+    name = 'set'
+    help = 'Set a QOM property value'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        cls.add_path_prop_arg(parser)
+        parser.add_argument(
+            'value',
+            metavar='<value>',
+            action='store',
+            help='new QOM property value'
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path, self.prop = args.path_prop.rsplit('.', 1)
+        self.value = args.value
+
+    def run(self) -> int:
+        rsp = self.qmp.command(
+            'qom-set',
+            path=self.path,
+            property=self.prop,
+            value=self.value
+        )
+        print(rsp)
+        return 0
+
+
+class QOMGet(QOMCommand):
+    """QOM Command - Get a property's current value."""
+    name = 'get'
+    help = 'Get a QOM property value'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        cls.add_path_prop_arg(parser)
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        try:
+            tmp = args.path_prop.rsplit('.', 1)
+        except ValueError as err:
+            raise ValueError('Invalid format for <path>.<property>') from err
+        self.path = tmp[0]
+        self.prop = tmp[1]
+
+    def run(self) -> int:
+        rsp = self.qmp.command(
+            'qom-get',
+            path=self.path,
+            property=self.prop
+        )
+        if isinstance(rsp, dict):
+            for key, value in rsp.items():
+                print(f"{key}: {value}")
+        else:
+            print(rsp)
+        return 0
+
+
+class QOMList(QOMCommand):
+    """QOM Command - List the properties at a given path."""
+    name = 'list'
+    help = 'List QOM properties at a given path'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'path',
+            metavar='<path>',
+            action='store',
+            help='QOM path',
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path = args.path
+
+    def run(self) -> int:
+        rsp = self.qom_list(self.path)
+        for item in rsp:
+            if item.child:
+                print(f"{item.name}/")
+            elif item.link:
+                print(f"@{item.name}/")
+            else:
+                print(item.name)
+        return 0
+
+
+class QOMTree(QOMCommand):
+    """QOM Command - Show the full tree below a given path."""
+    name = 'tree'
+    help = 'Show QOM tree from a given path'
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'path',
+            metavar='<path>',
+            action='store',
+            help='QOM path',
+            nargs='?',
+            default='/'
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.path = args.path
+
+    def _list_node(self, path: str) -> None:
+        print(path)
+        items = self.qom_list(path)
+        for item in items:
+            if item.child:
+                continue
+            try:
+                rsp = self.qmp.command('qom-get', path=path,
+                                       property=item.name)
+                print(f"  {item.name}: {rsp} ({item.type})")
+            except QMPResponseError as err:
+                print(f"  {item.name}: <EXCEPTION: {err!s}> ({item.type})")
+        print('')
+        for item in items:
+            if not item.child:
+                continue
+            if path == '/':
+                path = ''
+            self._list_node(f"{path}/{item.name}")
+
+    def run(self) -> int:
+        self._list_node(self.path)
+        return 0
+
+
+def main() -> int:
+    """QOM script main entry point."""
+    parser = argparse.ArgumentParser(
+        description='Query and manipulate QOM data'
+    )
+    subparsers = parser.add_subparsers(
+        title='QOM commands',
+        dest='command'
+    )
+
+    for command in QOMCommand.__subclasses__():
+        command.register(subparsers)
+
+    args = parser.parse_args()
+
+    if args.command is None:
+        parser.error('Command not specified.')
+        return 1
+
+    assert issubclass(args.cmd_class, QOMCommand)
+    cmd = args.cmd_class(args)
+    assert isinstance(cmd, QOMCommand)
+    return cmd.run()
+
+
+if __name__ == '__main__':
+    sys.exit(main())
python/qemu/qmp/qom_common.py new file +153 -0

153 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#2 lore raw

PATCH 2/15 python/qmp: add qom script entry points

In reply to John Snow

Add the 'qom', 'qom-set', 'qom-get', 'qom-list', and 'qom-tree' scripts to the qemu.qmp package. When you install this package, these scripts will become available on your command line.

---
 python/setup.cfg | 8 ++++++++
 1 file changed, 8 insertions(+)
python/setup.cfg +8 -0
index 08def52372..506944ef6b 100755
@@ -18,6 +18,14 @@ classifiers =
 python_requires = >= 3.6
 packages = find_namespace:
 
+[options.entry_points]
+console_scripts =
+    qom = qemu.qmp.qom:main
+    qom-set = qemu.qmp.qom:QOMSet.entry_point
+    qom-get = qemu.qmp.qom:QOMGet.entry_point
+    qom-list = qemu.qmp.qom:QOMList.entry_point
+    qom-tree = qemu.qmp.qom:QOMTree.entry_point
+
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 
Signature · 1 line
2.26.2
Reviewed-by on v1 2/15
#3 lore raw

Re: [PATCH 02/15] python/qmp: add qom script entry points

Quoted 8 lines · John Snow

Add the 'qom', 'qom-set', 'qom-get', 'qom-list', and 'qom-tree' scripts to the qemu.qmp package. When you install this package, these scripts will become available on your command line.

---
  python/setup.cfg | 8 ++++++++
  1 file changed, 8 insertions(+)
author
#4 lore raw

PATCH 3/15 scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/

In reply to John Snow

Redirect to the new qom scripts. These forwarders can be deleted eventually when there has been more time for the dust on the Python packaging to settle and people understand how to find these commands.

Note: You can run these by setting $PYTHONPATH in your shell and then running "python3 -m qemu.qmp.qom", or you can install the qemu namespace package and use the "qom" or "qom-set" scripts.

I've written how to install the package elsewhere, but for the sake of git-blame, cd to ./python, and then do:

- pip3 install [--user] [-e] .

--user will install to your local user install (will not work inside of
a venv), omitting this flag installs to your system-wide packages (outside of a venv) or to your current virtual environment (inside the venv).

When installing to a venv or to your system-wide packages, "qom" should be in your $PATH already. If you do a user install, you may need to add ~/.local/bin to your $PATH if you haven't already.

-e installs in editable mode: the installed package is effectively just
a symlink to this folder; so changes to your git working tree are reflected in the installed package.

Alternatively to the above, If you have pipenv installed (pip3 install --user pipenv), you may also invoke 'pipenv shell' to enter a pipenv-managed virtual environment (as a shell process that you may leave with ctrt+d) that has 'qom' already in $PATH.

---
 scripts/qmp/qom-get  | 66 +++------------------------------------
 scripts/qmp/qom-list | 63 +++----------------------------------
 scripts/qmp/qom-set  | 63 +++----------------------------------
 scripts/qmp/qom-tree | 74 +++-----------------------------------------
 4 files changed, 16 insertions(+), 250 deletions(-)
scripts/qmp/qom-get +4 -62
index 666df71832..e4f3e0c013 100755
@@ -1,69 +1,11 @@
 #!/usr/bin/env python3
-##
-# QEMU Object Model test tools
-#
-# Copyright IBM, Corp. 2011
-#
-# Authors:
-#  Anthony Liguori   <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or later.  See
-# the COPYING file in the top-level directory.
-##
 
-import sys
 import os
+import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.qmp.qom import QOMGet
 
-cmd, args = sys.argv[0], sys.argv[1:]
-socket_path = None
-path = None
-prop = None
 
-def usage():
-    return '''environment variables:
-    QMP_SOCKET=<path | addr:port>
-usage:
-    %s [-h] [-s <QMP socket path | addr:port>] <path>.<property>
-''' % cmd
-
-def usage_error(error_msg = "unspecified error"):
-    sys.stderr.write('%s\nERROR: %s\n' % (usage(), error_msg))
-    exit(1)
-
-if len(args) > 0:
-    if args[0] == "-h":
-        print(usage())
-        exit(0);
-    elif args[0] == "-s":
-        try:
-            socket_path = args[1]
-        except:
-            usage_error("missing argument: QMP socket path or address");
-        args = args[2:]
-
-if not socket_path:
-    if 'QMP_SOCKET' in os.environ:
-        socket_path = os.environ['QMP_SOCKET']
-    else:
-        usage_error("no QMP socket path or address given");
-
-if len(args) > 0:
-    try:
-        path, prop = args[0].rsplit('.', 1)
-    except:
-        usage_error("invalid format for path/property/value")
-else:
-    usage_error("not enough arguments")
-
-srv = QEMUMonitorProtocol(socket_path)
-srv.connect()
-
-rsp = srv.command('qom-get', path=path, property=prop)
-if type(rsp) == dict:
-    for i in rsp.keys():
-        print('%s: %s' % (i, rsp[i]))
-else:
-    print(rsp)
+if __name__ == '__main__':
+    sys.exit(QOMGet.entry_point())
scripts/qmp/qom-list +4 -59

70 lines not on this page · raw message

scripts/qmp/qom-set +4 -59

70 lines not on this page · raw message

scripts/qmp/qom-tree +4 -70

81 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v1 3/15
#5 lore raw

Re: [PATCH 03/15] scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/

Quoted 31 lines · John Snow

Redirect to the new qom scripts. These forwarders can be deleted eventually when there has been more time for the dust on the Python packaging to settle and people understand how to find these commands.

Note: You can run these by setting $PYTHONPATH in your shell and then running "python3 -m qemu.qmp.qom", or you can install the qemu namespace package and use the "qom" or "qom-set" scripts.

I've written how to install the package elsewhere, but for the sake of git-blame, cd to ./python, and then do:

- pip3 install [--user] [-e] .

--user will install to your local user install (will not work inside of
a venv), omitting this flag installs to your system-wide packages (outside of a venv) or to your current virtual environment (inside the venv).

When installing to a venv or to your system-wide packages, "qom" should be in your $PATH already. If you do a user install, you may need to add ~/.local/bin to your $PATH if you haven't already.

-e installs in editable mode: the installed package is effectively just
a symlink to this folder; so changes to your git working tree are reflected in the installed package.

Alternatively to the above, If you have pipenv installed (pip3 install --user pipenv), you may also invoke 'pipenv shell' to enter a pipenv-managed virtual environment (as a shell process that you may leave with ctrt+d) that has 'qom' already in $PATH.

Quoted 7 lines
---
  scripts/qmp/qom-get  | 66 +++------------------------------------
  scripts/qmp/qom-list | 63 +++----------------------------------
  scripts/qmp/qom-set  | 63 +++----------------------------------
  scripts/qmp/qom-tree | 74 +++-----------------------------------------
  4 files changed, 16 insertions(+), 250 deletions(-)
author
#6 lore raw

PATCH 4/15 scripts/qom-fuse: apply isort rules

In reply to John Snow

Hint: you can use symlinks to create qom_fuse.py in python/qemu/qmp/ and point to scripts/qom-fuse to apply the standard linting rules to this script.

---
 scripts/qmp/qom-fuse | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
scripts/qmp/qom-fuse +9 -3
index 7c7cff8edf..62deb9adb1 100755
@@ -13,14 +13,20 @@
 # the COPYING file in the top-level directory.
 ##
 
-import fuse, stat
-from fuse import FUSE, FuseOSError, Operations
-import os, posix, sys
 from errno import *
+import os
+import posix
+import stat
+import sys
+
+import fuse
+from fuse import FUSE, FuseOSError, Operations
+
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu.qmp import QEMUMonitorProtocol
 
+
 fuse.fuse_python_api = (0, 2)
 
 class QOMFS(Operations):
Signature · 1 line
2.26.2
Reviewed-by on v1 4/15
#7 lore raw

Re: [PATCH 04/15] scripts/qom-fuse: apply isort rules

Quoted 8 lines · John Snow

Hint: you can use symlinks to create qom_fuse.py in python/qemu/qmp/ and point to scripts/qom-fuse to apply the standard linting rules to this script.

---
  scripts/qmp/qom-fuse | 12 +++++++++---
  1 file changed, 9 insertions(+), 3 deletions(-)
author
#8 lore raw

PATCH 5/15 scripts/qom-fuse: apply flake8 rules

In reply to John Snow

flake8 still has one warning because of the sys.path hack, but that will be going away by the end of this patch series.

---
 scripts/qmp/qom-fuse | 81 +++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 38 deletions(-)
scripts/qmp/qom-fuse +43 -38
index 62deb9adb1..ca30e92867 100755
@@ -9,13 +9,12 @@
 #  Anthony Liguori   <[email protected]>
 #  Markus Armbruster <[email protected]>
 #
-# This work is licensed under the terms of the GNU GPL, version 2 or later.  See
-# the COPYING file in the top-level directory.
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
 ##
 
-from errno import *
+from errno import ENOENT, EPERM
 import os
-import posix
 import stat
 import sys
 
@@ -29,6 +28,7 @@ from qemu.qmp import QEMUMonitorProtocol
 
 fuse.fuse_python_api = (0, 2)
 
+
 class QOMFS(Operations):
     def __init__(self, qmp):
         self.qmp = qmp
@@ -45,7 +45,7 @@ class QOMFS(Operations):
 
     def is_object(self, path):
         try:
-            items = self.qmp.command('qom-list', path=path)
+            self.qmp.command('qom-list', path=path)
             return True
         except:
             return False
@@ -85,7 +85,7 @@ class QOMFS(Operations):
             path = '/'
         try:
             data = self.qmp.command('qom-get', path=path, property=prop)
-            data += '\n' # make values shell friendly
+            data += '\n'  # make values shell friendly
         except:
             raise FuseOSError(EPERM)
 
@@ -104,38 +104,44 @@ class QOMFS(Operations):
 
     def getattr(self, path, fh=None):
         if self.is_link(path):
-            value = { 'st_mode': 0o755 | stat.S_IFLNK,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 2,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o755 | stat.S_IFLNK,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 2,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         elif self.is_object(path):
-            value = { 'st_mode': 0o755 | stat.S_IFDIR,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 2,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o755 | stat.S_IFDIR,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 2,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         elif self.is_property(path):
-            value = { 'st_mode': 0o644 | stat.S_IFREG,
-                      'st_ino': self.get_ino(path),
-                      'st_dev': 0,
-                      'st_nlink': 1,
-                      'st_uid': 1000,
-                      'st_gid': 1000,
-                      'st_size': 4096,
-                      'st_atime': 0,
-                      'st_mtime': 0,
-                      'st_ctime': 0 }
+            value = {
+                'st_mode': 0o644 | stat.S_IFREG,
+                'st_ino': self.get_ino(path),
+                'st_dev': 0,
+                'st_nlink': 1,
+                'st_uid': 1000,
+                'st_gid': 1000,
+                'st_size': 4096,
+                'st_atime': 0,
+                'st_mtime': 0,
+                'st_ctime': 0
+            }
         else:
             raise FuseOSError(ENOENT)
         return value
@@ -146,8 +152,7 @@ class QOMFS(Operations):
         for item in self.qmp.command('qom-list', path=path):
             yield str(item['name'])
 
+
 if __name__ == '__main__':
-    import os
-
     fuse = FUSE(QOMFS(QEMUMonitorProtocol(os.environ['QMP_SOCKET'])),
                 sys.argv[1], foreground=True)
Signature · 1 line
2.26.2
author
#9 lore raw

PATCH 6/15 python: Add 'fh' to known-good variable names

In reply to John Snow

fd and fh are fine: we often use these for "file descriptor" or "file handle" accordingly. It is rarely the case that you need to enforce a more semantically meaningful name beyond "This is the file we are using right now."

While we're here: add comments for all of the non-standard pylint names. (And the underscore.)

---
 python/setup.cfg | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
python/setup.cfg +4 -3
index 506944ef6b..58cf48aaed 100755
@@ -56,9 +56,10 @@ good-names=i,
            k,
            ex,
            Run,
-           _,
-           fd,
-           c,
+           _,   # By convention: Unused variable
+           fh,  # fh = open(...)
+           fd,  # fd = os.open(...)
+           c,   # for c in string: ...
 
 [pylint.similarities]
 # Ignore imports when computing similarities.
Signature · 1 line
2.26.2
author
#10 lore raw

PATCH 7/15 scripts/qom-fuse: Apply pylint rules

In reply to John Snow

- Catch specific exceptions from QMP
- Reraise errors with explicit context
- method parameters should match parent's names

---
 scripts/qmp/qom-fuse | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
scripts/qmp/qom-fuse +8 -8
index ca30e92867..805e99c8ec 100755
@@ -23,7 +23,7 @@ from fuse import FUSE, FuseOSError, Operations
 
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.qmp import QEMUMonitorProtocol, QMPResponseError
 
 
 fuse.fuse_python_api = (0, 2)
@@ -47,7 +47,7 @@ class QOMFS(Operations):
         try:
             self.qmp.command('qom-list', path=path)
             return True
-        except:
+        except QMPResponseError:
             return False
 
     def is_property(self, path):
@@ -59,7 +59,7 @@ class QOMFS(Operations):
                 if item['name'] == prop:
                     return True
             return False
-        except:
+        except QMPResponseError:
             return False
 
     def is_link(self, path):
@@ -73,10 +73,10 @@ class QOMFS(Operations):
                         return True
                     return False
             return False
-        except:
+        except QMPResponseError:
             return False
 
-    def read(self, path, length, offset, fh):
+    def read(self, path, size, offset, fh):
         if not self.is_property(path):
             return -ENOENT
 
@@ -86,13 +86,13 @@ class QOMFS(Operations):
         try:
             data = self.qmp.command('qom-get', path=path, property=prop)
             data += '\n'  # make values shell friendly
-        except:
-            raise FuseOSError(EPERM)
+        except QMPResponseError as err:
+            raise FuseOSError(EPERM) from err
 
         if offset > len(data):
             return ''
 
-        return bytes(data[offset:][:length], encoding='utf-8')
+        return bytes(data[offset:][:size], encoding='utf-8')
 
     def readlink(self, path):
         if not self.is_link(path):
Signature · 1 line
2.26.2
author
#12 lore raw

PATCH 8/15 scripts/qom-fuse: Add docstrings

In reply to John Snow

The methods inherited from fuse don't need docstrings; that's up to fusepy to handle.

---
 scripts/qmp/qom-fuse | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
scripts/qmp/qom-fuse +20 -2
index 805e99c8ec..5b80da9df6 100755
@@ -1,7 +1,20 @@
 #!/usr/bin/env python3
+"""
+QEMU Object Model FUSE filesystem tool
+
+This script offers a simple FUSE filesystem within which the QOM tree
+may be browsed, queried and edited using traditional shell tooling.
+
+This script requires the 'fusepy' python package;
+you may install it by using ``pip3 install --user fusepy``.
+
+ENV:
+    QMP_SOCKET: Path to the QMP server socket
+
+Usage:
+    qom-fuse /mount/to/here
+"""
 ##
-# QEMU Object Model test tools
-#
 # Copyright IBM, Corp. 2012
 # Copyright (C) 2020 Red Hat, Inc.
 #
@@ -30,6 +43,7 @@ fuse.fuse_python_api = (0, 2)
 
 
 class QOMFS(Operations):
+    """QOMFS implements fuse.Operations to provide a QOM filesystem."""
     def __init__(self, qmp):
         self.qmp = qmp
         self.qmp.connect()
@@ -37,6 +51,7 @@ class QOMFS(Operations):
         self.ino_count = 1
 
     def get_ino(self, path):
+        """Get an inode number for a given QOM path."""
         if path in self.ino_map:
             return self.ino_map[path]
         self.ino_map[path] = self.ino_count
@@ -44,6 +59,7 @@ class QOMFS(Operations):
         return self.ino_map[path]
 
     def is_object(self, path):
+        """Is the given QOM path an object?"""
         try:
             self.qmp.command('qom-list', path=path)
             return True
@@ -51,6 +67,7 @@ class QOMFS(Operations):
             return False
 
     def is_property(self, path):
+        """Is the given QOM path a property?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
@@ -63,6 +80,7 @@ class QOMFS(Operations):
             return False
 
     def is_link(self, path):
+        """Is the given QOM path a link?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
Signature · 1 line
2.26.2
author
#13 lore raw

PATCH 9/15 scripts/qom-fuse: Convert to QOMCommand

In reply to John Snow

Move qom-fuse onto the QOMCommand base established in python/qemu/qmp/qom_common.py. The interface doesn't change incompatibly, "qom-fuse mountpoint" still works as an invocation, and QMP_SOCKET is still used as the environment variable.

---
 scripts/qmp/qom-fuse | 59 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 46 insertions(+), 13 deletions(-)
scripts/qmp/qom-fuse +46 -13
index 5b80da9df6..f9bf85f382 100755
@@ -8,11 +8,19 @@ may be browsed, queried and edited using traditional shell tooling.
 This script requires the 'fusepy' python package;
 you may install it by using ``pip3 install --user fusepy``.
 
-ENV:
-    QMP_SOCKET: Path to the QMP server socket
 
-Usage:
-    qom-fuse /mount/to/here
+usage: qom-fuse [-h] [--socket SOCKET] <mount>
+
+Mount a QOM tree as a FUSE filesystem
+
+positional arguments:
+  <mount>               Mount point
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --socket SOCKET, -s SOCKET
+                        QMP socket path or address (addr:port). May also be
+                        set via QMP_SOCKET environment variable.
 """
 ##
 # Copyright IBM, Corp. 2012
@@ -26,30 +34,56 @@ Usage:
 # See the COPYING file in the top-level directory.
 ##
 
+import argparse
 from errno import ENOENT, EPERM
 import os
 import stat
 import sys
+from typing import Dict
 
 import fuse
 from fuse import FUSE, FuseOSError, Operations
 
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol, QMPResponseError
+from qemu.qmp import QMPResponseError
+from qemu.qmp.qom_common import QOMCommand
 
 
 fuse.fuse_python_api = (0, 2)
 
 
-class QOMFS(Operations):
-    """QOMFS implements fuse.Operations to provide a QOM filesystem."""
-    def __init__(self, qmp):
-        self.qmp = qmp
-        self.qmp.connect()
-        self.ino_map = {}
+class QOMFuse(QOMCommand, Operations):
+    """
+    QOMFuse implements both fuse.Operations and QOMCommand.
+
+    Operations implements the FS, and QOMCommand implements the CLI command.
+    """
+    name = 'fuse'
+    help = 'Mount a QOM tree as a FUSE filesystem'
+    fuse: FUSE
+
+    @classmethod
+    def configure_parser(cls, parser: argparse.ArgumentParser) -> None:
+        super().configure_parser(parser)
+        parser.add_argument(
+            'mount',
+            metavar='<mount>',
+            action='store',
+            help="Mount point",
+        )
+
+    def __init__(self, args: argparse.Namespace):
+        super().__init__(args)
+        self.mount = args.mount
+        self.ino_map: Dict[str, int] = {}
         self.ino_count = 1
 
+    def run(self) -> int:
+        print(f"Mounting QOMFS to '{self.mount}'", file=sys.stderr)
+        self.fuse = FUSE(self, self.mount, foreground=True)
+        return 0
+
     def get_ino(self, path):
         """Get an inode number for a given QOM path."""
         if path in self.ino_map:
@@ -172,5 +206,4 @@ class QOMFS(Operations):
 
 
 if __name__ == '__main__':
-    fuse = FUSE(QOMFS(QEMUMonitorProtocol(os.environ['QMP_SOCKET'])),
-                sys.argv[1], foreground=True)
+    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.26.2
author
#14 lore raw

PATCH 10/15 scripts/qom-fuse: use QOMCommand.qom_list()

In reply to John Snow

the qom_list method provides a type-safe object that's easier to type check, so switch to using it.

---
 scripts/qmp/qom-fuse | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
scripts/qmp/qom-fuse +8 -10
index f9bf85f382..b1030273ef 100755
@@ -95,7 +95,7 @@ class QOMFuse(QOMCommand, Operations):
     def is_object(self, path):
         """Is the given QOM path an object?"""
         try:
-            self.qmp.command('qom-list', path=path)
+            self.qom_list(path)
             return True
         except QMPResponseError:
             return False
@@ -106,8 +106,8 @@ class QOMFuse(QOMCommand, Operations):
         if path == '':
             path = '/'
         try:
-            for item in self.qmp.command('qom-list', path=path):
-                if item['name'] == prop:
+            for item in self.qom_list(path):
+                if item.name == prop:
                     return True
             return False
         except QMPResponseError:
@@ -119,11 +119,9 @@ class QOMFuse(QOMCommand, Operations):
         if path == '':
             path = '/'
         try:
-            for item in self.qmp.command('qom-list', path=path):
-                if item['name'] == prop:
-                    if item['type'].startswith('link<'):
-                        return True
-                    return False
+            for item in self.qom_list(path):
+                if item.name == prop and item.link:
+                    return True
             return False
         except QMPResponseError:
             return False
@@ -201,8 +199,8 @@ class QOMFuse(QOMCommand, Operations):
     def readdir(self, path, fh):
         yield '.'
         yield '..'
-        for item in self.qmp.command('qom-list', path=path):
-            yield str(item['name'])
+        for item in self.qom_list(path):
+            yield item.name
 
 
 if __name__ == '__main__':
Signature · 1 line
2.26.2
author
#15 lore raw

PATCH 11/15 scripts/qom-fuse: ensure QOMFuse.read always returns bytes

In reply to John Snow

- Use FuseOSError to signal ENOENT instead of returning it
- Wrap qom-get in str(), as we don't always know its type
- The empty return should be b'', not ''.

---
 scripts/qmp/qom-fuse | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/qmp/qom-fuse +3 -3
index b1030273ef..b120b93391 100755
@@ -128,19 +128,19 @@ class QOMFuse(QOMCommand, Operations):
 
     def read(self, path, size, offset, fh):
         if not self.is_property(path):
-            return -ENOENT
+            raise FuseOSError(ENOENT)
 
         path, prop = path.rsplit('/', 1)
         if path == '':
             path = '/'
         try:
-            data = self.qmp.command('qom-get', path=path, property=prop)
+            data = str(self.qmp.command('qom-get', path=path, property=prop))
             data += '\n'  # make values shell friendly
         except QMPResponseError as err:
             raise FuseOSError(EPERM) from err
 
         if offset > len(data):
-            return ''
+            return b''
 
         return bytes(data[offset:][:size], encoding='utf-8')
 
Signature · 1 line
2.26.2
Reviewed-by on v1 11/15
#16 lore raw

Re: [PATCH 11/15] scripts/qom-fuse: ensure QOMFuse.read always returns bytes

Quoted 8 lines · John Snow

- Use FuseOSError to signal ENOENT instead of returning it
- Wrap qom-get in str(), as we don't always know its type
- The empty return should be b'', not ''.

---
  scripts/qmp/qom-fuse | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
author
#17 lore raw

PATCH 12/15 scripts/qom-fuse: add static type hints

In reply to John Snow

Because fusepy does not have type hints, add some targeted warning suppressions.

NOTE: Prior to this change, running 'mypy qemu' from the python directory worked OK, but only coincidentally. Going forward, you will need to run 'mypy -p qemu' instead. These invocation forms will be codified in a CI test soon.

---
 python/setup.cfg     |  8 ++++++++
 scripts/qmp/qom-fuse | 26 +++++++++++++++++---------
 2 files changed, 25 insertions(+), 9 deletions(-)
python/setup.cfg +8 -0
index 58cf48aaed..9592ac91df 100755
@@ -35,6 +35,14 @@ python_version = 3.6
 warn_unused_configs = True
 namespace_packages = True
 
+[mypy-qemu.qmp.qom_fuse]
+# fusepy has no type stubs:
+allow_subclassing_any = True
+
+[mypy-fuse]
+# fusepy has no type stubs:
+ignore_missing_imports = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
scripts/qmp/qom-fuse +17 -9
index b120b93391..22be366349 100755
@@ -39,7 +39,14 @@ from errno import ENOENT, EPERM
 import os
 import stat
 import sys
-from typing import Dict
+from typing import (
+    IO,
+    Dict,
+    Iterator,
+    Mapping,
+    Optional,
+    Union,
+)
 
 import fuse
 from fuse import FUSE, FuseOSError, Operations
@@ -84,7 +91,7 @@ class QOMFuse(QOMCommand, Operations):
         self.fuse = FUSE(self, self.mount, foreground=True)
         return 0
 
-    def get_ino(self, path):
+    def get_ino(self, path: str) -> int:
         """Get an inode number for a given QOM path."""
         if path in self.ino_map:
             return self.ino_map[path]
@@ -92,7 +99,7 @@ class QOMFuse(QOMCommand, Operations):
         self.ino_count += 1
         return self.ino_map[path]
 
-    def is_object(self, path):
+    def is_object(self, path: str) -> bool:
         """Is the given QOM path an object?"""
         try:
             self.qom_list(path)
@@ -100,7 +107,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def is_property(self, path):
+    def is_property(self, path: str) -> bool:
         """Is the given QOM path a property?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
@@ -113,7 +120,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def is_link(self, path):
+    def is_link(self, path: str) -> bool:
         """Is the given QOM path a link?"""
         path, prop = path.rsplit('/', 1)
         if path == '':
@@ -126,7 +133,7 @@ class QOMFuse(QOMCommand, Operations):
         except QMPResponseError:
             return False
 
-    def read(self, path, size, offset, fh):
+    def read(self, path: str, size: int, offset: int, fh: IO[bytes]) -> bytes:
         if not self.is_property(path):
             raise FuseOSError(ENOENT)
 
@@ -144,7 +151,7 @@ class QOMFuse(QOMCommand, Operations):
 
         return bytes(data[offset:][:size], encoding='utf-8')
 
-    def readlink(self, path):
+    def readlink(self, path: str) -> Union[bool, str]:
         if not self.is_link(path):
             return False
         path, prop = path.rsplit('/', 1)
@@ -152,7 +159,8 @@ class QOMFuse(QOMCommand, Operations):
         return prefix + str(self.qmp.command('qom-get', path=path,
                                              property=prop))
 
-    def getattr(self, path, fh=None):
+    def getattr(self, path: str,
+                fh: Optional[IO[bytes]] = None) -> Mapping[str, object]:
         if self.is_link(path):
             value = {
                 'st_mode': 0o755 | stat.S_IFLNK,
@@ -196,7 +204,7 @@ class QOMFuse(QOMCommand, Operations):
             raise FuseOSError(ENOENT)
         return value
 
-    def readdir(self, path, fh):
+    def readdir(self, path: str, fh: IO[bytes]) -> Iterator[str]:
         yield '.'
         yield '..'
         for item in self.qom_list(path):
Signature · 1 line
2.26.2
author
#18 lore raw

PATCH 13/15 scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py

In reply to John Snow

Move qom-fuse over to the python package now that it passes the linter. Update the import paradigms so that it passes.

---
 scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
 rename scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py (95%)
 mode change 100755 => 100644
python/qemu/qmp/qom_fuse.py renamed from scripts/qmp/qom-fuse +2 -10
old mode 100755
new mode 100644
similarity index 95%
rename from scripts/qmp/qom-fuse
rename to python/qemu/qmp/qom_fuse.py
index 22be366349..b0adcc1f8e
@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """
 QEMU Object Model FUSE filesystem tool
 
@@ -36,7 +35,6 @@
 
 import argparse
 from errno import ENOENT, EPERM
-import os
 import stat
 import sys
 from typing import (
@@ -51,10 +49,8 @@
 import fuse
 from fuse import FUSE, FuseOSError, Operations
 
-
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QMPResponseError
-from qemu.qmp.qom_common import QOMCommand
+from . import QMPResponseError
+from .qom_common import QOMCommand
 
 
 fuse.fuse_python_api = (0, 2)
@@ -209,7 +205,3 @@ def readdir(self, path: str, fh: IO[bytes]) -> Iterator[str]:
         yield '..'
         for item in self.qom_list(path):
             yield item.name
-
-
-if __name__ == '__main__':
-    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.26.2
Reviewed-by on v1 13/15
#19 lore raw

Re: [PATCH 13/15] scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py

Quoted 9 lines · John Snow

Move qom-fuse over to the python package now that it passes the linter. Update the import paradigms so that it passes.

---
  scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py | 12 ++----------
  1 file changed, 2 insertions(+), 10 deletions(-)

rename scripts/qmp/qom-fuse => python/qemu/qmp/qom_fuse.py (95%) mode change 100755 => 100644

author
#20 lore raw

PATCH 14/15 scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py

In reply to John Snow

By leaving the script absent for a commit, git-blame travels to the new file instead of staying on the shim.

---
 scripts/qmp/qom-fuse | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 scripts/qmp/qom-fuse
scripts/qmp/qom-fuse new file +11 -0
new file mode 100755
index 0000000000..a58c8ef979
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp.qom_fuse import QOMFuse
+
+
+if __name__ == '__main__':
+    sys.exit(QOMFuse.entry_point())
Signature · 1 line
2.26.2
Reviewed-by on v1 14/15
#21 lore raw

Re: [PATCH 14/15] scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py

Quoted 8 lines · John Snow

By leaving the script absent for a commit, git-blame travels to the new file instead of staying on the shim.

---
  scripts/qmp/qom-fuse | 11 +++++++++++
  1 file changed, 11 insertions(+)

create mode 100755 scripts/qmp/qom-fuse

author
#22 lore raw

PATCH 15/15 python: add fuse command to 'qom' tools

In reply to John Snow
---
 python/qemu/qmp/qom.py | 12 +++++++++++-
 python/setup.cfg       |  1 +
 2 files changed, 12 insertions(+), 1 deletion(-)
python/qemu/qmp/qom.py +11 -1
index 912d1809e6..52e06a949a 100644
@@ -1,7 +1,7 @@
 """
 QEMU Object Model testing tools.
 
-usage: qom.py [-h] {set,get,list,tree} ...
+usage: qom.py [-h] {set,get,list,tree,fuse} ...
 
 Query and manipulate QOM data
 
@@ -14,6 +14,7 @@
     get                Get a QOM property value
     list               List QOM properties at a given path
     tree               Show QOM tree from a given path
+    fuse               Mount a QOM tree as a FUSE filesystem
 """
 ##
 # Copyright John Snow 2020, for Red Hat, Inc.
@@ -36,6 +37,15 @@
 from .qom_common import QOMCommand
 
 
+try:
+    from .qom_fuse import QOMFuse
+except ImportError:
+    # either fusepy isn't installed, or qom_fuse itself is broken.
+    pass
+else:
+    assert issubclass(QOMFuse, QOMCommand)
+
+
 class QOMSet(QOMCommand):
     """QOM Command - Set a property to a given value."""
     name = 'set'
python/setup.cfg +1 -0
index 9592ac91df..a766a5af4d 100755
@@ -25,6 +25,7 @@ console_scripts =
     qom-get = qemu.qmp.qom:QOMGet.entry_point
     qom-list = qemu.qmp.qom:QOMList.entry_point
     qom-tree = qemu.qmp.qom:QOMTree.entry_point
+    qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point
 
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
Signature · 1 line
2.26.2
author NAK
#23 lore raw
In reply to John Snow
Quoted 3 lines · John Snow

Based-on: <[email protected]>
[PATCH v3 00/15] python: create installable package

Thanks for the early look. This is superseded by:

[PATCH v2 00/72] python: move scripts/qmp to python/qemu/qmp

Quoted 52 lines

This is a bit of a demonstration of the direction I want to take our python tooling, and how it *might* work.

By moving items from ./scripts/*.py over to ./python/qemu/* somewhere, they can be checked with the same isort/flake8/pylint/mypy tooling as everything else. This will help prevent regressions.

I would like to, over time, move all applicable python scripts from ./scripts to ./python. That will be a long, gradual stream of changes, but the more we do it, the better off we'll be for these tools.

Reviewer notes:

- I just rewrote qom-xxx entirely, though it is based on the original scripts. Doing it brick by brick was too slow.

- I added a symlink to the qom-fuse file under the python/ tree so I could check it with the usual linters. This causes some future knowledge to bleed through in a few places; notably I update the python setup.cfg several times in the middle of the series where it doesn't seem like that should have an effect.

- qom-fuse disappears from the tree for a single commit, but that preserves git-blame history. Best I could do.

John Snow (15):
python/qmp: Add qom script rewrites
python/qmp: add qom script entry points
scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/ scripts/qom-fuse: apply isort rules
scripts/qom-fuse: apply flake8 rules
python: Add 'fh' to known-good variable names scripts/qom-fuse: Apply pylint rules
scripts/qom-fuse: Add docstrings
scripts/qom-fuse: Convert to QOMCommand
scripts/qom-fuse: use QOMCommand.qom_list() scripts/qom-fuse: ensure QOMFuse.read always returns bytes scripts/qom-fuse: add static type hints
scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py
python: add fuse command to 'qom' tools

  python/qemu/qmp/qom.py        | 217 ++++++++++++++++++++++++++++++++++
  python/qemu/qmp/qom_common.py | 153 ++++++++++++++++++++++++
  python/qemu/qmp/qom_fuse.py   | 207 ++++++++++++++++++++++++++++++++
  python/setup.cfg              |  24 +++-
  scripts/qmp/qom-fuse          | 144 +---------------------
  scripts/qmp/qom-get           |  66 +----------
  scripts/qmp/qom-list          |  63 +---------
  scripts/qmp/qom-set           |  63 +---------
  scripts/qmp/qom-tree          |  74 +-----------
  9 files changed, 618 insertions(+), 393 deletions(-)
Quoted 4 lines

create mode 100644 python/qemu/qmp/qom.py create mode 100644 python/qemu/qmp/qom_common.py create mode 100644 python/qemu/qmp/qom_fuse.py

NACK