Skip to contentSkip to conversation

python: a few improvements to qmp-shell

PATCH In review Daniel P. Berrangé 2 participants 6 messages Jan 28, 2022 – Feb 8, 2022

lore
Share & tools
lore
#0 lore raw

v3 cover python: a few improvements to qmp-shell

This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.

In v3:

- Add qmp-shell-wrap to setup.cfg entry points

In v2:

- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warnings

Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file

 python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  4 ++
 scripts/qmp/qmp-shell-wrap    | 11 +++++
 3 files changed, 96 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/2 python: introduce qmp-shell-wrap convenience tool

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
 python/qemu/aqmp/qmp_shell.py | 67 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  1 +
 scripts/qmp/qmp-shell-wrap    | 11 ++++++
 3 files changed, 75 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/aqmp/qmp_shell.py +63 -4
index d11bf54b00..185f5066ef 100644
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -167,8 +168,10 @@ class QMPShell(QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -409,8 +412,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -533,5 +538,59 @@ def main() -> None:
             pass
 
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and
+    start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = ["qemu-system-x86_64"]
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {sockpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    try:
+        with shell_class(address, args.pretty, args.verbose, True) as qemu:
+            with Popen(cmd):
+
+                try:
+                    qemu.accept()
+                except qmp.QMPConnectError:
+                    die("Didn't get QMP greeting message")
+                except qmp.QMPCapabilitiesError:
+                    die("Couldn't negotiate capabilities")
+                except OSError as err:
+                    die(f"Couldn't connect to {sockpath}: {err!s}")
+
+                for _ in qemu.repl():
+                    pass
+    finally:
+        os.unlink(sockpath)
+
+
 if __name__ == '__main__':
     main()
python/setup.cfg +1 -0
index 3fb18f845d..84c0f33baf 100644
@@ -68,6 +68,7 @@ console_scripts =
     qom-fuse = qemu.utils.qom_fuse:QOMFuse.entry_point [fuse]
     qemu-ga-client = qemu.utils.qemu_ga_client:main
     qmp-shell = qemu.aqmp.qmp_shell:main
+    qmp-shell-wrap = qemu.aqmp.qmp_shell:main_wrap
     aqmp-tui = qemu.aqmp.aqmp_tui:main [tui]
 
 [flake8]
scripts/qmp/qmp-shell-wrap new file +11 -0
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/2 python: support recording QMP session to a file

In reply to Daniel P. Berrangé

When running QMP commands with very large response payloads, it is often not easy to spot the info you want. If we can save the response to a file then tools like 'grep' or 'jq' can be used to extract information.

For convenience of processing, we merge the QMP command and response dictionaries together:

  {
      "arguments": {},
      "execute": "query-kvm",
      "return": {
          "enabled": false,
          "present": true
      }
  }

Example usage

  $ ./scripts/qmp/qmp-shell-wrap -l q.log -p -- ./build/qemu-system-x86_64 -display none
  Welcome to the QMP low-level shell!
  Connected
  (QEMU) query-kvm
  {
      "return": {
          "enabled": false,
          "present": true
      }
  }
  (QEMU) query-mice
  {
      "return": [
          {
              "absolute": false,
              "current": true,
              "index": 2,
              "name": "QEMU PS/2 Mouse"
          }
      ]
  }
 $ jq --slurp '. | to_entries[] | select(.value.execute == "query-kvm") |
               .value.return.enabled' < q.log
   false
---
 python/qemu/aqmp/qmp_shell.py | 29 ++++++++++++++++++++++-------
 python/setup.cfg              |  3 +++
 2 files changed, 25 insertions(+), 7 deletions(-)
python/qemu/aqmp/qmp_shell.py +22 -7
index 185f5066ef..85988d9f2e 100644
@@ -89,6 +89,7 @@
 from subprocess import Popen
 import sys
 from typing import (
+    IO,
     Iterator,
     List,
     NoReturn,
@@ -170,7 +171,8 @@ class QMPShell(QEMUMonitorProtocol):
     def __init__(self, address: SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
+                 server: bool = False,
+                 logfile: Optional[str] = None):
         super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
@@ -180,6 +182,10 @@ def __init__(self, address: SocketAddrT,
                                       '.qmp-shell_history')
         self.pretty = pretty
         self.verbose = verbose
+        self.logfile = None
+
+        if logfile is not None:
+            self.logfile = open(logfile, "w", encoding='utf-8')
 
     def close(self) -> None:
         # Hook into context manager of parent to save shell history.
@@ -320,11 +326,11 @@ def _build_cmd(self, cmdline: str) -> Optional[QMPMessage]:
         self._cli_expr(cmdargs[1:], qmpcmd['arguments'])
         return qmpcmd
 
-    def _print(self, qmp_message: object) -> None:
+    def _print(self, qmp_message: object, fh: IO[str] = sys.stdout) -> None:
         jsobj = json.dumps(qmp_message,
                            indent=4 if self.pretty else None,
                            sort_keys=self.pretty)
-        print(str(jsobj))
+        print(str(jsobj), file=fh)
 
     def _execute_cmd(self, cmdline: str) -> bool:
         try:
@@ -347,6 +353,9 @@ def _execute_cmd(self, cmdline: str) -> bool:
             print('Disconnected')
             return False
         self._print(resp)
+        if self.logfile is not None:
+            cmd = {**qmpcmd, **resp}
+            self._print(cmd, fh=self.logfile)
         return True
 
     def connect(self, negotiate: bool = True) -> None:
@@ -414,8 +423,9 @@ class HMPShell(QMPShell):
     def __init__(self, address: SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
-        super().__init__(address, pretty, verbose, server)
+                 server: bool = False,
+                 logfile: Optional[str] = None):
+        super().__init__(address, pretty, verbose, server, logfile)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -508,6 +518,8 @@ def main() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     default_server = os.environ.get('QMP_SOCKET')
     parser.add_argument('qmp_server', action='store',
@@ -526,7 +538,7 @@ def main() -> None:
         parser.error(f"Bad port number: {args.qmp_server}")
         return  # pycharm doesn't know error() is noreturn
 
-    with shell_class(address, args.pretty, args.verbose) as qemu:
+    with shell_class(address, args.pretty, args.verbose, args.logfile) as qemu:
         try:
             qemu.connect(negotiate=not args.skip_negotiation)
         except ConnectError as err:
@@ -550,6 +562,8 @@ def main_wrap() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     parser.add_argument('command', nargs=argparse.REMAINDER,
                         help='QEMU command line to invoke')
@@ -574,7 +588,8 @@ def main_wrap() -> None:
         return  # pycharm doesn't know error() is noreturn
 
     try:
-        with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        with shell_class(address, args.pretty, args.verbose,
+                         True, args.logfile) as qemu:
             with Popen(cmd):
 
                 try:
python/setup.cfg +3 -0
index 84c0f33baf..9aeb74aed4 100644
@@ -114,7 +114,10 @@ ignore_missing_imports = True
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
 disable=consider-using-f-string,
+        consider-using-with,
+        too-many-arguments,
         too-many-function-args,  # mypy handles this with less false positives.
+        too-many-instance-attributes,
         no-member,  # mypy also handles this better.
 
 [pylint.basic]
Signature · 1 line
2.34.1
#3 lore raw
In reply to Daniel P. Berrangé
Quoted 27 lines · Daniel P. Berrangé

This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.

In v3:

- Add qmp-shell-wrap to setup.cfg entry points

In v2:

- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warnings

Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file

Quoted 10 lines
 python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  4 ++
 scripts/qmp/qmp-shell-wrap    | 11 +++++
 3 files changed, 96 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap

--
2.34.1

Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.

--js

(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3 setup.py install' does indeed cause issues here. I have a patch I'll send soon that discourages the direct setup.py invocation to avoid frustration in the future.)

#4 lore raw
Quoted 39 lines · John Snow
Quoted 27 lines · Daniel P. Berrangé

This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.

In v3:

- Add qmp-shell-wrap to setup.cfg entry points

In v2:

- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warnings

Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file

Quoted 10 lines
 python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  4 ++
 scripts/qmp/qmp-shell-wrap    | 11 +++++
 3 files changed, 96 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap

--
2.34.1

Quoted 8 lines

Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.

--js

(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3 setup.py install' does indeed cause issues here. I have a patch I'll send soon that discourages the direct setup.py invocation to avoid frustration in the future.)

I've only ever used pip to install from pypi or remote git archives. How do you use it to install from your local git checkout

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw
Quoted 47 lines · Daniel P. Berrangé
Quoted 1 line · John Snow

On Fri, Jan 28, 2022 at 11:12 AM Daniel P. Berrangé <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines

This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU

command

Quoted 27 lines
Quoted 27 lines

line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.

In v3:

- Add qmp-shell-wrap to setup.cfg entry points

In v2:

- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warnings

Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file

 python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  4 ++
 scripts/qmp/qmp-shell-wrap    | 11 +++++
 3 files changed, 96 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
Quoted 6 lines
Quoted 4 lines

--
2.34.1

Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged

these.

Quoted 3 lines

--js

Quoted 8 lines
Quoted 4 lines

(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3 setup.py install' does indeed cause issues here. I have a patch I'll send soon that discourages the direct setup.py invocation to avoid frustration in the future.)

I've only ever used pip to install from pypi or remote git archives. How do you use it to install from your local git checkout

cd ~/src/qemu/python
pip install [-e] [--user] .

See python/README.rst: https://gitlab.com/jsnow/qemu/-/tree/master/python

I've also sent a patch that disables the broken method (and explains why it's broken), see

https://patchew.org/QEMU/[email protected]/

thanks!
--js

Quoted 11 lines

Regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|