Skip to contentSkip to conversation

python: a few improvements to qmp-shell

PATCH Superseded Daniel P. Berrangé 4 participants 12 messages Jan 17, 2022 – Jan 21, 2022

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover python: a few improvements to qmp-shell

This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.

Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file

 python/qemu/qmp/qmp_shell.py | 80 ++++++++++++++++++++++++++++++++----
 scripts/qmp/qmp-shell-wrap   | 11 +++++
 2 files changed, 84 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
Signature · 1 line
2.33.1
#1 lore raw

PATCH 1/2 python: introduce qmp-shell-wrap convenience tool

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
 python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
 scripts/qmp/qmp-shell-wrap   | 11 +++++++
 2 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/qmp/qmp_shell.py +57 -4
index e7d7eb18f1..12f7d28afc 100644
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -162,8 +163,10 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -404,8 +407,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -529,6 +534,54 @@ def main() -> None:
         for _ in qemu.repl():
             pass
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = "qemu-system-x86_64"
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {socketpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        qemuproc = Popen(cmd)
+
+        try:
+            qemu.accept()
+        except qmp.QMPConnectError:
+            die("Didn't get QMP greeting message")
+        except qmp.QMPCapabilitiesError:
+            die("Couldn't negotiate capabilities")
+        except OSError as err:
+            die(f"Couldn't connect to {sockpath}: {err!s}")
+
+        for _ in qemu.repl():
+            pass
 
 if __name__ == '__main__':
     main()
scripts/qmp/qmp-shell-wrap new file +11 -0
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
Signature · 1 line
2.33.1
#2 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 141 lines · Daniel P. Berrangé

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
 python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
 scripts/qmp/qmp-shell-wrap   | 11 +++++++
 2 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/qmp/qmp_shell.py +57 -4
index e7d7eb18f1..12f7d28afc 100644
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -162,8 +163,10 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -404,8 +407,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -529,6 +534,54 @@ def main() -> None:
         for _ in qemu.repl():
             pass
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = "qemu-system-x86_64"
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {socketpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        qemuproc = Popen(cmd)
+
+        try:
+            qemu.accept()
+        except qmp.QMPConnectError:
+            die("Didn't get QMP greeting message")
+        except qmp.QMPCapabilitiesError:
+            die("Couldn't negotiate capabilities")
+        except OSError as err:
+            die(f"Couldn't connect to {sockpath}: {err!s}")
+
+        for _ in qemu.repl():
+            pass
 
 if __name__ == '__main__':
     main()
scripts/qmp/qmp-shell-wrap new file +11 -1
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
--
Quoted 2 lines

2.33.1

Adds some new failures to the python linters; try "make check-dev" in the python sub-dir.

... Though, due to a bug in avocado, this helpfully doesn't actually show you the failure output right now ...

making this little edit should fix that, sorry for the inconvenience here.

python/avocado.cfg +1 -1
index c7722e7ecd..a460420059 100644
@@ -1,5 +1,5 @@
 [run]
-test_runner = runner
+test_runner = nrunner
 
 
#3 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 65 lines · John Snow
Quoted 48 lines · Daniel P. Berrangé

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

  # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

  # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
  # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
  python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
  scripts/qmp/qmp-shell-wrap   | 11 +++++++
  2 files changed, 68 insertions(+), 4 deletions(-)

create mode 100755 scripts/qmp/qmp-shell-wrap

scripts/qmp/qmp-shell-wrap new file +11 -1
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
--
Quoted 2 lines

2.33.1

Adds some new failures to the python linters; try "make check-dev" in the python sub-dir.

... Though, due to a bug in avocado, this helpfully doesn't actually show you the failure output right now ...

making this little edit should fix that, sorry for the inconvenience here.

python/avocado.cfg +0 -0
index c7722e7ecd..a460420059 100644

python/avocado.cfg line 2

Quoted 3 lines
python/avocado.cfg +1 -1
@@ -1,5 +1,5 @@
  [run]
-test_runner = runner
+test_runner = nrunner

Cc'ing Beraldo, Willian once told me the nrunner switch was scheduled for QEMU next release.

#4 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 61 lines · Philippe Mathieu-Daudé
Quoted 60 lines · John Snow
Quoted 48 lines · Daniel P. Berrangé

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

  # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

  # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
  # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
  python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
  scripts/qmp/qmp-shell-wrap   | 11 +++++++
  2 files changed, 68 insertions(+), 4 deletions(-)

create mode 100755 scripts/qmp/qmp-shell-wrap

scripts/qmp/qmp-shell-wrap new file +11 -1
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
--
Quoted 2 lines

2.33.1

Adds some new failures to the python linters; try "make check-dev" in the python sub-dir.

... Though, due to a bug in avocado, this helpfully doesn't actually show you the failure output right now ...

making this little edit should fix that, sorry for the inconvenience here.

Quoted 11 lines

python/avocado.cfg line 2

Quoted 8 lines
python/avocado.cfg +1 -1
index c7722e7ecd..a460420059 100644
@@ -1,5 +1,5 @@
  [run]
-test_runner = runner
+test_runner = nrunner

Cc'ing Beraldo, Willian once told me the nrunner switch was scheduled for QEMU next release.

Thanks Philippe, for the Cc.

We solved a few issues on the nrunner side recently and implemented a set of changes regarding the logging logic aimed to improve the debugging process.

Currently, our legacy runner ('runner') is in "dismantle mode," and its usage with the latest versions of Avocado is discouraged.

Because of the nrunner's new architecture and how Coverage works [1], we spotted an issue when using nrunner + Coverage [2]. We are also investigating a possible alternative [3] in parallel.

[1] - https://coverage.readthedocs.io/en/stable/subprocess.html
[2] - https://github.com/avocado-framework/avocado/issues/4765
[3] - https://gitlab.com/beraldoleal/qemu/-/commit/c7aeb34505aab4357421890575cc9ddd7e83d04f

--
Beraldo

#5 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

In reply to John Snow
Quoted 139 lines · John Snow
Quoted 138 lines · Daniel P. Berrangé

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
 python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
 scripts/qmp/qmp-shell-wrap   | 11 +++++++
 2 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/qmp/qmp_shell.py +57 -4
index e7d7eb18f1..12f7d28afc 100644
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -162,8 +163,10 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -404,8 +407,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -529,6 +534,54 @@ def main() -> None:
         for _ in qemu.repl():
             pass
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = "qemu-system-x86_64"
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {socketpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        qemuproc = Popen(cmd)
+
+        try:
+            qemu.accept()
+        except qmp.QMPConnectError:
+            die("Didn't get QMP greeting message")
+        except qmp.QMPCapabilitiesError:
+            die("Couldn't negotiate capabilities")
+        except OSError as err:
+            die(f"Couldn't connect to {sockpath}: {err!s}")
+
+        for _ in qemu.repl():
+            pass
 
 if __name__ == '__main__':
     main()
scripts/qmp/qmp-shell-wrap new file +9 -0
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
Quoted 8 lines
Quoted 5 lines
scripts/qmp/qmp-shell-wrap new file +2 -1
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
--

2.33.1

Adds some new failures to the python linters; try "make check-dev" in the python sub-dir.

It would be nice to just have this integrated into 'make check' so we don't need to remember to run a special command.

Quoted 2 lines

... Though, due to a bug in avocado, this helpfully doesn't actually show you the failure output right now ...

Urgh.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 150 lines · Daniel P. Berrangé
Quoted 139 lines · John Snow
Quoted 138 lines · Daniel P. Berrangé

With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.

---
 python/qemu/qmp/qmp_shell.py | 61 +++++++++++++++++++++++++++++++++---
 scripts/qmp/qmp-shell-wrap   | 11 +++++++
 2 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/qmp/qmp_shell.py +57 -4
index e7d7eb18f1..12f7d28afc 100644
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -162,8 +163,10 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -404,8 +407,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: qmp.SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -529,6 +534,54 @@ def main() -> None:
         for _ in qemu.repl():
             pass
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = "qemu-system-x86_64"
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {socketpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        qemuproc = Popen(cmd)
+
+        try:
+            qemu.accept()
+        except qmp.QMPConnectError:
+            die("Didn't get QMP greeting message")
+        except qmp.QMPCapabilitiesError:
+            die("Couldn't negotiate capabilities")
+        except OSError as err:
+            die(f"Couldn't connect to {sockpath}: {err!s}")
+
+        for _ in qemu.repl():
+            pass
 
 if __name__ == '__main__':
     main()
scripts/qmp/qmp-shell-wrap new file +9 -0
new file mode 100755
index 0000000000..9e94da114f
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
Quoted 8 lines
Quoted 5 lines
scripts/qmp/qmp-shell-wrap new file +2 -1
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
--

2.33.1

Adds some new failures to the python linters; try "make check-dev" in the python sub-dir.

Quoted 2 lines

It would be nice to just have this integrated into 'make check' so we don't need to remember to run a special command.

The CI will run it, but 'make check' doesn't. To add it to make check, I need to figure out how to insert a venv-building step into 'make check' such that the venv gets deposited into the build dir instead of the source dir.
I think I may also need yet another set of package dependencies that pin on precise dependencies for testing purposes to prevent random regressions during 'make check' when nobody has touched the Python code.

Overall, I felt like maybe it was more hassle than it was worth if I can just nudge people touching the python to run a 'make check-dev' every so often.

Patches welcome, etc. My overall strategy with the python tests so far has been:

(1) Keep python tests fully separate from the QEMU build system to allow them to be split out into new repositories easily.
(2) Use the pipenv test to lock the very oldest dependencies the code and tests support, using the very oldest python we support (3.6) This test is used as the gating test in GitLab CI, as it is very repeatable and the GitLab CI setup ensures I can always have the exact Python packages it requires available.
(3) Use the tox test to test against a wide variety of Python interpreters (3.6 through 3.10 inclusive) using the very latest python packages to detect regressions on cutting-edge environments
(4) Use the widest possible range of versions for dependent packages in setup.cfg such that QEMU packages are unlikely to cause versioning conflicts in environments that decide to integrate our code.

Overall, I test on 3.6 through 3.10, and against the "oldest" and "newest" dependencies. It's a good, wide matrix.

However, It's #4 there that runs me into trouble with tests that are guaranteed to pass -- the linters update all the time and cause new problems. I use pipenv to lock to specific versions, but that tool wants to run against Python 3.6 *explicitly*, so it isn't suitable for a generic purpose 'make check' because not everyone will have a Python 3.6 interpreter available. I need something kind of halfway between, where I can lock against specific versions but not against the Python interpreter version, and that's what could be used for a decent 'make check' test.

Of course, I don't want to maintain like 10 versions of a dependent packages list, either.

(I really, really wish pip had an --use-oldest flag for dependency resolution, but it doesn't.)

Quoted 6 lines
Quoted 3 lines

... Though, due to a bug in avocado, this helpfully doesn't actually show you the failure output right now ...

Urgh.

It regressed during winter vacation. I sent a small patch to enable the new runner [1], and I have been discussing with Beraldo how to re-enable the Coverage.py tests under the new Avocado.

[1] https://lists.gnu.org/archive/html/qemu-devel/2022-01/msg03614.html

#7 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 47 lines · John Snow
Quoted 2 lines · Daniel P. Berrangé

It would be nice to just have this integrated into 'make check' so we don't need to remember to run a special command.

The CI will run it, but 'make check' doesn't. To add it to make check, I need to figure out how to insert a venv-building step into 'make check' such that the venv gets deposited into the build dir instead of the source dir.
I think I may also need yet another set of package dependencies that pin on precise dependencies for testing purposes to prevent random regressions during 'make check' when nobody has touched the Python code.

Overall, I felt like maybe it was more hassle than it was worth if I can just nudge people touching the python to run a 'make check-dev' every so often.

Patches welcome, etc. My overall strategy with the python tests so far has been:

(1) Keep python tests fully separate from the QEMU build system to allow them to be split out into new repositories easily.
(2) Use the pipenv test to lock the very oldest dependencies the code and tests support, using the very oldest python we support (3.6) This test is used as the gating test in GitLab CI, as it is very repeatable and the GitLab CI setup ensures I can always have the exact Python packages it requires available.
(3) Use the tox test to test against a wide variety of Python interpreters (3.6 through 3.10 inclusive) using the very latest python packages to detect regressions on cutting-edge environments
(4) Use the widest possible range of versions for dependent packages in setup.cfg such that QEMU packages are unlikely to cause versioning conflicts in environments that decide to integrate our code.

Overall, I test on 3.6 through 3.10, and against the "oldest" and "newest" dependencies. It's a good, wide matrix.

However, It's #4 there that runs me into trouble with tests that are guaranteed to pass -- the linters update all the time and cause new problems. I use pipenv to lock to specific versions, but that tool wants to run against Python 3.6 *explicitly*, so it isn't suitable for a generic purpose 'make check' because not everyone will have a Python 3.6 interpreter available. I need something kind of halfway between, where I can lock against specific versions but not against the Python interpreter version, and that's what could be used for a decent 'make check' test.

Quoted 5 lines

Of course, I don't want to maintain like 10 versions of a dependent packages list, either.

(I really, really wish pip had an --use-oldest flag for dependency resolution, but it doesn't.)

Could we simply use a virtualenv for all QEMU testing tasks (packages consumed by QEMU tests), and only deal with installed Python packages for regular non-testing QEMU uses (things exposed via pyqemu that we want stable)?

#8 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 51 lines · Philippe Mathieu-Daudé
Quoted 50 lines · John Snow
Quoted 2 lines · Daniel P. Berrangé

It would be nice to just have this integrated into 'make check' so we don't need to remember to run a special command.

The CI will run it, but 'make check' doesn't. To add it to make check, I need to figure out how to insert a venv-building step into 'make check' such that the venv gets deposited into the build dir instead of the source dir.
I think I may also need yet another set of package dependencies that pin on precise dependencies for testing purposes to prevent random regressions during 'make check' when nobody has touched the Python code.

Overall, I felt like maybe it was more hassle than it was worth if I can just nudge people touching the python to run a 'make check-dev' every so often.

Patches welcome, etc. My overall strategy with the python tests so far has been:

(1) Keep python tests fully separate from the QEMU build system to allow them to be split out into new repositories easily.
(2) Use the pipenv test to lock the very oldest dependencies the code and tests support, using the very oldest python we support (3.6) This test is used as the gating test in GitLab CI, as it is very repeatable and the GitLab CI setup ensures I can always have the exact Python packages it requires available.
(3) Use the tox test to test against a wide variety of Python interpreters (3.6 through 3.10 inclusive) using the very latest python packages to detect regressions on cutting-edge environments
(4) Use the widest possible range of versions for dependent packages in setup.cfg such that QEMU packages are unlikely to cause versioning conflicts in environments that decide to integrate our code.

Overall, I test on 3.6 through 3.10, and against the "oldest" and "newest" dependencies. It's a good, wide matrix.

However, It's #4 there that runs me into trouble with tests that are guaranteed to pass -- the linters update all the time and cause new problems. I use pipenv to lock to specific versions, but that tool wants to run against Python 3.6 *explicitly*, so it isn't suitable for a generic purpose 'make check' because not everyone will have a Python 3.6 interpreter available. I need something kind of halfway between, where I can lock against specific versions but not against the Python interpreter version, and that's what could be used for a decent 'make check' test.

Of course, I don't want to maintain like 10 versions of a dependent packages list, either.

Quoted 7 lines
Quoted 2 lines

(I really, really wish pip had an --use-oldest flag for dependency resolution, but it doesn't.)

Could we simply use a virtualenv for all QEMU testing tasks (packages consumed by QEMU tests), and only deal with installed Python packages for regular non-testing QEMU uses (things exposed via pyqemu that we want stable)?

I don't especially like the idea of using virtualenv. It is a good thing that we're testing on the distro python packages, because that is the scenario that our contributors/developers will actually use these tools in. We're got a well defined set of target platforms that QEMU aims for that we need to cover in testing. If we also want to do tests against general python using a virtualenv in CI pipelines thats fine, but doesn't replace the need to testing against our explicit platform targets, its just additive.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw

Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

Quoted 53 lines · Daniel P. Berrangé
Quoted 51 lines · Philippe Mathieu-Daudé
Quoted 50 lines · John Snow
Quoted 2 lines · Daniel P. Berrangé

It would be nice to just have this integrated into 'make check' so we don't need to remember to run a special command.

The CI will run it, but 'make check' doesn't. To add it to make check, I need to figure out how to insert a venv-building step into 'make check' such that the venv gets deposited into the build dir instead of the source dir.
I think I may also need yet another set of package dependencies that pin on precise dependencies for testing purposes to prevent random regressions during 'make check' when nobody has touched the Python code.

Overall, I felt like maybe it was more hassle than it was worth if I can just nudge people touching the python to run a 'make check-dev' every so often.

Patches welcome, etc. My overall strategy with the python tests so far has been:

(1) Keep python tests fully separate from the QEMU build system to allow them to be split out into new repositories easily.
(2) Use the pipenv test to lock the very oldest dependencies the code and tests support, using the very oldest python we support (3.6) This test is used as the gating test in GitLab CI, as it is very repeatable and the GitLab CI setup ensures I can always have the exact Python packages it requires available.
(3) Use the tox test to test against a wide variety of Python interpreters (3.6 through 3.10 inclusive) using the very latest python packages to detect regressions on cutting-edge environments
(4) Use the widest possible range of versions for dependent packages in setup.cfg such that QEMU packages are unlikely to cause versioning conflicts in environments that decide to integrate our code.

Overall, I test on 3.6 through 3.10, and against the "oldest" and "newest" dependencies. It's a good, wide matrix.

However, It's #4 there that runs me into trouble with tests that are guaranteed to pass -- the linters update all the time and cause new problems. I use pipenv to lock to specific versions, but that tool wants to run against Python 3.6 *explicitly*, so it isn't suitable for a generic purpose 'make check' because not everyone will have a Python 3.6 interpreter available. I need something kind of halfway between, where I can lock against specific versions but not against the Python interpreter version, and that's what could be used for a decent 'make check' test.

Of course, I don't want to maintain like 10 versions of a dependent packages list, either.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

(I really, really wish pip had an --use-oldest flag for dependency resolution, but it doesn't.)

Could we simply use a virtualenv for all QEMU testing tasks (packages consumed by QEMU tests), and only deal with installed Python packages for regular non-testing QEMU uses (things exposed via pyqemu that we want stable)?

There's no python packages we need except those that are for testing. Everything we need for the build is self-contained. "Python packages regular non-testing QEMU uses" is an empty set. We do ship *one* self-contained Python script, qemu-trace-stap. That lives in scripts/qemu-trace-stap and we don't need to mess with it right now.

Now, in the process of setting up the python/ directory, Cleber and I had discussed how to handle virtual environments for that directory -- as it has its own requirements, its own goals, etc. We decided at the time it would be best not to mix the dependencies of the test system with the dependencies of the burgeoning QEMU library. Let me elaborate on why we believe that to be the right decision:

In summary, the Python in use in the QEMU tree can be found in these categories:

- Used during the build itself, Not shipped with QEMU. Zero external dependencies.
- Used for testing, Not shipped with QEMU. Avocado tests use a scant few external dependencies.
- Not used for building nor testing, Shipped with QEMU. (This is to my knowledge a single script with no dependencies.)

Now, there are two kinds of tests that involves python code:

- Tests that are used for testing QEMU that incidentally use or are written in Python
- Tests that are used for testing the Python code itself

The first category of test there makes sense to be tied to a QEMU build. That's what Avocado tests are, and they use their own virtual environment for the purpose.
The second category of test does not make sense to be tied to a QEMU build, because it does not test QEMU. This is what the python/ CI tests behave like.

The different in purpose and motivation between those two categories of tests is why where are presently two different virtual environment regimes in the tree, and it's very likely that making any attempt to combine them is a very bad idea.
For example, /python/qemu/aqmp/ is on its way to being forked out as its own standalone library. This library needs its own set of requirements for being installed, tested, etc. tests/avocado/, on the other hand, is an application that will consume the QMP library, and it will likely have its own requirements.

Put another way: mixing the requirements of library and application together simply because they're in the same git tree at the moment is not good.

Quoted 11 lines

I don't especially like the idea of using virtualenv. It is a good thing that we're testing on the distro python packages, because that is the scenario that our contributors/developers will actually use these tools in. We're got a well defined set of target platforms that QEMU aims for that we need to cover in testing. If we also want to do tests against general python using a virtualenv in CI pipelines thats fine, but doesn't replace the need to testing against our explicit platform targets, its just additive.

I don't especially like the idea of using virtualenv.

Well, we already are using them in multiple places. tests/requirements.txt is used to instantiate a per-build VENV that is used to run the Avocado tests, and the python/ tests use several different ones in order to achieve a wide testing matrix for testing itself.

Quoted 1 line

It is a good thing that we're testing on the distro python packages

I mean, virtual environments don't download new copies of Python. No matter what, we're testing against "distro python". All the virtual environment does is allow us to pull very specific python package dependencies. For example, the avocado tests mandate that we use version 88.1 of avocado-framework. I assume most distributions don't even *have* avocado bundled in their repo. For sure, absolutely nobody will have packaged "qemu.qmp" by the time I need to start using it. Python testing packages (mypy, pylint, avocado, et al) move so fast that trying to maintain compatibility with every last one that ships as part of a distro repository is impossible (I've tried).

Most of all, though, I don't see the benefit in trying to maintain compatibility with a random ancient version of 'pylint' that someone might have as part of their platform. I see no reason why downloading a more modern pylint into a cached folder somewhere hurts our ability to test QEMU against our supported build platforms ... especially if said package-grab indeed works on all of those build platforms.

Quoted 1 line

because that is the scenario that our contributors/developers will actually use these tools in

Not if the virtual environment is part of the test invocation script... like it already is. Then everyone is using a very tightly controlled version of the testing environment with the exact correct dependencies. Mind you, I am drawing a distinction between build scripts, test scripts, and tools. The argument I see you making here is something that I believe applies to libraries and tools. That's indeed exactly why the testing regime for python/ is completely separate and fairly different from the QEMU build process, and covers a wide spectrum of dependent versions precisely to guarantee maximum compatibility.

Quoted 1 line

We're got a well defined set of target platforms that QEMU aims for that we need to cover in testing.

And we do! If we use a VENV to pull in some pure python code, who cares? The point is that QEMU is getting built and tested on those platforms. It matters extremely little if the test environment uses tighter bounds for thinks like a QMP library. This is barely different from initializing git submodules at the start of a build. I really feel like you are being artificially harsh here.

Quoted 2 lines

If we also want to do tests against general python using a virtualenv in CI pipelines thats fine, but doesn't replace the need to testing against our explicit platform targets, its just additive.

I don't understand this paragraph, unless you believe that virtual environments also replace the Python interpreter. Any use of a venv to run python tests is still gonna run those tests on these target platforms you wanna make sure work. I don't understand your apprehension.

--js

#10 lore raw

PATCH 2/2 python: support recording QMP session to a file

In reply to Daniel P. Berrangé

When running QMP commands with very large response payloads, it is often not easy to spot the info you want. If we can save the response to a file then tools like 'grep' or 'jq' can be used to extract information.

For convenience of processing, we merge the QMP command and response dictionaries together:

  {
      "arguments": {},
      "execute": "query-kvm",
      "return": {
          "enabled": false,
          "present": true
      }
  }

Example usage

  $ ./scripts/qmp/qmp-shell-wrap -l q.log -p -- ./build/qemu-system-x86_64 -display none
  Welcome to the QMP low-level shell!
  Connected
  (QEMU) query-kvm
  {
      "return": {
          "enabled": false,
          "present": true
      }
  }
  (QEMU) query-mice
  {
      "return": [
          {
              "absolute": false,
              "current": true,
              "index": 2,
              "name": "QEMU PS/2 Mouse"
          }
      ]
  }
 $ jq --slurp '. | to_entries[] | select(.value.execute == "query-kvm") |
               .value.return.enabled' < q.log
   false
---
 python/qemu/qmp/qmp_shell.py | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
python/qemu/qmp/qmp_shell.py +20 -7
index 12f7d28afc..a50c3f1bc7 100644
@@ -165,7 +165,8 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     def __init__(self, address: qmp.SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
+                 server: bool = False,
+                 logfile: str = None):
         super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
@@ -175,6 +176,10 @@ def __init__(self, address: qmp.SocketAddrT,
                                       '.qmp-shell_history')
         self.pretty = pretty
         self.verbose = verbose
+        self.logfile = None
+
+        if logfile is not None:
+            self.logfile = open(logfile, "w")
 
     def close(self) -> None:
         # Hook into context manager of parent to save shell history.
@@ -315,11 +320,11 @@ def _build_cmd(self, cmdline: str) -> Optional[QMPMessage]:
         self._cli_expr(cmdargs[1:], qmpcmd['arguments'])
         return qmpcmd
 
-    def _print(self, qmp_message: object) -> None:
+    def _print(self, qmp_message: object, fh=sys.stdout) -> None:
         jsobj = json.dumps(qmp_message,
                            indent=4 if self.pretty else None,
                            sort_keys=self.pretty)
-        print(str(jsobj))
+        print(str(jsobj), file=fh)
 
     def _execute_cmd(self, cmdline: str) -> bool:
         try:
@@ -342,6 +347,9 @@ def _execute_cmd(self, cmdline: str) -> bool:
             print('Disconnected')
             return False
         self._print(resp)
+        if self.logfile is not None:
+            cmd = {**qmpcmd, **resp}
+            self._print(cmd, fh=self.logfile)
         return True
 
     def connect(self, negotiate: bool = True) -> None:
@@ -409,8 +417,9 @@ class HMPShell(QMPShell):
     def __init__(self, address: qmp.SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
-        super().__init__(address, pretty, verbose, server)
+                 server: bool = False,
+                 logfile: str = None):
+        super().__init__(address, pretty, verbose, server, logfile)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -503,6 +512,8 @@ def main() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     default_server = os.environ.get('QMP_SOCKET')
     parser.add_argument('qmp_server', action='store',
@@ -521,7 +532,7 @@ def main() -> None:
         parser.error(f"Bad port number: {args.qmp_server}")
         return  # pycharm doesn't know error() is noreturn
 
-    with shell_class(address, args.pretty, args.verbose) as qemu:
+    with shell_class(address, args.pretty, args.verbose, args.logfile) as qemu:
         try:
             qemu.connect(negotiate=not args.skip_negotiation)
         except qmp.QMPConnectError:
@@ -545,6 +556,8 @@ def main_wrap() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     parser.add_argument('command', nargs=argparse.REMAINDER,
                         help='QEMU command line to invoke')
@@ -568,7 +581,7 @@ def main_wrap() -> None:
         parser.error(f"Bad port number: {socketpath}")
         return  # pycharm doesn't know error() is noreturn
 
-    with shell_class(address, args.pretty, args.verbose, True) as qemu:
+    with shell_class(address, args.pretty, args.verbose, True, args.logfile) as qemu:
         qemuproc = Popen(cmd)
 
         try:
Signature · 1 line
2.33.1
Reviewed-by Philippe Mathieu-Daudé on v1 2/2
#11 lore raw

Re: [PATCH 2/2] python: support recording QMP session to a file

Quoted 8 lines · Daniel P. Berrangé

When running QMP commands with very large response payloads, it is often not easy to spot the info you want. If we can save the response to a file then tools like 'grep' or 'jq' can be used to extract information.

---
 python/qemu/qmp/qmp_shell.py | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)