python: a few improvements to qmp-shell
v3 cover python: a few improvements to qmp-shell
This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.
First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.
Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.
In v3:
- Add qmp-shell-wrap to setup.cfg entry points
In v2:
- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warnings
Daniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file
python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++--- python/setup.cfg | 4 ++ scripts/qmp/qmp-shell-wrap | 11 +++++ 3 files changed, 96 insertions(+), 7 deletions(-) create mode 100755 scripts/qmp/qmp-shell-wrap
Signature · 1 line
2.34.1
PATCH v3 1/2 python: introduce qmp-shell-wrap convenience tool
With the current 'qmp-shell' tool developers must first spawn QEMU with a suitable -qmp arg and then spawn qmp-shell in a separate terminal pointing to the right socket.
With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and just pass the QEMU command and arguments they want. The program will listen on a UNIX socket and tell QEMU to connect QMP to that.
For example, this:
# qmp-shell-wrap -- qemu-system-x86_64 -display none
Is roughly equivalent of running:
# qemu-system-x86_64 -display none -qmp qmp-shell-1234 & # qmp-shell qmp-shell-1234
Except that 'qmp-shell-wrap' switches the socket peers around so that it is the UNIX socket server and QEMU is the socket client. This makes QEMU reliably go away when qmp-shell-wrap exits, closing the server socket.
--- python/qemu/aqmp/qmp_shell.py | 67 ++++++++++++++++++++++++++++++++--- python/setup.cfg | 1 + scripts/qmp/qmp-shell-wrap | 11 ++++++ 3 files changed, 75 insertions(+), 4 deletions(-) create mode 100755 scripts/qmp/qmp-shell-wrap
python/qemu/aqmp/qmp_shell.py +63 -4
@@ -86,6 +86,7 @@ import os import re import readline +from subprocess import Popen import sys from typing import ( Iterator,
@@ -167,8 +168,10 @@ class QMPShell(QEMUMonitorProtocol): :param verbose: Echo outgoing QMP messages to console. """ def __init__(self, address: SocketAddrT, - pretty: bool = False, verbose: bool = False): - super().__init__(address) + pretty: bool = False, + verbose: bool = False, + server: bool = False): + super().__init__(address, server=server) self._greeting: Optional[QMPMessage] = None self._completer = QMPCompleter() self._transmode = False
@@ -409,8 +412,10 @@ class HMPShell(QMPShell): :param verbose: Echo outgoing QMP messages to console. """ def __init__(self, address: SocketAddrT, - pretty: bool = False, verbose: bool = False): - super().__init__(address, pretty, verbose) + pretty: bool = False, + verbose: bool = False, + server: bool = False): + super().__init__(address, pretty, verbose, server) self._cpu_index = 0 def _cmd_completion(self) -> None:
@@ -533,5 +538,59 @@ def main() -> None: pass +def main_wrap() -> None: + """ + qmp-shell-wrap entry point: parse command line arguments and + start the REPL. + """ + parser = argparse.ArgumentParser() + parser.add_argument('-H', '--hmp', action='store_true', + help='Use HMP interface') + parser.add_argument('-v', '--verbose', action='store_true', + help='Verbose (echo commands sent and received)') + parser.add_argument('-p', '--pretty', action='store_true', + help='Pretty-print JSON') + + parser.add_argument('command', nargs=argparse.REMAINDER, + help='QEMU command line to invoke') + + args = parser.parse_args() + + cmd = args.command + if len(cmd) != 0 and cmd[0] == '--': + cmd = cmd[1:] + if len(cmd) == 0: + cmd = ["qemu-system-x86_64"] + + sockpath = "qmp-shell-wrap-%d" % os.getpid() + cmd += ["-qmp", "unix:%s" % sockpath] + + shell_class = HMPShell if args.hmp else QMPShell + + try: + address = shell_class.parse_address(sockpath) + except qmp.QMPBadPortError: + parser.error(f"Bad port number: {sockpath}") + return # pycharm doesn't know error() is noreturn + + try: + with shell_class(address, args.pretty, args.verbose, True) as qemu: + with Popen(cmd): + + try: + qemu.accept() + except qmp.QMPConnectError: + die("Didn't get QMP greeting message") + except qmp.QMPCapabilitiesError: + die("Couldn't negotiate capabilities") + except OSError as err: + die(f"Couldn't connect to {sockpath}: {err!s}") + + for _ in qemu.repl(): + pass + finally: + os.unlink(sockpath) + + if __name__ == '__main__': main()
python/setup.cfg +1 -0
@@ -68,6 +68,7 @@ console_scripts = qom-fuse = qemu.utils.qom_fuse:QOMFuse.entry_point [fuse] qemu-ga-client = qemu.utils.qemu_ga_client:main qmp-shell = qemu.aqmp.qmp_shell:main + qmp-shell-wrap = qemu.aqmp.qmp_shell:main_wrap aqmp-tui = qemu.aqmp.aqmp_tui:main [tui] [flake8]
scripts/qmp/qmp-shell-wrap new file +11 -0
@@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +import os +import sys + +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) +from qemu.qmp import qmp_shell + + +if __name__ == '__main__': + qmp_shell.main_wrap()
Signature · 1 line
2.34.1
PATCH v3 2/2 python: support recording QMP session to a file
In reply to Daniel P. BerrangéWhen running QMP commands with very large response payloads, it is often not easy to spot the info you want. If we can save the response to a file then tools like 'grep' or 'jq' can be used to extract information.
For convenience of processing, we merge the QMP command and response dictionaries together:
{
"arguments": {},
"execute": "query-kvm",
"return": {
"enabled": false,
"present": true
}
}Example usage
$ ./scripts/qmp/qmp-shell-wrap -l q.log -p -- ./build/qemu-system-x86_64 -display none
Welcome to the QMP low-level shell!
Connected
(QEMU) query-kvm
{
"return": {
"enabled": false,
"present": true
}
}
(QEMU) query-mice
{
"return": [
{
"absolute": false,
"current": true,
"index": 2,
"name": "QEMU PS/2 Mouse"
}
]
} $ jq --slurp '. | to_entries[] | select(.value.execute == "query-kvm") |
.value.return.enabled' < q.log
false--- python/qemu/aqmp/qmp_shell.py | 29 ++++++++++++++++++++++------- python/setup.cfg | 3 +++ 2 files changed, 25 insertions(+), 7 deletions(-)
python/qemu/aqmp/qmp_shell.py +22 -7
@@ -89,6 +89,7 @@ from subprocess import Popen import sys from typing import ( + IO, Iterator, List, NoReturn,
@@ -170,7 +171,8 @@ class QMPShell(QEMUMonitorProtocol): def __init__(self, address: SocketAddrT, pretty: bool = False, verbose: bool = False, - server: bool = False): + server: bool = False, + logfile: Optional[str] = None): super().__init__(address, server=server) self._greeting: Optional[QMPMessage] = None self._completer = QMPCompleter()
@@ -180,6 +182,10 @@ def __init__(self, address: SocketAddrT, '.qmp-shell_history') self.pretty = pretty self.verbose = verbose + self.logfile = None + + if logfile is not None: + self.logfile = open(logfile, "w", encoding='utf-8') def close(self) -> None: # Hook into context manager of parent to save shell history.
@@ -320,11 +326,11 @@ def _build_cmd(self, cmdline: str) -> Optional[QMPMessage]: self._cli_expr(cmdargs[1:], qmpcmd['arguments']) return qmpcmd - def _print(self, qmp_message: object) -> None: + def _print(self, qmp_message: object, fh: IO[str] = sys.stdout) -> None: jsobj = json.dumps(qmp_message, indent=4 if self.pretty else None, sort_keys=self.pretty) - print(str(jsobj)) + print(str(jsobj), file=fh) def _execute_cmd(self, cmdline: str) -> bool: try:
@@ -347,6 +353,9 @@ def _execute_cmd(self, cmdline: str) -> bool: print('Disconnected') return False self._print(resp) + if self.logfile is not None: + cmd = {**qmpcmd, **resp} + self._print(cmd, fh=self.logfile) return True def connect(self, negotiate: bool = True) -> None:
@@ -414,8 +423,9 @@ class HMPShell(QMPShell): def __init__(self, address: SocketAddrT, pretty: bool = False, verbose: bool = False, - server: bool = False): - super().__init__(address, pretty, verbose, server) + server: bool = False, + logfile: Optional[str] = None): + super().__init__(address, pretty, verbose, server, logfile) self._cpu_index = 0 def _cmd_completion(self) -> None:
@@ -508,6 +518,8 @@ def main() -> None: help='Verbose (echo commands sent and received)') parser.add_argument('-p', '--pretty', action='store_true', help='Pretty-print JSON') + parser.add_argument('-l', '--logfile', + help='Save log of all QMP messages to PATH') default_server = os.environ.get('QMP_SOCKET') parser.add_argument('qmp_server', action='store',
@@ -526,7 +538,7 @@ def main() -> None: parser.error(f"Bad port number: {args.qmp_server}") return # pycharm doesn't know error() is noreturn - with shell_class(address, args.pretty, args.verbose) as qemu: + with shell_class(address, args.pretty, args.verbose, args.logfile) as qemu: try: qemu.connect(negotiate=not args.skip_negotiation) except ConnectError as err:
@@ -550,6 +562,8 @@ def main_wrap() -> None: help='Verbose (echo commands sent and received)') parser.add_argument('-p', '--pretty', action='store_true', help='Pretty-print JSON') + parser.add_argument('-l', '--logfile', + help='Save log of all QMP messages to PATH') parser.add_argument('command', nargs=argparse.REMAINDER, help='QEMU command line to invoke')
@@ -574,7 +588,8 @@ def main_wrap() -> None: return # pycharm doesn't know error() is noreturn try: - with shell_class(address, args.pretty, args.verbose, True) as qemu: + with shell_class(address, args.pretty, args.verbose, + True, args.logfile) as qemu: with Popen(cmd): try:
python/setup.cfg +3 -0
@@ -114,7 +114,10 @@ ignore_missing_imports = True # no Warning level messages displayed, use "--disable=all --enable=classes # --disable=W". disable=consider-using-f-string, + consider-using-with, + too-many-arguments, too-many-function-args, # mypy handles this with less false positives. + too-many-instance-attributes, no-member, # mypy also handles this better. [pylint.basic]
Signature · 1 line
2.34.1
Quoted 27 lines · Daniel P. Berrangé
This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.
First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.
Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.
In v3:
- Add qmp-shell-wrap to setup.cfg entry points
In v2:
- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warningsDaniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a file
Quoted 10 lines
python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++--- python/setup.cfg | 4 ++ scripts/qmp/qmp-shell-wrap | 11 +++++ 3 files changed, 96 insertions(+), 7 deletions(-) create mode 100755 scripts/qmp/qmp-shell-wrap--
2.34.1
Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.
--js
(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3 setup.py install' does indeed cause issues here. I have a patch I'll send soon that discourages the direct setup.py invocation to avoid frustration in the future.)
Quoted 39 lines · John Snow
Quoted 27 lines · Daniel P. Berrangé
This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.
First it introduces a 'qmp-shell-wrap' command that takes a QEMU command line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.
Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.
In v3:
- Add qmp-shell-wrap to setup.cfg entry points
In v2:
- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warningsDaniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a fileQuoted 10 lines
python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++--- python/setup.cfg | 4 ++ scripts/qmp/qmp-shell-wrap | 11 +++++ 3 files changed, 96 insertions(+), 7 deletions(-) create mode 100755 scripts/qmp/qmp-shell-wrap--
2.34.1
Quoted 8 lines
Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.
--js
(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3
setup.pyinstall' does indeed cause issues here. I have a patch I'll send soon that discourages the directsetup.pyinvocation to avoid frustration in the future.)
I've only ever used pip to install from pypi or remote git archives. How do you use it to install from your local git checkout
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 47 lines · Daniel P. Berrangé
Quoted 1 line · John Snow
On Fri, Jan 28, 2022 at 11:12 AM Daniel P. Berrangé <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
This makes the qmp-shell program a little more pleasant to use when you are just trying to spawn a throw-away QEMU process to query some info from.
First it introduces a 'qmp-shell-wrap' command that takes a QEMU
command
Quoted 27 lines
Quoted 27 lines
line instead of QMP socket, and spawns QEMU automatically, so its life is tied to that of the shell.
Second it adds ability to log QMP commands/responses to a file that can be queried with 'jq' to extract information. This is good for commands which return huge JSON docs.
In v3:
- Add qmp-shell-wrap to setup.cfg entry points
In v2:
- Unlink unix socket path on exit
- Fix default command name
- Deal with flake8/pylint warningsDaniel P. Berrangé (2):
python: introduce qmp-shell-wrap convenience tool
python: support recording QMP session to a filepython/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++--- python/setup.cfg | 4 ++ scripts/qmp/qmp-shell-wrap | 11 +++++ 3 files changed, 96 insertions(+), 7 deletions(-) create mode 100755 scripts/qmp/qmp-shell-wrapQuoted 6 lines
Quoted 4 lines
--
2.34.1Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged
these.
Quoted 3 lines
--js
Quoted 8 lines
Quoted 4 lines
(fwiw: using pip, it seems like the wrapper script works just fine. it appears as though using 'python3
setup.pyinstall' does indeed cause issues here. I have a patch I'll send soon that discourages the directsetup.pyinvocation to avoid frustration in the future.)I've only ever used pip to install from pypi or remote git archives. How do you use it to install from your local git checkout
cd ~/src/qemu/python
pip install [-e] [--user] .
See python/README.rst: https://gitlab.com/jsnow/qemu/-/tree/master/python
I've also sent a patch that disables the broken method (and explains why it's broken), see
https://patchew.org/QEMU/[email protected]/
thanks!
--js
Quoted 11 lines
Regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|