Skip to contentSkip to conversation

Introduce a Python module structure

PATCH In review Cleber Rosa 3 participants 4 messages Feb 6, 2019 – Feb 18, 2019

lore
Share & tools
lore
#0 lore raw

cover Introduce a Python module structure

The amount of Python code that is being reused by a now large number of different scripts and tests on QEMU urges for a better structure.

This addresses the feedback received on a previous RFC[1], but further changes that will really benefit from this change were not attempted here. Once, the module structure is present, I myself have plans to send some unittests to "QEMUMachine", to some parts of "qemu.qmp", and others.

Documentation, lint and style checkers are other possible candidates.

[1] - https://lists.gnu.org/archive/html/qemu-devel/2018-11/msg05478.html

---

Git Info:
- URI: https://github.com/clebergnu/qemu/tree/sent/python_module
- Remote: https://github.com/clebergnu/qemu
- Branch: sent/python_module

Travis CI Info:
- Job: https://travis-ci.org/clebergnu/qemu/builds/489616874

Cleber Rosa (1):
Introduce a Python module structure

 configure                                  |  1 +
 scripts/qemu.py => python/qemu/__init__.py | 11 ++++++-----
 {scripts/qmp => python/qemu}/qmp.py        |  0
 {scripts => python/qemu}/qtest.py          |  5 +++--
 scripts/device-crash-test                  |  2 ++
 scripts/qmp/__init__.py                    |  0
 scripts/qmp/qemu-ga-client                 |  5 ++++-
 scripts/qmp/qmp-shell                      |  4 +++-
 scripts/render_block_graph.py              |  2 ++
 tests/acceptance/avocado_qemu/__init__.py  |  5 ++---
 tests/acceptance/virtio_version.py         |  2 +-
 tests/migration/guestperf/engine.py        |  7 ++++---
 tests/qemu-iotests/235                     |  2 +-
 tests/qemu-iotests/238                     |  2 +-
 tests/qemu-iotests/iotests.py              |  4 ++--
 tests/vm/basevm.py                         |  2 +-
 16 files changed, 33 insertions(+), 21 deletions(-)
 rename scripts/qemu.py => python/qemu/__init__.py (98%)
 rename {scripts/qmp => python/qemu}/qmp.py (100%)
 rename {scripts => python/qemu}/qtest.py (98%)
 delete mode 100644 scripts/qmp/__init__.py
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 Introduce a Python module structure

This is a simple move of Python code that wraps common QEMU functionality, and are used by a number of different tests and scripts.

By treating that code as a real Python module, we can more easily:
* reuse code
* have a proper place for the module's own unittests
* apply a more consistent style
* generate documentation

---
 configure                                  |  1 +
 scripts/qemu.py => python/qemu/__init__.py | 11 ++++++-----
 {scripts/qmp => python/qemu}/qmp.py        |  0
 {scripts => python/qemu}/qtest.py          |  5 +++--
 scripts/device-crash-test                  |  2 ++
 scripts/qmp/__init__.py                    |  0
 scripts/qmp/qemu-ga-client                 |  5 ++++-
 scripts/qmp/qmp-shell                      |  4 +++-
 scripts/render_block_graph.py              |  2 ++
 tests/acceptance/avocado_qemu/__init__.py  |  5 ++---
 tests/acceptance/virtio_version.py         |  2 +-
 tests/migration/guestperf/engine.py        |  7 ++++---
 tests/qemu-iotests/235                     |  2 +-
 tests/qemu-iotests/238                     |  2 +-
 tests/qemu-iotests/iotests.py              |  4 ++--
 tests/vm/basevm.py                         |  2 +-
 16 files changed, 33 insertions(+), 21 deletions(-)
 rename scripts/qemu.py => python/qemu/__init__.py (98%)
 rename {scripts/qmp => python/qemu}/qmp.py (100%)
 rename {scripts => python/qemu}/qtest.py (98%)
 delete mode 100644 scripts/qmp/__init__.py
configure +1 -0
index 3d89870d99..c1479cf064 100755
@@ -7579,6 +7579,7 @@ LINKS="$LINKS pc-bios/qemu-icon.bmp"
 LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
 LINKS="$LINKS tests/acceptance tests/data"
 LINKS="$LINKS tests/qemu-iotests/check"
+LINKS="$LINKS python"
 for bios_file in \
     $source_path/pc-bios/*.bin \
     $source_path/pc-bios/*.lid \
python/qemu/__init__.py renamed from scripts/qemu.py +6 -5
similarity index 98%
rename from scripts/qemu.py
rename to python/qemu/__init__.py
index 0a5e02eb56..22dd605cfe 100644
@@ -16,12 +16,13 @@ import errno
 import logging
 import os
 import subprocess
-import qmp.qmp
 import re
 import shutil
 import socket
 import tempfile
 
+from . import qmp
+
 
 LOG = logging.getLogger(__name__)
 
@@ -58,7 +59,7 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     failures reported by the QEMU binary itself.
     """
 
-class MonitorResponseError(qmp.qmp.QMPError):
+class MonitorResponseError(qmp.QMPError):
     """
     Represents erroneous QMP monitor reply
     """
@@ -259,8 +260,8 @@ class QEMUMachine(object):
         self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log")
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
-        self._qmp = qmp.qmp.QEMUMonitorProtocol(self._vm_monitor,
-                                                server=True)
+        self._qmp = qmp.QEMUMonitorProtocol(self._vm_monitor,
+                                            server=True)
 
     def _post_launch(self):
         self._qmp.accept()
@@ -376,7 +377,7 @@ class QEMUMachine(object):
         """
         reply = self.qmp(cmd, conv_keys, **args)
         if reply is None:
-            raise qmp.qmp.QMPError("Monitor is closed")
+            raise qmp.QMPError("Monitor is closed")
         if "error" in reply:
             raise MonitorResponseError(reply)
         return reply["return"]
python/qemu/qmp.py renamed from scripts/qmp/qmp.py +0 -0
similarity index 100%
rename from scripts/qmp/qmp.py
rename to python/qemu/qmp.py
python/qemu/qtest.py renamed from scripts/qtest.py +3 -2
similarity index 98%
rename from scripts/qtest.py
rename to python/qemu/qtest.py
index afac3fe900..eb45824dd0 100644
@@ -13,7 +13,8 @@
 
 import socket
 import os
-import qemu
+
+from . import QEMUMachine
 
 
 class QEMUQtestProtocol(object):
@@ -79,7 +80,7 @@ class QEMUQtestProtocol(object):
         self._sock.settimeout(timeout)
 
 
-class QEMUQtestMachine(qemu.QEMUMachine):
+class QEMUQtestMachine(QEMUMachine):
     '''A QEMU VM'''
 
     def __init__(self, binary, args=None, name=None, test_dir="/var/tmp",
scripts/device-crash-test +2 -0
index 2a13fa4f84..a6748910ad 100755
@@ -25,6 +25,7 @@ check for crashes and unexpected errors.
 """
 from __future__ import print_function
 
+import os
 import sys
 import glob
 import logging
@@ -34,6 +35,7 @@ import random
 import argparse
 from itertools import chain
 
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
 from qemu import QEMUMachine
 
 logger = logging.getLogger('device-crash-test')
scripts/qmp/__init__.py deleted +0 -0
deleted file mode 100644
index e69de29bb2..0000000000
scripts/qmp/qemu-ga-client +4 -1
index e8cb7646a0..30cf8a9a0d 100755
@@ -37,10 +37,13 @@
 #
 
 from __future__ import print_function
+import os
+import sys
 import base64
 import random
 
-import qmp
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu import qmp
 
 
 class QemuGuestAgent(qmp.QEMUMonitorProtocol):
scripts/qmp/qmp-shell +3 -1
index 770140772d..9fec46e2ed 100755
@@ -66,7 +66,6 @@
 # sent to QEMU, which is useful for debugging and documentation generation.
 
 from __future__ import print_function
-import qmp
 import json
 import ast
 import readline
@@ -76,6 +75,9 @@ import errno
 import atexit
 import shlex
 
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu import qmp
+
 class QMPCompleter(list):
     def complete(self, text, state):
         for cmd in self:
scripts/render_block_graph.py +2 -0
index ed7e581b4f..3e9d282a49 100755
@@ -23,6 +23,8 @@ import sys
 import subprocess
 import json
 from graphviz import Digraph
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
 from qemu import MonitorResponseError
 
 
tests/acceptance/avocado_qemu/__init__.py +2 -3
index 1e54fd5932..28bfb8e9d3 100644
@@ -13,9 +13,8 @@ import sys
 
 import avocado
 
-SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
-SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
-sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
+SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
+sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
 
 from qemu import QEMUMachine
 
tests/acceptance/virtio_version.py +1 -1
index ce990250d8..072c50371e 100644
@@ -11,7 +11,7 @@ Check compatibility of virtio device types
 import sys
 import os
 
-sys.path.append(os.path.join(os.path.dirname(__file__), "..", "..", "scripts"))
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import QEMUMachine
 from avocado_qemu import Test
 
tests/migration/guestperf/engine.py +4 -3
index 398e3f2706..0e304660b8 100644
@@ -24,13 +24,14 @@ import re
 import sys
 import time
 
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', '..', 'scripts'))
-import qemu
-import qmp.qmp
 from guestperf.progress import Progress, ProgressStats
 from guestperf.report import Report
 from guestperf.timings import TimingRecord, Timings
 
+sys.path.append(os.path.join(os.path.dirname(__file__),
+                             '..', '..', '..', 'python'))
+import qemu
+
 
 class Engine(object):
 
tests/qemu-iotests/235 +1 -1
index d6edd97ab4..75c203b30c 100755
@@ -23,7 +23,7 @@ import os
 import iotests
 from iotests import qemu_img_create, qemu_io, file_path, log
 
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts'))
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 
 from qemu import QEMUMachine
 
tests/qemu-iotests/238 +1 -1
index f81ee1112f..688abc9acb 100755
@@ -23,7 +23,7 @@ import os
 import iotests
 from iotests import log
 
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts'))
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 
 from qemu import QEMUMachine
 
tests/qemu-iotests/iotests.py +2 -2
index b461f53abf..54a31044b4 100644
@@ -32,8 +32,8 @@ import atexit
 import io
 from collections import OrderedDict
 
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts'))
-import qtest
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu import qtest
 
 
 # This will not work if arguments contain spaces but is necessary if we
tests/vm/basevm.py +1 -1
index 5caf77d6b8..9fece647ef 100755
@@ -17,7 +17,7 @@ import sys
 import logging
 import time
 import datetime
-sys.path.append(os.path.join(os.path.dirname(__file__), "..", "..", "scripts"))
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import QEMUMachine, kvm_available
 import subprocess
 import hashlib
Signature · 1 line
2.20.1
Reviewed-by on v1 1/1
#2 lore raw
Quoted 25 lines · Cleber Rosa

This is a simple move of Python code that wraps common QEMU functionality, and are used by a number of different tests and scripts.

By treating that code as a real Python module, we can more easily:
* reuse code
* have a proper place for the module's own unittests
* apply a more consistent style
* generate documentation

---
 configure                                  |  1 +
 scripts/qemu.py => python/qemu/__init__.py | 11 ++++++-----
 {scripts/qmp => python/qemu}/qmp.py        |  0
 {scripts => python/qemu}/qtest.py          |  5 +++--
 scripts/device-crash-test                  |  2 ++
 scripts/qmp/__init__.py                    |  0
 scripts/qmp/qemu-ga-client                 |  5 ++++-
 scripts/qmp/qmp-shell                      |  4 +++-
 scripts/render_block_graph.py              |  2 ++
 tests/acceptance/avocado_qemu/__init__.py  |  5 ++---
 tests/acceptance/virtio_version.py         |  2 +-
 tests/migration/guestperf/engine.py        |  7 ++++---
 tests/qemu-iotests/235                     |  2 +-
Quoted 8 lines
 tests/qemu-iotests/238                     |  2 +-
 tests/qemu-iotests/iotests.py              |  4 ++--
 tests/vm/basevm.py                         |  2 +-
 16 files changed, 33 insertions(+), 21 deletions(-)
 rename scripts/qemu.py => python/qemu/__init__.py (98%)
 rename {scripts/qmp => python/qemu}/qmp.py (100%)
 rename {scripts => python/qemu}/qtest.py (98%)
 delete mode 100644 scripts/qmp/__init__.py
Reviewed-by on v1 1/1
#3 lore raw
In reply to Cleber Rosa
Quoted 11 lines · Cleber Rosa

This is a simple move of Python code that wraps common QEMU functionality, and are used by a number of different tests and scripts.

By treating that code as a real Python module, we can more easily:
* reuse code
* have a proper place for the module's own unittests
* apply a more consistent style
* generate documentation

Quoted 14 lines
---
 configure                                  |  1 +
 scripts/qemu.py => python/qemu/__init__.py | 11 ++++++-----
 {scripts/qmp => python/qemu}/qmp.py        |  0
 {scripts => python/qemu}/qtest.py          |  5 +++--
 scripts/device-crash-test                  |  2 ++
 scripts/qmp/__init__.py                    |  0
 scripts/qmp/qemu-ga-client                 |  5 ++++-
 scripts/qmp/qmp-shell                      |  4 +++-
 scripts/render_block_graph.py              |  2 ++
 tests/acceptance/avocado_qemu/__init__.py  |  5 ++---
 tests/acceptance/virtio_version.py         |  2 +-
 tests/migration/guestperf/engine.py        |  7 ++++---
 tests/qemu-iotests/235                     |  2 +-
Quoted 9 lines
 tests/qemu-iotests/238                     |  2 +-
 tests/qemu-iotests/iotests.py              |  4 ++--
 tests/vm/basevm.py                         |  2 +-
 16 files changed, 33 insertions(+), 21 deletions(-)
 rename scripts/qemu.py => python/qemu/__init__.py (98%)
 rename {scripts/qmp => python/qemu}/qmp.py (100%)
 rename {scripts => python/qemu}/qtest.py (98%)
 delete mode 100644 scripts/qmp/__init__.py

{...}

Quoted 1 line

Thanks,

Signature · 3 lines
Caio Carrara
Software Engineer, Virt Team - Red Hat
[email protected]