Skip to contentSkip to conversation

python 3.5 compatibility

PATCH In review Enrico Weigelt, metux IT consult 7 participants 9 messages Nov 27, 2020 – Dec 1, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 python 3.5 compatibility

Several stable distros are still on python-3.5, so qemu cannot be built there, w/o backporting the whole python stack. In order to make qemu build there, make it running w/ python-3.5. The changes are mostly removing the new type annotations (which, IMHO, just serve for documentation purpose) and replacing f-strings by classic string formatting.

---
 configure                          |  4 +--
 scripts/block-coroutine-wrapper.py | 56 ++++++++++++++++++--------------------
 scripts/qapi/commands.py           |  2 +-
 scripts/qapi/common.py             |  2 +-
 scripts/qapi/events.py             |  2 +-
 scripts/qapi/gen.py                | 10 +++----
 scripts/qapi/main.py               |  6 ++--
 scripts/qapi/schema.py             |  2 +-
 scripts/qapi/source.py             | 10 +++----
 9 files changed, 46 insertions(+), 48 deletions(-)
configure +2 -2
index 18c26e0389..e1119929fa 100755
@@ -1866,8 +1866,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
-  error_exit "Cannot use '$python', Python >= 3.6 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
+  error_exit "Cannot use '$python', Python >= 3.5 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
scripts/block-coroutine-wrapper.py +27 -29

96 lines not on this page · raw message

scripts/qapi/commands.py +1 -1
index 50978090b4..3b22a16759 100644
@@ -256,7 +256,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
             prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', None, __doc__)
         self._regy = QAPIGenCCode(None)
-        self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
+        self._visited_ret_types = {}
 
     def _begin_user_module(self, name: str) -> None:
         self._visited_ret_types[self._genc] = set()
scripts/qapi/common.py +1 -1
index 11b86beeab..d107d09d8e 100644
@@ -147,7 +147,7 @@ class Indentation:
         """Decrease the indentation level by ``amount``, default 4."""
         if self._level < amount:
             raise ArithmeticError(
-                f"Can't remove {amount:d} spaces from {self!r}")
+                "Can't remove %d spaces from %s" % (amount, self.r))
         self._level -= amount
 
 
scripts/qapi/events.py +1 -1
index 599f3d1f56..b4e6438b9d 100644
@@ -162,7 +162,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members: List[QAPISchemaEnumMember] = []
+        self._event_enum_members = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
     def _begin_user_module(self, name: str) -> None:
scripts/qapi/gen.py +5 -5
index b40f18eee3..666f50478c 100644
@@ -123,7 +123,7 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 class QAPIGenCCode(QAPIGen):
     def __init__(self, fname: Optional[str]):
         super().__init__(fname)
-        self._start_if: Optional[Tuple[List[str], str, str]] = None
+        self._start_if = None
 
     def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
@@ -243,10 +243,10 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc: Optional[QAPIGenC] = None
-        self._genh: Optional[QAPIGenH] = None
-        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
-        self._main_module: Optional[str] = None
+        self._genc = None
+        self._genh = None
+        self._module = {}
+        self._main_module = None
 
     @staticmethod
     def _is_user_module(name: Optional[str]) -> bool:
scripts/qapi/main.py +3 -3
index 42517210b8..2636227915 100644
@@ -79,8 +79,8 @@ def main() -> int:
 
     funny_char = invalid_prefix_char(args.prefix)
     if funny_char:
-        msg = f"funny character '{funny_char}' in argument of --prefix"
-        print(f"{sys.argv[0]}: {msg}", file=sys.stderr)
+        msg = ("funny character '%s' in argument of --prefix" % funny_char)
+        print("%s: %s" % (sys.argv[0], msg), file=sys.stderr)
         return 1
 
     try:
@@ -90,6 +90,6 @@ def main() -> int:
                  unmask=args.unmask,
                  builtins=args.builtins)
     except QAPIError as err:
-        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        print("%s: %s" % (sys.argv[0], str(err)), file=sys.stderr)
         return 1
     return 0
scripts/qapi/schema.py +1 -1
index 720449feee..027a517e47 100644
@@ -26,7 +26,7 @@ from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
-    meta: Optional[str] = None
+    meta = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
scripts/qapi/source.py +5 -5
index d7a79a9b8a..b7ff61b230 100644
@@ -22,9 +22,9 @@ class QAPISchemaPragma:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist: List[str] = []
+        self.returns_whitelist = []
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist: List[str] = []
+        self.name_case_whitelist = []
 
 
 class QAPISourceInfo:
@@ -35,11 +35,11 @@ class QAPISourceInfo:
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma: QAPISchemaPragma = (
+        self.pragma = (
             parent.pragma if parent else QAPISchemaPragma()
         )
-        self.defn_meta: Optional[str] = None
-        self.defn_name: Optional[str] = None
+        self.defn_meta = None
+        self.defn_name = None
 
     def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
Signature · 1 line
2.11.0
#1 lore raw
Quoted 6 lines · metux IT consult

Several stable distros are still on python-3.5, so qemu cannot be built there, w/o backporting the whole python stack. In order to make qemu build there, make it running w/ python-3.5. The changes are mostly removing the new type annotations (which, IMHO, just serve for documentation purpose) and replacing f-strings by classic string formatting.

We dropped Python 3.5 support deliberately in commit 1b11f28d05121fec43 on the basis that all our supported build platforms are on at least 3.6 and there were features we wanted there.

Could you say which "stable distros" you have in mind, and whether they are covered by our "supported build platforms" policy
https://www.qemu.org/docs/master/system/build-platforms.html ?

thanks
-- PMM

#2 lore raw

On 27.11.20 20:15, Peter Maydell wrote:

Hi,

Quoted 3 lines

Could you say which "stable distros" you have in mind, and whether they are covered by our "supported build platforms" policy
https://www.qemu.org/docs/master/system/build-platforms.html ?

I'm running on Devuan Ascii.

And packaging python-3.6 just for a few pieces of one application (qemu) is far too much work.

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#3 lore raw
Quoted 9 lines · metux IT consult

On 27.11.20 20:15, Peter Maydell wrote:

Hi,

Quoted 3 lines

Could you say which "stable distros" you have in mind, and whether they are covered by our "supported build platforms" policy
https://www.qemu.org/docs/master/system/build-platforms.html ?

I'm running on Devuan Ascii.

Which has oldstable status. Good for running the old and stable software packaged by it (such as QEMU 2.8), and old (and hopefully stable) software of similar vintage.

Devian ASCII appears to be a derivative of Debian 9 Stretch, which we no longer support, because it's EOL. docs/system/build-platforms.rst:

For distributions with long-lifetime releases, the project will aim to support the most recent major version at all times. Support for the previous major version will be dropped 2 years after the new major version is released, or when it reaches "end of life". For the purposes of identifying supported software versions, the project will look at RHEL, Debian, Ubuntu LTS, and SLES distros. Other long-lifetime distros will be assumed to ship similar software versions.

This policy balances cost and benefit of keeping bleeding-edge QEMU work on a wide variety of hosts. Years of balancing.

Quoted 2 lines

And packaging python-3.6 just for a few pieces of one application (qemu) is far too much work.

Have you considered upgrading to stable?

#4 lore raw
Quoted 3 lines · Markus Armbruster

Which has oldstable status. Good for running the old and stable software packaged by it (such as QEMU 2.8), and old (and hopefully stable) software of similar vintage.

It's still heavily used out in the field, and officially supported. But that's just one example.

Perhaps I should add some more details on the situation: I'm using (specially built) qemu with ptxdist/dkit - not the distro package. The idea w/ ptxdist is that you just pull the trigger and it builds everything needed for some project. Qemu is built specifically for the configured target. Perhaps you've noticed I'm also doing development of new qemu features - something that one doesn't want to do on old versions.

Quoted 1 line

Have you considered upgrading to stable?

This would solve the problem just for me alone, not for others out there, who're working w/ the BSP. And asking everybody (especially in enterprise environments) to do a full release upgrade just for one single tool (qemu) isn't someting that works easily.

If you insist in having python3.6 a hard requirement for qemu, you're putting me into the situation of having to do lots of backport work for quite long time (until everybody really did the upgrade). :(

This individual patch wasn't very complicated, but expect even more (and more tricky cases) on the horizon.

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#5 lore raw
Quoted 19 lines · metux IT consult
Quoted 3 lines · Markus Armbruster

Which has oldstable status. Good for running the old and stable software packaged by it (such as QEMU 2.8), and old (and hopefully stable) software of similar vintage.

It's still heavily used out in the field, and officially supported. But that's just one example.

Perhaps I should add some more details on the situation: I'm using (specially built) qemu with ptxdist/dkit - not the distro package. The idea w/ ptxdist is that you just pull the trigger and it builds everything needed for some project. Qemu is built specifically for the configured target. Perhaps you've noticed I'm also doing development of new qemu features - something that one doesn't want to do on old versions.

Quoted 1 line

Have you considered upgrading to stable?

Quoted 8 lines

This would solve the problem just for me alone, not for others out there, who're working w/ the BSP. And asking everybody (especially in enterprise environments) to do a full release upgrade just for one single tool (qemu) isn't someting that works easily.

If you insist in having python3.6 a hard requirement for qemu, you're putting me into the situation of having to do lots of backport work for quite long time (until everybody really did the upgrade). :(

python3.6 is just the one problem you've currently hit. In order to keep the burden of maintaining support for old software under control we have a well defined set of platforms that we target. When a particular version of a distro drops off our list, we will bump the minimum versions of any software we depend on. In this case you've hit python3.6 so far, but we are liable to bump other minimum versions too which will also impact this old distro. So I'm afraid it will be a loosing battle to stay on this old distro, while building cutting edge QEMU.

I appreciate this isn't the answer you want to hear, but we've defined our support platform matrix to try to balance multiple competing factors and unfortunately this means distros are going to periodically get dropped by new QEMU as they get older.

One possible way you can mitigate this is to make use of containers for your development and deployment. eg even tough you're using an old Devuan, you can use docker/podman to build and deploy QEMU inside a stable Devuan container.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw
In reply to Enrico Weigelt, metux IT consult

Hi

Quoted 4 lines · metux IT consult

On 27.11.20 20:15, Peter Maydell wrote:

Hi,

Quoted 9 lines
Quoted 3 lines

Could you say which "stable distros" you have in mind, and whether they are covered by our "supported build platforms" policy
https://www.qemu.org/docs/master/system/build-platforms.html ?

I'm running on Devuan Ascii.

And packaging python-3.6 just for a few pieces of one application (qemu) is far too much work.

It might be necessary to keep that distro alive, as python 3.5 is EOL:

https://www.python.org/downloads/release/python-3510/

(fwiw, upcoming meson version will also depend on >= 3.6)

Quoted 13 lines

--mtx

--
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287

Signature · 1 line
Marc-André Lureau
#7 lore raw
In reply to Enrico Weigelt, metux IT consult

Am 27.11.2020 um 19:36 hat Enrico Weigelt, metux IT consult geschrieben:

Quoted 5 lines

Several stable distros are still on python-3.5, so qemu cannot be built there, w/o backporting the whole python stack. In order to make qemu build there, make it running w/ python-3.5. The changes are mostly removing the new type annotations (which, IMHO, just serve for documentation purpose)

While type hints are valuable documentation, they are more than just that. They help to find and prevent avoidable bugs in the code. We are actively in the process of adding them to everything in the QAPI generator to improve maintainability rather than removing them.

Kevin

Quoted 15 lines

and replacing f-strings by classic
string formatting.

---
 configure                          |  4 +--
 scripts/block-coroutine-wrapper.py | 56 ++++++++++++++++++--------------------
 scripts/qapi/commands.py           |  2 +-
 scripts/qapi/common.py             |  2 +-
 scripts/qapi/events.py             |  2 +-
 scripts/qapi/gen.py                | 10 +++----
 scripts/qapi/main.py               |  6 ++--
 scripts/qapi/schema.py             |  2 +-
 scripts/qapi/source.py             | 10 +++----
 9 files changed, 46 insertions(+), 48 deletions(-)
#8 lore raw

On 30.11.20 10:44, Kevin Wolf wrote:

Hi,

Quoted 4 lines

While type hints are valuable documentation, they are more than just that. They help to find and prevent avoidable bugs in the code. We are actively in the process of adding them to everything in the QAPI generator to improve maintainability rather than removing them.

IOW: I'll have to do lots of backporting work, just to keep it running :(

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287