Skip to contentSkip to conversation

qapi: Fix code generation with Python 3.5

PATCH In review Markus Armbruster 8 participants 14 messages Jan 16, 2020 – Feb 7, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 qapi: Fix code generation with Python 3.5

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +1 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)
Signature · 1 line
2.21.1
#1 lore raw
Quoted 28 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -0
index 0bfc5256fb..5100110fa2 100644

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 8 lines
scripts/qapi/schema.py +1 -1
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

#2 lore raw
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
Quoted 8 lines

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 5 lines
scripts/qapi/schema.py +1 -0
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

No clue.

3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches

We support 3.5 because we support Debian 9.

We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.

#3 lore raw
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
Quoted 8 lines

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 5 lines
scripts/qapi/schema.py +1 -0
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

No clue.

Quoted 9 lines

3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches

We support 3.5 because we support Debian 9.

We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.

We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.

We should use a rule of "two years after successor, or End-of-Support, whichever comes first."

For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?

In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?

--js

#4 lore raw
Quoted 56 lines · John Snow
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
Quoted 8 lines

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 5 lines
scripts/qapi/schema.py +1 -0
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

No clue.

Quoted 9 lines

3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches

We support 3.5 because we support Debian 9.

We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.

Quoted 5 lines

We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.

We should use a rule of "two years after successor, or End-of-Support, whichever comes first."

Yes.

Quoted 2 lines

For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?

We need to define end-of-support for Debian: is it Debian proper or is it Debian LTS?

<https://wiki.debian.org/DebianOldStable>:

Q) How long will security updates be provided?

The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.

<https://wiki.debian.org/LTS>:

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.

Thus the Debian LTS team takes over security maintenance of the various releases once the Debian Security team stops its work.

Debian 10 "Buster" was released in July 2019. Debian 9 "Stretch" will receive security updates from Debian until mid 2020, i.e. just about when Python 3.5 reaches EOL. LTS will attempt to support it until June 2022.

I think we should give ourselves a bit more flexibility than the categorical "Support for the previous major version will be dropped 2 years after the new major version is released." At some point, the cost of supporting old hosts exceeds the utility. We should face this tradeoff.

Quoted 2 lines

In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?

June 2022 even.

We use Python at compile time with trusted input. The need for security maintenance is relatively low there. I'm not ready to vouch for "and we don't use Python for anything else".

#5 lore raw
Quoted 3 lines · Markus Armbruster
Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster

John Snow <[email protected]> writes:

[...]

Quoted 41 lines
Quoted 15 lines
Quoted 5 lines
Quoted 2 lines

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

No clue.

Quoted 9 lines

3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches

We support 3.5 because we support Debian 9.

We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.

Quoted 5 lines

We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.

We should use a rule of "two years after successor, or End-of-Support, whichever comes first."

Yes.

Quoted 2 lines

For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?

We need to define end-of-support for Debian: is it Debian proper or is it Debian LTS?

<https://wiki.debian.org/DebianOldStable>:

Q) How long will security updates be provided?

The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.

<https://wiki.debian.org/LTS>:

Quoted 7 lines

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.

Thus the Debian LTS team takes over security maintenance of the various releases once the Debian Security team stops its work.

As Debian LTS is maintained by a separate group, I interpret "Debian EOL" as not including LTS.

Supporting Debian 9 in 2020 is already being a burden. Supporting it until mid-2021 seems pointless.

Quoted 11 lines

Debian 10 "Buster" was released in July 2019. Debian 9 "Stretch" will receive security updates from Debian until mid 2020, i.e. just about when Python 3.5 reaches EOL. LTS will attempt to support it until June 2022.

I think we should give ourselves a bit more flexibility than the categorical "Support for the previous major version will be dropped 2 years after the new major version is released." At some point, the cost of supporting old hosts exceeds the utility. We should face this tradeoff.

Agreed.

Signature · 1 line
Eduardo
#6 lore raw
In reply to John Snow
Quoted 59 lines · John Snow
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
  scripts/qapi/schema.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
          self.docs = parser.docs
          self._entity_list = []
          self._entity_dict = {}
-        self._module_dict = {}
Quoted 8 lines

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 5 lines
scripts/qapi/schema.py +1 -0
+        self._module_dict = OrderedDict()
          self._schema_dir = os.path.dirname(fname)
          self._make_module(None) # built-ins
          self._make_module(fname)

This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".

No clue.

Quoted 9 lines

3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches

We support 3.5 because we support Debian 9.

We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.

We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.

Quoted 8 lines

We should use a rule of "two years after successor, or End-of-Support, whichever comes first."

For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?

In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?

This broke 29 of the 32 Travis jobs we have:

https://travis-ci.org/qemu/qemu/builds/637999366

Since we started to use Travis CI, it catched quite some bugs...

I think it is important to add in the equation we also depend of our CIs.

#7 lore raw

[Qemu-devel] Xenial in Travis (was: qapi: Fix code generation with Python 3.5)

On 18/01/2020 09.29, Philippe Mathieu-Daudé wrote:
[...]

Quoted 7 lines

This broke 29 of the 32 Travis jobs we have:

https://travis-ci.org/qemu/qemu/builds/637999366

Since we started to use Travis CI, it catched quite some bugs...

I think it is important to add in the equation we also depend of our CIs.

Thinking about the state of our Travis CI ... The Travis CI jobs that broke are still running Ubuntu Xenial (v16.04). According to our support policy, we will stop supporting Xenial two years after v18.04 LTS has been released, i.e. in April this year. So we have to update our Travis CI jobs to a newer version with a newer Python very soon anyway - the question is here only whether we update to v18.04 (with the broken libssh version that we don't support in QEMU), or rather directly to v20.04 (which might take a while 'till it shows up as an option in Travis)...

Thomas

Tested-by on v1 1/1
#8 lore raw
In reply to Markus Armbruster
Quoted 28 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -0
index 0bfc5256fb..5100110fa2 100644

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 8 lines
scripts/qapi/schema.py +1 -1
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

Thanks, this fixes the problems on Travis for me!

Peter, could you maybe apply this directly to the master branch as a build fix?

#9 lore raw
Quoted 40 lines · Thomas Huth
Quoted 28 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -0
index 0bfc5256fb..5100110fa2 100644

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 8 lines
scripts/qapi/schema.py +1 -1
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)

Thanks, this fixes the problems on Travis for me!

Quoted 4 lines

Peter, could you maybe apply this directly to the master branch as a build fix?

The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/. Peter, want me to respin for that?

Reviewed-by Tested-by
#10 lore raw
Quoted 42 lines · Markus Armbruster
Quoted 40 lines · Thomas Huth
Quoted 28 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c
---
  scripts/qapi/schema.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -0
index 0bfc5256fb..5100110fa2 100644

scripts/qapi/schema.py class QAPISchema(object): · line 801

Quoted 8 lines
scripts/qapi/schema.py +1 -1
@@ -795,7 +795,7 @@ class QAPISchema(object):
          self.docs = parser.docs
          self._entity_list = []
          self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
          self._schema_dir = os.path.dirname(fname)
          self._make_module(None) # built-ins
          self._make_module(fname)

Thanks, this fixes the problems on Travis for me!

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 2 lines

Peter, could you maybe apply this directly to the master branch as a build fix?

The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/. Peter, want me to respin for that?

Since it is a single patch, it shouldn't be too much work to respin :)

I agree this patch is candidate for direct fix on /master.

Thanks,

Phil.

Tested-by BALATON Zoltan
#11 lore raw
Quoted 43 lines · Philippe Mathieu-Daudé
Quoted 42 lines · Markus Armbruster
Quoted 40 lines · Thomas Huth
Quoted 27 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

     for entity in self._entity_list:

effectively to

     for mod in self._module_dict.values():
         for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later.  Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c

---
  scripts/qapi/schema.py | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +0 -0
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
Quoted 9 lines

          self.docs = parser.docs self._entity_list = [] self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)

Thanks, this fixes the problems on Travis for me!

Quoted 4 lines
Quoted 1 line
Quoted 1 line

And per https://www.mail-archive.com/[email protected]/msg671745.html:

Quoted 15 lines
Quoted 5 lines
Quoted 2 lines

Peter, could you maybe apply this directly to the master branch as a build fix?

The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/.  Peter, want me to respin for that?

Since it is a single patch, it shouldn't be too much work to respin :)

I agree this patch is candidate for direct fix on /master.

Thanks,

Phil.

Reviewed-by Tested-by on v1 1/1
#12 lore raw
In reply to Markus Armbruster
Quoted 10 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Quoted 8 lines

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c

Well that certainly clears up a bunch of red. Can we apply it directly as a build fix?

Quoted 17 lines
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +1 -1
index 0bfc5256fb..5100110fa2 100644
@@ -795,7 +795,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
-        self._module_dict = {}
+        self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(None) # built-ins
         self._make_module(fname)
Signature · 1 line
Alex Bennée
#13 lore raw
In reply to Markus Armbruster
Quoted 15 lines · Markus Armbruster

Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from

for entity in self._entity_list:

effectively to

for mod in self._module_dict.values():
for entity in mod._entity_list:

Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.

Quoted 5 lines

Fix by making self._module_dict an OrderedDict rather than a dict.

Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26c

--

Applied to master as a buildfix, thanks.

-- PMM