qapi: Fix code generation with Python 3.5
PATCH 1/1 qapi: Fix code generation with Python 3.5
Recent commit 3e7fb5811b "qapi: Fix code generation for empty modules" modules" switched QAPISchema.visit() from
for entity in self._entity_list:
effectively to
for mod in self._module_dict.values():
for entity in mod._entity_list:
Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/schema.py +1 -1
@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)
Signature · 1 line
2.21.1
Quoted 28 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-0
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 8 lines
scripts/qapi/schema.py +1
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)
This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {}
Quoted 8 lines
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 5 lines
scripts/qapi/schema.py +1
-0+ self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
No clue.
3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branches
We support 3.5 because we support Debian 9.
We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {}Quoted 8 lines
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 5 lines
scripts/qapi/schema.py +1
-0+ self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
No clue.
Quoted 9 lines
3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branchesWe support 3.5 because we support Debian 9.
We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.
We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.
We should use a rule of "two years after successor, or End-of-Support, whichever comes first."
For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?
In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?
--js
Quoted 56 lines · John Snow
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {}Quoted 8 lines
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 5 lines
scripts/qapi/schema.py +1
-0+ self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
No clue.
Quoted 9 lines
3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branchesWe support 3.5 because we support Debian 9.
We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.
Quoted 5 lines
We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.
We should use a rule of "two years after successor, or End-of-Support, whichever comes first."
Yes.
Quoted 2 lines
For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?
We need to define end-of-support for Debian: is it Debian proper or is it Debian LTS?
<https://wiki.debian.org/DebianOldStable>:
Q) How long will security updates be provided?
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
<https://wiki.debian.org/LTS>:
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
Thus the Debian LTS team takes over security maintenance of the various releases once the Debian Security team stops its work.
Debian 10 "Buster" was released in July 2019. Debian 9 "Stretch" will receive security updates from Debian until mid 2020, i.e. just about when Python 3.5 reaches EOL. LTS will attempt to support it until June 2022.
I think we should give ourselves a bit more flexibility than the categorical "Support for the previous major version will be dropped 2 years after the new major version is released." At some point, the cost of supporting old hosts exceeds the utility. We should face this tradeoff.
Quoted 2 lines
In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?
June 2022 even.
We use Python at compile time with trusted input. The need for security maintenance is relatively low there. I'm not ready to vouch for "and we don't use Python for anything else".
Quoted 3 lines · Markus Armbruster
Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster
John Snow <[email protected]> writes:
[...]
Quoted 41 lines
Quoted 15 lines
Quoted 5 lines
Quoted 2 lines
This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
No clue.
Quoted 9 lines
3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branchesWe support 3.5 because we support Debian 9.
We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.
Quoted 5 lines
We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.
We should use a rule of "two years after successor, or End-of-Support, whichever comes first."
Yes.
Quoted 2 lines
For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?
We need to define end-of-support for Debian: is it Debian proper or is it Debian LTS?
<https://wiki.debian.org/DebianOldStable>:
Q) How long will security updates be provided?
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
Quoted 7 lines
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
Thus the Debian LTS team takes over security maintenance of the various releases once the Debian Security team stops its work.
As Debian LTS is maintained by a separate group, I interpret "Debian EOL" as not including LTS.
Supporting Debian 9 in 2020 is already being a burden. Supporting it until mid-2021 seems pointless.
Quoted 11 lines
Debian 10 "Buster" was released in July 2019. Debian 9 "Stretch" will receive security updates from Debian until mid 2020, i.e. just about when Python 3.5 reaches EOL. LTS will attempt to support it until June 2022.
I think we should give ourselves a bit more flexibility than the categorical "Support for the previous major version will be dropped 2 years after the new major version is released." At some point, the cost of supporting old hosts exceeds the utility. We should face this tradeoff.
Agreed.
Signature · 1 line
Eduardo
Quoted 59 lines · John Snow
Quoted 45 lines · Markus Armbruster
Quoted 32 lines · John Snow
Quoted 31 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {}Quoted 8 lines
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 5 lines
scripts/qapi/schema.py +1
-0+ self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)This problem has bitten me *many* times. I'm wondering if there's a prescription that isn't just "Wait until we can stipulate 3.6+".
No clue.
Quoted 9 lines
3.5 EOL is scheduled for 2020-09-13.
https://devguide.python.org/#status-of-python-branchesWe support 3.5 because we support Debian 9.
We'd normally drop support for Debian 9 two years after Debian 10, i.e. July 2021. Assuming Debian supports it that far. Whether they can truly support Python 3.5 after uptstream EOL seems doubtful.
We should decide whether we consider Debian LTS to be adequately supported, yes-or-no.
Quoted 8 lines
We should use a rule of "two years after successor, or End-of-Support, whichever comes first."
For Debian, is end of support three years after it comes out, or is it when the LTS is EOL?
In this specific case, do we trust that Debian 9 LTS will continue to patch Python3.5 all the way up until July 2021?
This broke 29 of the 32 Travis jobs we have:
https://travis-ci.org/qemu/qemu/builds/637999366
Since we started to use Travis CI, it catched quite some bugs...
I think it is important to add in the equation we also depend of our CIs.
[Qemu-devel] Xenial in Travis (was: qapi: Fix code generation with Python 3.5)
On 18/01/2020 09.29, Philippe Mathieu-Daudé wrote:
[...]
Quoted 7 lines
This broke 29 of the 32 Travis jobs we have:
https://travis-ci.org/qemu/qemu/builds/637999366
Since we started to use Travis CI, it catched quite some bugs...
I think it is important to add in the equation we also depend of our CIs.
Thinking about the state of our Travis CI ... The Travis CI jobs that broke are still running Ubuntu Xenial (v16.04). According to our support policy, we will stop supporting Xenial two years after v18.04 LTS has been released, i.e. in April this year. So we have to update our Travis CI jobs to a newer version with a newer Python very soon anyway - the question is here only whether we update to v18.04 (with the broken libssh version that we don't support in QEMU), or rather directly to v20.04 (which might take a while 'till it shows up as an option in Travis)...
Thomas
Quoted 28 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-0
scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 8 lines
scripts/qapi/schema.py +1
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)
Thanks, this fixes the problems on Travis for me!
Peter, could you maybe apply this directly to the master branch as a build fix?
Quoted 40 lines · Thomas Huth
Quoted 28 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-0scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 8 lines
scripts/qapi/schema.py +1
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)Thanks, this fixes the problems on Travis for me!
Quoted 4 lines
Tested-by Thomas Huth <[email protected]>Peter, could you maybe apply this directly to the master branch as a build fix?
The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/. Peter, want me to respin for that?
Quoted 42 lines · Markus Armbruster
Quoted 40 lines · Thomas Huth
Quoted 28 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-0scripts/qapi/schema.py class QAPISchema(object): · line 801
Quoted 8 lines
scripts/qapi/schema.py +1
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)Thanks, this fixes the problems on Travis for me!
Quoted 1 line
Quoted 1 line
Tested-by Thomas Huth <[email protected]>
Quoted 5 lines
Quoted 2 lines
Peter, could you maybe apply this directly to the master branch as a build fix?
The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/. Peter, want me to respin for that?
Since it is a single patch, it shouldn't be too much work to respin :)
I agree this patch is candidate for direct fix on /master.
Thanks,
Phil.
Philippe Mathieu-Daudé
Tested-by BALATON ZoltanQuoted 43 lines · Philippe Mathieu-Daudé
Quoted 42 lines · Markus Armbruster
Quoted 40 lines · Thomas Huth
Quoted 27 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>---
scripts/qapi/schema.py| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +0
-0@@ -795,7 +795,7 @@ class QAPISchema(object):Quoted 9 lines
self.docs = parser.docs self._entity_list = [] self._entity_dict = {}
- self._module_dict = {}
+ self._module_dict =OrderedDict()self._schema_dir =os.path.dirname(fname)self._make_module(None)# built-insself._make_module(fname)Thanks, this fixes the problems on Travis for me!
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Tested-by Thomas Huth <[email protected]>Reviewed-by Philippe Mathieu-Daudé <[email protected]>
And per https://www.mail-archive.com/[email protected]/msg671745.html:
Quoted 15 lines
Quoted 5 lines
Quoted 2 lines
Peter, could you maybe apply this directly to the master branch as a build fix?
The commit message isn't quite right: s/Visits in the same order/Visits modules in the same order/. Peter, want me to respin for that?
Since it is a single patch, it shouldn't be too much work to respin :)
I agree this patch is candidate for direct fix on /master.
Thanks,
Phil.
Quoted 10 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:
Quoted 8 lines
Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>
Well that certainly clears up a bunch of red. Can we apply it directly as a build fix?
Quoted 17 lines
--- scripts/qapi/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/schema.py +1
-1@@ -795,7 +795,7 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} - self._module_dict = {} + self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) self._make_module(None) # built-ins self._make_module(fname)
Signature · 1 line
Alex Bennée
Quoted 15 lines · Markus Armbruster
Recent commit
3e7fb5811b"qapi: Fix code generation for empty modules" modules" switchedQAPISchema.visit()fromfor entity in self._entity_list:
effectively to
for mod in
self._module_dict.values():
for entity in mod._entity_list:Visits in the same order as long as .values() is in insertion order. That's the case only for Python 3.6 and later. Before, it's in some arbitrary order, which results in broken generated code.
Quoted 5 lines
Fix by making self._module_dict an OrderedDict rather than a dict.
Fixes 3e7fb5811baab213dcc7149c3aa69442d683c26cSigned-off-by Markus Armbruster <[email protected]>--
Applied to master as a buildfix, thanks.
-- PMM