python: don't sort the manifest in create_manifest
PATCH 1/3 python: don't sort the manifest in create_manifest
Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages)
The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected.
This is a backport of the same changes done by Ross Burton for python3
--- .../python/python/create_manifest2.py | 16 +- .../python/python/python2-manifest.json | 1658 ++++++++++---------- 2 files changed, 839 insertions(+), 835 deletions(-)
meta/recipes-devtools/python/python/create_manifest2.py +9 -7
@@ -37,6 +37,7 @@ import sys import subprocess import json import os +import collections # Hack to get native python search path (for folders), not fond of it but it works for now pivot='recipe-sysroot-native'
@@ -45,7 +46,7 @@ for p in sys.path: nativelibfolder=p[:p.find(pivot)+len(pivot)] # Empty dict to hold the whole manifest -new_manifest = {} +new_manifest = collections.OrderedDict() # Check for repeated files, folders and wildcards allfiles=[]
@@ -63,7 +64,7 @@ def isFolder(value): # Read existing JSON manifest with open('python2-manifest.json') as manifest: - old_manifest=json.load(manifest) + old_manifest = json.load(manifest, object_pairs_hook=collections.OrderedDict) # First pass to get core-package functionality, because we base everything on the fact that core is actually working
@@ -124,13 +125,14 @@ for key in old_manifest: for key in old_manifest: # Use an empty dict as data structure to hold data for each package and fill it up - new_manifest[key]={} - new_manifest[key]['files']=[] + new_manifest[key] = collections.OrderedDict() + new_manifest[key]['summary'] = old_manifest[key]['summary'] new_manifest[key]['rdepends']=[] + new_manifest[key]['files'] = [] + # All packages should depend on core if key != 'core': - new_manifest[key]['rdepends'].append('core') - new_manifest[key]['summary']=old_manifest[key]['summary'] + new_manifest[key]['rdepends'].append('core') # Handle special cases, we assume that when they were manually added # to the manifest we knew what we were doing.
@@ -274,4 +276,4 @@ for key in new_manifest: # Create the manifest from the data structure that was built with open('python2-manifest.json.new','w') as outfile: - json.dump(new_manifest,outfile,sort_keys=True, indent=4, separators=(',', ': ')) + json.dump(new_manifest,outfile, indent=4)
meta/recipes-devtools/python/python/python2-manifest.json +830 -828
1,878 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 1/3 python: don't sort the manifest in create_manifest
Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages)
The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected.
This is a backport of the same changes done by Ross Burton for python3
Changes since v1:
- Moved distutils-staticdev above distutils so packaging rules work as expected.
--- .../python/python/create_manifest2.py | 16 +- .../python/python/python2-manifest.json | 1660 ++++++++++---------- 2 files changed, 840 insertions(+), 836 deletions(-)
meta/recipes-devtools/python/python/create_manifest2.py +9 -7
@@ -37,6 +37,7 @@ import sys import subprocess import json import os +import collections # Hack to get native python search path (for folders), not fond of it but it works for now pivot='recipe-sysroot-native'
@@ -45,7 +46,7 @@ for p in sys.path: nativelibfolder=p[:p.find(pivot)+len(pivot)] # Empty dict to hold the whole manifest -new_manifest = {} +new_manifest = collections.OrderedDict() # Check for repeated files, folders and wildcards allfiles=[]
@@ -63,7 +64,7 @@ def isFolder(value): # Read existing JSON manifest with open('python2-manifest.json') as manifest: - old_manifest=json.load(manifest) + old_manifest = json.load(manifest, object_pairs_hook=collections.OrderedDict) # First pass to get core-package functionality, because we base everything on the fact that core is actually working
@@ -124,13 +125,14 @@ for key in old_manifest: for key in old_manifest: # Use an empty dict as data structure to hold data for each package and fill it up - new_manifest[key]={} - new_manifest[key]['files']=[] + new_manifest[key] = collections.OrderedDict() + new_manifest[key]['summary'] = old_manifest[key]['summary'] new_manifest[key]['rdepends']=[] + new_manifest[key]['files'] = [] + # All packages should depend on core if key != 'core': - new_manifest[key]['rdepends'].append('core') - new_manifest[key]['summary']=old_manifest[key]['summary'] + new_manifest[key]['rdepends'].append('core') # Handle special cases, we assume that when they were manually added # to the manifest we knew what we were doing.
@@ -274,4 +276,4 @@ for key in new_manifest: # Create the manifest from the data structure that was built with open('python2-manifest.json.new','w') as outfile: - json.dump(new_manifest,outfile,sort_keys=True, indent=4, separators=(',', ': ')) + json.dump(new_manifest,outfile, indent=4)
meta/recipes-devtools/python/python/python2-manifest.json +831 -829
1,878 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 2/3 python: respect package order in manifest
In reply to Andrew GeisslerDon't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.
This is a backport of the same changes done by Ross Burton for python3
--- meta/recipes-devtools/python/python_2.7.15.bb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-devtools/python/python_2.7.15.bb +2 -4
@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules" INCLUDE_PYCS ?= "1" python(){ - import json + import collections, json filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json') # This python changes the datastore based on the contents of a file, so mark
@@ -215,7 +215,7 @@ python(){ bb.parse.mark_dependency(d, filename) with open(filename) as manifest_file: - python_manifest=json.load(manifest_file) + python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict) include_pycs = d.getVar('INCLUDE_PYCS')
@@ -248,8 +248,6 @@ python(){ d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value) d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary']) - # We need to ensure staticdev packages match for files first so we sort in reverse - newpackages.sort(reverse=True) # Prepending so to avoid python-misc getting everything packages = newpackages + packages d.setVar('PACKAGES', ' '.join(packages))
Signature · 1 line
2.7.4
Re: [PATCH 2/3] python: respect package order in manifest
Close, but:
ERROR: python-2.7.15-r1 do_package_qa: QA Issue: non -staticdev
package contains static .a library: python-distutils path '/work/corei7-64-poky-linux/python/2.7.15-r1/packages-split/python-distutils/usr/lib/python2.7/config/libpython2.7.a' [staticdev]
Ross
Quoted 45 lines · Andrew Geissler
Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.
This is a backport of the same changes done by Ross Burton for python3
Signed-off-by Andrew Geissler <[email protected]>--- meta/recipes-devtools/python/python_2.7.15.bb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)meta/recipes-devtools/python/python_2.7.15.bb +2
-5@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules" INCLUDE_PYCS ?= "1" python(){ - import json + import collections, json filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json') # This python changes the datastore based on the contents of a file, so mark@@ -215,7 +215,7 @@ python(){ bb.parse.mark_dependency(d, filename) with open(filename) as manifest_file: - python_manifest=json.load(manifest_file) + python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict) include_pycs = d.getVar('INCLUDE_PYCS')@@ -248,8 +248,6 @@ python(){ d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value) d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary']) - # We need to ensure staticdev packages match for files first so we sort in reverse - newpackages.sort(reverse=True) # Prepending so to avoid python-misc getting everything packages = newpackages + packages d.setVar('PACKAGES', ' '.join(packages)) --2.7.4
Re: [PATCH 2/3] python: respect package order in manifest
Quoted 7 lines · Ross
Close, but:
ERROR: python-2.7.15-r1
do_package_qa: QA Issue: non -staticdev
package contains static .a library: python-distutils path '/work/corei7-64-poky-linux/python/2.7.15-r1/packages-split/python-distutils/usr/lib/python2.7/config/libpython2.7.a' [staticdev]
Ahh shoot, I missed that reorder change in your patch set 1 changes. I'll get an update up.
Quoted 48 lines
Ross
Quoted 45 lines · Andrew Geissler
Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.
This is a backport of the same changes done by Ross Burton for python3
Signed-off-by Andrew Geissler <[email protected]>--- meta/recipes-devtools/python/python_2.7.15.bb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)meta/recipes-devtools/python/python_2.7.15.bb +2
-5@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules" INCLUDE_PYCS ?= "1" python(){ - import json + import collections, json filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json') # This python changes the datastore based on the contents of a file, so mark@@ -215,7 +215,7 @@ python(){ bb.parse.mark_dependency(d, filename) with open(filename) as manifest_file: - python_manifest=json.load(manifest_file) + python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict) include_pycs = d.getVar('INCLUDE_PYCS')@@ -248,8 +248,6 @@ python(){ d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value) d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary']) - # We need to ensure staticdev packages match for files first so we sort in reverse - newpackages.sort(reverse=True) # Prepending so to avoid python-misc getting everything packages = newpackages + packages d.setVar('PACKAGES', ' '.join(packages)) --2.7.4
PATCH 3/3 python: consolidate tests
In reply to Andrew GeisslerCurrently the bulk of the tests in python-tests, some more in python-sqlite3-tests, and others in their parent module (such as python-ctypes). This is pointless space usage if we're not planning on running the tests, so consolidate all the tests into python-tests.
This is a backport of the same changes done by Ross Burton for python3
--- .../python/python/python2-manifest.json | 35 ++++++++++------------ 1 file changed, 15 insertions(+), 20 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +15 -20
@@ -1,4 +1,18 @@ { + "tests": { + "summary": "Python test suite", + "rdepends": [ + "core", + "modules" + ], + "files": [ + "${libdir}/python2.7/*/test", + "${libdir}/python2.7/*/tests", + "${libdir}/python2.7/idlelib/idle_test/", + "${libdir}/python2.7/test" + ], + "cached": [] + }, "2to3": { "summary": "Python automated Python 2 to 3 code translator", "rdepends": [
@@ -873,16 +887,6 @@ "${libdir}/python2.7/lib-dynload/_sqlite3.so" ] }, - "sqlite3-tests": { - "summary": "Python Sqlite3 database support tests", - "rdepends": [ - "core", - "tests" - ], - "files": [ - "${libdir}/python2.7/sqlite3/test" - ] - }, "stringold": { "summary": "Python string APIs [deprecated]", "rdepends": [
@@ -925,15 +929,6 @@ "${libdir}/python2.7/tty.py" ] }, - "tests": { - "summary": "Python tests", - "rdepends": [ - "core" - ], - "files": [ - "${libdir}/python2.7/test" - ] - }, "textutils": { "summary": "Python option parsin", "rdepends": [
@@ -1046,4 +1041,4 @@ "${libdir}/python2.7/lib-dynload/zlib.so" ] } -} +}
Signature · 1 line
2.7.4
✗ patchtest: failure for "python: don't sort the manifes..." and 2 more (rev2)
In reply to Andrew Geissler== Series Details ==
Series: "python: don't sort the manifes..." and 2 more (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/14085/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at 6a1e550838)
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe