Skip to contentSkip to conversation

python: don't sort the manifest in create_manifest

PATCH Superseded Andrew Geissler 3 participants 7 messages Sep 14, 2018 – Sep 19, 2018

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 python: don't sort the manifest in create_manifest

Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages)

The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected.

This is a backport of the same changes done by Ross Burton for python3

---
 .../python/python/create_manifest2.py              |   16 +-
 .../python/python/python2-manifest.json            | 1658 ++++++++++----------
 2 files changed, 839 insertions(+), 835 deletions(-)
meta/recipes-devtools/python/python/create_manifest2.py +9 -7
index e745045..1af1443 100644
@@ -37,6 +37,7 @@ import sys
 import subprocess
 import json
 import os
+import collections
 
 # Hack to get native python search path (for folders), not fond of it but it works for now
 pivot='recipe-sysroot-native'
@@ -45,7 +46,7 @@ for p in sys.path:
     nativelibfolder=p[:p.find(pivot)+len(pivot)]
 
 # Empty dict to hold the whole manifest
-new_manifest = {}
+new_manifest = collections.OrderedDict()
 
 # Check for repeated files, folders and wildcards
 allfiles=[]
@@ -63,7 +64,7 @@ def isFolder(value):
 
 # Read existing JSON manifest
 with open('python2-manifest.json') as manifest:
-  old_manifest=json.load(manifest)
+  old_manifest = json.load(manifest, object_pairs_hook=collections.OrderedDict)
 
 
 # First pass to get core-package functionality, because we base everything on the fact that core is actually working
@@ -124,13 +125,14 @@ for key in old_manifest:
 
 for key in old_manifest:
     # Use an empty dict as data structure to hold data for each package and fill it up
-    new_manifest[key]={}
-    new_manifest[key]['files']=[]
+    new_manifest[key] = collections.OrderedDict()
+    new_manifest[key]['summary'] = old_manifest[key]['summary']
     new_manifest[key]['rdepends']=[]
+    new_manifest[key]['files'] = []
+
     # All packages should depend on core
     if key != 'core':
-         new_manifest[key]['rdepends'].append('core')
-    new_manifest[key]['summary']=old_manifest[key]['summary']
+        new_manifest[key]['rdepends'].append('core')
 
     # Handle special cases, we assume that when they were manually added 
     # to the manifest we knew what we were doing.
@@ -274,4 +276,4 @@ for key in new_manifest:
 
 # Create the manifest from the data structure that was built
 with open('python2-manifest.json.new','w') as outfile:
-    json.dump(new_manifest,outfile,sort_keys=True, indent=4, separators=(',', ': '))
+    json.dump(new_manifest,outfile, indent=4)
meta/recipes-devtools/python/python/python2-manifest.json +830 -828

1,878 lines not on this page · raw message

Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/3 python: don't sort the manifest in create_manifest

Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages)

The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Moved distutils-staticdev above distutils so packaging rules work as expected.

---
 .../python/python/create_manifest2.py              |   16 +-
 .../python/python/python2-manifest.json            | 1660 ++++++++++----------
 2 files changed, 840 insertions(+), 836 deletions(-)
meta/recipes-devtools/python/python/create_manifest2.py +9 -7
index e745045..1af1443 100644
@@ -37,6 +37,7 @@ import sys
 import subprocess
 import json
 import os
+import collections
 
 # Hack to get native python search path (for folders), not fond of it but it works for now
 pivot='recipe-sysroot-native'
@@ -45,7 +46,7 @@ for p in sys.path:
     nativelibfolder=p[:p.find(pivot)+len(pivot)]
 
 # Empty dict to hold the whole manifest
-new_manifest = {}
+new_manifest = collections.OrderedDict()
 
 # Check for repeated files, folders and wildcards
 allfiles=[]
@@ -63,7 +64,7 @@ def isFolder(value):
 
 # Read existing JSON manifest
 with open('python2-manifest.json') as manifest:
-  old_manifest=json.load(manifest)
+  old_manifest = json.load(manifest, object_pairs_hook=collections.OrderedDict)
 
 
 # First pass to get core-package functionality, because we base everything on the fact that core is actually working
@@ -124,13 +125,14 @@ for key in old_manifest:
 
 for key in old_manifest:
     # Use an empty dict as data structure to hold data for each package and fill it up
-    new_manifest[key]={}
-    new_manifest[key]['files']=[]
+    new_manifest[key] = collections.OrderedDict()
+    new_manifest[key]['summary'] = old_manifest[key]['summary']
     new_manifest[key]['rdepends']=[]
+    new_manifest[key]['files'] = []
+
     # All packages should depend on core
     if key != 'core':
-         new_manifest[key]['rdepends'].append('core')
-    new_manifest[key]['summary']=old_manifest[key]['summary']
+        new_manifest[key]['rdepends'].append('core')
 
     # Handle special cases, we assume that when they were manually added 
     # to the manifest we knew what we were doing.
@@ -274,4 +276,4 @@ for key in new_manifest:
 
 # Create the manifest from the data structure that was built
 with open('python2-manifest.json.new','w') as outfile:
-    json.dump(new_manifest,outfile,sort_keys=True, indent=4, separators=(',', ': '))
+    json.dump(new_manifest,outfile, indent=4)
meta/recipes-devtools/python/python/python2-manifest.json +831 -829

1,878 lines not on this page · raw message

Signature · 1 line
2.7.4
#2 lore raw

PATCH 2/3 python: respect package order in manifest

In reply to Andrew Geissler

Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.

This is a backport of the same changes done by Ross Burton for python3

---
 meta/recipes-devtools/python/python_2.7.15.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-devtools/python/python_2.7.15.bb +2 -4
index 43d9ff5..472c1da 100644
@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules"
 INCLUDE_PYCS ?= "1"
 
 python(){
-    import json
+    import collections, json
 
     filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json')
     # This python changes the datastore based on the contents of a file, so mark
@@ -215,7 +215,7 @@ python(){
     bb.parse.mark_dependency(d, filename)
 
     with open(filename) as manifest_file:
-        python_manifest=json.load(manifest_file)
+        python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict)
 
     include_pycs = d.getVar('INCLUDE_PYCS')
 
@@ -248,8 +248,6 @@ python(){
             d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value)
         d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary'])
 
-    # We need to ensure staticdev packages match for files first so we sort in reverse
-    newpackages.sort(reverse=True)
     # Prepending so to avoid python-misc getting everything
     packages = newpackages + packages
     d.setVar('PACKAGES', ' '.join(packages))
Signature · 1 line
2.7.4
#3 lore raw

Re: [PATCH 2/3] python: respect package order in manifest

Close, but:

ERROR: python-2.7.15-r1 do_package_qa: QA Issue: non -staticdev
package contains static .a library: python-distutils path '/work/corei7-64-poky-linux/python/2.7.15-r1/packages-split/python-distutils/usr/lib/python2.7/config/libpython2.7.a' [staticdev]

Ross

Quoted 45 lines · Andrew Geissler

Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.

This is a backport of the same changes done by Ross Burton for python3

---
 meta/recipes-devtools/python/python_2.7.15.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-devtools/python/python_2.7.15.bb +2 -5
index 43d9ff5..472c1da 100644
@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules"
 INCLUDE_PYCS ?= "1"
 
 python(){
-    import json
+    import collections, json
 
     filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json')
     # This python changes the datastore based on the contents of a file, so mark
@@ -215,7 +215,7 @@ python(){
     bb.parse.mark_dependency(d, filename)
 
     with open(filename) as manifest_file:
-        python_manifest=json.load(manifest_file)
+        python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict)
 
     include_pycs = d.getVar('INCLUDE_PYCS')
 
@@ -248,8 +248,6 @@ python(){
             d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value)
         d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary'])
 
-    # We need to ensure staticdev packages match for files first so we sort in reverse
-    newpackages.sort(reverse=True)
     # Prepending so to avoid python-misc getting everything
     packages = newpackages + packages
     d.setVar('PACKAGES', ' '.join(packages))
--

2.7.4

#4 lore raw

Re: [PATCH 2/3] python: respect package order in manifest

Quoted 7 lines · Ross

Close, but:

ERROR: python-2.7.15-r1 do_package_qa: QA Issue: non -staticdev
package contains static .a library: python-distutils path '/work/corei7-64-poky-linux/python/2.7.15-r1/packages-split/python-distutils/usr/lib/python2.7/config/libpython2.7.a' [staticdev]

Ahh shoot, I missed that reorder change in your patch set 1 changes. I'll get an update up.

Quoted 48 lines

Ross

Quoted 45 lines · Andrew Geissler

Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.

This is a backport of the same changes done by Ross Burton for python3

---
 meta/recipes-devtools/python/python_2.7.15.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-devtools/python/python_2.7.15.bb +2 -5
index 43d9ff5..472c1da 100644
@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules"
 INCLUDE_PYCS ?= "1"
 
 python(){
-    import json
+    import collections, json
 
     filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json')
     # This python changes the datastore based on the contents of a file, so mark
@@ -215,7 +215,7 @@ python(){
     bb.parse.mark_dependency(d, filename)
 
     with open(filename) as manifest_file:
-        python_manifest=json.load(manifest_file)
+        python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict)
 
     include_pycs = d.getVar('INCLUDE_PYCS')
 
@@ -248,8 +248,6 @@ python(){
             d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value)
         d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary'])
 
-    # We need to ensure staticdev packages match for files first so we sort in reverse
-    newpackages.sort(reverse=True)
     # Prepending so to avoid python-misc getting everything
     packages = newpackages + packages
     d.setVar('PACKAGES', ' '.join(packages))
--

2.7.4

#5 lore raw

PATCH 3/3 python: consolidate tests

In reply to Andrew Geissler

Currently the bulk of the tests in python-tests, some more in python-sqlite3-tests, and others in their parent module (such as python-ctypes). This is pointless space usage if we're not planning on running the tests, so consolidate all the tests into python-tests.

This is a backport of the same changes done by Ross Burton for python3

---
 .../python/python/python2-manifest.json            | 35 ++++++++++------------
 1 file changed, 15 insertions(+), 20 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +15 -20
index cacba5b..45fa371 100644
@@ -1,4 +1,18 @@
 {
+    "tests": {
+        "summary": "Python test suite",
+        "rdepends": [
+            "core",
+            "modules"
+        ],
+        "files": [
+            "${libdir}/python2.7/*/test",
+            "${libdir}/python2.7/*/tests",
+            "${libdir}/python2.7/idlelib/idle_test/",
+            "${libdir}/python2.7/test"
+        ],
+        "cached": []
+    },
     "2to3": {
         "summary": "Python automated Python 2 to 3 code translator", 
         "rdepends": [
@@ -873,16 +887,6 @@
             "${libdir}/python2.7/lib-dynload/_sqlite3.so"
         ]
     }, 
-    "sqlite3-tests": {
-        "summary": "Python Sqlite3 database support tests", 
-        "rdepends": [
-            "core", 
-            "tests"
-        ], 
-        "files": [
-            "${libdir}/python2.7/sqlite3/test"
-        ]
-    }, 
     "stringold": {
         "summary": "Python string APIs [deprecated]", 
         "rdepends": [
@@ -925,15 +929,6 @@
             "${libdir}/python2.7/tty.py"
         ]
     }, 
-    "tests": {
-        "summary": "Python tests", 
-        "rdepends": [
-            "core"
-        ], 
-        "files": [
-            "${libdir}/python2.7/test"
-        ]
-    }, 
     "textutils": {
         "summary": "Python option parsin", 
         "rdepends": [
@@ -1046,4 +1041,4 @@
             "${libdir}/python2.7/lib-dynload/zlib.so"
         ]
     }
-}
\ No newline at end of file
+}
Signature · 1 line
2.7.4
#6 lore raw

✗ patchtest: failure for "python: don't sort the manifes..." and 2 more (rev2)

In reply to Andrew Geissler

== Series Details ==

Series: "python: don't sort the manifes..." and 2 more (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/14085/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at 6a1e550838)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe