Skip to contentSkip to conversation

python: don't sort the manifest in create_manifest

PATCH New Andrew Geissler 1 participant 3 messages Sep 19, 2018

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/3 python: don't sort the manifest in create_manifest

Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages)

The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Moved distutils-staticdev above distutils so packaging rules work as expected.
Changes since v2:
- Rebase

---
 .../python/python/create_manifest2.py              |   16 +-
 .../python/python/python2-manifest.json            | 1660 ++++++++++----------
 2 files changed, 840 insertions(+), 836 deletions(-)
meta/recipes-devtools/python/python/create_manifest2.py +9 -7
index e745045..1af1443 100644
@@ -37,6 +37,7 @@ import sys
 import subprocess
 import json
 import os
+import collections
 
 # Hack to get native python search path (for folders), not fond of it but it works for now
 pivot='recipe-sysroot-native'
@@ -45,7 +46,7 @@ for p in sys.path:
     nativelibfolder=p[:p.find(pivot)+len(pivot)]
 
 # Empty dict to hold the whole manifest
-new_manifest = {}
+new_manifest = collections.OrderedDict()
 
 # Check for repeated files, folders and wildcards
 allfiles=[]
@@ -63,7 +64,7 @@ def isFolder(value):
 
 # Read existing JSON manifest
 with open('python2-manifest.json') as manifest:
-  old_manifest=json.load(manifest)
+  old_manifest = json.load(manifest, object_pairs_hook=collections.OrderedDict)
 
 
 # First pass to get core-package functionality, because we base everything on the fact that core is actually working
@@ -124,13 +125,14 @@ for key in old_manifest:
 
 for key in old_manifest:
     # Use an empty dict as data structure to hold data for each package and fill it up
-    new_manifest[key]={}
-    new_manifest[key]['files']=[]
+    new_manifest[key] = collections.OrderedDict()
+    new_manifest[key]['summary'] = old_manifest[key]['summary']
     new_manifest[key]['rdepends']=[]
+    new_manifest[key]['files'] = []
+
     # All packages should depend on core
     if key != 'core':
-         new_manifest[key]['rdepends'].append('core')
-    new_manifest[key]['summary']=old_manifest[key]['summary']
+        new_manifest[key]['rdepends'].append('core')
 
     # Handle special cases, we assume that when they were manually added 
     # to the manifest we knew what we were doing.
@@ -274,4 +276,4 @@ for key in new_manifest:
 
 # Create the manifest from the data structure that was built
 with open('python2-manifest.json.new','w') as outfile:
-    json.dump(new_manifest,outfile,sort_keys=True, indent=4, separators=(',', ': '))
+    json.dump(new_manifest,outfile, indent=4)
meta/recipes-devtools/python/python/python2-manifest.json +831 -829

1,878 lines not on this page · raw message

Signature · 1 line
2.7.4
#1 lore raw

PATCH v3 2/3 python: respect package order in manifest

Don't sort the manifest when using it to generate packaging rules, so ordering can be used to have complex packaging rules.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Rebase

---
 meta/recipes-devtools/python/python_2.7.15.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-devtools/python/python_2.7.15.bb +2 -4
index 43d9ff5..472c1da 100644
@@ -207,7 +207,7 @@ RPROVIDES_${PN} += "${PN}-modules"
 INCLUDE_PYCS ?= "1"
 
 python(){
-    import json
+    import collections, json
 
     filename = os.path.join(d.getVar('THISDIR'), 'python', 'python2-manifest.json')
     # This python changes the datastore based on the contents of a file, so mark
@@ -215,7 +215,7 @@ python(){
     bb.parse.mark_dependency(d, filename)
 
     with open(filename) as manifest_file:
-        python_manifest=json.load(manifest_file)
+        python_manifest=json.load(manifest_file, object_pairs_hook=collections.OrderedDict)
 
     include_pycs = d.getVar('INCLUDE_PYCS')
 
@@ -248,8 +248,6 @@ python(){
             d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value)
         d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary'])
 
-    # We need to ensure staticdev packages match for files first so we sort in reverse
-    newpackages.sort(reverse=True)
     # Prepending so to avoid python-misc getting everything
     packages = newpackages + packages
     d.setVar('PACKAGES', ' '.join(packages))
Signature · 1 line
2.7.4
#2 lore raw

PATCH v3 3/3 python: consolidate tests

In reply to Andrew Geissler

Currently the bulk of the tests in python-tests, some more in python-sqlite3-tests, and others in their parent module (such as python-ctypes). This is pointless space usage if we're not planning on running the tests, so consolidate all the tests into python-tests.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Rebase

---
 .../python/python/python2-manifest.json            | 33 +++++++++-------------
 1 file changed, 14 insertions(+), 19 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +14 -19
index 4a29c56..1e7004e 100644
@@ -1,4 +1,18 @@
 {
+    "tests": {
+        "summary": "Python test suite",
+        "rdepends": [
+            "core",
+            "modules"
+        ],
+        "files": [
+            "${libdir}/python2.7/*/test",
+            "${libdir}/python2.7/*/tests",
+            "${libdir}/python2.7/idlelib/idle_test/",
+            "${libdir}/python2.7/test"
+        ],
+        "cached": []
+    },
     "2to3": {
         "summary": "Python automated Python 2 to 3 code translator", 
         "rdepends": [
@@ -873,16 +887,6 @@
             "${libdir}/python2.7/lib-dynload/_sqlite3.so"
         ]
     }, 
-    "sqlite3-tests": {
-        "summary": "Python Sqlite3 database support tests", 
-        "rdepends": [
-            "core", 
-            "tests"
-        ], 
-        "files": [
-            "${libdir}/python2.7/sqlite3/test"
-        ]
-    }, 
     "stringold": {
         "summary": "Python string APIs [deprecated]", 
         "rdepends": [
@@ -925,15 +929,6 @@
             "${libdir}/python2.7/tty.py"
         ]
     }, 
-    "tests": {
-        "summary": "Python tests", 
-        "rdepends": [
-            "core"
-        ], 
-        "files": [
-            "${libdir}/python2.7/test"
-        ]
-    }, 
     "textutils": {
         "summary": "Python option parsin", 
         "rdepends": [
Signature · 1 line
2.7.4