Skip to contentSkip to conversation

python/python3: Move test files to test package

PATCH In review Andrew Geissler 4 participants 14 messages Sep 13, 2018 – Sep 14, 2018

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 python/python3: Move test files to test package

A quick search in the python package shows the following test directories being included in the python image:
./Python-2.7.14/Lib/bsddb/test ./Python-2.7.14/Lib/sqlite3/test ./Python-2.7.14/Lib/ctypes/test ./Python-2.7.14/Lib/unittest/test ./Python-2.7.14/Lib/lib-tk/test ./Python-2.7.14/Lib/email/test

Move the bsddb, ctypes, and email packages to the python-tests package to conserve some space.

Packages not include in this patch:
- sqlite3 has it's own -tests package
- unittest is already a test package
- lib-tk (tkinter) seems to have its own special installation and does not allow the manifest override for its test directory
- bsddb is no longer in python3 so update for it was not needed there

Just the ctypes and email change saved us ~150KB of read-only filesystem space in our embedded env.

Changes since v1:
- Used manifest file instead of python recipe to move files to -tests package

---
 meta/recipes-devtools/python/python/python2-manifest.json  | 5 ++++-
 meta/recipes-devtools/python/python3/python3-manifest.json | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +4 -1
index 723e513..8db2afe 100644
@@ -925,7 +925,10 @@
     },
     "tests": {
         "files": [
-            "${libdir}/python2.7/test"
+            "${libdir}/python2.7/test",
+            "${libdir}/python2.7/email/test",
+            "${libdir}/python2.7/ctypes/test",
+            "${libdir}/python2.7/bsddb/test"
         ],
         "rdepends": [
             "core"
meta/recipes-devtools/python/python3/python3-manifest.json +3 -1
index 90a41fb..963140f 100644
@@ -1040,7 +1040,9 @@
     "tests": {
         "cached": [],
         "files": [
-            "${libdir}/python${PYTHON_MAJMIN}/test"
+            "${libdir}/python${PYTHON_MAJMIN}/test",
+            "${libdir}/python${PYTHON_MAJMIN}/email/test",
+            "${libdir}/python${PYTHON_MAJMIN}/ctypes/test"
         ],
         "rdepends": [
             "core",
Signature · 1 line
2.7.4
#1 lore raw

A little oe-pkgdata-util[1] managed to give me a good review of some missing entries here.

python3-asyncio: /usr/lib/python3.5/asyncio/test_utils.py
python3-2to3: /usr/lib/python3.5/lib2to3/tests/test_*.py
python3-tkinter: /usr/lib/python3.5/tkinter/test/
python3-unittest: /usr/lib/python3.5/unittest/test/ (the test suite for unittest should be in python3-test)
python3-distutils: /usr/lib/python3.5/distutils/tests/test_*

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Ross

[1] oe-pkgdata-util find-path '/usr/lib/python*/**/test_*.py'

Quoted 66 lines · Andrew Geissler

A quick search in the python package shows the following test directories being included in the python image:
./Python-2.7.14/Lib/bsddb/test ./Python-2.7.14/Lib/sqlite3/test ./Python-2.7.14/Lib/ctypes/test ./Python-2.7.14/Lib/unittest/test ./Python-2.7.14/Lib/lib-tk/test ./Python-2.7.14/Lib/email/test

Move the bsddb, ctypes, and email packages to the python-tests package to conserve some space.

Packages not include in this patch:
- sqlite3 has it's own -tests package
- unittest is already a test package
- lib-tk (tkinter) seems to have its own special installation and does not allow the manifest override for its test directory
- bsddb is no longer in python3 so update for it was not needed there

Just the ctypes and email change saved us ~150KB of read-only filesystem space in our embedded env.

Changes since v1:
- Used manifest file instead of python recipe to move files to -tests package

---
 meta/recipes-devtools/python/python/python2-manifest.json  | 5 ++++-
 meta/recipes-devtools/python/python3/python3-manifest.json | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +4 -1
index 723e513..8db2afe 100644
@@ -925,7 +925,10 @@
     },
     "tests": {
         "files": [
-            "${libdir}/python2.7/test"
+            "${libdir}/python2.7/test",
+            "${libdir}/python2.7/email/test",
+            "${libdir}/python2.7/ctypes/test",
+            "${libdir}/python2.7/bsddb/test"
         ],
         "rdepends": [
             "core"
meta/recipes-devtools/python/python3/python3-manifest.json +3 -2
index 90a41fb..963140f 100644
@@ -1040,7 +1040,9 @@
     "tests": {
         "cached": [],
         "files": [
-            "${libdir}/python${PYTHON_MAJMIN}/test"
+            "${libdir}/python${PYTHON_MAJMIN}/test",
+            "${libdir}/python${PYTHON_MAJMIN}/email/test",
+            "${libdir}/python${PYTHON_MAJMIN}/ctypes/test"
         ],
         "rdepends": [
             "core",
--

2.7.4

#2 lore raw
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Ross

#3 lore raw
Quoted 4 lines · Ross
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Quoted 8 lines

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

So I went and did this.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest

Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.

Ross

#4 lore raw
Quoted 15 lines · Ross
Quoted 4 lines · Ross
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Quoted 8 lines

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Quoted 7 lines

So I went and did this.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest

Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.

Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?

Quoted 2 lines

Ross

#5 lore raw
Quoted 2 lines · Andrew Geissler

Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?

Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.

Ross

#6 lore raw

To be fair, I have also not had a chance to update the py2 manifest script with the latest changes I made to py3 either, but I think applying these to create_manifest2 should be trivial.

Alejandro

Quoted 8 lines · Ross
Quoted 2 lines · Andrew Geissler

Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?

Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.

Ross

#7 lore raw
In reply to Burton, Ross
Quoted 8 lines · Ross
Quoted 2 lines · Andrew Geissler

Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?

Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.

Sure, I'll get some patches tested out and uploaded based on your Py3 changes.

Quoted 2 lines

Ross

#8 lore raw
In reply to Andrew Geissler

Sorry about the last email, I just saw this after sending the other one, yes I believe this might work, it would give us more control on how we handle packaging, although in a way it assumes the user knows what its doing which is not always the case :)

Cheers,

Alejandro

Quoted 24 lines · Andrew Geissler
Quoted 12 lines · Ross
Quoted 11 lines · Ross
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Quoted 7 lines

So I went and did this.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest

Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.

Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?

Quoted 1 line

Ross

Acked-by Alejandro Enedino Hernandez Samaniego
#9 lore raw
In reply to Burton, Ross

Thanks Ross,

Quoted 21 lines · Ross
Quoted 11 lines · Ross
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

So I went and did this.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest

Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.

Ross

#10 lore raw
In reply to Burton, Ross

Hey Ross,

I believe that if we want to go with this approach we would need to add these packages as "special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such as lib/python/email shouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly  with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files on do_install()

Cheers,

Alejandro

Quoted 13 lines · Ross
Quoted 2 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Ross

#11 lore raw

We should probably do another dig at ridding oe-core of py2 :)

Alex

2018-09-14 18:35 GMT+02:00 Alejandro Hernandez <[email protected]>:

Quoted 43 lines

Hey Ross,

I believe that if we want to go with this approach we would need to add these packages as "special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such as lib/python/email shouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files on do_install()

Cheers,

Alejandro

Quoted 11 lines · Ross
Quoted 3 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

Quoted 5 lines

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Ross

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#12 lore raw
Quoted 2 lines · Alexander Kanavin

We should probably do another dig at ridding oe-core of py2 :)

Uh-oh! Guess we better get the Py3 update on our list.

Quoted 48 lines

Alex

2018-09-14 18:35 GMT+02:00 Alejandro Hernandez <[email protected]>:

Quoted 43 lines

Hey Ross,

I believe that if we want to go with this approach we would need to add these packages as "special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such as lib/python/email shouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files on do_install()

Cheers,

Alejandro

Quoted 11 lines · Ross
Quoted 3 lines · Ross

I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.

Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.

Quoted 5 lines

I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.

Ross

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#13 lore raw

2018-09-14 19:36 GMT+02:00 Andrew Geissler <[email protected]>:

Quoted 3 lines
Quoted 1 line

We should probably do another dig at ridding oe-core of py2 :)

Uh-oh! Guess we better get the Py3 update on our list.

I meant that oe-core will not depend on py2 for anything. The py2 recipe will be provided in oe-core for quite a while yet, but you should keep in mind the official message from upstream:

"Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support."

Alex