python/python3: Move test files to test package
PATCH v2 1/1 python/python3: Move test files to test package
A quick search in the python package shows the following test directories being included in the python image:
./Python-2.7.14/Lib/bsddb/test ./Python-2.7.14/Lib/sqlite3/test ./Python-2.7.14/Lib/ctypes/test ./Python-2.7.14/Lib/unittest/test ./Python-2.7.14/Lib/lib-tk/test ./Python-2.7.14/Lib/email/test
Move the bsddb, ctypes, and email packages to the python-tests package to conserve some space.
Packages not include in this patch:
- sqlite3 has it's own -tests package
- unittest is already a test package
- lib-tk (tkinter) seems to have its own special installation and does not allow the manifest override for its test directory
- bsddb is no longer in python3 so update for it was not needed there
Just the ctypes and email change saved us ~150KB of read-only filesystem space in our embedded env.
Changes since v1:
- Used manifest file instead of python recipe to move files to -tests package
--- meta/recipes-devtools/python/python/python2-manifest.json | 5 ++++- meta/recipes-devtools/python/python3/python3-manifest.json | 4 +++- 2 files changed, 7 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python/python2-manifest.json +4 -1
@@ -925,7 +925,10 @@ }, "tests": { "files": [ - "${libdir}/python2.7/test" + "${libdir}/python2.7/test", + "${libdir}/python2.7/email/test", + "${libdir}/python2.7/ctypes/test", + "${libdir}/python2.7/bsddb/test" ], "rdepends": [ "core"
meta/recipes-devtools/python/python3/python3-manifest.json +3 -1
@@ -1040,7 +1040,9 @@ "tests": { "cached": [], "files": [ - "${libdir}/python${PYTHON_MAJMIN}/test" + "${libdir}/python${PYTHON_MAJMIN}/test", + "${libdir}/python${PYTHON_MAJMIN}/email/test", + "${libdir}/python${PYTHON_MAJMIN}/ctypes/test" ], "rdepends": [ "core",
Signature · 1 line
2.7.4
A little oe-pkgdata-util[1] managed to give me a good review of some missing entries here.
python3-asyncio: /usr/lib/python3.5/asyncio/test_utils.py
python3-2to3: /usr/lib/python3.5/lib2to3/tests/test_*.py
python3-tkinter: /usr/lib/python3.5/tkinter/test/
python3-unittest: /usr/lib/python3.5/unittest/test/ (the test suite for unittest should be in python3-test)
python3-distutils: /usr/lib/python3.5/distutils/tests/test_*
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Ross
[1] oe-pkgdata-util find-path '/usr/lib/python*/**/test_*.py'
Quoted 66 lines · Andrew Geissler
A quick search in the python package shows the following test directories being included in the python image:
./Python-2.7.14/Lib/bsddb/test ./Python-2.7.14/Lib/sqlite3/test ./Python-2.7.14/Lib/ctypes/test ./Python-2.7.14/Lib/unittest/test ./Python-2.7.14/Lib/lib-tk/test ./Python-2.7.14/Lib/email/testMove the bsddb, ctypes, and email packages to the python-tests package to conserve some space.
Packages not include in this patch:
- sqlite3 has it's own -tests package
- unittest is already a test package
- lib-tk (tkinter) seems to have its own special installation and does not allow the manifest override for its test directory
- bsddb is no longer in python3 so update for it was not needed thereJust the ctypes and email change saved us ~150KB of read-only filesystem space in our embedded env.
Changes since v1:
- Used manifest file instead of python recipe to move files to -tests packageSigned-off-by Andrew Geissler <[email protected]>--- meta/recipes-devtools/python/python/python2-manifest.json | 5 ++++- meta/recipes-devtools/python/python3/python3-manifest.json | 4 +++- 2 files changed, 7 insertions(+), 2 deletions(-)meta/recipes-devtools/python/python/python2-manifest.json +4
-1@@ -925,7 +925,10 @@ }, "tests": { "files": [ - "${libdir}/python2.7/test" + "${libdir}/python2.7/test", + "${libdir}/python2.7/email/test", + "${libdir}/python2.7/ctypes/test", + "${libdir}/python2.7/bsddb/test" ], "rdepends": [ "core"meta/recipes-devtools/python/python3/python3-manifest.json +3
-2@@ -1040,7 +1040,9 @@ "tests": { "cached": [], "files": [ - "${libdir}/python${PYTHON_MAJMIN}/test" + "${libdir}/python${PYTHON_MAJMIN}/test", + "${libdir}/python${PYTHON_MAJMIN}/email/test", + "${libdir}/python${PYTHON_MAJMIN}/ctypes/test" ], "rdepends": [ "core", --2.7.4
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Ross
Quoted 4 lines · Ross
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Quoted 8 lines
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
So I went and did this.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest
Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.
Ross
Quoted 15 lines · Ross
Quoted 4 lines · Ross
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Quoted 8 lines
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Quoted 7 lines
So I went and did this.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest
Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.
Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?
Quoted 2 lines
Ross
Quoted 2 lines · Andrew Geissler
Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?
Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.
Ross
To be fair, I have also not had a chance to update the py2 manifest script with the latest changes I made to py3 either, but I think applying these to create_manifest2 should be trivial.
Alejandro
Quoted 8 lines · Ross
Quoted 2 lines · Andrew Geissler
Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?
Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.
Ross
Quoted 8 lines · Ross
Quoted 2 lines · Andrew Geissler
Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?
Struggling to be motivated for Py2 to be honest. Do you fancy duplicating the changes for Py2? Alternatively I'll get around to doing it next week.
Sure, I'll get some patches tested out and uploaded based on your Py3 changes.
Quoted 2 lines
Ross
Sorry about the last email, I just saw this after sending the other one, yes I believe this might work, it would give us more control on how we handle packaging, although in a way it assumes the user knows what its doing which is not always the case :)
Cheers,
Alejandro
Quoted 24 lines · Andrew Geissler
Quoted 12 lines · Ross
Quoted 11 lines · Ross
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Quoted 7 lines
So I went and did this.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest
Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.
Nice, thanks Ross! Looks good to me. Will you be back-porting to python2 as well?
Quoted 1 line
Ross
Alejandro Hernandez
Acked-by Alejandro Enedino Hernandez SamaniegoThanks Ross,
Quoted 21 lines · Ross
Quoted 11 lines · Ross
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
So I went and did this.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ross/manifest
Does anyone have a problem with this? Basically all the test code is merged into python3-test, including the previously separate sqlite3-test.
Ross
Hey Ross,
I believe that if we want to go with this approach we would need to add these packages as "special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such as lib/python/email shouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files on do_install()
Cheers,
Alejandro
Quoted 13 lines · Ross
Quoted 2 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Ross
We should probably do another dig at ridding oe-core of py2 :)
Alex
2018-09-14 18:35 GMT+02:00 Alejandro Hernandez <[email protected]>:
Quoted 43 lines
Hey Ross,
I believe that if we want to go with this approach we would need to add these packages as "
special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such aslib/python/emailshouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files ondo_install()Cheers,
Alejandro
Quoted 11 lines · Ross
Quoted 3 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
Quoted 5 lines
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Ross
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 2 lines · Alexander Kanavin
We should probably do another dig at ridding oe-core of py2 :)
Uh-oh! Guess we better get the Py3 update on our list.
Quoted 48 lines
Alex
2018-09-14 18:35 GMT+02:00 Alejandro Hernandez <[email protected]>:
Quoted 43 lines
Hey Ross,
I believe that if we want to go with this approach we would need to add these packages as "
special_packages" on the manifest script, that way the script will simply avoid checking dependencies for them and will pass them without modifications, this will also cause these packages to appear earlier on the list, it does require some tweaking because directories such aslib/python/emailshouldnt be on the email package itself, so we would need to take out such directories from the packages and rerun the script, that will repopulate the email package (ad others) correctly with the files needed instead of the directory, but to be quite honest it might be easier to go with the first approach and delete such files ondo_install()Cheers,
Alejandro
Quoted 11 lines · Ross
Quoted 3 lines · Ross
I suspect simply adding python3.5/*/test/ will catch the bulk of this now and in the future.
Gave this a go and annoyingly the manifest generation must sort the list because the glob works for everything apart from packages after tests alphabetically, so the tkinter and unittest tests don't get moved.
Quoted 5 lines
I guess we could change the manifest to put the tests package at the top of the list, and be sure to respect the order when generating the lists.
Ross
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
2018-09-14 19:36 GMT+02:00 Andrew Geissler <[email protected]>:
Quoted 3 lines
Quoted 1 line
We should probably do another dig at ridding oe-core of py2 :)
Uh-oh! Guess we better get the Py3 update on our list.
I meant that oe-core will not depend on py2 for anything. The py2 recipe will be provided in oe-core for quite a while yet, but you should keep in mind the official message from upstream:
"Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support."
Alex