python: Restructure python packaging and replace it with autopackaging
PATCH 1/3 python: Restructure python packaging and replace it with autopackaging
In reply to a message not in the archiveThe reason we have a manifest file for python is that our goal is to keep python-core as small as posible and add other python packages only when the user needs them, hence why we split upstream python into several packages.
Although our manifest file has several issues:
- Its unorganized and hard to read and understand it for an average human being.
- Git throws an error every single time a patch to the manifest is submitted.
- It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible.
- The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one.
This patch fixes those issues, while adding some additional features.
Features/Fixes:
- A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time.
- It provides an automatic manifest creation task (explained below), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity.
- Dependencies are also checked automagically for new packages (explained below).
- Fixes the manifest in the following ways:
* python-core should be base and all packages should depend on it , fixes lang, string, codecs, etc.
* Fixes packages with repeated files (e.g. bssdb and db, or netclient and mime, and many others).
- Removes the manifest from the python-native recipe (Why was it there in the first place?, native recipes do not get split).
- It creates a solution for users that want precompiled bytecode files (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not.
- Fixes python-core for dependencies, e.g.
When python is run on an image, it TRIES to import everything it needs, but it doesnt necessarily fails when it doesnt find something, so even if we didnt know, we had errors like (trimmed on purpose):
# trying /usr/lib/python2.7/_locale.so # trying /usr/lib/python2.7/lib-dynload/_locale.so # trying /usr/lib/python2.7/_sysconfigdata.so
while it didnt complain about _locale it should have imported it, after creating a new manifest with the automated script we get:
# trying /usr/lib/python2.7/lib-dynload/_locale.so
dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.soHow to use (after a new release of python, or maybe before every OE release):
- A new task called create_manifest was added to the python package, which may be invoked via:
$ bitbake python -c create_manifest
This task runs a script on native python on our HOST system, this script is called create_manifest.py and it in a very simplistic way it does the following:
1. Reads the JSON manifest file and creates a dictionary data structure with all of our python packages, their FILES, RDEPENDS and SUMMARY.
2. Loops through all of them and runs them asynchronously, determining every dependency that they have.
3. These module dependencies are then handled, to be able to know which packages contain those files and which should RDEPEND on one another.
4. The data structure that comes out of this, is then used to create a new manifest file which is automatically copied onto the user's python directory replacing the old one.
Create_manifest script features:
- Handles modules which dont exist anymore (new release for example).
- Handles modules that are builtin.
- Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
- This method works for both python modules and shared libraries used by python.
- Deals with packages which include folders.
- Deals with packages which include FILES with a wildcard.
- The manifest can be constructed on a multilib environment as well.
How to add a new package:
- If a user wants to add a new package all that has to be done is modify the python2-manifest.json file, and add the required file(s) to the FILES list, the script should handle all the rest. Real example:
"webbrowser": {
"files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
"rdepends": [],
"summary": "Python Web browser support"} Run bitbake python -c create_manifest and the resulting manifest should be completed after a few seconds, showing something like:
"webbrowser": {
"files": ["${libdir}/python2.7/webbrowser.py"],
"rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
"summary": "Python Web browser support"}Known errors/issues:
- Some special packages are handled differently: core, misc, modules,dev, staticdev.
All these should be handled manually, because they either include binaries, static libraries, include files, etc. Specifically static libraries are not not supported by this method and have to be handled by the user.
- After the manifest files changes, if not building from a clean environment bitbake throws an error complaining about the basehash from the package task mismatching (which makes sense, since the JSON file was parsed and it changed the PACKAGES variable), this can be solved by either building on a clean environment or by deleting the cache file located at tmp/cache/default-glibc/... This error should only happen once and when a the manifest file changed.
- The change should be transparent to the user, other than the fact that now we CANT build python-foo (it was pretty dumb anyway, since what building python-foo actually did was building the whole python package anyway), but doing IMAGE_INSTALL_append = " python-foo" would create an image with the requested package with no issues.
[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]
--- .../python/python-2.7-manifest.inc | 287 ------ .../python/python-native_2.7.13.bb | 3 - .../python/python/create_manifest2.py | 274 ++++++ .../python/python/get_module_deps2.py | 110 +++ .../python/python/python2-manifest.json | 1033 ++++++++++++++++++++ .../python/python/sitecustomize.py | 8 - meta/recipes-devtools/python/python_2.7.13.bb | 82 +- scripts/contrib/python/generate-manifest-2.7.py | 418 -------- 8 files changed, 1497 insertions(+), 718 deletions(-) delete mode 100644 meta/recipes-devtools/python/python-2.7-manifest.inc create mode 100644 meta/recipes-devtools/python/python/create_manifest2.py create mode 100644 meta/recipes-devtools/python/python/get_module_deps2.py create mode 100644 meta/recipes-devtools/python/python/python2-manifest.json delete mode 100755 scripts/contrib/python/generate-manifest-2.7.py
meta/recipes-devtools/python/python-2.7-manifest.inc deleted +0 -287
@@ -1,287 +0,0 @@ - -# WARNING: This file is AUTO GENERATED: Manual edits will be lost next time I regenerate the file. -# Generator: '../../../scripts/contrib/python/generate-manifest-2.7.py' Version 20110222.2 (C) 2002-2010 Michael 'Mickey' Lauer <[email protected]> - - - -PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib " - -PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-modules" - -SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator" -RDEPENDS_${PN}-2to3="${PN}-core" -FILES_${PN}-2to3="${bindir}/2to3 ${libdir}/python2.7/lib2to3 " - -SUMMARY_${PN}-argparse="Python command line argument parser" -RDEPENDS_${PN}-argparse="${PN}-core ${PN}-codecs ${PN}-textutils" -FILES_${PN}-argparse="${libdir}/python2.7/argparse.* " - -SUMMARY_${PN}-audio="Python Audio Handling" -RDEPENDS_${PN}-audio="${PN}-core" -FILES_${PN}-audio="${libdir}/python2.7/wave.* ${libdir}/python2.7/chunk.* ${libdir}/python2.7/sndhdr.* ${libdir}/python2.7/lib-dynload/ossaudiodev.so ${libdir}/python2.7/lib-dynload/audioop.so ${libdir}/python2.7/audiodev.* ${libdir}/python2.7/sunaudio.* ${libdir}/python2.7/sunau.* ${libdir}/python2.7/toaiff.* " - -SUMMARY_${PN}-bsddb="Python bindings for the Berkeley Database" -RDEPENDS_${PN}-bsddb="${PN}-core" -FILES_${PN}-bsddb="${libdir}/python2.7/bsddb ${libdir}/python2.7/lib-dynload/_bsddb.so " - -SUMMARY_${PN}-codecs="Python codecs, encodings & i18n support" -RDEPENDS_${PN}-codecs="${PN}-core ${PN}-lang" -FILES_${PN}-codecs="${libdir}/python2.7/codecs.* ${libdir}/python2.7/encodings ${libdir}/python2.7/gettext.* ${libdir}/python2.7/locale.* ${libdir}/python2.7/lib-dynload/_locale.so ${libdir}/python2.7/lib-dynload/_codecs* ${libdir}/python2.7/lib-dynload/_multibytecodec.so ${libdir}/python2.7/lib-dynload/unicodedata.so ${libdir}/python2.7/stringprep.* ${libdir}/python2.7/xdrlib.* " - -SUMMARY_${PN}-compile="Python bytecode compilation support" -RDEPENDS_${PN}-compile="${PN}-core" -FILES_${PN}-compile="${libdir}/python2.7/py_compile.* ${libdir}/python2.7/compileall.* " - -SUMMARY_${PN}-compiler="Python compiler support" -RDEPENDS_${PN}-compiler="${PN}-core" -FILES_${PN}-compiler="${libdir}/python2.7/compiler " - -SUMMARY_${PN}-compression="Python high-level compression support" -RDEPENDS_${PN}-compression="${PN}-core ${PN}-zlib" -FILES_${PN}-compression="${libdir}/python2.7/gzip.* ${libdir}/python2.7/zipfile.* ${libdir}/python2.7/tarfile.* ${libdir}/python2.7/lib-dynload/bz2.so " - -SUMMARY_${PN}-contextlib="Python utilities for with-statementcontexts." -RDEPENDS_${PN}-contextlib="${PN}-core" -FILES_${PN}-contextlib="${libdir}/python${PYTHON_MAJMIN}/contextlib.* " - -SUMMARY_${PN}-core="Python interpreter and core modules" -RDEPENDS_${PN}-core="${PN}-lang ${PN}-re" -FILES_${PN}-core="${libdir}/python2.7/__future__.* ${libdir}/python2.7/_abcoll.* ${libdir}/python2.7/abc.* ${libdir}/python2.7/ast.* ${libdir}/python2.7/copy.* ${libdir}/python2.7/copy_reg.* ${libdir}/python2.7/ConfigParser.* ${libdir}/python2.7/genericpath.* ${libdir}/python2.7/getopt.* ${libdir}/python2.7/linecache.* ${libdir}/python2.7/new.* ${libdir}/python2.7/os.* ${libdir}/python2.7/posixpath.* ${libdir}/python2.7/struct.* ${libdir}/python2.7/warnings.* ${libdir}/python2.7/site.* ${libdir}/python2.7/stat.* ${libdir}/python2.7/UserDict.* ${libdir}/python2.7/UserList.* ${libdir}/python2.7/UserString.* ${libdir}/python2.7/lib-dynload/binascii.so ${libdir}/python2.7/lib-dynload/_struct.so ${libdir}/python2.7/lib-dynload/time.so ${libdir}/python2.7/lib-dynload/xreadlines.so ${libdir}/python2.7/types.* ${libdir}/python2.7/platform.* ${bindir}/python* ${libdir}/python2.7/_weakrefset.* ${libdir}/python2.7/sysconfig.* ${libdir}/python2.7/_sysconfigdata.* ${includedir}/python${PYTHON_MAJMIN}/pyconfig*.h ${libdir}/python${PYTHON_MAJMIN}/sitecustomize.py " - -SUMMARY_${PN}-crypt="Python basic cryptographic and hashing support" -RDEPENDS_${PN}-crypt="${PN}-core" -FILES_${PN}-crypt="${libdir}/python2.7/hashlib.* ${libdir}/python2.7/md5.* ${libdir}/python2.7/sha.* ${libdir}/python2.7/lib-dynload/crypt.so ${libdir}/python2.7/lib-dynload/_hashlib.so ${libdir}/python2.7/lib-dynload/_sha256.so ${libdir}/python2.7/lib-dynload/_sha512.so " - -SUMMARY_${PN}-ctypes="Python C types support" -RDEPENDS_${PN}-ctypes="${PN}-core" -FILES_${PN}-ctypes="${libdir}/python2.7/ctypes ${libdir}/python2.7/lib-dynload/_ctypes.so ${libdir}/python2.7/lib-dynload/_ctypes_test.so " - -SUMMARY_${PN}-curses="Python curses support" -RDEPENDS_${PN}-curses="${PN}-core" -FILES_${PN}-curses="${libdir}/python2.7/curses ${libdir}/python2.7/lib-dynload/_curses.so ${libdir}/python2.7/lib-dynload/_curses_panel.so " - -SUMMARY_${PN}-datetime="Python calendar and time support" -RDEPENDS_${PN}-datetime="${PN}-core ${PN}-codecs" -FILES_${PN}-datetime="${libdir}/python2.7/_strptime.* ${libdir}/python2.7/calendar.* ${libdir}/python2.7/lib-dynload/datetime.so " - -SUMMARY_${PN}-db="Python file-based database support" -RDEPENDS_${PN}-db="${PN}-core" -FILES_${PN}-db="${libdir}/python2.7/anydbm.* ${libdir}/python2.7/dumbdbm.* ${libdir}/python2.7/whichdb.* " - -SUMMARY_${PN}-debugger="Python debugger" -RDEPENDS_${PN}-debugger="${PN}-core ${PN}-io ${PN}-lang ${PN}-re ${PN}-stringold ${PN}-shell ${PN}-pprint" -FILES_${PN}-debugger="${libdir}/python2.7/bdb.* ${libdir}/python2.7/pdb.* " - -SUMMARY_${PN}-dev="Python development package" -RDEPENDS_${PN}-dev="${PN}-core" -FILES_${PN}-dev="${includedir} ${libdir}/lib*${SOLIBSDEV} ${libdir}/*.la ${libdir}/*.a ${libdir}/*.o ${libdir}/pkgconfig ${base_libdir}/*.a ${base_libdir}/*.o ${datadir}/aclocal ${datadir}/pkgconfig ${libdir}/python2.7/config/Makefile " - -SUMMARY_${PN}-difflib="Python helpers for computing deltas between objects" -RDEPENDS_${PN}-difflib="${PN}-lang ${PN}-re" -FILES_${PN}-difflib="${libdir}/python2.7/difflib.* " - -SUMMARY_${PN}-distutils="Python Distribution Utilities" -RDEPENDS_${PN}-distutils="${PN}-core ${PN}-email" -FILES_${PN}-distutils="${libdir}/python2.7/config ${libdir}/python2.7/distutils " - -SUMMARY_${PN}-distutils-staticdev="Python distribution utilities (static libraries)" -RDEPENDS_${PN}-distutils-staticdev="${PN}-distutils" -FILES_${PN}-distutils-staticdev="${libdir}/python2.7/config/lib*.a " - -SUMMARY_${PN}-doctest="Python framework for running examples in docstrings" -RDEPENDS_${PN}-doctest="${PN}-core ${PN}-lang ${PN}-io ${PN}-re ${PN}-unittest ${PN}-debugger ${PN}-difflib" -FILES_${PN}-doctest="${libdir}/python2.7/doctest.* " - -SUMMARY_${PN}-email="Python email support" -RDEPENDS_${PN}-email="${PN}-core ${PN}-io ${PN}-re ${PN}-mime ${PN}-audio ${PN}-image ${PN}-netclient" -FILES_${PN}-email="${libdir}/python2.7/imaplib.* ${libdir}/python2.7/email " - -SUMMARY_${PN}-fcntl="Python's fcntl interface" -RDEPENDS_${PN}-fcntl="${PN}-core" -FILES_${PN}-fcntl="${libdir}/python2.7/lib-dynload/fcntl.so " - -SUMMARY_${PN}-gdbm="Python GNU database support" -RDEPENDS_${PN}-gdbm="${PN}-core" -FILES_${PN}-gdbm="${libdir}/python2.7/lib-dynload/gdbm.so " - -SUMMARY_${PN}-hotshot="Python hotshot performance profiler" -RDEPENDS_${PN}-hotshot="${PN}-core" -FILES_${PN}-hotshot="${libdir}/python2.7/hotshot ${libdir}/python2.7/lib-dynload/_hotshot.so " - -SUMMARY_${PN}-html="Python HTML processing support" -RDEPENDS_${PN}-html="${PN}-core" -FILES_${PN}-html="${libdir}/python2.7/formatter.* ${libdir}/python2.7/htmlentitydefs.* ${libdir}/python2.7/htmllib.* ${libdir}/python2.7/markupbase.* ${libdir}/python2.7/sgmllib.* ${libdir}/python2.7/HTMLParser.* " - -SUMMARY_${PN}-idle="Python Integrated Development Environment" -RDEPENDS_${PN}-idle="${PN}-core ${PN}-tkinter" -FILES_${PN}-idle="${bindir}/idle ${libdir}/python2.7/idlelib " - -SUMMARY_${PN}-image="Python graphical image handling" -RDEPENDS_${PN}-image="${PN}-core" -FILES_${PN}-image="${libdir}/python2.7/colorsys.* ${libdir}/python2.7/imghdr.* ${libdir}/python2.7/lib-dynload/imageop.so ${libdir}/python2.7/lib-dynload/rgbimg.so " - -SUMMARY_${PN}-importlib="Python import implementation library" -RDEPENDS_${PN}-importlib="${PN}-core" -FILES_${PN}-importlib="${libdir}/python2.7/importlib " - -SUMMARY_${PN}-io="Python low-level I/O" -RDEPENDS_${PN}-io="${PN}-core ${PN}-math ${PN}-textutils ${PN}-netclient ${PN}-contextlib" -FILES_${PN}-io="${libdir}/python2.7/lib-dynload/_socket.so ${libdir}/python2.7/lib-dynload/_io.so ${libdir}/python2.7/lib-dynload/_ssl.so ${libdir}/python2.7/lib-dynload/select.so ${libdir}/python2.7/lib-dynload/termios.so ${libdir}/python2.7/lib-dynload/cStringIO.so ${libdir}/python2.7/pipes.* ${libdir}/python2.7/socket.* ${libdir}/python2.7/ssl.* ${libdir}/python2.7/tempfile.* ${libdir}/python2.7/StringIO.* ${libdir}/python2.7/io.* ${libdir}/python2.7/_pyio.* " - -SUMMARY_${PN}-json="Python JSON support" -RDEPENDS_${PN}-json="${PN}-core ${PN}-math ${PN}-re ${PN}-codecs" -FILES_${PN}-json="${libdir}/python2.7/json ${libdir}/python2.7/lib-dynload/_json.so " - -SUMMARY_${PN}-lang="Python low-level language support" -RDEPENDS_${PN}-lang="${PN}-core" -FILES_${PN}-lang="${libdir}/python2.7/lib-dynload/_bisect.so ${libdir}/python2.7/lib-dynload/_collections.so ${libdir}/python2.7/lib-dynload/_heapq.so ${libdir}/python2.7/lib-dynload/_weakref.so ${libdir}/python2.7/lib-dynload/_functools.so ${libdir}/python2.7/lib-dynload/array.so ${libdir}/python2.7/lib-dynload/itertools.so ${libdir}/python2.7/lib-dynload/operator.so ${libdir}/python2.7/lib-dynload/parser.so ${libdir}/python2.7/atexit.* ${libdir}/python2.7/bisect.* ${libdir}/python2.7/code.* ${libdir}/python2.7/codeop.* ${libdir}/python2.7/collections.* ${libdir}/python2.7/dis.* ${libdir}/python2.7/functools.* ${libdir}/python2.7/heapq.* ${libdir}/python2.7/inspect.* ${libdir}/python2.7/keyword.* ${libdir}/python2.7/opcode.* ${libdir}/python2.7/symbol.* ${libdir}/python2.7/repr.* ${libdir}/python2.7/token.* ${libdir}/python2.7/tokenize.* ${libdir}/python2.7/traceback.* ${libdir}/python2.7/weakref.* " - -SUMMARY_${PN}-logging="Python logging support" -RDEPENDS_${PN}-logging="${PN}-core ${PN}-io ${PN}-lang ${PN}-pickle ${PN}-stringold" -FILES_${PN}-logging="${libdir}/python2.7/logging " - -SUMMARY_${PN}-mailbox="Python mailbox format support" -RDEPENDS_${PN}-mailbox="${PN}-core ${PN}-mime" -FILES_${PN}-mailbox="${libdir}/python2.7/mailbox.* " - -SUMMARY_${PN}-math="Python math support" -RDEPENDS_${PN}-math="${PN}-core ${PN}-crypt" -FILES_${PN}-math="${libdir}/python2.7/lib-dynload/cmath.so ${libdir}/python2.7/lib-dynload/math.so ${libdir}/python2.7/lib-dynload/_random.so ${libdir}/python2.7/random.* ${libdir}/python2.7/sets.* " - -SUMMARY_${PN}-mime="Python MIME handling APIs" -RDEPENDS_${PN}-mime="${PN}-core ${PN}-io" -FILES_${PN}-mime="${libdir}/python2.7/mimetools.* ${libdir}/python2.7/uu.* ${libdir}/python2.7/quopri.* ${libdir}/python2.7/rfc822.* ${libdir}/python2.7/MimeWriter.* " - -SUMMARY_${PN}-mmap="Python memory-mapped file support" -RDEPENDS_${PN}-mmap="${PN}-core ${PN}-io" -FILES_${PN}-mmap="${libdir}/python2.7/lib-dynload/mmap.so " - -SUMMARY_${PN}-multiprocessing="Python multiprocessing support" -RDEPENDS_${PN}-multiprocessing="${PN}-core ${PN}-io ${PN}-lang ${PN}-pickle ${PN}-threading ${PN}-ctypes ${PN}-mmap" -FILES_${PN}-multiprocessing="${libdir}/python2.7/lib-dynload/_multiprocessing.so ${libdir}/python2.7/multiprocessing " - -SUMMARY_${PN}-netclient="Python Internet Protocol clients" -RDEPENDS_${PN}-netclient="${PN}-core ${PN}-crypt ${PN}-datetime ${PN}-io ${PN}-lang ${PN}-logging ${PN}-mime" -FILES_${PN}-netclient="${libdir}/python2.7/*Cookie*.* ${libdir}/python2.7/base64.* ${libdir}/python2.7/cookielib.* ${libdir}/python2.7/ftplib.* ${libdir}/python2.7/gopherlib.* ${libdir}/python2.7/hmac.* ${libdir}/python2.7/httplib.* ${libdir}/python2.7/mimetypes.* ${libdir}/python2.7/nntplib.* ${libdir}/python2.7/poplib.* ${libdir}/python2.7/smtplib.* ${libdir}/python2.7/telnetlib.* ${libdir}/python2.7/urllib.* ${libdir}/python2.7/urllib2.* ${libdir}/python2.7/urlparse.* ${libdir}/python2.7/uuid.* ${libdir}/python2.7/rfc822.* ${libdir}/python2.7/mimetools.* " - -SUMMARY_${PN}-netserver="Python Internet Protocol servers" -RDEPENDS_${PN}-netserver="${PN}-core ${PN}-netclient ${PN}-shell ${PN}-threading" -FILES_${PN}-netserver="${libdir}/python2.7/cgi.* ${libdir}/python2.7/*HTTPServer.* ${libdir}/python2.7/SocketServer.* " - -SUMMARY_${PN}-numbers="Python number APIs" -RDEPENDS_${PN}-numbers="${PN}-core ${PN}-lang ${PN}-re" -FILES_${PN}-numbers="${libdir}/python2.7/decimal.* ${libdir}/python2.7/fractions.* ${libdir}/python2.7/numbers.* " - -SUMMARY_${PN}-pickle="Python serialisation/persistence support" -RDEPENDS_${PN}-pickle="${PN}-core ${PN}-codecs ${PN}-io ${PN}-re" -FILES_${PN}-pickle="${libdir}/python2.7/pickle.* ${libdir}/python2.7/shelve.* ${libdir}/python2.7/lib-dynload/cPickle.so ${libdir}/python2.7/pickletools.* " - -SUMMARY_${PN}-pkgutil="Python package extension utility support" -RDEPENDS_${PN}-pkgutil="${PN}-core" -FILES_${PN}-pkgutil="${libdir}/python2.7/pkgutil.* " - -SUMMARY_${PN}-plistlib="Generate and parse Mac OS X .plist files" -RDEPENDS_${PN}-plistlib="${PN}-core ${PN}-datetime ${PN}-io" -FILES_${PN}-plistlib="${libdir}/python2.7/plistlib.* " - -SUMMARY_${PN}-pprint="Python pretty-print support" -RDEPENDS_${PN}-pprint="${PN}-core ${PN}-io" -FILES_${PN}-pprint="${libdir}/python2.7/pprint.* " - -SUMMARY_${PN}-profile="Python basic performance profiling support" -RDEPENDS_${PN}-profile="${PN}-core ${PN}-textutils" -FILES_${PN}-profile="${libdir}/python2.7/profile.* ${libdir}/python2.7/pstats.* ${libdir}/python2.7/cProfile.* ${libdir}/python2.7/lib-dynload/_lsprof.so " - -SUMMARY_${PN}-pydoc="Python interactive help support" -RDEPENDS_${PN}-pydoc="${PN}-core ${PN}-lang ${PN}-stringold ${PN}-re" -FILES_${PN}-pydoc="${bindir}/pydoc ${libdir}/python2.7/pydoc.* ${libdir}/python2.7/pydoc_data " - -SUMMARY_${PN}-re="Python Regular Expression APIs" -RDEPENDS_${PN}-re="${PN}-core" -FILES_${PN}-re="${libdir}/python2.7/re.* ${libdir}/python2.7/sre.* ${libdir}/python2.7/sre_compile.* ${libdir}/python2.7/sre_constants* ${libdir}/python2.7/sre_parse.* " - -SUMMARY_${PN}-readline="Python readline support" -RDEPENDS_${PN}-readline="${PN}-core" -FILES_${PN}-readline="${libdir}/python2.7/lib-dynload/readline.so ${libdir}/python2.7/rlcompleter.* " - -SUMMARY_${PN}-resource="Python resource control interface" -RDEPENDS_${PN}-resource="${PN}-core" -FILES_${PN}-resource="${libdir}/python2.7/lib-dynload/resource.so " - -SUMMARY_${PN}-robotparser="Python robots.txt parser" -RDEPENDS_${PN}-robotparser="${PN}-core ${PN}-netclient" -FILES_${PN}-robotparser="${libdir}/python2.7/robotparser.* " - -SUMMARY_${PN}-shell="Python shell-like functionality" -RDEPENDS_${PN}-shell="${PN}-core ${PN}-re" -FILES_${PN}-shell="${libdir}/python2.7/cmd.* ${libdir}/python2.7/commands.* ${libdir}/python2.7/dircache.* ${libdir}/python2.7/fnmatch.* ${libdir}/python2.7/glob.* ${libdir}/python2.7/popen2.* ${libdir}/python2.7/shlex.* ${libdir}/python2.7/shutil.* " - -SUMMARY_${PN}-smtpd="Python Simple Mail Transport Daemon" -RDEPENDS_${PN}-smtpd="${PN}-core ${PN}-netserver ${PN}-email ${PN}-mime" -FILES_${PN}-smtpd="${bindir}/smtpd.* ${libdir}/python2.7/smtpd.* " - -SUMMARY_${PN}-sqlite3="Python Sqlite3 database support" -RDEPENDS_${PN}-sqlite3="${PN}-core ${PN}-datetime ${PN}-lang ${PN}-crypt ${PN}-io ${PN}-threading ${PN}-zlib" -FILES_${PN}-sqlite3="${libdir}/python2.7/lib-dynload/_sqlite3.so ${libdir}/python2.7/sqlite3/dbapi2.* ${libdir}/python2.7/sqlite3/__init__.* ${libdir}/python2.7/sqlite3/dump.* " - -SUMMARY_${PN}-sqlite3-tests="Python Sqlite3 database support tests" -RDEPENDS_${PN}-sqlite3-tests="${PN}-core ${PN}-sqlite3" -FILES_${PN}-sqlite3-tests="${libdir}/python2.7/sqlite3/test " - -SUMMARY_${PN}-stringold="Python string APIs [deprecated]" -RDEPENDS_${PN}-stringold="${PN}-core ${PN}-re" -FILES_${PN}-stringold="${libdir}/python2.7/lib-dynload/strop.so ${libdir}/python2.7/string.* ${libdir}/python2.7/stringold.* " - -SUMMARY_${PN}-subprocess="Python subprocess support" -RDEPENDS_${PN}-subprocess="${PN}-core ${PN}-io ${PN}-re ${PN}-fcntl ${PN}-pickle" -FILES_${PN}-subprocess="${libdir}/python2.7/subprocess.* " - -SUMMARY_${PN}-syslog="Python syslog interface" -RDEPENDS_${PN}-syslog="${PN}-core" -FILES_${PN}-syslog="${libdir}/python2.7/lib-dynload/syslog.so " - -SUMMARY_${PN}-terminal="Python terminal controlling support" -RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io" -FILES_${PN}-terminal="${libdir}/python2.7/pty.* ${libdir}/python2.7/tty.* " - -SUMMARY_${PN}-tests="Python tests" -RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules" -FILES_${PN}-tests="${libdir}/python2.7/test " - -SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support" -RDEPENDS_${PN}-textutils="${PN}-core ${PN}-io ${PN}-re ${PN}-stringold" -FILES_${PN}-textutils="${libdir}/python2.7/lib-dynload/_csv.so ${libdir}/python2.7/csv.* ${libdir}/python2.7/optparse.* ${libdir}/python2.7/textwrap.* " - -SUMMARY_${PN}-threading="Python threading & synchronization support" -RDEPENDS_${PN}-threading="${PN}-core ${PN}-lang" -FILES_${PN}-threading="${libdir}/python2.7/_threading_local.* ${libdir}/python2.7/dummy_thread.* ${libdir}/python2.7/dummy_threading.* ${libdir}/python2.7/mutex.* ${libdir}/python2.7/threading.* ${libdir}/python2.7/Queue.* " - -SUMMARY_${PN}-tkinter="Python Tcl/Tk bindings" -RDEPENDS_${PN}-tkinter="${PN}-core" -FILES_${PN}-tkinter="${libdir}/python2.7/lib-dynload/_tkinter.so ${libdir}/python2.7/lib-tk " - -SUMMARY_${PN}-unittest="Python unit testing framework" -RDEPENDS_${PN}-unittest="${PN}-core ${PN}-stringold ${PN}-lang ${PN}-io ${PN}-difflib ${PN}-pprint ${PN}-shell" -FILES_${PN}-unittest="${libdir}/python2.7/unittest/ " - -SUMMARY_${PN}-unixadmin="Python Unix administration support" -RDEPENDS_${PN}-unixadmin="${PN}-core" -FILES_${PN}-unixadmin="${libdir}/python2.7/lib-dynload/nis.so ${libdir}/python2.7/lib-dynload/grp.so ${libdir}/python2.7/lib-dynload/pwd.so ${libdir}/python2.7/getpass.* " - -SUMMARY_${PN}-xml="Python basic XML support" -RDEPENDS_${PN}-xml="${PN}-core ${PN}-re" -FILES_${PN}-xml="${libdir}/python2.7/lib-dynload/_elementtree.so ${libdir}/python2.7/lib-dynload/pyexpat.so ${libdir}/python2.7/xml ${libdir}/python2.7/xmllib.* " - -SUMMARY_${PN}-xmlrpc="Python XML-RPC support" -RDEPENDS_${PN}-xmlrpc="${PN}-core ${PN}-xml ${PN}-netserver ${PN}-lang" -FILES_${PN}-xmlrpc="${libdir}/python2.7/xmlrpclib.* ${libdir}/python2.7/SimpleXMLRPCServer.* ${libdir}/python2.7/DocXMLRPCServer.* " - -SUMMARY_${PN}-zlib="Python zlib compression support" -RDEPENDS_${PN}-zlib="${PN}-core" -FILES_${PN}-zlib="${libdir}/python2.7/lib-dynload/zlib.so " - -SUMMARY_${PN}-modules="All Python modules" -RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib " -ALLOW_EMPTY_${PN}-modules = "1" - -
meta/recipes-devtools/python/python-native_2.7.13.bb +0 -3
@@ -1,5 +1,4 @@ require python.inc - EXTRANATIVEPATH += "bzip2-native" DEPENDS = "openssl-native bzip2-replacement-native zlib-native readline-native sqlite3-native expat-native" PR = "${INC_PR}.1"
@@ -25,8 +24,6 @@ FILESEXTRAPATHS =. "${FILE_DIRNAME}/${PN}:" inherit native -require python-native-${PYTHON_MAJMIN}-manifest.inc - EXTRA_OECONF_append = " --bindir=${bindir}/${PN} --with-system-expat=${STAGING_DIR_HOST}" EXTRA_OEMAKE = '\
meta/recipes-devtools/python/python/create_manifest2.py new file +274 -0
274 lines not on this page · raw message
meta/recipes-devtools/python/python/get_module_deps2.py new file +110 -0
110 lines not on this page · raw message
meta/recipes-devtools/python/python/python2-manifest.json new file +1033 -0
1,034 lines not on this page · raw message
meta/recipes-devtools/python/python/sitecustomize.py +0 -8
@@ -27,19 +27,11 @@ def __enableReadlineSupport(): except IOError: pass -def __enableDefaultEncoding(): - import sys - try: - sys.setdefaultencoding( "utf8" ) - except LookupError: - pass - import sys try: import rlcompleter, readline except ImportError: pass else: - __enableDefaultEncoding() __registerExitHandler() __enableReadlineSupport()
meta/recipes-devtools/python/python_2.7.13.bb +80 -2
96 lines not on this page · raw message
scripts/contrib/python/generate-manifest-2.7.py deleted +0 -418
418 lines not on this page · raw message
Signature · 1 line
2.12.3
Quoted 6 lines · On Thu, 17 Aug 2017 03:24:23 -0700 Alejandro Hernandez…
- It creates a solution for users that want precompiled bytecode files (*.pyc)
INCLUDE_PYCS= "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not.
A serious problem with not shipping precompiled pyc and/or pyo files is that, once generated on first run, package managers won't uninstall them.
This can cause many undesired effects, because python programs will still be able to import these files, but dependencies like graphics or shared libraries may have been deleted by the package manager.
Also, not being able to uninstall something is bad on its own.
So, if anything, I believe INCLUDE_PYCS = "1" would be the only sane default, unless no package management was involved at all.
And no read-only filesystem, for what it's worth.
Best regards,
Andreas
P.S.: I didn't see patches 2 and 3. Is it just an incorrect subject?
Hey Andreas,
Quoted 16 lines · Andreas Oberritter
Quoted 6 lines · On Thu, 17 Aug 2017 03:24:23 -0700 Alejandro Hernandez…
- It creates a solution for users that want precompiled bytecode files (*.pyc)
INCLUDE_PYCS= "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not.A serious problem with not shipping precompiled pyc and/or pyo files is that, once generated on first run, package managers won't uninstall them.
This can cause many undesired effects, because python programs will still be able to import these files, but dependencies like graphics or shared libraries may have been deleted by the package manager.
Quoted 6 lines
Also, not being able to uninstall something is bad on its own.
So, if anything, I believe
INCLUDE_PYCS= "1" would be the only sane default, unless no package management was involved at all.And no read-only filesystem, for what it's worth.
Interesting, I didn't know that happened, sounds like a fair argument, I will make sure to set INCLUDE_PYCS = "1" by default to avoid problems like this, and let the user set it to 0 if thats what they want.
Quoted 5 lines
Best regards,
Andreas
P.S.: I didn't see patches 2 and 3. Is it just an incorrect subject?
Yes, there's 2 other patches, which are used for python3, but somehow I forgot to send them, it was 4 am my apologies, will send them in a bit.
Regards,
Alejandro