support profile-optimized build for Python
v2 cover support profile-optimized build for Python
This patchset makes it possible to make a PGO (profile-guided-optimization) build of python. This version of the patchset is almost identical to v1 submitted back in February, with these changes:
- rebased on top of latest oe-core master
- exclude profile data for Modules/posixmodule of Python 2.7 as it was not working correctly
[YOCTO #9338]
The following changes since commit 123962018251dfb1d6ca5aa5c0d02534007de3ab:
build-appliance-image: Update to master head revision (2017-05-01 08:56:43 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338
Markus Lehtonen (20):
python-native: support profile optimized build
python: add python-profile-opt recipe
python: remove path hack from setup.py
python-profile-opt: rename libpython
devtools/images: add python-pgo-image
python: make profile-optimized build possible
python-pgo-image: exclude tests from the default profile target
python: add python-tools subpackage
python-pgo-image: switch python default profile task to pybench
python-pgo-image: enable sstate for do_profile
python3-native: support profile optimized build
python3: fix depends of python-tests
python3: add python-profile-opt3 recipe
python-pgo-image: profiling for python3
python3: remove two setup.py cross-compile hacks
python3: support profile optimized build
python3: fix profile-optimized build of modules
python-pgo-image: exclude tests from the python3 profile target
python3: add python3-tools subpackage
python-pgo-image: change python3 profile target to pybench
meta/recipes-devtools/images/python-pgo-image.bb | 121 +++++++++++++++++++++ .../python/python-2.7-manifest.inc | 8 +- .../python/python-3.5-manifest.inc | 10 +- ...x-.so-loading-when-when-running-profile-t.patch | 26 +++++ .../python/python-native_2.7.13.bb | 10 ++ meta/recipes-devtools/python/python-profile-opt3 | 1 + .../python/python-profile-opt3_3.5.2.bb | 15 +++ .../python/python-profile-opt_2.7.13.bb | 13 +++ .../01-use-proper-tools-for-cross-build.patch | 10 -- ...rename-libpython-to-libpython-profile-opt.patch | 94 ++++++++++++++++ .../python/python3-native_3.5.2.bb | 9 ++ .../python/python3/000-cross-compile.patch | 9 -- ...efile-add-install_generate_profile-target.patch | 25 +++++ ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++ .../python3-setup.py-no-host-headers-libs.patch | 33 ------ ...name-libpython3-to-libpython-profile-opt3.patch | 111 +++++++++++++++++++ .../setup.py-find-libraries-in-staging-dirs.patch | 30 ----- meta/recipes-devtools/python/python3_3.5.2.bb | 74 +++++++++---- meta/recipes-devtools/python/python_2.7.13.bb | 54 +++++++-- 19 files changed, 590 insertions(+), 119 deletions(-) create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb create mode 100644 meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch create mode 120000 meta/recipes-devtools/python/python-profile-opt3 create mode 100644 meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb create mode 100644 meta/recipes-devtools/python/python-profile-opt_2.7.13.bb create mode 100644 meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
Signature · 1 line
2.12.0
PATCH v2 1/20 python-native: support profile optimized build
Make it possible to build python-native with profile directed optimization enabled. The feature is enabled by setting PYTHON_NATIVE_PROFILE_OPT = "1" in local.conf.
The profile task to run may be specified with PYTHON_NATIVE_PROFILE_TASK variable in local.conf, e.g.PYTHON_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- ...x-.so-loading-when-when-running-profile-t.patch | 26 ++++++++++++++++++++++ .../python/python-native_2.7.13.bb | 10 +++++++++ 2 files changed, 36 insertions(+) create mode 100644 meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch
meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch new file +26 -0
@@ -0,0 +1,26 @@ +Makefile: fix .so loading when when running profile task + +Use correct search path when doing profiling for profile-optimized +build. This way we load the correct Python libraries, i.e. the ones from +our build directory, and not the ones from the host os. + +Upstream-Status: Pending + +Signed-off-by: Markus Lehtonen <[email protected]> + +diff --git a/Makefile.pre.in b/Makefile.pre.in +index 2affc41..4355e7f 100644 +--- a/Makefile.pre.in ++++ b/Makefile.pre.in +@@ -458,7 +458,7 @@ build_all_generate_profile: + + run_profile_task: + : # FIXME: can't run for a cross build +- $(LLVM_PROF_FILE) ./$(BUILDPYTHON) $(PROFILE_TASK) || true ++ $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true + + build_all_merge_profile: + $(LLVM_PROF_MERGER) +-- +2.6.6 +
meta/recipes-devtools/python/python-native_2.7.13.bb +10 -0
@@ -17,6 +17,7 @@ SRC_URI += "\ file://builddir.patch \ file://parallel-makeinst-create-bindir.patch \ file://revert_use_of_sysconfigdata.patch \ + file://Makefile-fix-.so-loading-when-when-running-profile-t.patch \ " S = "${WORKDIR}/Python-${PV}"
@@ -35,10 +36,19 @@ EXTRA_OEMAKE = '\ STAGING_INCDIR=${STAGING_INCDIR_NATIVE} \ ' +PYTHON_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON_NATIVE_PROFILE_OPT', True) == '1' else ''}" + do_configure_append() { autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi } +do_compile() { + if [ -n "${PYTHON_NATIVE_PROFILE_TASK}" ]; then + sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON_NATIVE_PROFILE_TASK},g' Makefile + fi + oe_runmake ${PYTHON_NATIVE_MAKE_TARGET} +} + do_install() { oe_runmake 'DESTDIR=${D}' install install -d ${D}${bindir}/${PN}
Signature · 1 line
2.12.0
PATCH v2 2/20 python: add python-profile-opt recipe
In reply to Markus LehtonenThis patch adds a new recipe that builds a special version of python that produces profile data for profile guided optimization. It will install under /opt in order to prevent clash with the "normal" python. The new recipe directly includes the base python recipe so that they are build in as similar way as possible and this hopefully decreases the recipe maintenance burden, too.
Normally, python profile-guided-optimization is done simply by doing "make profile-opt" which first builds python with profile instrumentation enabled, then runs a profile task to get the profile data, and last, re-builds python with profile data guiding the optimization. However, in our cross-build environment this gets a lot trickier. We need to split out the steps as building is done on the build host but we need to run the second step (i.e. run the profile task) on the target hardware. This patch enables the first step, i.e. building python with profile instrumentation enabled
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- .../python/python-profile-opt_2.7.13.bb | 17 ++++++++++++++ meta/recipes-devtools/python/python_2.7.13.bb | 26 +++++++++++++++------- 2 files changed, 35 insertions(+), 8 deletions(-) create mode 100644 meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
meta/recipes-devtools/python/python-profile-opt_2.7.13.bb new file +17 -0
@@ -0,0 +1,17 @@ +require python_${PV}.bb + +PROVIDES += "libpython2" +RPROVIDES_${PN}-core += "libpython2" + +# Use special prefix in order to prevent clash with normal python +STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include" +STAGING_LIBDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/${baselib}" +TARGET_CFLAGS += "-I${STAGING_INCDIR_DEFAULT}" +TARGET_CPPFLAGS += "-I${STAGING_INCDIR_DEFAULT}" +prefix = "/opt" +exec_prefix = "/opt" + +PYTHON_MAKE_TARGET = "build_all_generate_profile" + +# Force pkgdata not to clash with python recipe +PKG_libpython-profile-opt2 = "libpython-profile-opt2.7-1.0"
meta/recipes-devtools/python/python_2.7.13.bb +18 -8
@@ -37,6 +37,11 @@ CONFIGUREOPTS += " --with-system-ffi " EXTRA_OECONF += "ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no" +# These are needed in order to build with modified prefix (python-prorile-opt +# recipe) +STAGING_INCDIR_DEFAULT ?= "${STAGING_INCDIR}" +STAGING_LIBDIR_DEFAULT ?= "${STAGING_LIBDIR}" + do_configure_append() { rm -f ${S}/Makefile.orig autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -45,12 +50,12 @@ do_configure_append() { do_compile() { # regenerate platform specific files, because they depend on system headers cd ${S}/Lib/plat-linux2 - include=${STAGING_INCDIR} ${STAGING_BINDIR_NATIVE}/python-native/python \ + include=${STAGING_INCDIR_DEFAULT} ${STAGING_BINDIR_NATIVE}/python-native/python \ ${S}/Tools/scripts/h2py.py -i '(u_long)' \ - ${STAGING_INCDIR}/dlfcn.h \ - ${STAGING_INCDIR}/linux/cdrom.h \ - ${STAGING_INCDIR}/netinet/in.h \ - ${STAGING_INCDIR}/sys/types.h + ${STAGING_INCDIR_DEFAULT}/dlfcn.h \ + ${STAGING_INCDIR_DEFAULT}/linux/cdrom.h \ + ${STAGING_INCDIR_DEFAULT}/netinet/in.h \ + ${STAGING_INCDIR_DEFAULT}/sys/types.h sed -e 's,${STAGING_DIR_HOST},,g' -i *.py cd -
@@ -60,7 +65,7 @@ do_compile() { if [ ! -f Makefile.orig ]; then install -m 0644 Makefile Makefile.orig fi - sed -i -e 's#^LDFLAGS=.*#LDFLAGS=${LDFLAGS} -L. -L${STAGING_LIBDIR}#g' \ + sed -i -e 's#^LDFLAGS=.*#LDFLAGS=${LDFLAGS} -L. -L${STAGING_LIBDIR_DEFAULT}#g' \ -e 's,libdir=${libdir},libdir=${STAGING_LIBDIR},g' \ -e 's,libexecdir=${libexecdir},libexecdir=${STAGING_DIR_HOST}${libexecdir},g' \ -e 's,^LIBDIR=.*,LIBDIR=${STAGING_LIBDIR},g' \
@@ -74,13 +79,17 @@ do_compile() { export CROSS_COMPILE="${TARGET_PREFIX}" export PYTHONBUILDDIR="${B}" + # This is only used in PGO profiling by python-profile-opt package + if [ "${PYTHON_MAKE_TARGET}" = "build_all_generate_profile" ]; then + export EXTRA_CFLAGS="-fprofile-dir=./python-pgo-profiles/" + fi oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \ HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python-native/python \ STAGING_LIBDIR=${STAGING_LIBDIR} \ STAGING_INCDIR=${STAGING_INCDIR} \ STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \ - OPT="${CFLAGS}" + OPT="${CFLAGS}" ${PYTHON_MAKE_TARGET} } do_install() {
@@ -99,7 +108,8 @@ do_install() { STAGING_LIBDIR=${STAGING_LIBDIR} \ STAGING_INCDIR=${STAGING_INCDIR} \ STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \ - DESTDIR=${D} LIBDIR=${libdir} + DESTDIR=${D} LIBDIR=${libdir} ${PYTHON_MAKE_TARGET} + oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \ HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python-native/python \
Signature · 1 line
2.12.0
PATCH v2 3/20 python: remove path hack from setup.py
In reply to Markus LehtonenThere should be no need to hack the library and include dirs in setup.py. The script detects them from compiler output. We need to remove this is because python-profile-opt recipe is built with a non-standard prefix setting (i.e. /opt) and the hack breaks the discovery of system libraries and headers.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- .../python/python/01-use-proper-tools-for-cross-build.patch | 10 ---------- 1 file changed, 10 deletions(-)
meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.patch +0 -10
@@ -85,13 +85,3 @@ Index: Python-2.7.13/setup.py assert not self.inplace basename, tail = os.path.splitext(ext_filename) newname = basename + "_failed" + tail -@@ -574,6 +575,9 @@ class PyBuildExt(build_ext): - - # XXX Omitted modules: gl, pure, dl, SGI-specific modules - -+ lib_dirs = [ os.getenv("STAGING_LIBDIR"), os.getenv("STAGING_BASELIBDIR") ] -+ inc_dirs = [ os.getenv("STAGING_INCDIR") ] -+ - # - # The following modules are all pretty straightforward, and compile - # on pretty much any POSIXish platform.
Signature · 1 line
2.12.0
PATCH v2 4/20 python-profile-opt: rename libpython
In reply to Markus LehtonenThe python library needs to be renamed so that the automatic dependency generation/checking mechanism in bitbake does not get confused. Otherwise python-profile-opt will depend on libpython from the "normal" python package.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- .../python/python-profile-opt_2.7.13.bb | 6 +- ...rename-libpython-to-libpython-profile-opt.patch | 94 ++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-) create mode 100644 meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch
meta/recipes-devtools/python/python-profile-opt_2.7.13.bb +1 -5
@@ -1,7 +1,6 @@ require python_${PV}.bb -PROVIDES += "libpython2" -RPROVIDES_${PN}-core += "libpython2" +SRC_URI += "file://rename-libpython-to-libpython-profile-opt.patch" # Use special prefix in order to prevent clash with normal python STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include"
@@ -12,6 +11,3 @@ prefix = "/opt" exec_prefix = "/opt" PYTHON_MAKE_TARGET = "build_all_generate_profile" - -# Force pkgdata not to clash with python recipe -PKG_libpython-profile-opt2 = "libpython-profile-opt2.7-1.0"
meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.12.0
PATCH v2 5/20 devtools/images: add python-pgo-image
In reply to Markus LehtonenThis is a special image for profiling Python in order to utilize profile-guided-optimization. Profile data can be obtained by by running bitbake python-pgo-image -c profile. It will be located in the directory pointed by ${PYTHON_PROFILE_DIR}. The profile task that is run can be altered by specifying ${PYTHON_PROFILE_TASK}.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 62 ++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
meta/recipes-devtools/images/python-pgo-image.bb new file +62 -0
@@ -0,0 +1,62 @@ +SUMMARY = "Minimal image for doing Python profiling (for PGO)" + +IMAGE_FEATURES += "ssh-server-dropbear" +IMAGE_INSTALL = "packagegroup-core-boot python-profile-opt python-profile-opt-tests" + +LICENSE = "MIT" + +inherit core-image + +PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data" +PYTHON_PROFILE_TASK ?= "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" + +# We need these because we're utilizing the runtime test helpers from oeqa +TEST_TARGET ?= "qemu" +TEST_QEMUBOOT_TIMEOUT ?= "1000" +TEST_LOG_DIR ?= "${WORKDIR}/qemulogs" +FIND_ROOTFS = "1" + +python do_profile() { + from oeqa.targetcontrol import get_target_controller + from oe.utils import getstatusoutput + + target = get_target_controller(d) + target.deploy() + try: + # Boot target + bootparams = None + if d.getVar('VIRTUAL-RUNTIME_init_manager', True) == 'systemd': + bootparams = 'systemd.log_level=debug systemd.log_target=console' + target.start(extra_bootparams=bootparams) + + # Run profile task + profile_cmd = 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python %s' % d.getVar("PYTHON_PROFILE_TASK", True) + ret, output = target.run(profile_cmd, timeout=7200) + if ret: + bb.fatal("Failed to run profile task on target: %s" % output) + ret, output = target.run('tar czf pgo-data.tgz -C /home/root/python-pgo-profiles/ .') + if ret: + bb.fatal("Failed to archive profile data on target: %s" % output) + + # Retrieve and unpack profile data + profile_dir = d.getVar("PYTHON_PROFILE_DIR", True) + target.copy_from('/home/root/pgo-data.tgz', profile_dir) + + profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz') + ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, profile_dir)) + os.unlink(profile_tarball) + if ret: + bb.fatal("Failed to unpack python profile data: %s" % output) + finally: + target.stop() + + # Exclude files causing problems in cross-profiling + excludes = [os.path.join(profile_dir, 'Modules', 'posixmodule.gcda')] + for path in excludes: + if os.path.exists(path): + os.unlink(path) +} + +addtask profile after do_build +do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot" +do_profile[cleandirs] = "${PYTHON_PROFILE_DIR}"
Signature · 1 line
2.12.0
PATCH v2 6/20 python: make profile-optimized build possible
In reply to Markus LehtonenThis patch makes it possible to build python with profile directed optimization. That is, feed python build process with profile data to guide optimization. This is the third (and the last) step in profile directed optimization for Python.
In order to do a profile-optimized build you need to set PYTHON_PROFILE_OPT = "1" in your local.conf. Be sure to get the profile data for python, preferably by running
"bitbake python-pgo-image -c profile".
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python_2.7.13.bb | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python_2.7.13.bb +24 -1
@@ -42,6 +42,17 @@ EXTRA_OECONF += "ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no" STAGING_INCDIR_DEFAULT ?= "${STAGING_INCDIR}" STAGING_LIBDIR_DEFAULT ?= "${STAGING_LIBDIR}" +# Automatic profile guided optimization +PYTHON_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON_PROFILE_OPT', True) == '1' else ''}" +PYTHON_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python/pgo-data' if d.getVar('PYTHON_PROFILE_OPT', True) == '1' else ''}" +python () { + if (d.getVar('PYTHON_PROFILE_OPT', True) == '1' and + d.getVar('PYTHON_MAKE_TARGET', True) == 'build_all_use_profile'): + profile_dir = d.getVar('PYTHON_PROFILE_DIR', True) + bb.utils.mkdirhier(profile_dir) + d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir) +} + do_configure_append() { rm -f ${S}/Makefile.orig autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -79,9 +90,15 @@ do_compile() { export CROSS_COMPILE="${TARGET_PREFIX}" export PYTHONBUILDDIR="${B}" - # This is only used in PGO profiling by python-profile-opt package if [ "${PYTHON_MAKE_TARGET}" = "build_all_generate_profile" ]; then + # This is only used in PGO profiling by python-profile-opt package export EXTRA_CFLAGS="-fprofile-dir=./python-pgo-profiles/" + else + if [ -n "${PYTHON_PROFILE_DIR}" ]; then + export EXTRA_CFLAGS="-fprofile-dir=${PYTHON_PROFILE_DIR}" + # Remove non-optimized build artefacts + oe_runmake clean + fi fi oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \
@@ -99,6 +116,11 @@ do_install() { export CROSS_COMPILE="${TARGET_PREFIX}" export PYTHONBUILDDIR="${B}" + # This only has effect if we build with -fprofile-use, e.g. when make + # target is build_all_use_profile + if [ -n "${PYTHON_PROFILE_DIR}" ]; then + export EXTRA_CFLAGS="-fprofile-dir=${PYTHON_PROFILE_DIR}" + fi # After swizzling the makefile, we need to run the build again. # install can race with the build so we have to run this first, then install
@@ -146,6 +168,7 @@ py_package_preprocess () { python -m py_compile ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py } + require python-${PYTHON_MAJMIN}-manifest.inc # manual dependency additions
Signature · 1 line
2.12.0
PATCH v2 7/20 python-pgo-image: exclude tests from the default profile target
In reply to Markus LehtonenExclue tests that are segfaulting or otherwise failing (on qemux86 target), and thus, causing the profile task to fail.
Also, disable tests that are taking very long to execute. In qemux86 target on my i7-3770K:
test_lib2to3: 1908.8s
test_io: 1278.4s
test_itertools: 436.4s
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/recipes-devtools/images/python-pgo-image.bb +8 -1
@@ -8,7 +8,14 @@ LICENSE = "MIT" inherit core-image PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data" -PYTHON_PROFILE_TASK ?= "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" +PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" +# Exclude tests that are segfaulting on qemux86 target +PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc" +# Exclude tests that are failing on qemux86 +PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings" +# Exclude tests that are taking very long on qemux86 +PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools" +PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}" # We need these because we're utilizing the runtime test helpers from oeqa TEST_TARGET ?= "qemu"
Signature · 1 line
2.12.0
PATCH v2 8/20 python: add python-tools subpackage
In reply to Markus LehtonenUseful in developing Python, e.g. in benchmarking.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python-2.7-manifest.inc | 8 +++++--- meta/recipes-devtools/python/python_2.7.13.bb | 5 +++++ 2 files changed, 10 insertions(+), 3 deletions(-)
meta/recipes-devtools/python/python-2.7-manifest.inc +5 -3
@@ -4,9 +4,9 @@ -PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib " +PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-tools" -PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-modules" +PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-modules ${PN}-tools" SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator" RDEPENDS_${PN}-2to3="${PN}-core"
@@ -284,4 +284,6 @@ SUMMARY_${PN}-modules="All Python modules" RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib " ALLOW_EMPTY_${PN}-modules = "1" - +SUMMARY_${PN}-tools="Useful scripts from Python" +RDEPENDS_${PN}-tools="${PN}-modules" +FILES_${PN}-tools="${docdir}/${PN}/Tools"
meta/recipes-devtools/python/python_2.7.13.bb +5 -0
@@ -148,6 +148,11 @@ do_install() { fi oe_multilib_header python${PYTHON_MAJMIN}/pyconfig.h + + # Install tools + install -d -m 755 ${D}/${docdir}/${PN} + cp -r ${S}/Tools ${D}/${docdir}/${PN}/ + find ${D}/${docdir}/${PN}/ -name Makefile -delete } do_install_append_class-nativesdk () {
Signature · 1 line
2.12.0
PATCH v2 9/20 python-pgo-image: switch python default profile task to pybench
In reply to Markus LehtonenSwitch to pybench in order to reduce the time taken by profiling.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
meta/recipes-devtools/images/python-pgo-image.bb +10 -8
@@ -1,20 +1,22 @@ SUMMARY = "Minimal image for doing Python profiling (for PGO)" IMAGE_FEATURES += "ssh-server-dropbear" -IMAGE_INSTALL = "packagegroup-core-boot python-profile-opt python-profile-opt-tests" +IMAGE_INSTALL = "packagegroup-core-boot" +IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools" LICENSE = "MIT" inherit core-image PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data" -PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" -# Exclude tests that are segfaulting on qemux86 target -PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc" -# Exclude tests that are failing on qemux86 -PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings" -# Exclude tests that are taking very long on qemux86 -PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools" +#PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" +## Exclude tests that are segfaulting on qemux86 target +#PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc" +## Exclude tests that are failing on qemux86 +#PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings" +## Exclude tests that are taking very long on qemux86 +#PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools" +PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck" PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}" # We need these because we're utilizing the runtime test helpers from oeqa
Signature · 1 line
2.12.0
PATCH v2 10/20 python-pgo-image: enable sstate for do_profile
In reply to Markus LehtonenThere are caveats however, basically because of the circular dependency between python-pgo-image and python (python is a build dep of python-pgo-image, and at the same time, depends on the profile data generated by python-pgo-image). In practice the sstate data is only used when no profile data is yet present. Existing profile data (i.e. non-empty PYTHON_PROFILE_DIR) changes signature hash of python which, in turn, changes signature of python-pgo-image and causes a sstate signature mismatch and rebuild of python-pgo-image. Thus, in order to utilize sstate data the user should run "bitbake python-pgo-image -c clean" (or otherwise wipe pre-existing profile data) before running "bitbake python-pgo-image -c profile".
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
meta/recipes-devtools/images/python-pgo-image.bb +13 -2
@@ -9,6 +9,7 @@ LICENSE = "MIT" inherit core-image PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data" +PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data" #PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess" ## Exclude tests that are segfaulting on qemux86 target #PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc"
@@ -48,7 +49,7 @@ python do_profile() { bb.fatal("Failed to archive profile data on target: %s" % output) # Retrieve and unpack profile data - profile_dir = d.getVar("PYTHON_PROFILE_DIR", True) + profile_dir = d.getVar("PROFILE_DATA_WORKDIR", True) target.copy_from('/home/root/pgo-data.tgz', profile_dir) profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz')
@@ -68,4 +69,14 @@ python do_profile() { addtask profile after do_build do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot" -do_profile[cleandirs] = "${PYTHON_PROFILE_DIR}" +do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}" + + +python do_profile_setscene () { + sstate_setscene(d) +} + +SSTATETASKS += "do_profile" +do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}" +do_profile[sstate-outputdirs] = "${PYTHON_PROFILE_DIR}" +addtask do_profile_setscene
Signature · 1 line
2.12.0
PATCH v2 11/20 python3-native: support profile optimized build
In reply to Markus LehtonenThis feature is virtually identical to that of python-native. Profile guided optimization is enabled by defining PYTHON3_NATIVE_PROFILE_OPT = "1"
in local.conf. In addition, the profile task may be defined with PYTHON3_NATIVE_PROFILE_TASK.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python3-native_3.5.2.bb | 9 +++++++++ meta/recipes-devtools/python/python3/000-cross-compile.patch | 9 --------- 2 files changed, 9 insertions(+), 9 deletions(-)
meta/recipes-devtools/python/python3-native_3.5.2.bb +9 -0
@@ -56,10 +56,19 @@ EXTRA_OEMAKE = '\ # No ctypes option for python 3 PYTHONLSBOPTS = "" +PYTHON3_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON3_NATIVE_PROFILE_OPT', True) == '1' else ''}" + do_configure_append() { autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi } +do_compile() { + if [ -n "${PYTHON3_NATIVE_PROFILE_TASK}" ]; then + sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON3_NATIVE_PROFILE_TASK},g' Makefile + fi + oe_runmake ${PYTHON3_NATIVE_MAKE_TARGET} +} + do_install() { install -d ${D}${libdir}/pkgconfig oe_runmake 'DESTDIR=${D}' install
meta/recipes-devtools/python/python3/000-cross-compile.patch +0 -9
@@ -29,15 +29,6 @@ Index: Python-3.5.2/Makefile.pre.in PSRCS= \ Parser/acceler.c \ -@@ -509,7 +511,7 @@ build_all_generate_profile: - - run_profile_task: - : # FIXME: can't run for a cross build -- $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true -+ $(LLVM_PROF_FILE) $(RUNSHARED) $(HOSTPYTHON) $(PROFILE_TASK) || true - - build_all_merge_profile: - $(LLVM_PROF_MERGER) @@ -792,7 +794,7 @@ $(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGEN) @$(MKDIR_P) Include # Avoid copying the file onto itself for an in-tree build
Signature · 1 line
2.12.0
PATCH v2 12/20 python3: fix depends of python-tests
In reply to Markus LehtonenSimilar to an earlier fix for Python 2.7. Make the tests subpackage depend on all modules as test.regrtest uses most (if not all) of them.
Some tests also depend on libgcc so add that as a runtime dependency as well.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python-3.5-manifest.inc +1 -1
@@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io" FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* " SUMMARY_${PN}-tests="Python tests" -RDEPENDS_${PN}-tests="${PN}-core" +RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc" FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ " SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
Signature · 1 line
2.12.0
PATCH v2 13/20 python3: add python-profile-opt3 recipe
In reply to Markus LehtonenSimilar to python-profile-opt for Python 2.7, this is a special version of Python3 used for profile guided optimization. It is used for generating profile data on the target device. Libpython3 is renamed to libpython-profile-opt3 in order to prevent soname clash with "normal" libpython3.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python-profile-opt3 | 1 + .../python/python-profile-opt3_3.5.2.bb | 15 +++ ...efile-add-install_generate_profile-target.patch | 25 +++++ ...name-libpython3-to-libpython-profile-opt3.patch | 111 +++++++++++++++++++++ meta/recipes-devtools/python/python3_3.5.2.bb | 46 +++++---- 5 files changed, 178 insertions(+), 20 deletions(-) create mode 120000 meta/recipes-devtools/python/python-profile-opt3 create mode 100644 meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
meta/recipes-devtools/python/python-profile-opt3 new file +1 -0
@@ -0,0 +1 @@ +python3
meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb new file +15 -0
@@ -0,0 +1,15 @@ +require python3_${PV}.bb + +SRC_URI += "file://rename-libpython3-to-libpython-profile-opt3.patch \ + file://Makefile-add-install_generate_profile-target.patch \ + " + +# Use special prefix in order to prevent clash with the normal python3 package +STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include" +STAGING_LIBDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/${baselib}" +TARGET_CFLAGS += "-I${STAGING_INCDIR_DEFAULT}" +TARGET_CPPFLAGS += "-I${STAGING_INCDIR_DEFAULT}" +prefix = "/opt" +exec_prefix = "/opt" + +PYTHON3_MAKE_TARGET = "build_all_generate_profile"
meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch new file +25 -0
@@ -0,0 +1,25 @@ +Makefile: add install_generate_profile target + +Making it possible to install the binaries with profile generation +enabled. We need this as linking is also performed at the install stage. + +Upstream-Status: Inappropriate [oe-specific] + +Signed-off-by: Markus Lehtonen <[email protected]> + +diff --git a/Makefile.pre.in b/Makefile.pre.in +index 2b87614..826155b 100644 +--- a/Makefile.pre.in ++++ b/Makefile.pre.in +@@ -1046,6 +1046,8 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \ + quicktest: all platform + $(TESTRUNNER) $(QUICKTESTOPTS) + ++install_generate_profile: ++ $(MAKE) install LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)" + + install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@ + if test "x$(ENSUREPIP)" != "xno" ; then \ +-- +2.6.6 +
meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch new file +111 -0
111 lines not on this page · raw message
meta/recipes-devtools/python/python3_3.5.2.bb +26 -20
96 lines not on this page · raw message
Signature · 1 line
2.12.0
PATCH v2 14/20 python-pgo-image: profiling for python3
In reply to Markus LehtonenAdd a new 'do_profile3' task for python-pgo-image that runs profiling task for python3 and retrieves the profile data, similarly to 'do_profile' for python2. Profile data will be copied into a directory pointed by PYTHON3_PROFILE_DIR on the host system. The profile task may be specified with PYTHON3_PROFILE_TASK.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 57 +++++++++++++++++++----- 1 file changed, 45 insertions(+), 12 deletions(-)
meta/recipes-devtools/images/python-pgo-image.bb +45 -12
@@ -3,6 +3,7 @@ SUMMARY = "Minimal image for doing Python profiling (for PGO)" IMAGE_FEATURES += "ssh-server-dropbear" IMAGE_INSTALL = "packagegroup-core-boot" IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools" +IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests" LICENSE = "MIT"
@@ -20,13 +21,18 @@ PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data" PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck" PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}" +PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data" +PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess" + +PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}" + # We need these because we're utilizing the runtime test helpers from oeqa TEST_TARGET ?= "qemu" TEST_QEMUBOOT_TIMEOUT ?= "1000" TEST_LOG_DIR ?= "${WORKDIR}/qemulogs" FIND_ROOTFS = "1" -python do_profile() { +def run_profile(d, profile_bin, profile_task, tgt_in_dir, host_out_dir): from oeqa.targetcontrol import get_target_controller from oe.utils import getstatusoutput
@@ -40,28 +46,34 @@ python do_profile() { target.start(extra_bootparams=bootparams) # Run profile task - profile_cmd = 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python %s' % d.getVar("PYTHON_PROFILE_TASK", True) - ret, output = target.run(profile_cmd, timeout=7200) + ret, output = target.run(profile_bin + ' ' + profile_task, timeout=7200) if ret: bb.fatal("Failed to run profile task on target: %s" % output) - ret, output = target.run('tar czf pgo-data.tgz -C /home/root/python-pgo-profiles/ .') + ret, output = target.run('tar czf pgo-data.tgz -C %s .' % tgt_in_dir) if ret: bb.fatal("Failed to archive profile data on target: %s" % output) # Retrieve and unpack profile data - profile_dir = d.getVar("PROFILE_DATA_WORKDIR", True) - target.copy_from('/home/root/pgo-data.tgz', profile_dir) + target.copy_from('/home/root/pgo-data.tgz', host_out_dir) - profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz') - ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, profile_dir)) + profile_tarball = os.path.join(host_out_dir, 'pgo-data.tgz') + ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, host_out_dir)) os.unlink(profile_tarball) if ret: bb.fatal("Failed to unpack python profile data: %s" % output) finally: target.stop() + +# Profile task for Python2 +python do_profile() { + outdir = os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python') + run_profile(d, 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python', + d.getVar('PYTHON_PROFILE_TASK', True), + '/home/root/python-pgo-profiles/', + outdir) # Exclude files causing problems in cross-profiling - excludes = [os.path.join(profile_dir, 'Modules', 'posixmodule.gcda')] + excludes = [os.path.join(outdir, 'Modules', 'posixmodule.gcda')] for path in excludes: if os.path.exists(path): os.unlink(path)
@@ -69,14 +81,35 @@ python do_profile() { addtask profile after do_build do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot" -do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}" - +do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}/python" python do_profile_setscene () { sstate_setscene(d) } SSTATETASKS += "do_profile" -do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}" +do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python" do_profile[sstate-outputdirs] = "${PYTHON_PROFILE_DIR}" addtask do_profile_setscene + + +# Profile task for Python3 +python do_profile3() { + run_profile(d, 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python3', + d.getVar('PYTHON3_PROFILE_TASK', True), + '/home/root/python3-pgo-profiles/', + os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python3')) +} + +addtask profile3 after do_build +do_profile3[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot" +do_profile3[cleandirs] = "${PROFILE_DATA_WORKDIR}/python3" + +python do_profile3_setscene () { + sstate_setscene(d) +} + +SSTATETASKS += "do_profile3" +do_profile3[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python3" +do_profile3[sstate-outputdirs] = "${PYTHON3_PROFILE_DIR}" +addtask do_profile3_setscene
Signature · 1 line
2.12.0
PATCH v2 15/20 python3: remove two setup.py cross-compile hacks
In reply to Markus LehtonenRemove two unneeded hacks. The first hack ("setup.py: no host headers libs" patch) is not needed because we use cross-compiler (e.g. i586-oe-linux-gcc) which has not been configured with any host system include or library directories, and thus, we don't get any host system directories when running "gcc -E -v".
The second hack becomes useless after the first hack has been removed and we get the standard include and lib directories normally from gcc.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- .../python3-setup.py-no-host-headers-libs.patch | 33 ---------------------- .../setup.py-find-libraries-in-staging-dirs.patch | 30 -------------------- meta/recipes-devtools/python/python3_3.5.2.bb | 2 -- 3 files changed, 65 deletions(-) delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch deleted +0 -33
@@ -1,33 +0,0 @@ -From 43238e1ac13e32984d015c92a5841f3de1fe1d15 Mon Sep 17 00:00:00 2001 -From: Jackie Huang <[email protected]> -Date: Tue, 18 Nov 2014 00:07:07 -0500 -Subject: [PATCH] setup.py: no host headers libs - -When we are cross-compiling, setup.py should never look in /usr -or /usr/local to find headers or libraries. - -Upstream-Status: Inappropriate [Cross compile specific] - -Signed-off-by: Jackie Huang <[email protected]> ---- - setup.py | 3 --- - 1 file changed, 3 deletions(-) - -diff --git a/setup.py b/setup.py -index f020b28..e8339cd 100644 ---- a/setup.py -+++ b/setup.py -@@ -444,10 +444,7 @@ class PyBuildExt(build_ext): - if not cross_compiling: - add_dir_to_list(self.compiler.library_dirs, os.path.join('/usr/local', sys.lib)) - add_dir_to_list(self.compiler.include_dirs, '/usr/local/include') -- # only change this for cross builds for 3.3, issues on Mageia -- if cross_compiling: - self.add_gcc_paths() -- if not cross_compiling: - self.add_multiarch_paths() - - # Add paths specified in the environment variables LDFLAGS and --- -2.0.0 -
meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch deleted +0 -30
@@ -1,30 +0,0 @@ -Patch setup.py so that the detect_modules() function looks for required -libraries and headers in STAGING_LIBDIR / STAGING_INCDIR. - -Without this patch, several extension modules are not built, even though -their dependencies are present in the compiler's search paths. -The result is the following warning, and ultimately incomplete packages: - -| The necessary bits to build these optional modules were not found: -| _bz2 _curses_panel _dbm -| _gdbm _lzma _sqlite3 -| nis readline zlib -| To find the necessary bits, look in setup.py in detect_modules() for the module's name. - -Upstream-Status: Inappropriate [Cross compile specific] - -Signed-off-by: Dominic Sacré <[email protected]> - -Index: Python-3.4.3/setup.py -=================================================================== ---- Python-3.4.3.orig/setup.py -+++ Python-3.4.3/setup.py -@@ -521,6 +521,8 @@ class PyBuildExt(build_ext): - else: - lib_dirs = self.compiler.library_dirs[:] - inc_dirs = self.compiler.include_dirs[:] -+ lib_dirs.append(os.environ.get('STAGING_LIBDIR')) -+ inc_dirs.append(os.environ.get('STAGING_INCDIR')) - exts = [] - missing = [] -
meta/recipes-devtools/python/python3_3.5.2.bb +0 -2
@@ -30,10 +30,8 @@ SRC_URI += "\ file://unixccompiler.patch \ file://avoid-ncursesw-include-path.patch \ file://python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \ - file://python3-setup.py-no-host-headers-libs.patch \ file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \ file://setup.py-check-cross_compiling-when-get-FLAGS.patch \ - file://setup.py-find-libraries-in-staging-dirs.patch \ file://configure.ac-fix-LIBPL.patch \ file://python3-fix-CVE-2016-1000110.patch \ file://upstream-random-fixes.patch \
Signature · 1 line
2.12.0
PATCH v2 16/20 python3: support profile optimized build
In reply to Markus LehtonenMake it possible to do profile-optimized build. The implementation is virtually identical to how we do it in Python 2.7. Basically, you need to define PYTHON3_PROFILE_OPT = "1" in your local.conf and have profile data available in the location pointed to by PYTHON3_PROFILE_DIR. Profile data can be obtainen e.g. by running bitbake python-pgo-image -c profile3.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python3_3.5.2.bb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
meta/recipes-devtools/python/python3_3.5.2.bb +20 -0
@@ -78,6 +78,17 @@ export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dy # No ctypes option for python 3 PYTHONLSBOPTS = "" +# Automatic profile guided optimization +PYTHON3_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}" +PYTHON3_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}" +python () { + if (d.getVar('PYTHON3_PROFILE_OPT', True) == '1' and + d.getVar('PYTHON3_MAKE_TARGET', True) == 'build_all_use_profile'): + profile_dir = d.getVar('PYTHON3_PROFILE_DIR', True) + bb.utils.mkdirhier(profile_dir) + d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir) +} + do_configure_append() { rm -f ${S}/Makefile.orig autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -117,6 +128,10 @@ do_compile() { if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then # This is only used in PGO profiling by python-profile-opt package export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/" + elif [ -n "${PYTHON3_PROFILE_DIR}" ]; then + export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}" + # Remove non-optimized build artefacts + oe_runmake clean fi oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
@@ -137,6 +152,11 @@ do_install() { install -d ${D}${libdir}/pkgconfig install -d ${D}${libdir}/python${PYTHON_MAJMIN}/config + # This only has effect if we build with -fprofile-use, e.g. when make + # target is build_all_use_profile + if [ -n "${PYTHON3_PROFILE_DIR}" ]; then + export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}" + fi # rerun the build once again with original makefile this time # run install in a separate step to avoid compile/install race oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
Signature · 1 line
2.12.0
PATCH v2 17/20 python3: fix profile-optimized build of modules
In reply to Markus LehtonenWithout this the pgo-related compiler flags are not used in cross-builds.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++++++++++++++ meta/recipes-devtools/python/python3_3.5.2.bb | 1 + 2 files changed, 57 insertions(+) create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch new file +56 -0
@@ -0,0 +1,56 @@ +From 0fd8b986888ddf1995b503edf46ac827e83114d9 Mon Sep 17 00:00:00 2001 +From: Markus Lehtonen <[email protected]> +Date: Wed, 23 Nov 2016 16:08:04 +0200 +Subject: [PATCH] Use correct CFLAGS for extensions when cross-compiling + +Take PY_CFLAGS_NODIST into account, like in native build. This is needed +in order to to profile-optimized build. Also, pass EXTRA_CFLAGS to +profile-optimized build. + +Upstream-Status: Pending + +Signed-off-by: Markus Lehtonen <[email protected]> +--- + Makefile.pre.in | 4 ++-- + setup.py | 3 ++- + 2 files changed, 4 insertions(+), 3 deletions(-) + +diff --git a/Makefile.pre.in b/Makefile.pre.in +index bace00e..9635aa4 100644 +--- a/Makefile.pre.in ++++ b/Makefile.pre.in +@@ -509,7 +509,7 @@ profile-opt: + $(MAKE) profile-removal + + build_all_generate_profile: +- $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)" ++ $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)" + + run_profile_task: + : # FIXME: can't run for a cross build +@@ -519,7 +519,7 @@ build_all_merge_profile: + $(LLVM_PROF_MERGER) + + build_all_use_profile: +- $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@" ++ $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@" + + # Compile and run with gcov + .PHONY=coverage coverage-lcov coverage-report +diff --git a/setup.py b/setup.py +index c1ce87e..72d37cf 100644 +--- a/setup.py ++++ b/setup.py +@@ -271,7 +271,8 @@ class PyBuildExt(build_ext): + # compilers + if compiler is not None: + if cross_compiling: +- (ccshared,cflags) = (os.environ.get('CCSHARED') or '', os.environ.get('CFLAGS') or '') ++ (ccshared,cflags) = (os.environ.get('CCSHARED') or '', ++ (os.environ.get('CFLAGS') or '') + ' ' + sysconfig.get_config_var('PY_CFLAGS_NODIST')) + else: + (ccshared,cflags) = sysconfig.get_config_vars('CCSHARED','CFLAGS') + args['compiler_so'] = compiler + ' ' + ccshared + ' ' + cflags +-- +2.6.6 +
meta/recipes-devtools/python/python3_3.5.2.bb +1 -0
@@ -35,6 +35,7 @@ SRC_URI += "\ file://configure.ac-fix-LIBPL.patch \ file://python3-fix-CVE-2016-1000110.patch \ file://upstream-random-fixes.patch \ + file://Use-correct-CFLAGS-for-extensions-when-cross-compili.patch \ " SRC_URI[md5sum] = "8906efbacfcdc7c3c9198aeefafd159e" SRC_URI[sha256sum] = "0010f56100b9b74259ebcd5d4b295a32324b58b517403a10d1a2aa7cb22bca40"
Signature · 1 line
2.12.0
PATCH v2 18/20 python-pgo-image: exclude tests from the python3 profile target
In reply to Markus LehtonenExlude failing tests (on qemux86 target) which would cause the profile task to fail.
Also, disable some very long running tests. Running in qemux86 target on my i7-3770K:
test_lib2to3: 1604.5s
test_buffer: 1562.8s
test_pickle: 1527.0s
test_io: 1517.8s
test_threading: 1384.1s
test_asyncio: 1380.9s
test_urllib2_localnet: 1276.8s
test_itertools: 944.7s
test_tuple: 847.7s
test_trace: 756.7s
test_tarfile: 716.7s
test_unicodedata: 709.9s
test_decimal: 615.1s
test_buffer: 1562.8s
test_pickle: 1527.0s
test_io: 1517.8s
test_threading: 1384.1s
test_asyncio: 1380.9s
test_urllib2_localnet: 1276.8s
test_itertools: 944.7s
test_tuple: 847.7s
test_trace: 756.7s
test_tarfile: 716.7s
test_unicodedata: 709.9s
test_decimal: 615.1s
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 5 +++++ 1 file changed, 5 insertions(+)
meta/recipes-devtools/images/python-pgo-image.bb +5 -0
@@ -23,6 +23,11 @@ PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}" PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data" PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess" +# Exclude tests that are failing on qemux86 +PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode" +# Exclude tests that are taking very long on qemux86 +PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal" +PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec" PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
Signature · 1 line
2.12.0
PATCH v2 19/20 python3: add python3-tools subpackage
In reply to Markus Lehtonen[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +++++--- meta/recipes-devtools/python/python3_3.5.2.bb | 5 +++++ 2 files changed, 10 insertions(+), 3 deletions(-)
meta/recipes-devtools/python/python-3.5-manifest.inc +5 -3
@@ -4,9 +4,9 @@ -PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc " +PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-tools " -PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules" +PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules ${PN}-tools" SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator" RDEPENDS_${PN}-2to3="${PN}-core"
@@ -280,4 +280,6 @@ SUMMARY_${PN}-modules="All Python modules" RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc " ALLOW_EMPTY_${PN}-modules = "1" - +SUMMARY_${PN}-tools="Useful scripts from Python" +RDEPENDS_${PN}-tools="${PN}-modules" +FILES_${PN}-tools="${docdir}/${PN}/Tools"
meta/recipes-devtools/python/python3_3.5.2.bb +5 -0
@@ -195,6 +195,11 @@ do_install() { fi oe_multilib_header python${PYTHON_BINABI}/pyconfig.h + + # Install tools + install -d -m 755 ${D}/${docdir}/${PN} + cp -r ${S}/Tools ${D}/${docdir}/${PN}/ + find ${D}/${docdir}/${PN}/ -name Makefile -delete } do_install_append_class-nativesdk () {
Signature · 1 line
2.12.0
PATCH v2 20/20 python-pgo-image: change python3 profile target to pybench
In reply to Markus LehtonenRunning test.regrtest was still taking ridiculously long, i.e. around 4 hours in qemux86 on my desktop machine. Change to pybench which "only" takes around 55 minutes.
[YOCTO #9338]
Signed-off-by Markus Lehtonen <[email protected]>
--- meta/recipes-devtools/images/python-pgo-image.bb | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
meta/recipes-devtools/images/python-pgo-image.bb +6 -5
@@ -3,7 +3,7 @@ SUMMARY = "Minimal image for doing Python profiling (for PGO)" IMAGE_FEATURES += "ssh-server-dropbear" IMAGE_INSTALL = "packagegroup-core-boot" IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools" -IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests" +IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests python-profile-opt3-tools" LICENSE = "MIT"
@@ -22,12 +22,13 @@ PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/p PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}" PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data" -PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess" +#PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess" # Exclude tests that are failing on qemux86 -PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode" +#PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode" # Exclude tests that are taking very long on qemux86 -PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal" -PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec" +#PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal" +#PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec" +PYTHON3_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt3/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck" PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
Signature · 1 line
2.12.0
Quoted 6 lines · Markus Lehtonen
This patchset makes it possible to make a PGO (profile-guided-optimization) build of python. This version of the patchset is almost identical to v1 submitted back in February, with these changes:
- rebased on top of latest oe-core master
- exclude profile data for Modules/posixmodule of Python 2.7 as it was not working correctly
How about doing this only for Python 3.x? We'd like to stick to our resolution of deprecating 2.x, which means not introducing any new dependencies or images that need it.
Also, python-pgo-image commits should probably be all squashed into one (with comments within files replacing the commit messages).
Alex
On 16/05/2017, 16.23, "Alexander Kanavin" <[email protected]> wrote:
On 05/16/2017 04:18 PM, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided-optimization) > build of python. This version of the patchset is almost identical to v1 > submitted back in February, with these changes:
> - rebased on top of latest oe-core master
> - exclude profile data for Modules/posixmodule of Python 2.7 as it was not
> working correctly
How about doing this only for Python 3.x? We'd like to stick to our resolution of deprecating 2.x, which means not introducing any new dependencies or images that need it.
You're probably right. There's not much use in churning soon-to-be-deprecated Python 2.x. I will submit a new patchset for Python 3.x only. Python 2.x could be added later if deprecation wouldn't happen for some reason.
Also, python-pgo-image commits should probably be all squashed into one (with comments within files replacing the commit messages).
Point taken.
Thanks,
Markus
Quoted 4 lines · Markus Lehtonen
You're probably right. There's not much use in churning soon-to-be-deprecated Python 2.x. I will submit a new patchset for Python 3.x only. Python 2.x could be added later if deprecation wouldn't happen for some reason.
The deprecation is not a single point in time, it's an ongoing process. A lot of work was done last spring, and some more work has been done this spring. The situation with what still needs py 2.x will be revisited again in a year's time. I just want to avoid regressions on this front :)
Alex
Quoted 8 lines · Markus Lehtonen
This patchset makes it possible to make a PGO (profile-guided- optimization) build of python. This version of the patchset is almost identical to v1 submitted back in February, with these changes:
- rebased on top of latest oe-core master - exclude profile data for Modules/posixmodule of Python 2.7 as it was not working correctly[YOCTO #9338]
I'm wondering if recipe specific sysroots might have made this problem a bit easier?
It should now be possible to build the two packages with conflicting files and as long as you exclude the recipe from the shlibs code and set the RPROVIDES correctly, it should all work without as many invasive changes?
Of course I haven't tested that...
Cheers,
Richard
On 16/05/2017, 16.45, "Richard Purdie" <[email protected]> wrote:
On Tue, 2017-05-16 at 16:18 +0300, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided- > optimization) build of python. This version of the patchset is almost > identical to v1 submitted back in February, with these changes:
> - rebased on top of latest oe-core master - exclude profile data for > Modules/posixmodule of Python 2.7 as it was not working correctly >
> [YOCTO #9338]
>
I'm wondering if recipe specific sysroots might have made this problem a bit easier?
It should now be possible to build the two packages with conflicting files and as long as you exclude the recipe from the shlibs code and set the RPROVIDES correctly, it should all work without as many invasive changes?
Of course I haven't tested that...
I'll try that out. That, accompanied with dropping Python 2.x support (thanks Alexander) would make the patchset significantly simpler.
Cheers,
MArkus