python3: Reproducible build fixes
cover python3: Reproducible build fixes
Two fixes to make the Python3 build reproducible
Joshua Watt (2):
python3: Reformat sysconfig
python3: Disable PGO for reproducible builds
.../python/python3/reformat_sysconfig.py | 21 +++++++++++++++++++ meta/recipes-devtools/python/python3_3.7.3.bb | 20 +++++++++++++++++- 2 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
Signature · 1 line
2.21.0
PATCH 1/2 python3: Reformat sysconfig
Reformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at 80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:
'A': 'B is really'
' long'
This causes a problem for reproducibility however because there might be lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping from pprint remains.
To correct this, reformat the entire sysconfig file by re-printing using pprint with an (effectively) unlimited line length.
--- .../python/python3/reformat_sysconfig.py | 21 +++++++++++++++++++ meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++ 2 files changed, 31 insertions(+) create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
meta/recipes-devtools/python/python3/reformat_sysconfig.py new file +21 -0
@@ -0,0 +1,21 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: MIT +# +# Copyright 2019 by Garmin Ltd. or its subsidiaries +# +# A script to reformat python sysconfig + +import sys +import pprint +l = {} +g = {} +with open(sys.argv[1], 'r') as f: + exec(f.read(), g, l) + +with open(sys.argv[1], 'w') as f: + for k in sorted(l.keys()): + f.write('%s = ' % k) + pprint.pprint(l[k], stream=f, width=sys.maxsize) + f.write('\n') +
meta/recipes-devtools/python/python3_3.7.3.bb +10 -0
@@ -27,6 +27,10 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ file://crosspythonpath.patch \ " +SRC_URI_append_class-target = " \ + file://reformat_sysconfig.py \ + " + SRC_URI_append_class-native = " \ file://0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch \ file://12-distutils-prefix-is-inside-staging-area.patch \
@@ -157,6 +161,12 @@ py_package_preprocess () { ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \ ${PKGD}/${bindir}/python${PYTHON_BINABI}-config + # Reformat _sysconfigdata after modifying it so that it remains + # reproducible + for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do + python3 ${WORKDIR}/reformat_sysconfig.py $c + done + # Recompile _sysconfigdata after modifying it cd ${PKGD} sysconfigfile=`find . -name _sysconfigdata_*.py`
Signature · 1 line
2.21.0
PATCH 2/2 python3: Disable PGO for reproducible builds
In reply to Joshua WattEnabling PGO for python current causes it to not be reproducible when building, so disable it for now.
--- meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python3_3.7.3.bb +9 -1
@@ -79,8 +79,16 @@ CACHED_CONFIGUREVARS = " \ ac_cv_file__dev_ptc=no \ ac_cv_working_tzset=yes \ " +python() { + # PGO currently causes builds to not be reproducible, so disable it for + # now. See YOCTO #13407 + if bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', True, False, d) and d.getVar('BUILD_REPRODUCIBLE_BINARIES') != '1': + d.setVar('PACKAGECONFIG_PGO', 'pgo') + else: + d.setVar('PACKAGECONFIG_PGO', '') +} -PACKAGECONFIG_class-target ??= "readline ${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'pgo', '', d)}" +PACKAGECONFIG_class-target ??= "readline ${PACKAGECONFIG_PGO}" PACKAGECONFIG_class-native ??= "readline" PACKAGECONFIG_class-nativesdk ??= "readline" PACKAGECONFIG[readline] = ",,readline"
Signature · 1 line
2.21.0
✗ patchtest: failure for python3: Reproducible build fixes
In reply to Joshua Watt== Series Details ==
Series: python3: Reproducible build fixes
Revision: 1
URL : https://patchwork.openembedded.org/series/18272/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue Errors in your Python code were encountered [test_pylint]
Suggested fix Correct the lines introduced by your patch
Output Please, fix the listed issues:
meta/recipes-devtools/python/python3/reformat_sysconfig.py does not exist
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
Re: [PATCH 1/2] python3: Reformat sysconfig
In reply to Joshua WattQuoted 16 lines · Joshua Watt
Reformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at
80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:'A': 'B is really'
' long'This causes a problem for reproducibility however because there might be
lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping
from pprint remains.
Quoted 5 lines
To correct this, reformat the entire sysconfig file by re-printing using
pprint with an (effectively) unlimited line length.Signed-off-by Joshua Watt <[email protected]>
Fails on nativesdk:
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/969
Have aborted that build.
Cheers,
Richard
PATCH v2 1/1 python3: Reformat sysconfig
In reply to Joshua WattReformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at 80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:
'A': 'B is really'
' long'
This causes a problem for reproducibility however because there might be lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping from pprint remains.
To correct this, reformat the entire sysconfig file by re-printing using pprint with an (effectively) unlimited line length.
--- .../python/python3/reformat_sysconfig.py | 21 +++++++++++++++++++ meta/recipes-devtools/python/python3_3.7.3.bb | 7 +++++++ 2 files changed, 28 insertions(+) create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
meta/recipes-devtools/python/python3/reformat_sysconfig.py new file +21 -0
@@ -0,0 +1,21 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: MIT +# +# Copyright 2019 by Garmin Ltd. or its subsidiaries +# +# A script to reformat python sysconfig + +import sys +import pprint +l = {} +g = {} +with open(sys.argv[1], 'r') as f: + exec(f.read(), g, l) + +with open(sys.argv[1], 'w') as f: + for k in sorted(l.keys()): + f.write('%s = ' % k) + pprint.pprint(l[k], stream=f, width=sys.maxsize) + f.write('\n') +
meta/recipes-devtools/python/python3_3.7.3.bb +7 -0
@@ -25,6 +25,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ file://0001-Makefile-fix-Issue36464-parallel-build-race-problem.patch \ file://0001-bpo-36852-proper-detection-of-mips-architecture-for-.patch \ file://crosspythonpath.patch \ + file://reformat_sysconfig.py \ " SRC_URI_append_class-native = " \
@@ -165,6 +166,12 @@ py_package_preprocess () { ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \ ${PKGD}/${bindir}/python${PYTHON_BINABI}-config + # Reformat _sysconfigdata after modifying it so that it remains + # reproducible + for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do + python3 ${WORKDIR}/reformat_sysconfig.py $c + done + # Recompile _sysconfigdata after modifying it cd ${PKGD} sysconfigfile=`find . -name _sysconfigdata_*.py`
Signature · 1 line
2.21.0
✗ patchtest: failure for python3: Reproducible build fixes (rev2)
In reply to Joshua Watt== Series Details ==
Series: python3: Reproducible build fixes (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/18272/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at c724a2feae)
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe