Skip to contentSkip to conversation

python3: Reproducible build fixes

PATCH Superseded Joshua Watt 3 participants 7 messages Jun 20, 2019 – Jun 21, 2019

lore
Share & tools
lore
#0 lore raw

cover python3: Reproducible build fixes

Two fixes to make the Python3 build reproducible

Joshua Watt (2):
python3: Reformat sysconfig
python3: Disable PGO for reproducible builds

 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb | 20 +++++++++++++++++-
 2 files changed, 40 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/2 python3: Reformat sysconfig

Reformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at 80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:

'A': 'B is really'

    ' long'

This causes a problem for reproducibility however because there might be lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping from pprint remains.

To correct this, reformat the entire sysconfig file by re-printing using pprint with an (effectively) unlimited line length.

---
 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
meta/recipes-devtools/python/python3/reformat_sysconfig.py new file +21 -0
new file mode 100644
index 00000000000..c4164313e8b
@@ -0,0 +1,21 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+#
+# A script to reformat python sysconfig
+
+import sys
+import pprint
+l = {}
+g = {}
+with open(sys.argv[1], 'r') as f:
+    exec(f.read(), g, l)
+
+with open(sys.argv[1], 'w') as f:
+    for k in sorted(l.keys()):
+        f.write('%s = ' % k)
+        pprint.pprint(l[k], stream=f, width=sys.maxsize)
+        f.write('\n')
+
meta/recipes-devtools/python/python3_3.7.3.bb +10 -0
index 24442961421..d14c93880fb 100644
@@ -27,6 +27,10 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
 	   file://crosspythonpath.patch \
            "
 
+SRC_URI_append_class-target = " \
+           file://reformat_sysconfig.py \
+           "
+
 SRC_URI_append_class-native = " \
            file://0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch \
            file://12-distutils-prefix-is-inside-staging-area.patch \
@@ -157,6 +161,12 @@ py_package_preprocess () {
                 ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \
                 ${PKGD}/${bindir}/python${PYTHON_BINABI}-config
 
+        # Reformat _sysconfigdata after modifying it so that it remains
+        # reproducible
+        for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+            python3 ${WORKDIR}/reformat_sysconfig.py $c
+        done
+
         # Recompile _sysconfigdata after modifying it
         cd ${PKGD}
         sysconfigfile=`find . -name _sysconfigdata_*.py`
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH 1/2] python3: Reformat sysconfig

Quoted 16 lines · Joshua Watt

Reformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at
80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:

'A': 'B is really'

    ' long'

This causes a problem for reproducibility however because there might be
lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping
from pprint remains.

Quoted 5 lines

To correct this, reformat the entire sysconfig file by re-printing using
pprint with an (effectively) unlimited line length.

Fails on nativesdk:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/969

Have aborted that build.

Cheers,

Richard

#3 lore raw

PATCH v2 1/1 python3: Reformat sysconfig

In reply to Joshua Watt

Reformats the sysconfig file when packaging. This file is output by using the python pprint function. This function will wrap long lines at 80 characters by default, and will even split strings at whitespace boundaries to do so, e.g.:

'A': 'B is really'

    ' long'

This causes a problem for reproducibility however because there might be lines of differing lengths depending on the build path. These non-reproducible paths are removed, but their effect on string wrapping from pprint remains.

To correct this, reformat the entire sysconfig file by re-printing using pprint with an (effectively) unlimited line length.

---
 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb |  7 +++++++
 2 files changed, 28 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py
meta/recipes-devtools/python/python3/reformat_sysconfig.py new file +21 -0
new file mode 100644
index 00000000000..c4164313e8b
@@ -0,0 +1,21 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+#
+# A script to reformat python sysconfig
+
+import sys
+import pprint
+l = {}
+g = {}
+with open(sys.argv[1], 'r') as f:
+    exec(f.read(), g, l)
+
+with open(sys.argv[1], 'w') as f:
+    for k in sorted(l.keys()):
+        f.write('%s = ' % k)
+        pprint.pprint(l[k], stream=f, width=sys.maxsize)
+        f.write('\n')
+
meta/recipes-devtools/python/python3_3.7.3.bb +7 -0
index 8e77dbe9595..3409d94ba08 100644
@@ -25,6 +25,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Makefile-fix-Issue36464-parallel-build-race-problem.patch \
 	   file://0001-bpo-36852-proper-detection-of-mips-architecture-for-.patch \
 	   file://crosspythonpath.patch \
+           file://reformat_sysconfig.py \
            "
 
 SRC_URI_append_class-native = " \
@@ -165,6 +166,12 @@ py_package_preprocess () {
                 ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \
                 ${PKGD}/${bindir}/python${PYTHON_BINABI}-config
 
+        # Reformat _sysconfigdata after modifying it so that it remains
+        # reproducible
+        for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+            python3 ${WORKDIR}/reformat_sysconfig.py $c
+        done
+
         # Recompile _sysconfigdata after modifying it
         cd ${PKGD}
         sysconfigfile=`find . -name _sysconfigdata_*.py`
Signature · 1 line
2.21.0
#4 lore raw

PATCH 2/2 python3: Disable PGO for reproducible builds

In reply to Joshua Watt

Enabling PGO for python current causes it to not be reproducible when building, so disable it for now.

---
 meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python3_3.7.3.bb +9 -1
index d14c93880fb..40e2229a165 100644
@@ -79,8 +79,16 @@ CACHED_CONFIGUREVARS = " \
                 ac_cv_file__dev_ptc=no \
                 ac_cv_working_tzset=yes \
 "
+python() {
+    # PGO currently causes builds to not be reproducible, so disable it for
+    # now. See YOCTO #13407
+    if bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', True, False, d) and d.getVar('BUILD_REPRODUCIBLE_BINARIES') != '1':
+        d.setVar('PACKAGECONFIG_PGO', 'pgo')
+    else:
+        d.setVar('PACKAGECONFIG_PGO', '')
+}
 
-PACKAGECONFIG_class-target ??= "readline ${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'pgo', '', d)}"
+PACKAGECONFIG_class-target ??= "readline ${PACKAGECONFIG_PGO}"
 PACKAGECONFIG_class-native ??= "readline"
 PACKAGECONFIG_class-nativesdk ??= "readline"
 PACKAGECONFIG[readline] = ",,readline"
Signature · 1 line
2.21.0
#5 lore raw

✗ patchtest: failure for python3: Reproducible build fixes

In reply to Joshua Watt

== Series Details ==

Series: python3: Reproducible build fixes
Revision: 1
URL : https://patchwork.openembedded.org/series/18272/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Errors in your Python code were encountered [test_pylint]
Suggested fix Correct the lines introduced by your patch
Output Please, fix the listed issues:
meta/recipes-devtools/python/python3/reformat_sysconfig.py does not exist

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
#6 lore raw

✗ patchtest: failure for python3: Reproducible build fixes (rev2)

In reply to Joshua Watt

== Series Details ==

Series: python3: Reproducible build fixes (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/18272/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at c724a2feae)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe