RFC: Enable world reproducibility testing
cover RFC: Enable world reproducibility testing
This patch series enables world reproducibility testing so that we can begin monitoring progress on reproducibile builds for dunfell. It also includes a version update for diffoscope to better enable debugging failed packages.
The main downside to merging this series is increased autobuilder load to run the world tests.
I've been testing this over the past couple of months and have updated the exclusion list for dunfell as I've encountered intemittent reproducibility issues (see selftest/reproducible:
adjust exclusion list for dunfell)
Most recent passing test of a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2024
Once this patch is merged reproducibility reports will be published at:
https://www.yoctoproject.org/reproducible-build-results/?branch=dunfell
The following changes since commit d044d9c0cb672c499059eb273e399ce4aee17e0d:
image,populate_sdk_base: move 'func' flag setting for sdk command vars (2021-04-02 04:21:56 -1000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
Alexander Kanavin (3):
selftest/reproducible: enable world reproducibility test selftest/reproducible: add an exclusion list for items that are not
yet reproducible
selftest/reproducible: track unusued entries in the exclusion list
Richard Purdie (2):
selftest/reproducible: Sort the unused exclusion list
diffoscope: Upgrade 136 -> 168
Steve Sakoman (1):
selftest/reproducible: adjust exclusion list for dunfell
meta/lib/oeqa/selftest/cases/reproducible.py | 80 ++++++++++++++++++-
.../{diffoscope_136.bb => diffoscope_168.bb} | 15 ++--
2 files changed, 84 insertions(+), 11 deletions(-)
rename meta/recipes-support/diffoscope/{diffoscope_136.bb => diffoscope_168.bb} (46%)Signature · 1 line
2.25.1
PATCH 1/6 selftest/reproducible: enable world reproducibility test
From: Alexander Kanavin <[email protected]>
Add systemd and pam distro features, and commercial license flag to include more recipes into the world set.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +3 -1
@@ -83,7 +83,7 @@ class ReproducibleTests(OESelftestTestCase): package_classes = ['deb', 'ipk'] # targets are the things we want to test the reproducibility of - targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline'] + targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'world'] # sstate targets are things to pull from sstate to potentially cut build/debugging time sstate_targets = [] save_results = False
@@ -156,6 +156,8 @@ class ReproducibleTests(OESelftestTestCase): PACKAGE_CLASSES = "{package_classes}" INHIBIT_PACKAGE_STRIP = "1" TMPDIR = "{tmpdir}" + LICENSE_FLAGS_WHITELIST = "commercial" + DISTRO_FEATURES_append = ' systemd pam' ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes), tmpdir=tmpdir)
Signature · 1 line
2.25.1
PATCH 2/6 selftest/reproducible: add an exclusion list for items that are not yet reproducible
In reply to Steve SakomanFrom: Alexander Kanavin <[email protected]>
Hopefully over time this list will be reduced to an empty one.
Non-reproducible excluded packages are not given to diffoscope and do not cause a failure, but still saved side-by-side with non-reproducible failing ones to make investigation easier.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 82 +++++++++++++++++++- 1 file changed, 79 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +79 -3
@@ -17,6 +17,72 @@ import stat import os import datetime +# For sample packages, see: +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-0t7wr_oo/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-4s9ejwyp/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-haiwdlbr/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-hwds3mcl/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201203-sua0pzvc/ +# (both packages/ and packages-excluded/) +exclude_packages = [ + 'acpica-src', + 'babeltrace2-ptest', + 'bootchart2-doc', + 'cups', + 'cwautomacros', + 'dtc', + 'efivar', + 'epiphany', + 'gcr', + 'git', + 'glide', + 'go-dep', + 'go-helloworld', + 'go-runtime', + 'go_', + 'groff', + 'gst-devtools', + 'gstreamer1.0-python', + 'gtk-doc', + 'igt-gpu-tools', + 'kernel-devsrc', + 'libaprutil', + 'libcap-ng', + 'libhandy-1-src', + 'libid3tag', + 'libproxy', + 'libsecret-dev', + 'libsecret-src', + 'lttng-tools-dbg', + 'lttng-tools-ptest', + 'ltp', + 'meson', + 'ovmf-shell-efi', + 'parted-ptest', + 'perf', + 'python3-cython', + 'qemu', + 'quilt-ptest', + 'rsync', + 'ruby', + 'spirv-tools-dev', + 'swig', + 'syslinux-misc', + 'systemd-bootchart', + 'valgrind-ptest', + 'vim', + 'watchdog', + 'xmlto', + 'xorg-minimal-fonts' + ] + +def is_excluded(package): + package_name = os.path.basename(package) + for i in exclude_packages: + if package_name.startswith(i): + return True + return False + MISSING = 'MISSING' DIFFERENT = 'DIFFERENT' SAME = 'SAME'
@@ -39,6 +105,7 @@ class PackageCompareResults(object): self.total = [] self.missing = [] self.different = [] + self.different_excluded = [] self.same = [] def add_result(self, r):
@@ -46,7 +113,10 @@ class PackageCompareResults(object): if r.status == MISSING: self.missing.append(r) elif r.status == DIFFERENT: - self.different.append(r) + if is_excluded(r.reference): + self.different_excluded.append(r) + else: + self.different.append(r) else: self.same.append(r)
@@ -54,10 +124,11 @@ class PackageCompareResults(object): self.total.sort() self.missing.sort() self.different.sort() + self.different_excluded.sort() self.same.sort() def __str__(self): - return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total)) + return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total)) def compare_file(reference, test, diffutils_sysroot): result = CompareResult()
@@ -226,6 +297,7 @@ class ReproducibleTests(OESelftestTestCase): self.write_package_list(package_class, 'missing', result.missing) self.write_package_list(package_class, 'different', result.different) + self.write_package_list(package_class, 'different_excluded', result.different_excluded) self.write_package_list(package_class, 'same', result.same) if self.save_results:
@@ -233,8 +305,12 @@ class ReproducibleTests(OESelftestTestCase): self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)])) self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)])) + for d in result.different_excluded: + self.copy_file(d.reference, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.reference)])) + self.copy_file(d.test, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.test)])) + if result.missing or result.different: - fails.append("The following %s packages are missing or different: %s" % + fails.append("The following %s packages are missing or different and not in exclusion list: %s" % (c, '\n'.join(r.test for r in (result.missing + result.different)))) # Clean up empty directories
Signature · 1 line
2.25.1
PATCH 3/6 selftest/reproducible: adjust exclusion list for dunfell
In reply to Steve SakomanSigned-off-be: Steve Sakoman <[email protected]>
--- meta/lib/oeqa/selftest/cases/reproducible.py | 32 ++++++-------------- 1 file changed, 10 insertions(+), 22 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +10 -22
@@ -27,11 +27,8 @@ import datetime exclude_packages = [ 'acpica-src', 'babeltrace2-ptest', + 'bind', 'bootchart2-doc', - 'cups', - 'cwautomacros', - 'dtc', - 'efivar', 'epiphany', 'gcr', 'git',
@@ -40,40 +37,31 @@ exclude_packages = [ 'go-helloworld', 'go-runtime', 'go_', - 'groff', - 'gst-devtools', 'gstreamer1.0-python', - 'gtk-doc', - 'igt-gpu-tools', + 'hwlatdetect', 'kernel-devsrc', 'libaprutil', 'libcap-ng', - 'libhandy-1-src', - 'libid3tag', + 'libjson', 'libproxy', - 'libsecret-dev', - 'libsecret-src', + 'lsb-release', 'lttng-tools-dbg', 'lttng-tools-ptest', 'ltp', - 'meson', 'ovmf-shell-efi', 'parted-ptest', 'perf', - 'python3-cython', + 'piglit', + 'pybootchartgui', 'qemu', - 'quilt-ptest', + "rpm", 'rsync', 'ruby', - 'spirv-tools-dev', - 'swig', - 'syslinux-misc', + 'stress-ng', 'systemd-bootchart', - 'valgrind-ptest', + 'systemtap', 'vim', - 'watchdog', - 'xmlto', - 'xorg-minimal-fonts' + 'webkitgtk', ] def is_excluded(package):
Signature · 1 line
2.25.1
PATCH 4/6 selftest/reproducible: track unusued entries in the exclusion list
In reply to Steve SakomanFrom: Alexander Kanavin <[email protected]>
This helps with trimming down the list, and towards 100% reproducibility :)
--- meta/lib/oeqa/selftest/cases/reproducible.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +10 -4
@@ -68,8 +68,8 @@ def is_excluded(package): package_name = os.path.basename(package) for i in exclude_packages: if package_name.startswith(i): - return True - return False + return i + return None MISSING = 'MISSING' DIFFERENT = 'DIFFERENT'
@@ -95,14 +95,17 @@ class PackageCompareResults(object): self.different = [] self.different_excluded = [] self.same = [] + self.active_exclusions = set() def add_result(self, r): self.total.append(r) if r.status == MISSING: self.missing.append(r) elif r.status == DIFFERENT: - if is_excluded(r.reference): + exclusion = is_excluded(r.reference) + if exclusion: self.different_excluded.append(r) + self.active_exclusions.add(exclusion) else: self.different.append(r) else:
@@ -116,7 +119,10 @@ class PackageCompareResults(object): self.same.sort() def __str__(self): - return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total)) + return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions()) + + def unused_exclusions(self): + return set(exclude_packages) - self.active_exclusions def compare_file(reference, test, diffutils_sysroot): result = CompareResult()
Signature · 1 line
2.25.1
PATCH 5/6 selftest/reproducible: Sort the unused exclusion list
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
This makes comparisions between lists easier.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +1 -1
@@ -122,7 +122,7 @@ class PackageCompareResults(object): return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions()) def unused_exclusions(self): - return set(exclude_packages) - self.active_exclusions + return sorted(set(exclude_packages) - self.active_exclusions) def compare_file(reference, test, diffutils_sysroot): result = CompareResult()
Signature · 1 line
2.25.1
PATCH 6/6 diffoscope: Upgrade 136 -> 168
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Needed to support reproducible build testing on autobuilder
---
.../{diffoscope_136.bb => diffoscope_168.bb} | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
rename meta/recipes-support/diffoscope/{diffoscope_136.bb => diffoscope_168.bb} (46%)meta/recipes-support/diffoscope/diffoscope_168.bb renamed from meta/recipes-support/diffoscope/diffoscope_136.bb +8 -7
@@ -1,10 +1,5 @@ SUMMARY = "in-depth comparison of files, archives, and directories" -DESCRIPTION = "Tries to get to the bottom of what makes files or directories \ -different. It will recursively unpack archives of many kinds and transform \ -various binary formats into more human-readable form to compare them. \ -It can compare two tarballs, ISO images, or PDF just as easily." HOMEPAGE = "https://diffoscope.org/" -BUGTRACKER = "https://salsa.debian.org/reproducible-builds/diffoscope/-/issues" LICENSE = "GPL-3.0+" LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
@@ -12,12 +7,18 @@ PYPI_PACKAGE = "diffoscope" inherit pypi setuptools3 -SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73" -SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15" +SRC_URI[sha256sum] = "c6f1dc3e75b7e2e5ceac4f857fbd2ee0ddb3f0169c2b39ea9187af34208e98de" RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic" # Dependencies don't build for musl COMPATIBLE_HOST_libc-musl = 'null' +do_install_append_class-native() { + create_wrapper ${D}${bindir}/diffoscope \ + MAGIC=${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc \ + RPM_CONFIGDIR=${STAGING_LIBDIR_NATIVE}/rpm \ + RPM_ETCCONFIGDIR=${STAGING_DIR_NATIVE} +} + BBCLASSEXTEND = "native"
Signature · 1 line
2.25.1
Quoted 10 lines · Steve Sakoman
This patch series enables world reproducibility testing so that we can begin monitoring progress on reproducibile builds for dunfell. It also includes a version update for diffoscope to better enable debugging failed packages.
The main downside to merging this series is increased autobuilder load to run the world tests.
I've been testing this over the past couple of months and have updated the exclusion list for dunfell as I've encountered intemittent reproducibility issues (see selftest/reproducible:
adjust exclusion list for dunfell)
Much as I hate loading up the autobuilder, improving reproducibility in dunfell where we can relatively easily is a good thing.
It could be we just don't test it on as many targets as in master?
[which means I'm in favour of these patches]
Cheers,
Richard