Patch review
cover Patch review
Please review this next set of patches for dunfell and have comments back by end of day Wednesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1700
The following changes since commit 02870c7fbaaa1c3869ecb439f5c58fcf40a533be:
binutils: fix CVE-2020-16592/16598 (2020-12-14 05:58:27 -1000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
Alexander Kanavin (2):
selftest/reproducible: enable world reproducibility test selftest/reproducible: add an exclusion list for items that are not
yet reproducible
Richard Purdie (5):
grub: Fix build reproducibility issue
grub: Add second fix for determinism issue
u-boot-tools: Fix reproducibility issue
groff: Fix reproducibility issue
man-db: Avoid reproducibility failures after fixing groff-native
Steve Sakoman (1):
selftest/reproducible: add packages to exclusion list for dunfell
meta/lib/oeqa/selftest/cases/reproducible.py | 96 ++++++++++++++++++- meta/recipes-bsp/grub/files/determinism.patch | 40 ++++++++ meta/recipes-bsp/grub/grub2.inc | 1 + meta/recipes-bsp/u-boot/u-boot-tools.inc | 15 +++ meta/recipes-extended/groff/groff_1.22.4.bb | 2 +- meta/recipes-extended/man-db/man-db_2.9.0.bb | 5 + 6 files changed, 154 insertions(+), 5 deletions(-) create mode 100644 meta/recipes-bsp/grub/files/determinism.patch
Signature · 1 line
2.17.1
PATCH 1/8 grub: Fix build reproducibility issue
From: Richard Purdie <[email protected]>
We're seeing reproducibility issue on the autobuilder due to changing module dependency ordering. Add some sorting to an awk script to fix this.
--- meta/recipes-bsp/grub/files/determinism.patch | 24 +++++++++++++++++++ meta/recipes-bsp/grub/grub2.inc | 1 + 2 files changed, 25 insertions(+) create mode 100644 meta/recipes-bsp/grub/files/determinism.patch
meta/recipes-bsp/grub/files/determinism.patch new file +24 -0
@@ -0,0 +1,24 @@ +The output in moddep.lst generated from syminfo.lst using genmoddep.awk is +not deterministic since the order of the dependencies on each line can vary +depending on how awk sorts the values in the array. + +Be deterministic in the output by sorting the dependencies on each line. + +Upstream-Status: Pending +Richard Purdie <[email protected]> + +Index: grub-2.04/grub-core/genmoddep.awk +=================================================================== +--- grub-2.04.orig/grub-core/genmoddep.awk ++++ grub-2.04/grub-core/genmoddep.awk +@@ -59,7 +59,9 @@ END { + } + modlist = "" + depcount[mod] = 0 +- for (depmod in uniqmods) { ++ n = asorti(uniqmods, w) ++ for (i = 1; i <= n; i++) { ++ depmod = w[i] + modlist = modlist " " depmod; + inverse_dependencies[depmod] = inverse_dependencies[depmod] " " mod + depcount[mod]++
meta/recipes-bsp/grub/grub2.inc +1 -0
@@ -27,6 +27,7 @@ SRC_URI = "${GNU_MIRROR}/grub/grub-${PV}.tar.gz \ file://script-Remove-unused-fields-from-grub_script_functio.patch \ file://CVE-2020-15706-script-Avoid-a-use-after-free-when-redefining-a-func.patch \ file://CVE-2020-15707-linux-Fix-integer-overflows-in-initrd-size-handling.patch \ + file://determinism.patch \ " SRC_URI[md5sum] = "5ce674ca6b2612d8939b9e6abed32934" SRC_URI[sha256sum] = "f10c85ae3e204dbaec39ae22fa3c5e99f0665417e91c2cb49b7e5031658ba6ea"
Signature · 1 line
2.17.1
PATCH 2/8 grub: Add second fix for determinism issue
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
There is a second list sorting problem in a generator script within grub, add a sort() of a list to resolve this.
--- meta/recipes-bsp/grub/files/determinism.patch | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
meta/recipes-bsp/grub/files/determinism.patch +16 -0
@@ -4,6 +4,10 @@ depending on how awk sorts the values in the array. Be deterministic in the output by sorting the dependencies on each line. +Also, the output of the SOURCES lines in grub-core/Makefile.core.am, generated +from grub-core/Makefile.core.def with gentpl.py is not deterministic due to +missing sorting of the list used to generate it. Add such a sort. + Upstream-Status: Pending Richard Purdie <[email protected]>
@@ -22,3 +26,15 @@ Index: grub-2.04/grub-core/genmoddep.awk modlist = modlist " " depmod; inverse_dependencies[depmod] = inverse_dependencies[depmod] " " mod depcount[mod]++ +Index: grub-2.04/gentpl.py +=================================================================== +--- grub-2.04.orig/gentpl.py ++++ grub-2.04/gentpl.py +@@ -568,6 +568,7 @@ def foreach_platform_value(defn, platfor + for group in RMAP[platform]: + for value in defn.find_all(group + suffix): + r.append(closure(value)) ++ r.sort() + return ''.join(r) + + def platform_conditional(platform, closure):
Signature · 1 line
2.17.1
PATCH 3/8 u-boot-tools: Fix reproducibility issue
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
If you build on a system with git < 2.14 from scratch (e.g. debian9), the tree will be marked as "dirty" and the version will include "-dirty", leading to a reproducibility problem. The issue is the inode count for Licnses/README changing due to do_populate_lic hardlinking a copy of the file. We avoid this by ensuring the index is updated with a "git diff" before the u-boot machinery tries to determine the version.
build$ ../git/scripts/setlocalversion ../git
""
build$ ln ../git/
build$ ln ../git/README ../foo
build$ ../git/scripts/setlocalversion ../git
""-dirty
(i.e. creating a hardlink dirties the index)
--- meta/recipes-bsp/u-boot/u-boot-tools.inc | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
meta/recipes-bsp/u-boot/u-boot-tools.inc +15 -0
@@ -23,6 +23,21 @@ SED_CONFIG_EFI_armeb = '' SED_CONFIG_EFI_aarch64 = '' do_compile () { + # Yes, this is crazy. If you build on a system with git < 2.14 from scratch, the tree will + # be marked as "dirty" and the version will include "-dirty", leading to a reproducibility problem. + # The issue is the inode count for Licnses/README changing due to do_populate_lic hardlinking a + # copy of the file. We avoid this by ensuring the index is updated with a "git diff" before the + # u-boot machinery tries to determine the version. + # + # build$ ../git/scripts/setlocalversion ../git + # "" + # build$ ln ../git/ + # build$ ln ../git/README ../foo + # build$ ../git/scripts/setlocalversion ../git + # ""-dirty + # (i.e. creating a hardlink dirties the index) + cd ${S}; git diff; cd ${B} + oe_runmake -C ${S} sandbox_defconfig O=${B} # Disable CONFIG_CMD_LICENSE, license.h is not used by tools and
Signature · 1 line
2.17.1
PATCH 4/8 groff: Fix reproducibility issue
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
groff chooses a default papersize depending on the value from /etc/papersize and failing that, the search domain in /etc/resolv.conf based on the comment in configure:
"""
If the top-level domain is two letters and it's not 'us' or 'ca' then they probably use A4 paper.
"""
Oddly, my system sets to "a4" in /etc/papersize which means it defaults to "letter" since its != "A4".
These defaults ripple through to cause the output of man-db to change depending on which default value was selected.
To resolve this, set a default of "A4" since that covers the larger population of the two default values.
--- meta/recipes-extended/groff/groff_1.22.4.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-extended/groff/groff_1.22.4.bb +1 -1
@@ -28,7 +28,7 @@ MULTILIB_SCRIPTS = "${PN}:${bindir}/gpinyin ${PN}:${bindir}/groffer ${PN}:${bind EXTRA_OECONF = "--without-x --without-doc" PARALLEL_MAKE = "" -CACHED_CONFIGUREVARS += "ac_cv_path_PERL='/usr/bin/env perl' ac_cv_path_BASH_PROG='no'" +CACHED_CONFIGUREVARS += "ac_cv_path_PERL='/usr/bin/env perl' ac_cv_path_BASH_PROG='no' PAGE=A4" do_install_append() { # Some distros have both /bin/perl and /usr/bin/perl, but we set perl location
Signature · 1 line
2.17.1
PATCH 5/8 man-db: Avoid reproducibility failures after fixing groff-native
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Add some temporary bumps to versions to change the output hash so we really can take advantage of the groff-native reproducibility fix.
--- meta/recipes-extended/man-db/man-db_2.9.0.bb | 5 +++++ 1 file changed, 5 insertions(+)
meta/recipes-extended/man-db/man-db_2.9.0.bb +5 -0
@@ -21,6 +21,11 @@ inherit gettext pkgconfig autotools systemd EXTRA_OECONF = "--with-pager=less --with-systemdsystemunitdir=${systemd_unitdir}/system" EXTRA_AUTORECONF += "-I ${S}/gl/m4" +# Can be dropped when the output next changes, avoids failures after +# reproducibility issues +PR = "r1" +HASHEQUIV_HASH_VERSION .= ".1" + do_install() { autotools_do_install
Signature · 1 line
2.17.1
PATCH 6/8 selftest/reproducible: enable world reproducibility test
In reply to Steve SakomanFrom: Alexander Kanavin <[email protected]>
Add systemd and pam distro features, and commercial license flag to include more recipes into the world set.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +3 -1
@@ -79,7 +79,7 @@ def compare_file(reference, test, diffutils_sysroot): class ReproducibleTests(OESelftestTestCase): package_classes = ['deb', 'ipk'] - images = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline'] + images = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'world'] save_results = False if 'OEQA_DEBUGGING_SAVED_OUTPUT' in os.environ: save_results = os.environ['OEQA_DEBUGGING_SAVED_OUTPUT']
@@ -150,6 +150,8 @@ class ReproducibleTests(OESelftestTestCase): PACKAGE_CLASSES = "{package_classes}" INHIBIT_PACKAGE_STRIP = "1" TMPDIR = "{tmpdir}" + LICENSE_FLAGS_WHITELIST = "commercial" + DISTRO_FEATURES_append = ' systemd pam' ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes), tmpdir=tmpdir)
Signature · 1 line
2.17.1
PATCH 7/8 selftest/reproducible: add an exclusion list for items that are not yet reproducible
In reply to Steve SakomanFrom: Alexander Kanavin <[email protected]>
Hopefully over time this list will be reduced to an empty one.
Non-reproducible excluded packages are not given to diffoscope and do not cause a failure, but still saved side-by-side with non-reproducible failing ones to make investigation easier.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 82 +++++++++++++++++++- 1 file changed, 79 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +79 -3
@@ -17,6 +17,72 @@ import stat import os import datetime +# For sample packages, see: +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-0t7wr_oo/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-4s9ejwyp/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-haiwdlbr/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-hwds3mcl/ +# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201203-sua0pzvc/ +# (both packages/ and packages-excluded/) +exclude_packages = [ + 'acpica-src', + 'babeltrace2-ptest', + 'bootchart2-doc', + 'cups', + 'cwautomacros', + 'dtc', + 'efivar', + 'epiphany', + 'gcr', + 'git', + 'glide', + 'go-dep', + 'go-helloworld', + 'go-runtime', + 'go_', + 'groff', + 'gst-devtools', + 'gstreamer1.0-python', + 'gtk-doc', + 'igt-gpu-tools', + 'kernel-devsrc', + 'libaprutil', + 'libcap-ng', + 'libhandy-1-src', + 'libid3tag', + 'libproxy', + 'libsecret-dev', + 'libsecret-src', + 'lttng-tools-dbg', + 'lttng-tools-ptest', + 'ltp', + 'meson', + 'ovmf-shell-efi', + 'parted-ptest', + 'perf', + 'python3-cython', + 'qemu', + 'quilt-ptest', + 'rsync', + 'ruby', + 'spirv-tools-dev', + 'swig', + 'syslinux-misc', + 'systemd-bootchart', + 'valgrind-ptest', + 'vim', + 'watchdog', + 'xmlto', + 'xorg-minimal-fonts' + ] + +def is_excluded(package): + package_name = os.path.basename(package) + for i in exclude_packages: + if package_name.startswith(i): + return True + return False + MISSING = 'MISSING' DIFFERENT = 'DIFFERENT' SAME = 'SAME'
@@ -39,6 +105,7 @@ class PackageCompareResults(object): self.total = [] self.missing = [] self.different = [] + self.different_excluded = [] self.same = [] def add_result(self, r):
@@ -46,7 +113,10 @@ class PackageCompareResults(object): if r.status == MISSING: self.missing.append(r) elif r.status == DIFFERENT: - self.different.append(r) + if is_excluded(r.reference): + self.different_excluded.append(r) + else: + self.different.append(r) else: self.same.append(r)
@@ -54,10 +124,11 @@ class PackageCompareResults(object): self.total.sort() self.missing.sort() self.different.sort() + self.different_excluded.sort() self.same.sort() def __str__(self): - return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total)) + return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total)) def compare_file(reference, test, diffutils_sysroot): result = CompareResult()
@@ -211,6 +282,7 @@ class ReproducibleTests(OESelftestTestCase): self.write_package_list(package_class, 'missing', result.missing) self.write_package_list(package_class, 'different', result.different) + self.write_package_list(package_class, 'different_excluded', result.different_excluded) self.write_package_list(package_class, 'same', result.same) if self.save_results:
@@ -218,8 +290,12 @@ class ReproducibleTests(OESelftestTestCase): self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)])) self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)])) + for d in result.different_excluded: + self.copy_file(d.reference, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.reference)])) + self.copy_file(d.test, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.test)])) + if result.missing or result.different: - fails.append("The following %s packages are missing or different: %s" % + fails.append("The following %s packages are missing or different and not in exclusion list: %s" % (c, '\n'.join(r.test for r in (result.missing + result.different)))) # Clean up empty directories
Signature · 1 line
2.17.1
PATCH 8/8 selftest/reproducible: add packages to exclusion list for dunfell
In reply to Steve SakomanSigned-off-be: Steve Sakoman <[email protected]>
--- meta/lib/oeqa/selftest/cases/reproducible.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
meta/lib/oeqa/selftest/cases/reproducible.py +10 -0
@@ -27,6 +27,7 @@ import datetime exclude_packages = [ 'acpica-src', 'babeltrace2-ptest', + 'bind', 'bootchart2-doc', 'cups', 'cwautomacros',
@@ -41,15 +42,18 @@ exclude_packages = [ 'go-runtime', 'go_', 'groff', + 'grub-efi', 'gst-devtools', 'gstreamer1.0-python', 'gtk-doc', + 'hwlatdetect', 'igt-gpu-tools', 'kernel-devsrc', 'libaprutil', 'libcap-ng', 'libhandy-1-src', 'libid3tag', + 'libjson', 'libproxy', 'libsecret-dev', 'libsecret-src',
@@ -60,18 +64,24 @@ exclude_packages = [ 'ovmf-shell-efi', 'parted-ptest', 'perf', + 'piglit', + 'pybootchartgui', 'python3-cython', 'qemu', 'quilt-ptest', + "rpm", 'rsync', 'ruby', 'spirv-tools-dev', + 'stress-ng', 'swig', 'syslinux-misc', 'systemd-bootchart', + 'systemtap', 'valgrind-ptest', 'vim', 'watchdog', + 'webkitgtk', 'xmlto', 'xorg-minimal-fonts' ]
Signature · 1 line
2.17.1