Skip to contentSkip to conversation

Patch review

PATCH New Steve Sakoman 1 participant 9 messages Dec 28, 2020

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this next set of patches for dunfell and have comments back by end of day Wednesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1700

The following changes since commit 02870c7fbaaa1c3869ecb439f5c58fcf40a533be:

binutils: fix CVE-2020-16592/16598 (2020-12-14 05:58:27 -1000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Alexander Kanavin (2):
selftest/reproducible: enable world reproducibility test selftest/reproducible: add an exclusion list for items that are not
yet reproducible

Richard Purdie (5):
grub: Fix build reproducibility issue
grub: Add second fix for determinism issue
u-boot-tools: Fix reproducibility issue
groff: Fix reproducibility issue
man-db: Avoid reproducibility failures after fixing groff-native

Steve Sakoman (1):
selftest/reproducible: add packages to exclusion list for dunfell

 meta/lib/oeqa/selftest/cases/reproducible.py  | 96 ++++++++++++++++++-
 meta/recipes-bsp/grub/files/determinism.patch | 40 ++++++++
 meta/recipes-bsp/grub/grub2.inc               |  1 +
 meta/recipes-bsp/u-boot/u-boot-tools.inc      | 15 +++
 meta/recipes-extended/groff/groff_1.22.4.bb   |  2 +-
 meta/recipes-extended/man-db/man-db_2.9.0.bb  |  5 +
 6 files changed, 154 insertions(+), 5 deletions(-)
 create mode 100644 meta/recipes-bsp/grub/files/determinism.patch
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/8 grub: Fix build reproducibility issue

From: Richard Purdie <[email protected]>

We're seeing reproducibility issue on the autobuilder due to changing module dependency ordering. Add some sorting to an awk script to fix this.

(cherry picked from commit 925ddd5edccbfec52ff45c1b54ab2ae1bfe0d57c)
---
 meta/recipes-bsp/grub/files/determinism.patch | 24 +++++++++++++++++++
 meta/recipes-bsp/grub/grub2.inc               |  1 +
 2 files changed, 25 insertions(+)
 create mode 100644 meta/recipes-bsp/grub/files/determinism.patch
meta/recipes-bsp/grub/files/determinism.patch new file +24 -0
new file mode 100644
index 0000000000..c4b1d3a2a8
@@ -0,0 +1,24 @@
+The output in moddep.lst generated from syminfo.lst using genmoddep.awk is
+not deterministic since the order of the dependencies on each line can vary
+depending on how awk sorts the values in the array.
+
+Be deterministic in the output by sorting the dependencies on each line.
+
+Upstream-Status: Pending
+Richard Purdie <[email protected]>
+
+Index: grub-2.04/grub-core/genmoddep.awk
+===================================================================
+--- grub-2.04.orig/grub-core/genmoddep.awk
++++ grub-2.04/grub-core/genmoddep.awk
+@@ -59,7 +59,9 @@ END {
+     }
+     modlist = ""
+     depcount[mod] = 0
+-    for (depmod in uniqmods) {
++    n = asorti(uniqmods, w)
++    for (i = 1; i <= n; i++) {
++      depmod = w[i]
+       modlist = modlist " " depmod;
+       inverse_dependencies[depmod] = inverse_dependencies[depmod] " " mod
+       depcount[mod]++
meta/recipes-bsp/grub/grub2.inc +1 -0
index 4ec7d0b0fc..2c55852ef0 100644
@@ -27,6 +27,7 @@ SRC_URI = "${GNU_MIRROR}/grub/grub-${PV}.tar.gz \
            file://script-Remove-unused-fields-from-grub_script_functio.patch \
            file://CVE-2020-15706-script-Avoid-a-use-after-free-when-redefining-a-func.patch \
            file://CVE-2020-15707-linux-Fix-integer-overflows-in-initrd-size-handling.patch \
+           file://determinism.patch \
 "
 SRC_URI[md5sum] = "5ce674ca6b2612d8939b9e6abed32934"
 SRC_URI[sha256sum] = "f10c85ae3e204dbaec39ae22fa3c5e99f0665417e91c2cb49b7e5031658ba6ea"
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/8 grub: Add second fix for determinism issue

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

There is a second list sorting problem in a generator script within grub, add a sort() of a list to resolve this.

(cherry picked from commit cb5e96e05930eaff4d679166416d6c84d6e3236b)
---
 meta/recipes-bsp/grub/files/determinism.patch | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
meta/recipes-bsp/grub/files/determinism.patch +16 -0
index c4b1d3a2a8..fac80e7b54 100644
@@ -4,6 +4,10 @@ depending on how awk sorts the values in the array.
 
 Be deterministic in the output by sorting the dependencies on each line.
 
+Also, the output of the SOURCES lines in grub-core/Makefile.core.am, generated
+from grub-core/Makefile.core.def with gentpl.py is not deterministic due to
+missing sorting of the list used to generate it. Add such a sort.
+
 Upstream-Status: Pending
 Richard Purdie <[email protected]>
 
@@ -22,3 +26,15 @@ Index: grub-2.04/grub-core/genmoddep.awk
        modlist = modlist " " depmod;
        inverse_dependencies[depmod] = inverse_dependencies[depmod] " " mod
        depcount[mod]++
+Index: grub-2.04/gentpl.py
+===================================================================
+--- grub-2.04.orig/gentpl.py
++++ grub-2.04/gentpl.py
+@@ -568,6 +568,7 @@ def foreach_platform_value(defn, platfor
+     for group in RMAP[platform]:
+         for value in defn.find_all(group + suffix):
+             r.append(closure(value))
++    r.sort()
+     return ''.join(r)
+ 
+ def platform_conditional(platform, closure):
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/8 u-boot-tools: Fix reproducibility issue

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

If you build on a system with git < 2.14 from scratch (e.g. debian9), the tree will be marked as "dirty" and the version will include "-dirty", leading to a reproducibility problem. The issue is the inode count for Licnses/README changing due to do_populate_lic hardlinking a copy of the file. We avoid this by ensuring the index is updated with a "git diff" before the u-boot machinery tries to determine the version.

build$ ../git/scripts/setlocalversion ../git
""
build$ ln ../git/
build$ ln ../git/README ../foo
build$ ../git/scripts/setlocalversion ../git
""-dirty

(i.e. creating a hardlink dirties the index)

(cherry picked from commit a4de915af9e2c4d365527ad09286483b2ee904fc)
---
 meta/recipes-bsp/u-boot/u-boot-tools.inc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
meta/recipes-bsp/u-boot/u-boot-tools.inc +15 -0
index 8ae290acc6..4ed936a70d 100644
@@ -23,6 +23,21 @@ SED_CONFIG_EFI_armeb = ''
 SED_CONFIG_EFI_aarch64 = ''
 
 do_compile () {
+	# Yes, this is crazy. If you build on a system with git < 2.14 from scratch, the tree will
+	# be marked as "dirty" and the version will include "-dirty", leading to a reproducibility problem.
+	# The issue is the inode count for Licnses/README changing due to do_populate_lic hardlinking a
+	# copy of the file. We avoid this by ensuring the index is updated with a "git diff" before the
+	# u-boot machinery tries to determine the version.
+	#
+	# build$ ../git/scripts/setlocalversion ../git
+	# ""
+	# build$ ln ../git/
+	# build$ ln ../git/README ../foo
+	# build$ ../git/scripts/setlocalversion ../git
+	# ""-dirty
+	# (i.e. creating a hardlink dirties the index)
+	cd ${S}; git diff; cd ${B}
+
 	oe_runmake -C ${S} sandbox_defconfig O=${B}
 
 	# Disable CONFIG_CMD_LICENSE, license.h is not used by tools and
Signature · 1 line
2.17.1
#4 lore raw

PATCH 4/8 groff: Fix reproducibility issue

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

groff chooses a default papersize depending on the value from /etc/papersize and failing that, the search domain in /etc/resolv.conf based on the comment in configure:

"""
If the top-level domain is two letters and it's not 'us' or 'ca' then they probably use A4 paper.
"""

Oddly, my system sets to "a4" in /etc/papersize which means it defaults to "letter" since its != "A4".

These defaults ripple through to cause the output of man-db to change depending on which default value was selected.

To resolve this, set a default of "A4" since that covers the larger population of the two default values.

(cherry picked from commit 6cb511bfbd18258c782ee18f054a2590e4daaddc)
---
 meta/recipes-extended/groff/groff_1.22.4.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-extended/groff/groff_1.22.4.bb +1 -1
index e398478349..0867452ce7 100644
@@ -28,7 +28,7 @@ MULTILIB_SCRIPTS = "${PN}:${bindir}/gpinyin ${PN}:${bindir}/groffer ${PN}:${bind
 EXTRA_OECONF = "--without-x --without-doc"
 PARALLEL_MAKE = ""
 
-CACHED_CONFIGUREVARS += "ac_cv_path_PERL='/usr/bin/env perl' ac_cv_path_BASH_PROG='no'"
+CACHED_CONFIGUREVARS += "ac_cv_path_PERL='/usr/bin/env perl' ac_cv_path_BASH_PROG='no' PAGE=A4"
 
 do_install_append() {
 	# Some distros have both /bin/perl and /usr/bin/perl, but we set perl location
Signature · 1 line
2.17.1
#5 lore raw

PATCH 5/8 man-db: Avoid reproducibility failures after fixing groff-native

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Add some temporary bumps to versions to change the output hash so we really can take advantage of the groff-native reproducibility fix.

(cherry picked from commit af9bd5a03ecf45e92868bc49b1de5c27d94eff6f)
---
 meta/recipes-extended/man-db/man-db_2.9.0.bb | 5 +++++
 1 file changed, 5 insertions(+)
meta/recipes-extended/man-db/man-db_2.9.0.bb +5 -0
index 5b017e8023..493fe817e5 100644
@@ -21,6 +21,11 @@ inherit gettext pkgconfig autotools systemd
 EXTRA_OECONF = "--with-pager=less --with-systemdsystemunitdir=${systemd_unitdir}/system"
 EXTRA_AUTORECONF += "-I ${S}/gl/m4"
 
+# Can be dropped when the output next changes, avoids failures after
+# reproducibility issues
+PR = "r1"
+HASHEQUIV_HASH_VERSION .= ".1"
+
 do_install() {
 	autotools_do_install
 
Signature · 1 line
2.17.1
#6 lore raw

PATCH 6/8 selftest/reproducible: enable world reproducibility test

In reply to Steve Sakoman

From: Alexander Kanavin <[email protected]>

Add systemd and pam distro features, and commercial license flag to include more recipes into the world set.

(cherry picked from commit ed8f94942c07784f9f6eef5c00b75aa9b398ea5e)
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +3 -1
index 5d3959be77..89924f3b9e 100644
@@ -79,7 +79,7 @@ def compare_file(reference, test, diffutils_sysroot):
 
 class ReproducibleTests(OESelftestTestCase):
     package_classes = ['deb', 'ipk']
-    images = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline']
+    images = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'world']
     save_results = False
     if 'OEQA_DEBUGGING_SAVED_OUTPUT' in os.environ:
         save_results = os.environ['OEQA_DEBUGGING_SAVED_OUTPUT']
@@ -150,6 +150,8 @@ class ReproducibleTests(OESelftestTestCase):
             PACKAGE_CLASSES = "{package_classes}"
             INHIBIT_PACKAGE_STRIP = "1"
             TMPDIR = "{tmpdir}"
+            LICENSE_FLAGS_WHITELIST = "commercial"
+            DISTRO_FEATURES_append = ' systemd pam'
             ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes),
                         tmpdir=tmpdir)
 
Signature · 1 line
2.17.1
#7 lore raw

PATCH 7/8 selftest/reproducible: add an exclusion list for items that are not yet reproducible

In reply to Steve Sakoman

From: Alexander Kanavin <[email protected]>

Hopefully over time this list will be reduced to an empty one.

Non-reproducible excluded packages are not given to diffoscope and do not cause a failure, but still saved side-by-side with non-reproducible failing ones to make investigation easier.

(cherry picked from commit 406bd0d48d8f90e2c836f7d3e204f21d5f13c833)
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 82 +++++++++++++++++++-
 1 file changed, 79 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +79 -3
index 89924f3b9e..181c08b4c8 100644
@@ -17,6 +17,72 @@ import stat
 import os
 import datetime
 
+# For sample packages, see:
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-0t7wr_oo/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-4s9ejwyp/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-haiwdlbr/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-hwds3mcl/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201203-sua0pzvc/
+# (both packages/ and packages-excluded/)
+exclude_packages = [
+	'acpica-src',
+	'babeltrace2-ptest',
+	'bootchart2-doc',
+	'cups',
+	'cwautomacros',
+	'dtc',
+	'efivar',
+	'epiphany',
+	'gcr',
+	'git',
+	'glide',
+	'go-dep',
+	'go-helloworld',
+	'go-runtime',
+	'go_',
+	'groff',
+	'gst-devtools',
+	'gstreamer1.0-python',
+	'gtk-doc',
+	'igt-gpu-tools',
+        'kernel-devsrc',
+	'libaprutil',
+	'libcap-ng',
+	'libhandy-1-src',
+	'libid3tag',
+	'libproxy',
+	'libsecret-dev',
+	'libsecret-src',
+	'lttng-tools-dbg',
+	'lttng-tools-ptest',
+	'ltp',
+	'meson',
+	'ovmf-shell-efi',
+	'parted-ptest',
+	'perf',
+	'python3-cython',
+	'qemu',
+	'quilt-ptest',
+	'rsync',
+	'ruby',
+	'spirv-tools-dev',
+	'swig',
+	'syslinux-misc',
+	'systemd-bootchart',
+	'valgrind-ptest',
+	'vim',
+	'watchdog',
+	'xmlto',
+	'xorg-minimal-fonts'
+	]
+
+def is_excluded(package):
+    package_name = os.path.basename(package)
+    for i in exclude_packages:
+        if package_name.startswith(i):
+            return True
+    return False
+
 MISSING = 'MISSING'
 DIFFERENT = 'DIFFERENT'
 SAME = 'SAME'
@@ -39,6 +105,7 @@ class PackageCompareResults(object):
         self.total = []
         self.missing = []
         self.different = []
+        self.different_excluded = []
         self.same = []
 
     def add_result(self, r):
@@ -46,7 +113,10 @@ class PackageCompareResults(object):
         if r.status == MISSING:
             self.missing.append(r)
         elif r.status == DIFFERENT:
-            self.different.append(r)
+            if is_excluded(r.reference):
+                self.different_excluded.append(r)
+            else:
+                self.different.append(r)
         else:
             self.same.append(r)
 
@@ -54,10 +124,11 @@ class PackageCompareResults(object):
         self.total.sort()
         self.missing.sort()
         self.different.sort()
+        self.different_excluded.sort()
         self.same.sort()
 
     def __str__(self):
-        return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total))
+        return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total))
 
 def compare_file(reference, test, diffutils_sysroot):
     result = CompareResult()
@@ -211,6 +282,7 @@ class ReproducibleTests(OESelftestTestCase):
 
                 self.write_package_list(package_class, 'missing', result.missing)
                 self.write_package_list(package_class, 'different', result.different)
+                self.write_package_list(package_class, 'different_excluded', result.different_excluded)
                 self.write_package_list(package_class, 'same', result.same)
 
                 if self.save_results:
@@ -218,8 +290,12 @@ class ReproducibleTests(OESelftestTestCase):
                         self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
                         self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
 
+                    for d in result.different_excluded:
+                        self.copy_file(d.reference, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.reference)]))
+                        self.copy_file(d.test, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.test)]))
+
                 if result.missing or result.different:
-                    fails.append("The following %s packages are missing or different: %s" %
+                    fails.append("The following %s packages are missing or different and not in exclusion list: %s" %
                             (c, '\n'.join(r.test for r in (result.missing + result.different))))
 
         # Clean up empty directories
Signature · 1 line
2.17.1
#8 lore raw

PATCH 8/8 selftest/reproducible: add packages to exclusion list for dunfell

In reply to Steve Sakoman

Signed-off-be: Steve Sakoman <[email protected]>

---
 meta/lib/oeqa/selftest/cases/reproducible.py | 10 ++++++++++
 1 file changed, 10 insertions(+)
meta/lib/oeqa/selftest/cases/reproducible.py +10 -0
index 181c08b4c8..5ebf7875d9 100644
@@ -27,6 +27,7 @@ import datetime
 exclude_packages = [
 	'acpica-src',
 	'babeltrace2-ptest',
+	'bind',
 	'bootchart2-doc',
 	'cups',
 	'cwautomacros',
@@ -41,15 +42,18 @@ exclude_packages = [
 	'go-runtime',
 	'go_',
 	'groff',
+	'grub-efi',
 	'gst-devtools',
 	'gstreamer1.0-python',
 	'gtk-doc',
+	'hwlatdetect',
 	'igt-gpu-tools',
         'kernel-devsrc',
 	'libaprutil',
 	'libcap-ng',
 	'libhandy-1-src',
 	'libid3tag',
+	'libjson',
 	'libproxy',
 	'libsecret-dev',
 	'libsecret-src',
@@ -60,18 +64,24 @@ exclude_packages = [
 	'ovmf-shell-efi',
 	'parted-ptest',
 	'perf',
+	'piglit',
+	'pybootchartgui',
 	'python3-cython',
 	'qemu',
 	'quilt-ptest',
+	"rpm",
 	'rsync',
 	'ruby',
 	'spirv-tools-dev',
+	'stress-ng',
 	'swig',
 	'syslinux-misc',
 	'systemd-bootchart',
+	'systemtap',
 	'valgrind-ptest',
 	'vim',
 	'watchdog',
+	'webkitgtk',
 	'xmlto',
 	'xorg-minimal-fonts'
 	]
Signature · 1 line
2.17.1