Patch review
cover Patch review
Please review this set of patches for dunfell and have comments back by end of day Wednesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2804
The following changes since commit 62cdc20a2186ecd54d3a7131ec8f6937aa0229ed:
uninative: Upgrade to 3.4 (2021-10-25 10:23:54 +0100)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
Joshua Watt (1):
classes/reproducible_build: Use atomic rename for SDE file
Richard Purdie (2):
rpm: Deterministically set vendor macro entry
reproducible_build: Work around caching issues
Steve Sakoman (1):
selftest/reproducible: adjust exclusion list for dunfell
meta/classes/reproducible_build.bbclass | 24 +++++++++++--------- meta/lib/oeqa/selftest/cases/reproducible.py | 5 ---- meta/recipes-devtools/rpm/rpm_4.14.2.1.bb | 3 ++- 3 files changed, 15 insertions(+), 17 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 rpm: Deterministically set vendor macro entry
From: Richard Purdie <[email protected]>
On an aarch64 build host, vendor is found to be "unknown", on x86 systems it is "pc". This filters through to the PLATFORM tag in target rpms.
We saw reproducibility test failures where the PLATFORM tags in noarch rpms were changing depending upon which host built them. Forcing the vendor value to a consistent one makes things deterministic.
--- meta/recipes-devtools/rpm/rpm_4.14.2.1.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/rpm/rpm_4.14.2.1.bb +2 -1
@@ -62,7 +62,8 @@ export PYTHON_ABI # OE-core patches autoreconf to additionally run gnu-configize, which fails with this recipe EXTRA_AUTORECONF_append = " --exclude=gnu-configize" -EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl" +# Vendor is detected differently on x86 and aarch64 hosts and can feed into target packages +EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl --with-vendor=pc" EXTRA_OECONF_append_libc-musl = " --disable-nls" # --sysconfdir prevents rpm from attempting to access machine-specific configuration in sysroot/etc; we need to have it in rootfs
Signature · 1 line
2.25.1
PATCH 2/4 reproducible_build: Work around caching issues
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
SOURCE_DATE_EPOCH can be expanded early in the parsing process before the class extensions are applied. This can mean the directory pointed to for the SDE can be incorrect until later in parsing. Cache the file name in the cached value and allow it to dynamically update.
This isn't ideal but avoding expansion of the variable likely isn't possible and I'm not sure how else to handle this. This works around the issue until a better solution can be found.
--- meta/classes/reproducible_build.bbclass | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/reproducible_build.bbclass +7 -4
@@ -91,11 +91,14 @@ python create_source_date_epoch_stamp() { } def get_source_date_epoch_value(d): - cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH') - if cached: + epochfile = d.getVar('SDE_FILE') + cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None) + if cached and efile == epochfile: return cached - epochfile = d.getVar('SDE_FILE') + if cached and epochfile != efile: + bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile)) + source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK')) if os.path.isfile(epochfile): with open(epochfile, 'r') as f:
@@ -113,7 +116,7 @@ def get_source_date_epoch_value(d): else: bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch)) - d.setVar('__CACHED_SOURCE_DATE_EPOCH', str(source_date_epoch)) + d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile)) return str(source_date_epoch) export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
Signature · 1 line
2.25.1
PATCH 3/4 classes/reproducible_build: Use atomic rename for SDE file
In reply to Steve SakomanFrom: Joshua Watt <[email protected]>
If an existing source date epoch file was found during do_unpack, it was deleted and a new one would be written in its place. This causes a race with check-before-use code in get_source_date_epoch_value. Resolve the problem by making do_unpack write the new source date epoch to a temporary file, then do an atomic rename to ensure it's always present, and change the check-before-use code to use a EAFP exception instead of checking for file existence.
[YOCTO #14384]
--- meta/classes/reproducible_build.bbclass | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
meta/classes/reproducible_build.bbclass +6 -7
@@ -77,17 +77,16 @@ python create_source_date_epoch_stamp() { import oe.reproducible epochfile = d.getVar('SDE_FILE') - # If it exists we need to regenerate as the sources may have changed - if os.path.isfile(epochfile): - bb.debug(1, "Deleting existing SOURCE_DATE_EPOCH from: %s" % epochfile) - os.remove(epochfile) + tmp_file = "%s.new" % epochfile source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch) bb.utils.mkdirhier(d.getVar('SDE_DIR')) - with open(epochfile, 'w') as f: + with open(tmp_file, 'w') as f: f.write(str(source_date_epoch)) + + os.rename(tmp_file, epochfile) } def get_source_date_epoch_value(d):
@@ -100,7 +99,7 @@ def get_source_date_epoch_value(d): bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile)) source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK')) - if os.path.isfile(epochfile): + try: with open(epochfile, 'r') as f: s = f.read() try:
@@ -113,7 +112,7 @@ def get_source_date_epoch_value(d): bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s) source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK')) bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch) - else: + except FileNotFoundError: bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch)) d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
Signature · 1 line
2.25.1
PATCH 4/4 selftest/reproducible: adjust exclusion list for dunfell
In reply to Steve SakomanSigned-off-be: Steve Sakoman <[email protected]>
--- meta/lib/oeqa/selftest/cases/reproducible.py | 5 ----- 1 file changed, 5 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +0 -5
@@ -31,7 +31,6 @@ exclude_packages = [ 'bootchart2-doc', 'epiphany', 'gcr', - 'git', 'glide', 'go-dep', 'go-helloworld',
@@ -44,7 +43,6 @@ exclude_packages = [ 'libcap-ng', 'libjson', 'libproxy', - 'lsb-release', 'lttng-tools-dbg', 'lttng-tools-ptest', 'ltp',
@@ -55,15 +53,12 @@ exclude_packages = [ 'pybootchartgui', 'qemu', 'quilt-ptest', - "rpm", 'rsync', 'ruby', 'stress-ng', 'systemd-bootchart', 'systemtap', 'valgrind-ptest', - 'vim', - 'webkitgtk', ] def is_excluded(package):
Signature · 1 line
2.25.1