Skip to contentSkip to conversation

Patch review

PATCH New Steve Sakoman 1 participant 5 messages Oct 25, 2021

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this set of patches for dunfell and have comments back by end of day Wednesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2804

The following changes since commit 62cdc20a2186ecd54d3a7131ec8f6937aa0229ed:

uninative: Upgrade to 3.4 (2021-10-25 10:23:54 +0100)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Joshua Watt (1):
classes/reproducible_build: Use atomic rename for SDE file

Richard Purdie (2):
rpm: Deterministically set vendor macro entry
reproducible_build: Work around caching issues

Steve Sakoman (1):
selftest/reproducible: adjust exclusion list for dunfell

 meta/classes/reproducible_build.bbclass      | 24 +++++++++++---------
 meta/lib/oeqa/selftest/cases/reproducible.py |  5 ----
 meta/recipes-devtools/rpm/rpm_4.14.2.1.bb    |  3 ++-
 3 files changed, 15 insertions(+), 17 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/4 rpm: Deterministically set vendor macro entry

From: Richard Purdie <[email protected]>

On an aarch64 build host, vendor is found to be "unknown", on x86 systems it is "pc". This filters through to the PLATFORM tag in target rpms.

We saw reproducibility test failures where the PLATFORM tags in noarch rpms were changing depending upon which host built them. Forcing the vendor value to a consistent one makes things deterministic.

(cherry picked from commit f6434075b2bdfc23c683d22281b674b1e6abde77)
---
 meta/recipes-devtools/rpm/rpm_4.14.2.1.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/rpm/rpm_4.14.2.1.bb +2 -1
index c93654aa8f..ab9f0e8e29 100644
@@ -62,7 +62,8 @@ export PYTHON_ABI
 # OE-core patches autoreconf to additionally run gnu-configize, which fails with this recipe
 EXTRA_AUTORECONF_append = " --exclude=gnu-configize"
 
-EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl"
+# Vendor is detected differently on x86 and aarch64 hosts and can feed into target packages
+EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl --with-vendor=pc"
 EXTRA_OECONF_append_libc-musl = " --disable-nls"
 
 # --sysconfdir prevents rpm from attempting to access machine-specific configuration in sysroot/etc; we need to have it in rootfs
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/4 reproducible_build: Work around caching issues

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

SOURCE_DATE_EPOCH can be expanded early in the parsing process before the class extensions are applied. This can mean the directory pointed to for the SDE can be incorrect until later in parsing. Cache the file name in the cached value and allow it to dynamically update.

This isn't ideal but avoding expansion of the variable likely isn't possible and I'm not sure how else to handle this. This works around the issue until a better solution can be found.

(cherry picked from commit 44dc97cd1223e4d2b635669627ec5f796838d42d)
---
 meta/classes/reproducible_build.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/reproducible_build.bbclass +7 -4
index f06e00d70d..43cf9dc894 100644
@@ -91,11 +91,14 @@ python create_source_date_epoch_stamp() {
 }
 
 def get_source_date_epoch_value(d):
-    cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
-    if cached:
+    epochfile = d.getVar('SDE_FILE')
+    cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
+    if cached and efile == epochfile:
         return cached
 
-    epochfile = d.getVar('SDE_FILE')
+    if cached and epochfile != efile:
+        bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
+
     source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
     if os.path.isfile(epochfile):
         with open(epochfile, 'r') as f:
@@ -113,7 +116,7 @@ def get_source_date_epoch_value(d):
     else:
         bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
 
-    d.setVar('__CACHED_SOURCE_DATE_EPOCH', str(source_date_epoch))
+    d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
     return str(source_date_epoch)
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/4 classes/reproducible_build: Use atomic rename for SDE file

In reply to Steve Sakoman

From: Joshua Watt <[email protected]>

If an existing source date epoch file was found during do_unpack, it was deleted and a new one would be written in its place. This causes a race with check-before-use code in get_source_date_epoch_value. Resolve the problem by making do_unpack write the new source date epoch to a temporary file, then do an atomic rename to ensure it's always present, and change the check-before-use code to use a EAFP exception instead of checking for file existence.

[YOCTO #14384]

(cherry picked from commit 0b5e3b33187bf78a2d62cc886463e4b27d6bd228)
---
 meta/classes/reproducible_build.bbclass | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
meta/classes/reproducible_build.bbclass +6 -7
index 43cf9dc894..62655c2a5b 100644
@@ -77,17 +77,16 @@ python create_source_date_epoch_stamp() {
     import oe.reproducible
 
     epochfile = d.getVar('SDE_FILE')
-    # If it exists we need to regenerate as the sources may have changed
-    if os.path.isfile(epochfile):
-        bb.debug(1, "Deleting existing SOURCE_DATE_EPOCH from: %s" % epochfile)
-        os.remove(epochfile)
+    tmp_file = "%s.new" % epochfile
 
     source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S'))
 
     bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
-    with open(epochfile, 'w') as f:
+    with open(tmp_file, 'w') as f:
         f.write(str(source_date_epoch))
+
+    os.rename(tmp_file, epochfile)
 }
 
 def get_source_date_epoch_value(d):
@@ -100,7 +99,7 @@ def get_source_date_epoch_value(d):
         bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
 
     source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
-    if os.path.isfile(epochfile):
+    try:
         with open(epochfile, 'r') as f:
             s = f.read()
             try:
@@ -113,7 +112,7 @@ def get_source_date_epoch_value(d):
                 bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s)
                 source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
         bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
-    else:
+    except FileNotFoundError:
         bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
 
     d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/4 selftest/reproducible: adjust exclusion list for dunfell

In reply to Steve Sakoman

Signed-off-be: Steve Sakoman <[email protected]>

---
 meta/lib/oeqa/selftest/cases/reproducible.py | 5 -----
 1 file changed, 5 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +0 -5
index 0e44ce4dbf..c8604a2054 100644
@@ -31,7 +31,6 @@ exclude_packages = [
 	'bootchart2-doc',
 	'epiphany',
 	'gcr',
-	'git',
 	'glide',
 	'go-dep',
 	'go-helloworld',
@@ -44,7 +43,6 @@ exclude_packages = [
 	'libcap-ng',
 	'libjson',
 	'libproxy',
-	'lsb-release',
 	'lttng-tools-dbg',
 	'lttng-tools-ptest',
 	'ltp',
@@ -55,15 +53,12 @@ exclude_packages = [
 	'pybootchartgui',
 	'qemu',
 	'quilt-ptest',
-	"rpm",
 	'rsync',
 	'ruby',
 	'stress-ng',
 	'systemd-bootchart',
 	'systemtap',
 	'valgrind-ptest',
-	'vim',
-	'webkitgtk',
 	]
 
 def is_excluded(package):
Signature · 1 line
2.25.1