Skip to contentSkip to conversation

Patch review

PATCH New Steve Sakoman 1 participant 15 messages Nov 11, 2021

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this set of patches for dunfell and have comments back by end of day Friday.

The following changes since commit 38fc0807eea14dc12610da4ba73c082d5a4b0744:

meta/scripts: Manual git url branch additions (2021-11-03 08:43:53 -1000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Jose Quaresma (1):
sstate: another fix for touching files inside pseudo

Joshua Watt (1):
oeqa: reproducible: Fix test not producing diffs

Khem Raj (1):
webkitgtk: Fix reproducibility in minibrowser

Marek Vasut (1):
piglit: upgrade to latest revision

Mark Hatle (1):
reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

Mingli Yu (1):
python3-magic: add the missing rdepends

Richard Purdie (6):
linunistring: Add missing gperf-native dependency
pseudo: Add in ability to flush database with shutdown request
pseudo: Add fcntl64 wrapper
mirrors: Add uninative mirror on kernel.org
sstate: Ensure SDE is accounted for in package task timestamps
sstate: Avoid deploy_source_date_epoch sstate when unneeded

Steve Sakoman (2):
python3-magic: add missing DEPENDS selftest/reproducible: add webkitgtk back to exclusion list for
dunfell

 meta/classes/mirrors.bbclass                  |  1 +
 meta/classes/reproducible_build.bbclass       | 53 ++++++++++++-------
 meta/classes/sstate.bbclass                   | 34 +++++++++---
 .../oeqa/selftest/cases/diffoscope/A/file.txt |  1 +
 .../oeqa/selftest/cases/diffoscope/B/file.txt |  1 +
 meta/lib/oeqa/selftest/cases/reproducible.py  | 29 +++++++++-
 meta/recipes-devtools/pseudo/pseudo_git.bb    |  2 +-
 .../python/python3-magic_0.4.15.bb            |  7 ++-
 ...ssing-include-for-htobe32-definition.patch | 27 ++++++++++
 ...file.py-make-test-lists-reproducible.patch | 31 +++++++++++
 ...gen_tcs-tes_input_tests.py-do-not-ha.patch | 44 +++++++++++++++
 ...lizer.py-make-.gz-files-reproducible.patch | 30 +++++++++++
 ...sort-the-file-list-before-working-on.patch | 28 ++++++++++
 ...t-shader.c-do-not-hardcode-build-pat.patch | 30 +++++++++++
 meta/recipes-graphics/piglit/piglit_git.bb    | 12 ++++-
 .../0001-MiniBrowser-Fix-reproduciblity.patch | 31 +++++++++++
 meta/recipes-sato/webkit/webkitgtk_2.28.4.bb  |  1 +
 .../libunistring/libunistring_0.9.10.bb       |  1 +
 18 files changed, 333 insertions(+), 30 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/diffoscope/A/file.txt
 create mode 100644 meta/lib/oeqa/selftest/cases/diffoscope/B/file.txt
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-Add-a-missing-include-for-htobe32-definition.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-framework-profile.py-make-test-lists-reproducible.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-generated_tests-gen_tcs-tes_input_tests.py-do-not-ha.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-serializer.py-make-.gz-files-reproducible.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-tests-shader.py-sort-the-file-list-before-working-on.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0002-tests-util-piglit-shader.c-do-not-hardcode-build-pat.patch
 create mode 100644 meta/recipes-sato/webkit/webkitgtk/0001-MiniBrowser-Fix-reproduciblity.patch
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/14 oeqa: reproducible: Fix test not producing diffs

From: Joshua Watt <[email protected]>

Diffoscope changed the --exclude-directory-metadata option to require an argument.

Add a test to validate that diffoscope is functioning as expected to ensure that future upgrades do not unintentionally break the reproducibility tests.

[YOCTO #14025]

(cherry picked from commit ea8fbcb7978ce48d7a9a83143d09402329535f86)
---
 .../oeqa/selftest/cases/diffoscope/A/file.txt |  1 +
 .../oeqa/selftest/cases/diffoscope/B/file.txt |  1 +
 meta/lib/oeqa/selftest/cases/reproducible.py  | 28 ++++++++++++++++++-
 3 files changed, 29 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/diffoscope/A/file.txt
 create mode 100644 meta/lib/oeqa/selftest/cases/diffoscope/B/file.txt
meta/lib/oeqa/selftest/cases/diffoscope/A/file.txt new file +1 -0
new file mode 100644
index 0000000000..f70f10e4db
@@ -0,0 +1 @@
+A
meta/lib/oeqa/selftest/cases/diffoscope/B/file.txt new file +1 -0
new file mode 100644
index 0000000000..223b7836fb
@@ -0,0 +1 @@
+B
meta/lib/oeqa/selftest/cases/reproducible.py +27 -1
index c8604a2054..60ddc76340 100644
@@ -139,6 +139,32 @@ def compare_file(reference, test, diffutils_sysroot):
     result.status = SAME
     return result
 
+def run_diffoscope(a_dir, b_dir, html_dir, **kwargs):
+    return runCmd(['diffoscope', '--no-default-limits', '--exclude-directory-metadata', 'yes', '--html-dir', html_dir, a_dir, b_dir],
+                **kwargs)
+
+class DiffoscopeTests(OESelftestTestCase):
+    diffoscope_test_files = os.path.join(os.path.dirname(os.path.abspath(__file__)), "diffoscope")
+
+    def test_diffoscope(self):
+        bitbake("diffoscope-native -c addto_recipe_sysroot")
+        diffoscope_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffoscope-native")
+
+        # Check that diffoscope doesn't return an error when the files compare
+        # the same (a general check that diffoscope is working)
+        with tempfile.TemporaryDirectory() as tmpdir:
+            run_diffoscope('A', 'A', tmpdir,
+                native_sysroot=diffoscope_sysroot, cwd=self.diffoscope_test_files)
+
+        # Check that diffoscope generates an index.html file when the files are
+        # different
+        with tempfile.TemporaryDirectory() as tmpdir:
+            r = run_diffoscope('A', 'B', tmpdir,
+                native_sysroot=diffoscope_sysroot, ignore_status=True, cwd=self.diffoscope_test_files)
+
+            self.assertNotEqual(r.status, 0, msg="diffoscope was successful when an error was expected")
+            self.assertTrue(os.path.exists(os.path.join(tmpdir, 'index.html')), "HTML index not found!")
+
 class ReproducibleTests(OESelftestTestCase):
     # Test the reproducibility of whatever is built between sstate_targets and targets
 
@@ -316,7 +342,7 @@ class ReproducibleTests(OESelftestTestCase):
                 # Copy jquery to improve the diffoscope output usability
                 self.copy_file(os.path.join(jquery_sysroot, 'usr/share/javascript/jquery/jquery.min.js'), os.path.join(package_html_dir, 'jquery.js'))
 
-                runCmd(['diffoscope', '--no-default-limits', '--exclude-directory-metadata', '--html-dir', package_html_dir, 'reproducibleA', 'reproducibleB'],
+                run_diffoscope('reproducibleA', 'reproducibleB', package_html_dir,
                         native_sysroot=diffoscope_sysroot, ignore_status=True, cwd=package_dir)
 
         if fails:
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/14 webkitgtk: Fix reproducibility in minibrowser

In reply to Steve Sakoman

From: Khem Raj <[email protected]>

(cherry picked from commit 8f08ca440b6c2ad3494808ffa4ec6091722c0339)
---
 .../0001-MiniBrowser-Fix-reproduciblity.patch | 31 +++++++++++++++++++
 meta/recipes-sato/webkit/webkitgtk_2.28.4.bb  |  1 +
 2 files changed, 32 insertions(+)
 create mode 100644 meta/recipes-sato/webkit/webkitgtk/0001-MiniBrowser-Fix-reproduciblity.patch
meta/recipes-sato/webkit/webkitgtk/0001-MiniBrowser-Fix-reproduciblity.patch new file +31 -0
new file mode 100644
index 0000000000..528dec8c8b
@@ -0,0 +1,31 @@
+From dcf9ae0dc0b4510eddbeeea09e11edfb123f95af Mon Sep 17 00:00:00 2001
+From: Khem Raj <[email protected]>
+Date: Sun, 2 May 2021 13:10:49 -0700
+Subject: [PATCH] MiniBrowser: Fix reproduciblity
+
+Do not emit references to source dir in generated sourcecode
+
+Upstream-Status: Submitted [https://bugs.webkit.org/show_bug.cgi?id=225283]
+Signed-off-by: Khem Raj <[email protected]>
+---
+ Tools/MiniBrowser/gtk/CMakeLists.txt | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/Tools/MiniBrowser/gtk/CMakeLists.txt b/Tools/MiniBrowser/gtk/CMakeLists.txt
+index 93b62521..482d3b00 100644
+--- a/Tools/MiniBrowser/gtk/CMakeLists.txt
++++ b/Tools/MiniBrowser/gtk/CMakeLists.txt
+@@ -48,8 +48,8 @@ add_custom_command(
+     OUTPUT ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.c
+            ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.h
+     MAIN_DEPENDENCY ${MINIBROWSER_DIR}/browser-marshal.list
+-    COMMAND glib-genmarshal --prefix=browser_marshal ${MINIBROWSER_DIR}/browser-marshal.list --body > ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.c
+-    COMMAND glib-genmarshal --prefix=browser_marshal ${MINIBROWSER_DIR}/browser-marshal.list --header > ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.h
++    COMMAND glib-genmarshal --prefix=browser_marshal ${MINIBROWSER_DIR}/browser-marshal.list --body --skip-source > ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.c
++    COMMAND glib-genmarshal --prefix=browser_marshal ${MINIBROWSER_DIR}/browser-marshal.list --header --skip-source > ${DERIVED_SOURCES_MINIBROWSER_DIR}/BrowserMarshal.h
+     VERBATIM)
+ 
+ if (DEVELOPER_MODE)
+-- 
+2.31.1
+
meta/recipes-sato/webkit/webkitgtk_2.28.4.bb +1 -0
index ceda2992d7..2e3f0aa682 100644
@@ -20,6 +20,7 @@ SRC_URI = "https://www.webkitgtk.org/releases/${BPN}-${PV}.tar.xz \
            file://0001-Fix-build-with-musl.patch \
            file://include_array.patch \
            file://0001-clang-11-fix-build-errors-due-to-WWc-11-narrowing.patch \
+           file://0001-MiniBrowser-Fix-reproduciblity.patch \
            "
 SRC_URI[sha256sum] = "821952e8c9303ed752f1fb1d4283f612c25249d00d705d2b79c2db1bc49c9464"
 
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/14 python3-magic: add the missing rdepends

In reply to Steve Sakoman

From: Mingli Yu <[email protected]>

Add the missing rdepends to fix below error:

 # python3
 [snip]
Quoted 1 line
Quoted 1 line
Quoted 1 line

import magic

[snip]
ModuleNotFoundError: No module named 'ctypes'
ModuleNotFoundError: No module named 'tempfile'

(cherry picked from commit 115791844124bdddfbaec9d75bb887ef35c41f20)
---
 meta/recipes-devtools/python/python3-magic_0.4.15.bb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python3-magic_0.4.15.bb +4 -1
index 698016ba4c..09b24fb7f3 100644
@@ -14,6 +14,9 @@ inherit pypi setuptools3
 SRC_URI[md5sum] = "e384c95a47218f66c6501cd6dd45ff59"
 SRC_URI[sha256sum] = "f3765c0f582d2dfc72c15f3b5a82aecfae9498bd29ca840d72f37d7bd38bfcd5"
 
-RDEPENDS_${PN} += "file"
+RDEPENDS_${PN} += "file \
+                   ${PYTHON_PN}-ctypes \
+                   ${PYTHON_PN}-io \
+                   ${PYTHON_PN}-shell"
 
 BBCLASSEXTEND = "native"
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/14 python3-magic: add missing DEPENDS

In reply to Steve Sakoman

Since file-native is ASSUME_PROVIDED magic.mgc is not being staged. As a result diffoscope-native is failing with:

magic.MagicException: b'could not find any valid magic files!

Fix this by adding dependency on file-replacement-native

---
 meta/recipes-devtools/python/python3-magic_0.4.15.bb | 2 ++
 1 file changed, 2 insertions(+)
meta/recipes-devtools/python/python3-magic_0.4.15.bb +2 -0
index 09b24fb7f3..b73310c808 100644
@@ -14,6 +14,8 @@ inherit pypi setuptools3
 SRC_URI[md5sum] = "e384c95a47218f66c6501cd6dd45ff59"
 SRC_URI[sha256sum] = "f3765c0f582d2dfc72c15f3b5a82aecfae9498bd29ca840d72f37d7bd38bfcd5"
 
+DEPENDS_append_class-native = " file-replacement-native"
+
 RDEPENDS_${PN} += "file \
                    ${PYTHON_PN}-ctypes \
                    ${PYTHON_PN}-io \
Signature · 1 line
2.25.1
#5 lore raw

PATCH 5/14 linunistring: Add missing gperf-native dependency

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

(cherry picked from commit 73d3efbaeb2f412ab8d3491d2da3f3124fc009f3)
---
 meta/recipes-support/libunistring/libunistring_0.9.10.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-support/libunistring/libunistring_0.9.10.bb +1 -0
index 97fac4ecfa..2197b6656d 100644
@@ -18,6 +18,7 @@ LIC_FILES_CHKSUM = "file://COPYING.LIB;md5=6a6a8e020838b23406c81b19c1d46df6 \
                     file://README;beginline=45;endline=65;md5=08287d16ba8d839faed8d2dc14d7d6a5 \
                     file://doc/libunistring.texi;md5=287fa6075f78a3c85c1a52b0a92547cd \
                    "
+DEPENDS = "gperf-native"
 
 SRC_URI = "${GNU_MIRROR}/libunistring/libunistring-${PV}.tar.gz \
            file://iconv-m4-remove-the-test-to-convert-euc-jp.patch \
Signature · 1 line
2.25.1
#6 lore raw

PATCH 6/14 pseudo: Add in ability to flush database with shutdown request

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Pulls in:
pseudo_db: Flush DB if there is a shutdown request
fcntl: Add support for fcntl F_GETPIPE_SZ and F_SETPIPE_SZ (test fix)

(cherry picked from commit 0882095d608ce3abbcc9814517434c21ea549063)
---
 meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
index f36dfa589f..6779db16d2 100644
@@ -13,7 +13,7 @@ SRC_URI:append:class-nativesdk = " \
     file://older-glibc-symbols.patch"
 SRC_URI[prebuilt.sha256sum] = "ed9f456856e9d86359f169f46a70ad7be4190d6040282b84c8d97b99072485aa"
 
-SRCREV = "21ff2fb690efbe57e7dd867c39aff36ab72a6ac5"
+SRCREV = "0cda3ba5f94aed8d50652a99ee9c502975aa2926"
 S = "${WORKDIR}/git"
 PV = "1.9.0+git${SRCPV}"
 
Signature · 1 line
2.25.1
#7 lore raw

PATCH 7/14 pseudo: Add fcntl64 wrapper

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Add fcntl64 wrapper which hopefully fixes issues seen in findutils and the find command in the libtool removal code when built with LFS compile flags on Gentoo.

(cherry picked from commit f26867fe4daec7299f59a82ae4a0d70cceb3e082)
---
 meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
index 6779db16d2..1a5d230c69 100644
@@ -13,7 +13,7 @@ SRC_URI:append:class-nativesdk = " \
     file://older-glibc-symbols.patch"
 SRC_URI[prebuilt.sha256sum] = "ed9f456856e9d86359f169f46a70ad7be4190d6040282b84c8d97b99072485aa"
 
-SRCREV = "0cda3ba5f94aed8d50652a99ee9c502975aa2926"
+SRCREV = "d34f2f6cedccf8488730001bcbde6bb7499f8814"
 S = "${WORKDIR}/git"
 PV = "1.9.0+git${SRCPV}"
 
Signature · 1 line
2.25.1
#8 lore raw

PATCH 8/14 piglit: upgrade to latest revision

In reply to Steve Sakoman

From: Marek Vasut <[email protected]>

Update piglit to latest git revision and update the branch name, since the original one is no longer updated. Make sure the VK tests are only enabled if VK is also enabled in PACKAGECONFIG, and that this is opt-in, otherwise older systems fail to build.

Cherry picked from squashed commits:
eb3a8d4c7b ("piglit: upgrade to latest revision") a27b06f73a ("piglit: upgrade to latest revision") bb091bc0be ("piglit: upgrade to latest revision") 394746d1cb ("piglit: upgrade to latest revision") 5aec8cff94 ("piglit: upgrade to latest revision") fc4c82773d ("piglit: fix reproducibility") 6fbec0f12a ("piglit: update to latest revision") 8d23a0d498 ("piglit: upgrade to latest revision") 5144d515fe ("piglit: upgrade to latest revision") dd085bd577 ("piglit: upgrade to latest revision") 9ba6df1b2c ("piglit: upgrade to latest revision") 1ccd71eb3e ("piglit: upgrade to latest revision")

---
 ...ssing-include-for-htobe32-definition.patch | 27 ++++++++++++
 ...file.py-make-test-lists-reproducible.patch | 31 +++++++++++++
 ...gen_tcs-tes_input_tests.py-do-not-ha.patch | 44 +++++++++++++++++++
 ...lizer.py-make-.gz-files-reproducible.patch | 30 +++++++++++++
 ...sort-the-file-list-before-working-on.patch | 28 ++++++++++++
 ...t-shader.c-do-not-hardcode-build-pat.patch | 30 +++++++++++++
 meta/recipes-graphics/piglit/piglit_git.bb    | 12 ++++-
 7 files changed, 200 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-Add-a-missing-include-for-htobe32-definition.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-framework-profile.py-make-test-lists-reproducible.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-generated_tests-gen_tcs-tes_input_tests.py-do-not-ha.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-serializer.py-make-.gz-files-reproducible.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0001-tests-shader.py-sort-the-file-list-before-working-on.patch
 create mode 100644 meta/recipes-graphics/piglit/piglit/0002-tests-util-piglit-shader.c-do-not-hardcode-build-pat.patch
meta/recipes-graphics/piglit/piglit/0001-Add-a-missing-include-for-htobe32-definition.patch new file +27 -0
new file mode 100644
index 0000000000..caa48e088d
@@ -0,0 +1,27 @@
+From d623e9797b7ee9b3739a8a4afe1a01f7e03754aa Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Sun, 1 Nov 2020 20:08:49 +0000
+Subject: [PATCH] Add a missing include for htobe32 definition
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ tests/spec/nv_copy_depth_to_color/nv_copy_depth_to_color.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/tests/spec/nv_copy_depth_to_color/nv_copy_depth_to_color.c b/tests/spec/nv_copy_depth_to_color/nv_copy_depth_to_color.c
+index 5f45e0c23..c755ee29a 100644
+--- a/tests/spec/nv_copy_depth_to_color/nv_copy_depth_to_color.c
++++ b/tests/spec/nv_copy_depth_to_color/nv_copy_depth_to_color.c
+@@ -34,6 +34,8 @@
+ 
+ #include "piglit-util-gl.h"
+ 
++#include <endian.h>
++
+ #define IMAGE_WIDTH 60
+ #define IMAGE_HEIGHT 60
+ 
+-- 
+2.17.1
+
meta/recipes-graphics/piglit/piglit/0001-framework-profile.py-make-test-lists-reproducible.patch new file +31 -0
new file mode 100644
index 0000000000..cc9482c047
@@ -0,0 +1,31 @@
+From 9086d42df1f3134bafcfe33ff16db7bbb9d9a0fd Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Mon, 30 Nov 2020 23:08:22 +0000
+Subject: [PATCH] framework/profile.py: make test lists reproducible
+
+These are created with os.walk, which yields different
+order depending on where it's run.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ framework/profile.py | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/framework/profile.py b/framework/profile.py
+index c210e535e..9b5d51d68 100644
+--- a/framework/profile.py
++++ b/framework/profile.py
+@@ -528,7 +528,11 @@ class TestProfile(object):
+                 else:
+                     opts[n] = self.test_list[n]
+         else:
+-            opts = self.test_list  # pylint: disable=redefined-variable-type
++            opts = collections.OrderedDict()
++            test_keys = list(self.test_list.keys())
++            test_keys.sort()
++            for k in test_keys:
++                opts[k] = self.test_list[k]
+ 
+         for k, v in self.filters.run(opts.items()):
+             yield k, v
meta/recipes-graphics/piglit/piglit/0001-generated_tests-gen_tcs-tes_input_tests.py-do-not-ha.patch new file +44 -0

44 lines not on this page · raw message

meta/recipes-graphics/piglit/piglit/0001-serializer.py-make-.gz-files-reproducible.patch new file +30 -0
new file mode 100644
index 0000000000..2efba6f866
@@ -0,0 +1,30 @@
+From 1919bb7f4072d73dcbb64d0e06eff5b04529c3db Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Mon, 16 Nov 2020 18:01:02 +0000
+Subject: [PATCH] serializer.py: make .gz files reproducible
+
+.gz format contains mtime of the compressed data, and
+SOURCE_DATE_EPOCH is the standard way to make it reproducuble.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ tests/serializer.py | 5 ++++-
+ 1 file changed, 4 insertions(+), 1 deletion(-)
+
+diff --git a/tests/serializer.py b/tests/serializer.py
+index bd14bc3db..bc5b45d7f 100644
+--- a/tests/serializer.py
++++ b/tests/serializer.py
+@@ -138,7 +138,10 @@ def serializer(name, profile, outfile):
+                 et.SubElement(env, 'env', name=k, value=v)
+ 
+     tree = et.ElementTree(root)
+-    with gzip.open(outfile, 'wb') as f:
++    reproducible_mtime = None
++    if 'SOURCE_DATE_EPOCH' in os.environ:
++        reproducible_mtime=os.environ['SOURCE_DATE_EPOCH']
++    with gzip.GzipFile(outfile, 'wb', mtime=reproducible_mtime) as f:
+         tree.write(f, encoding='utf-8', xml_declaration=True)
+ 
+ 
meta/recipes-graphics/piglit/piglit/0001-tests-shader.py-sort-the-file-list-before-working-on.patch new file +28 -0
new file mode 100644
index 0000000000..8321be8490
@@ -0,0 +1,28 @@
+From 5bf89c6a314952313b2b762fff0d5501fe57ac53 Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Wed, 2 Dec 2020 21:21:52 +0000
+Subject: [PATCH] tests/shader.py: sort the file list before working on it
+
+This allows later xml output to be reproducible.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ tests/shader.py | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/tests/shader.py b/tests/shader.py
+index 849273660..e6e65d1ba 100644
+--- a/tests/shader.py
++++ b/tests/shader.py
+@@ -52,7 +52,9 @@ for basedir in [TESTS_DIR, GENERATED_TESTS_DIR]:
+ for group, files in shader_tests.items():
+     assert group not in profile.test_list, 'duplicate group: {}'.format(group)
+ 
+-    # We'll end up with a list of tuples, split that into two lists
++    # This makes the xml output reproducible, as os.walk() order is random
++    files.sort()
++    # We'll end up with a list of tuples, split that into two list
+     files, installedfiles = list(zip(*files))
+     files = list(files)
+     installedfiles = list(installedfiles)
meta/recipes-graphics/piglit/piglit/0002-tests-util-piglit-shader.c-do-not-hardcode-build-pat.patch new file +30 -0
new file mode 100644
index 0000000000..16c7c5c803
@@ -0,0 +1,30 @@
+From 1c67250308a92d4991ed05d9d240090ab84accae Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Tue, 10 Nov 2020 17:13:50 +0000
+Subject: [PATCH 2/2] tests/util/piglit-shader.c: do not hardcode build path
+ into target binary
+
+This helps reproducibilty.
+
+Upstream-Status: Inappropriate [oe-core specific]
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ tests/util/piglit-shader.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/tests/util/piglit-shader.c b/tests/util/piglit-shader.c
+index 4fd68d21e..c9ea8295e 100644
+--- a/tests/util/piglit-shader.c
++++ b/tests/util/piglit-shader.c
+@@ -73,7 +73,7 @@ piglit_compile_shader(GLenum target, const char *filename)
+ 
+ 	source_dir = getenv("PIGLIT_SOURCE_DIR");
+ 	if (source_dir == NULL) {
+-		source_dir = SOURCE_DIR;
++		source_dir = ".";
+ 	}
+ 
+ 	snprintf(filename_with_path, FILENAME_MAX - 1,
+-- 
+2.17.1
+
meta/recipes-graphics/piglit/piglit_git.bb +10 -2
index b84d53f87c..9897ef1575 100644
@@ -6,13 +6,19 @@ BUGTRACKER = "https://gitlab.freedesktop.org/mesa/piglit/-/issues"
 LICENSE = "MIT & LGPLv2+ & GPLv3 & GPLv2+ & BSD-3-Clause"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b2beded7103a3d8a442a2a0391d607b0"
 
-SRC_URI = "git://gitlab.freedesktop.org/mesa/piglit.git;protocol=https;branch=master \
+SRC_URI = "git://gitlab.freedesktop.org/mesa/piglit.git;protocol=https;branch=main \
            file://0001-cmake-install-bash-completions-in-the-right-place.patch \
            file://0001-cmake-use-proper-WAYLAND_INCLUDE_DIRS-variable.patch \
+           file://0001-Add-a-missing-include-for-htobe32-definition.patch \
+           file://0001-generated_tests-gen_tcs-tes_input_tests.py-do-not-ha.patch \
+           file://0002-tests-util-piglit-shader.c-do-not-hardcode-build-pat.patch \
+           file://0001-serializer.py-make-.gz-files-reproducible.patch \
+           file://0001-framework-profile.py-make-test-lists-reproducible.patch \
+           file://0001-tests-shader.py-sort-the-file-list-before-working-on.patch \
            "
 UPSTREAM_CHECK_COMMITS = "1"
 
-SRCREV = "6126c2d4e476c7770d216ffa1932c10e2a5a7813"
+SRCREV = "83bc56abf2686e2cd9024a152e121ca4aa524985"
 # (when PV goes above 1.0 remove the trailing r)
 PV = "1.0+gitr${SRCPV}"
 
@@ -37,7 +43,9 @@ do_compile[dirs] =+ "${B}/temp/"
 PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'x11', d)}"
 PACKAGECONFIG[freeglut] = "-DPIGLIT_USE_GLUT=1,-DPIGLIT_USE_GLUT=0,freeglut,"
 PACKAGECONFIG[x11] = "-DPIGLIT_BUILD_GL_TESTS=ON,-DPIGLIT_BUILD_GL_TESTS=OFF,${X11_DEPS}, ${X11_RDEPS}"
+PACKAGECONFIG[vulkan] = "-DPIGLIT_BUILD_VK_TESTS=ON,-DPIGLIT_BUILD_VK_TESTS=OFF,vulkan-loader"
 
+export PIGLIT_BUILD_DIR = "../../../../git"
 
 do_configure_prepend() {
    if [ "${@bb.utils.contains('PACKAGECONFIG', 'freeglut', 'yes', 'no', d)}" = "no" ]; then
Signature · 1 line
2.25.1
#9 lore raw

PATCH 9/14 mirrors: Add uninative mirror on kernel.org

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

At the last nas outage, we realised that we don't have good mirrors of the uninative tarball if our main system can't be accessed. kernel.org mirrors some Yocto Project data so we've ensured uninative is there. Add the appropriate mirror url to make use of that.

---
 meta/classes/mirrors.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes/mirrors.bbclass +1 -0
index ebec56c83c..2ba50f174e 100644
@@ -61,6 +61,7 @@ ftp://.*/.*     http://sources.openembedded.org/ \n \
 npm://.*/?.*    http://sources.openembedded.org/ \n \
 ${CPAN_MIRROR}  http://cpan.metacpan.org/ \n \
 ${CPAN_MIRROR}  http://search.cpan.org/CPAN/ \n \
+https?$://downloads.yoctoproject.org/releases/uninative/ https://mirrors.kernel.org/yocto/uninative/ \n \
 "
 
 # Use MIRRORS to provide git repo fallbacks using the https protocol, for cases
Signature · 1 line
2.25.1
#10 lore raw

PATCH 10/14 sstate: another fix for touching files inside pseudo

In reply to Steve Sakoman

From: Jose Quaresma <[email protected]>

This patch is a fixup for 676757f "sstate: fix touching files inside pseudo"

running the 'id' command inside the sstate_unpack_package function shows that this funcion run inside the pseudo:

uid=0(root) gid=0(root) groups=0(root)

The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem.

As the documentation refers:
- the file test operator "-w" check if the file has write permission (for the user running the test).
- the file test operator "-O" check if you are owner of file

We can avoid this test running the touch and mask any return errors that we have.

(From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876)

(cherry picked from commit 5b9210d66c78bb3f79056e5586cea7b0edd714a9)
---
 meta/classes/sstate.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/sstate.bbclass +6 -6
index 38dc3bff30..930d87424f 100644
@@ -859,12 +859,12 @@ python sstate_report_unihash() {
 #
 sstate_unpack_package () {
 	tar -xvzf ${SSTATE_PKG}
-	# update .siginfo atime on local/NFS mirror
-	[ -O ${SSTATE_PKG}.siginfo ] && [ -w ${SSTATE_PKG}.siginfo ] && [ -h ${SSTATE_PKG}.siginfo ] && touch -a ${SSTATE_PKG}.siginfo
-	# Use "! -w ||" to return true for read only files
-	[ ! -w ${SSTATE_PKG} ] || touch --no-dereference ${SSTATE_PKG}
-	[ ! -w ${SSTATE_PKG}.sig ] || [ ! -e ${SSTATE_PKG}.sig ] || touch --no-dereference ${SSTATE_PKG}.sig
-	[ ! -w ${SSTATE_PKG}.siginfo ] || [ ! -e ${SSTATE_PKG}.siginfo ] || touch --no-dereference ${SSTATE_PKG}.siginfo
+	# update .siginfo atime on local/NFS mirror if it is a symbolic link
+	[ ! -h ${SSTATE_PKG}.siginfo ] || touch -a ${SSTATE_PKG}.siginfo 2>/dev/null || true
+	# update each symbolic link instead of any referenced file
+	touch --no-dereference ${SSTATE_PKG} 2>/dev/null || true
+	[ ! -e ${SSTATE_PKG}.sig ] || touch --no-dereference ${SSTATE_PKG}.sig 2>/dev/null || true
+	[ ! -e ${SSTATE_PKG}.siginfo ] || touch --no-dereference ${SSTATE_PKG}.siginfo 2>/dev/null || true
 }
 
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
Signature · 1 line
2.25.1
#11 lore raw

PATCH 11/14 sstate: Ensure SDE is accounted for in package task timestamps

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

When creating packages we build them with --clamp-mtime and use SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages reproducible. The data stored in sstate for do_package and the package task doesn't benefit from this though and have varying timestamps. This means their outhash varies and means hash equivalance isn't effective at all and doesn't work as intended/desired.

We could create the sstate archives with the same clamping however that would lead to different results depending on whether a task was installed from sstate or not. Making that differ is a path to madness. It also wouldn't fix the outhash of the task to be determninistic without clamping of the date in the hash calculation code.

Instead, iterate over the files in sstate output and clamp them at the code level. This isn't ideal but does make the file timestamps determnistic everywhere and means we don't have to change the hash calculation code.

This issue can be clearly seen looking at the do_package outhash for a recipe which you then re-run the package task for after adding something like whitespace to the install task. The outhash shouldn't change but currently does.

(cherry picked from commit c3b3cc4745811b48b9193f83889946b2e1788932)
---
 meta/classes/sstate.bbclass | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
meta/classes/sstate.bbclass +18 -0
index 930d87424f..50d44398f9 100644
@@ -640,10 +640,21 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import time
 
     tmpdir = d.getVar('TMPDIR')
 
+    fixtime = False
+    if ss['task'] == "package":
+        fixtime = True
+
+    def fixtimestamp(root, path):
+        f = os.path.join(root, path)
+        if os.lstat(f).st_mtime > sde:
+            os.utime(f, (sde, sde), follow_symlinks=False)
+
     sstatebuild = d.expand("${WORKDIR}/sstate-build-%s/" % ss['task'])
+    sde = int(d.getVar("SOURCE_DATE_EPOCH") or time.time())
     d.setVar("SSTATE_CURRTASK", ss['task'])
     bb.utils.remove(sstatebuild, recurse=True)
     bb.utils.mkdirhier(sstatebuild)
@@ -656,6 +667,8 @@ def sstate_package(ss, d):
         # to sstate tasks but there aren't many of these so better just avoid them entirely.
         for walkroot, dirs, files in os.walk(state[1]):
             for file in files + dirs:
+                if fixtime:
+                    fixtimestamp(walkroot, file)
                 srcpath = os.path.join(walkroot, file)
                 if not os.path.islink(srcpath):
                     continue
@@ -677,6 +690,11 @@ def sstate_package(ss, d):
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
         os.rename(plain, pdir)
+        if fixtime:
+            fixtimestamp(pdir, "")
+            for walkroot, dirs, files in os.walk(pdir):
+                for file in files + dirs:
+                    fixtimestamp(walkroot, file)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
Signature · 1 line
2.25.1
#12 lore raw

PATCH 12/14 sstate: Avoid deploy_source_date_epoch sstate when unneeded

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

This sstate task is only needed when depended upon, it can be skipped if there are no tasks running that directly depend upon it.

This reduced the number of sstate tasks in something like an image build.

(cherry picked from commit 601cee016da5c7505915e26641a085714de175ce)
---
 meta/classes/sstate.bbclass | 4 ++++
 1 file changed, 4 insertions(+)
meta/classes/sstate.bbclass +4 -0
index 50d44398f9..c2720cde92 100644
@@ -1060,6 +1060,10 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
     if taskdependees[task][1] == "do_populate_lic":
         return True
 
+    # We only need to trigger deploy_source_date_epoch through direct dependencies
+    if taskdependees[task][1] == "do_deploy_source_date_epoch":
+        return True
+
     # stash_locale and gcc_stash_builddir are never needed as a dependency for built objects
     if taskdependees[task][1] == "do_stash_locale" or taskdependees[task][1] == "do_gcc_stash_builddir":
         return True
Signature · 1 line
2.25.1
#13 lore raw

PATCH 13/14 reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

In reply to Steve Sakoman

From: Mark Hatle <[email protected]>

Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would fall back and select the default epoch (April 2011), but still perform the reproducible build actions. This resulted in binaries that had an unusually old date.

Simplify the functions and remove the anonymous python as no longer necessary.

Also improve the documentation to better explain what the class is doing and how a recipe can override the behavior if necessary.

(cherry picked from commit 1976013b026cfba94de32a13e994d92d7e9e39e5)
---
 meta/classes/reproducible_build.bbclass | 53 ++++++++++++++++---------
 1 file changed, 34 insertions(+), 19 deletions(-)
meta/classes/reproducible_build.bbclass +34 -19
index b44053d076..2b402b9966 100644
@@ -1,17 +1,38 @@
 # reproducible_build.bbclass
 #
-# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Sets the default SOURCE_DATE_EPOCH in each component's build environment.
+# The format is number of seconds since the system epoch.
+#
 # Upstream components (generally) respect this environment variable,
 # using it in place of the "current" date and time.
 # See https://reproducible-builds.org/specs/source-date-epoch/
 #
-# After sources are unpacked but before they are patched, we set a reproducible value for SOURCE_DATE_EPOCH.
-# This value should be reproducible for anyone who builds the same revision from the same sources.
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available (or set to 0) will use the fallback of
+# SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
 #
-# There are 4 ways we determine SOURCE_DATE_EPOCH:
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
 #
 # 1. Use the value from __source_date_epoch.txt file if this file exists.
-#    This file was most likely created in the previous build by one of the following methods 2,3,4.
+#    This file was most likely created in the previous build by one of the
+#    following methods 2,3,4.
 #    Alternatively, it can be provided by a recipe via SRC_URI.
 #
 # If the file does not exist:
@@ -22,20 +43,16 @@
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
 #    This works for well-kept repositories distributed via tarball.
 #
-# 4. Use the modification time of the youngest file in the source tree, if there is one.
+# 4. Use the modification time of the youngest file in the source tree, if
+#    there is one.
 #    This will be the newest file from the distribution tarball, if any.
 #
-# 5. Fall back to a fixed timestamp.
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
-# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
-# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
-#
-# If this file is found by other tasks, the value is exported in the SOURCE_DATE_EPOCH variable.
-# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
-inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
+inherit reproducible_build_simple
 
 SDE_DIR = "${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
@@ -89,6 +106,9 @@ python create_source_date_epoch_stamp() {
     os.rename(tmp_file, epochfile)
 }
 
+# Generate the stamp after do_unpack runs
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
 def get_source_date_epoch_value(d):
     epochfile = d.getVar('SDE_FILE')
     cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
@@ -116,8 +136,3 @@ def get_source_date_epoch_value(d):
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
-
-python () {
-    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
-        d.appendVarFlag("do_unpack", "postfuncs", " create_source_date_epoch_stamp")
-}
Signature · 1 line
2.25.1
#14 lore raw

PATCH 14/14 selftest/reproducible: add webkitgtk back to exclusion list for dunfell

In reply to Steve Sakoman

We are still getting occassional failures to reproduce

---
 meta/lib/oeqa/selftest/cases/reproducible.py | 1 +
 1 file changed, 1 insertion(+)
meta/lib/oeqa/selftest/cases/reproducible.py +1 -0
index 60ddc76340..4b606e7e64 100644
@@ -59,6 +59,7 @@ exclude_packages = [
 	'systemd-bootchart',
 	'systemtap',
 	'valgrind-ptest',
+	'webkitgtk',
 	]
 
 def is_excluded(package):
Signature · 1 line
2.25.1