Skip to contentSkip to conversation

oeqa: Run diffoscop on saved output

PATCH Applied Joshua Watt 5 participants 14 messages Feb 12, 2020 – Jun 10, 2020

lore
Share & tools
lore
#0 lore raw

cover oeqa: Run diffoscop on saved output

Adds recipes to build the diffoscope tool and run it when the OEQA reproducible build test saves output to a local path. This should make it much easier to debug reproducible build issues from the autobuilder, since the published output can be easily viewed on the website.

Joshua Watt (4):
python: Add libarchive-c recipe
python: Add magic recipe
recipes-support: Add diffoscope recipe
oeqa: reproducible: Run diffoscope on saved output

 meta/conf/distro/include/maintainers.inc      |  3 +++
 meta/lib/oeqa/selftest/cases/reproducible.py  | 24 +++++++++++++------
 .../python/python3-libarchive-c_2.9.bb        | 17 +++++++++++++
 .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++
 .../diffoscope/diffoscope_136.bb              | 15 ++++++++++++
 5 files changed, 71 insertions(+), 7 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
 create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/4 python: Add libarchive-c recipe

The libarchive python module is used by diffoscope tool to make build comparisons.

---
 meta/conf/distro/include/maintainers.inc        |  1 +
 .../python/python3-libarchive-c_2.9.bb          | 17 +++++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
meta/conf/distro/include/maintainers.inc +1 -0
index c68e9edf634..abbdb74a0d3 100644
@@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
 RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <[email protected]>"
+RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <[email protected]>"
meta/recipes-devtools/python/python3-libarchive-c_2.9.bb new file +17 -0
new file mode 100644
index 00000000000..4983ae527a1
@@ -0,0 +1,17 @@
+SUMMARY = "Python interface to libarchive"
+DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
+    dynamically load and access the C library."
+HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
+LICENSE = "CC0-1.0"
+LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
+
+PYPI_PACKAGE = "libarchive-c"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
+SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
+
+RDEPENDS_${PN} += "libarchive"
+
+BBCLASSEXTEND = "native"
Signature · 1 line
2.23.0
#3 lore raw

PATCH 2/4 python: Add magic recipe

In reply to Joshua Watt

The python-magic module is used by diffoscope tool to make build comparisons.

---
 meta/conf/distro/include/maintainers.inc      |  1 +
 .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
meta/conf/distro/include/maintainers.inc +1 -0
index abbdb74a0d3..171a43615e3 100644
@@ -580,6 +580,7 @@ RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kr
 RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <[email protected]>"
+RECIPE_MAINTAINER_pn-python3-magic = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <[email protected]>"
 RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <[email protected]>"
meta/recipes-devtools/python/python3-magic_0.4.15.bb new file +19 -0
new file mode 100644
index 00000000000..698016ba4c2
@@ -0,0 +1,19 @@
+SUMMARY = "File type identification using libmagic"
+DESCRIPTION = "This module uses ctypes to access the libmagic file type \
+    identification library. It makes use of the local magic database and supports \
+    both textual and MIME-type output."
+HOMEPAGE = "http://github.com/ahupp/python-magic"
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88"
+
+PYPI_PACKAGE = "python-magic"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "e384c95a47218f66c6501cd6dd45ff59"
+SRC_URI[sha256sum] = "f3765c0f582d2dfc72c15f3b5a82aecfae9498bd29ca840d72f37d7bd38bfcd5"
+
+RDEPENDS_${PN} += "file"
+
+BBCLASSEXTEND = "native"
Signature · 1 line
2.23.0
#4 lore raw

PATCH 3/4 recipes-support: Add diffoscope recipe

In reply to Joshua Watt

Diffoscope is the universal diff tool, capable of comparing many different formats.

---
 meta/conf/distro/include/maintainers.inc          |  1 +
 meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
meta/conf/distro/include/maintainers.inc +1 -0
index 171a43615e3..4a267faabce 100644
@@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <[email protected]>"
 RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <[email protected]>"
 RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <[email protected]>"
 RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <[email protected]>"
+RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <[email protected]>"
meta/recipes-support/diffoscope/diffoscope_136.bb new file +15 -0
new file mode 100644
index 00000000000..a00f281deb0
@@ -0,0 +1,15 @@
+SUMMARY = "in-depth comparison of files, archives, and directories"
+HOMEPAGE = "https://diffoscope.org/"
+LICENSE = "GPL-3.0+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+
+PYPI_PACKAGE = "diffoscope"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
+SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+
+RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
+
+BBCLASSEXTEND = "native"
Signature · 1 line
2.23.0
#9 lore raw

PATCH 4/4 oeqa: reproducible: Run diffoscope on saved output

In reply to Joshua Watt

If there are differing packages and they are being saved for review, automatically run diffoscope on them and include the output in the saved output. The output is currently done in HTML format since these are typically published on a webpage by the autobuilder.

---
 meta/lib/oeqa/selftest/cases/reproducible.py | 24 ++++++++++++++------
 1 file changed, 17 insertions(+), 7 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +17 -7
index f6433c9a02b..d3fd8c392b8 100644
@@ -1,7 +1,7 @@
 #
 # SPDX-License-Identifier: MIT
 #
-# Copyright 2019 by Garmin Ltd. or its subsidiaries
+# Copyright 2019-2020 by Garmin Ltd. or its subsidiaries
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars
@@ -167,10 +167,16 @@ class ReproducibleTests(OESelftestTestCase):
         return d
 
     def test_reproducible_builds(self):
+        def strip_topdir(s):
+            if s.startswith(self.topdir):
+                return s[len(self.topdir):]
+            return s
+
         # Build native utilities
         self.write_config('')
-        bitbake("diffutils-native -c addto_recipe_sysroot")
+        bitbake("diffoscope-native diffutils-native -c addto_recipe_sysroot")
         diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native")
+        diffoscope_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffoscope-native")
 
         if self.save_results:
             os.makedirs(self.save_results, exist_ok=True)
@@ -206,18 +212,22 @@ class ReproducibleTests(OESelftestTestCase):
 
                 if self.save_results:
                     for d in result.different:
-                        self.copy_file(d.reference, '/'.join([save_dir, d.reference]))
-                        self.copy_file(d.test, '/'.join([save_dir, d.test]))
+                        self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
+                        self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
 
                 if result.missing or result.different:
                     fails.append("The following %s packages are missing or different: %s" %
                             (c, '\n'.join(r.test for r in (result.missing + result.different))))
 
-        if fails:
-            self.fail('\n'.join(fails))
-
         # Clean up empty directories
         if self.save_results:
             if not os.listdir(save_dir):
                 os.rmdir(save_dir)
+            else:
+                self.logger.info('Running diffoscope')
+                runCmd(['diffoscope', '--no-default-limits', '--exclude-directory-metadata', '--html-dir', 'diff-html', 'reproducibleA', 'reproducibleB'],
+                        native_sysroot=diffoscope_sysroot, ignore_status=True, cwd=os.path.join(save_dir, 'packages'))
+
+        if fails:
+            self.fail('\n'.join(fails))
 
Signature · 1 line
2.23.0
#10 lore raw
In reply to Joshua Watt
Quoted 4 lines · Joshua Watt

Adds recipes to build the diffoscope tool and run it when the OEQA reproducible build test saves output to a local path. This should make it much easier to debug reproducible build issues from the autobuilder, since the published output can be easily viewed on the website.\

Thanks

Quoted 17 lines

Joshua Watt (4):
python: Add libarchive-c recipe
python: Add magic recipe
recipes-support: Add diffoscope recipe
oeqa: reproducible: Run diffoscope on saved output

 meta/conf/distro/include/maintainers.inc      |  3 +++
 meta/lib/oeqa/selftest/cases/reproducible.py  | 24 +++++++++++++------
 .../python/python3-libarchive-c_2.9.bb        | 17 +++++++++++++
 .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++
 .../diffoscope/diffoscope_136.bb              | 15 ++++++++++++
 5 files changed, 71 insertions(+), 7 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
 create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
#5 lore raw

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

In reply to Joshua Watt

Hi Joshua

I think it needs to be ignored on musl

15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/diffoscope_136.bb RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host
aarch64-yoe-linux-musl (not in COMPATIBLE_HOST)

Quoted 52 lines · Joshua Watt

Diffoscope is the universal diff tool, capable of comparing many different formats.

---
 meta/conf/distro/include/maintainers.inc          |  1 +
 meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
meta/conf/distro/include/maintainers.inc +1 -0
index 171a43615e3..4a267faabce 100644
@@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <[email protected]>"
 RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <[email protected]>"
 RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <[email protected]>"
 RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <[email protected]>"
+RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <[email protected]>"
meta/recipes-support/diffoscope/diffoscope_136.bb new file +15 -1
new file mode 100644
index 00000000000..a00f281deb0
@@ -0,0 +1,15 @@
+SUMMARY = "in-depth comparison of files, archives, and directories"
+HOMEPAGE = "https://diffoscope.org/"
+LICENSE = "GPL-3.0+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+
+PYPI_PACKAGE = "diffoscope"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
+SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+
+RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
+
+BBCLASSEXTEND = "native"
--

2.23.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#6 lore raw

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

Quoted 4 lines · Khem Raj

Hi Joshua

I think it needs to be ignored on musl

Quoted 8 lines

15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but

/mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/ diffoscope_136.bb
RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host
aarch64-yoe-linux-musl (not in COMPATIBLE_HOST

Would an RRECOMMENDS fix it?

Quoted 61 lines · Joshua Watt
Quoted 4 lines

Diffoscope is the universal diff tool, capable of comparing many different formats.

Review

Quoted 8 lines
---
 meta/conf/distro/include/maintainers.inc          |  1 +
 meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
(quoted diff) +0 -0

b/meta/conf/distro/include/maintainers.inc

meta/conf/distro/include/maintainers.inc RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <

Quoted 4 lines

index 171a43615e3..4a267faabce 100644

meta/conf/distro/include/maintainers.inc +0 -0
@@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <

[email protected]>"

meta/conf/distro/include/maintainers.inc RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi < · line 146

Quoted 1 line
meta/conf/distro/include/maintainers.inc +0 -0
 RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <

[email protected]>"

meta/conf/distro/include/maintainers.inc RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi < · line 147

Quoted 1 line
meta/conf/distro/include/maintainers.inc +0 -0
 RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <

[email protected]>"

Review

Quoted 6 lines
meta/conf/distro/include/maintainers.inc +1 -0
 RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <[email protected]>"
+RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <[email protected]>"
(quoted diff) +0 -0

b/meta/recipes-support/diffoscope/diffoscope_136.bb

Quoted 6 lines

new file mode 100644
index 00000000000..a00f281deb0

meta/recipes-support/diffoscope/diffoscope_136.bb new file +1 -0
@@ -0,0 +1,15 @@
+SUMMARY = "in-depth comparison of files, archives, and directories"

meta/recipes-support/diffoscope/diffoscope_136.bb line 11

Quoted 10 lines
meta/recipes-support/diffoscope/diffoscope_136.bb new file +10 -0
+HOMEPAGE = "https://diffoscope.org/"
+LICENSE = "GPL-3.0+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+
+PYPI_PACKAGE = "diffoscope"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
+SRC_URI[sha256sum] =

"0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"

meta/recipes-support/diffoscope/diffoscope_136.bb line 13

Quoted 2 lines
meta/recipes-support/diffoscope/diffoscope_136.bb new file +2 -0
+
+RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c

python3-magic"

Quoted 10 lines
meta/recipes-support/diffoscope/diffoscope_136.bb new file +2 -1
+
+BBCLASSEXTEND = "native"
--

2.23.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#7 lore raw

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

Quoted 7 lines · Joshua Watt
Quoted 4 lines · Khem Raj

Hi Joshua

I think it needs to be ignored on musl

Quoted 8 lines
Quoted 6 lines

15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded- core/meta/recipes-support/diffoscope/diffoscope_136.bb RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host aarch64-yoe-linux-musl (not in COMPATIBLE_HOST

Would an RRECOMMENDS fix it?

I merged your patch and one of mine to follow it adding a COMPATIBLE_HOST entry.

I see Khem is fixing some other software so we may end up not needing it but we can remove later.

Cheers,

Richard

#8 lore raw

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

Quoted 16 lines · Richard Purdie
Quoted 14 lines · Joshua Watt
Quoted 10 lines · Khem Raj

Hi Joshua

I think it needs to be ignored on musl

15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded- core/meta/recipes-support/diffoscope/diffoscope_136.bb RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host aarch64-yoe-linux-musl (not in COMPATIBLE_HOST

Quoted 8 lines
Quoted 1 line

Would an RRECOMMENDS fix it?

I merged your patch and one of mine to follow it adding a COMPATIBLE_HOST entry.

I see Khem is fixing some other software so we may end up not needing it but we can remove later.

right, we were unnecessarily masking squashfs-tools for musl

Quoted 4 lines

Cheers,

Richard

#11 lore raw
In reply to Joshua Watt
Quoted 7 lines · Joshua Watt

Adds recipes to build the diffoscope tool and run it when the OEQA reproducible build test saves output to a local path. This should make
it much easier to debug reproducible build issues from the autobuilder,
since the published output can be easily viewed on the website.

Quoted 5 lines

Joshua Watt (4):
python: Add libarchive-c recipe
python: Add magic recipe
recipes-support: Add diffoscope recipe
oeqa: reproducible: Run diffoscope on saved output

Thanks!

The first production use:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/

:)

I am a bit puzzled/concerned about how this hasn't been removed from the system yet as it should have been, unless its the hashequiv problem with timestamps continuing to cause problems. Need to fix my patch...

Cheers,

Richard

#12 lore raw
Quoted 14 lines · Richard Purdie
Quoted 7 lines · Joshua Watt

Adds recipes to build the diffoscope tool and run it when the OEQA reproducible build test saves output to a local path. This should make
it much easier to debug reproducible build issues from the autobuilder,
since the published output can be easily viewed on the website.

Quoted 5 lines

Joshua Watt (4):
python: Add libarchive-c recipe
python: Add magic recipe
recipes-support: Add diffoscope recipe
oeqa: reproducible: Run diffoscope on saved output

Quoted 9 lines

Thanks!

The first production use:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/

:)

That's great!

Quoted 4 lines

I am a bit puzzled/concerned about how this hasn't been removed from the system yet as it should have been, unless its the hashequiv problem with timestamps continuing to cause problems. Need to fix my patch...

It looks like a directory ordering issue to me, since the timestamps are the same. One way to really flush these out would be to use disorderfs (
https://salsa.debian.org/reproducible-builds/disorderfs) which permutates the order in which entries are listed in a directory, but I think we should try to get "wider" coverage (more recipes) before we try that.

Quoted 5 lines

Cheers,

Richard

#13 lore raw
Quoted 33 lines · Joshua Watt
Quoted 14 lines · Richard Purdie
Quoted 7 lines · Joshua Watt

Adds recipes to build the diffoscope tool and run it when the OEQA reproducible build test saves output to a local path. This should make
it much easier to debug reproducible build issues from the autobuilder,
since the published output can be easily viewed on the website.

Quoted 5 lines

Joshua Watt (4):
python: Add libarchive-c recipe
python: Add magic recipe
recipes-support: Add diffoscope recipe
oeqa: reproducible: Run diffoscope on saved output

Quoted 7 lines

Thanks!

The first production use:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/

:)

That's great!

Quoted 6 lines

I am a bit puzzled/concerned about how this hasn't been removed from
the system yet as it should have been, unless its the hashequiv problem
with timestamps continuing to cause problems. Need to fix my patch...

Quoted 6 lines

It looks like a directory ordering issue to me, since the timestamps are the same. One way to really flush these out would be to use disorderfs (https://salsa.debian.org/reproducible-builds/disorderfs)
which permutates the order in which entries are listed in a directory, but I think we should try to get "wider" coverage (more recipes) before we try that.

Its more complicated unfortunately. We should have fixed the directory order issue with:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=09c6a41b751d080f0c12fc4172a31d1dbe760b0b

However issues which shows signs of that bug still persisted in the reproducibility test results.

I theorised that since opkg-utils is in ABISAFE, the change wasn't accounted for. I therefore merged:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5fd3eb4bda651bddf4e4dc76d456333c70df4ddc

However with current master, we're seeing the above reproducible build failure. I checked in a build and its using this sstate object for comparison:

pub/sstate/b0/ca/sstate\:matchbox-config-gtk\:core2-64-poky-linux\:0.2\:r0\:core2-64\:3\:b0ca3b3559d37ba8de8ade8b282baa5082e612ef6590d3ff94fcebe38e9f17b9_package_write_ipk.tgz.siginfo

which is interesting in that its from 2nd Feb (predating both commits above) and doesn't show the above change to package_ipk.bbclass.

The change to package_ipk means the task's own basehash has to have changed which means it *has* to have rerun. Somehow hashequiv has then marked it as equivalent.

I guess the issue may be that in some builds, this did generate output that was equivalent and things have then raced so this "broken" sstate object has persisted.

The question then becomes, how to we get this out the cache?! I'm not entirely sure how we can :(

(obviously I can cleansstate a few things which I'll do now but its not good this can't recover even through metadata changes)

Cheers,

Richard

#2 lore raw

Re: [OE-core] [PATCH 1/4] python: Add libarchive-c recipe

In reply to Joshua Watt

Hi Joshua,

I encounter below error when use the libarchive module, could you help to take a look?

# python3
Python 3.8.2 (default, Jun  8 2020, 06:35:32)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 1 line
Quoted 1 line
Quoted 1 line

import libarchive

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.8/site-packages/libarchive/__init__.py",
line 1, in <module>
from .entry import ArchiveEntry
File "/usr/lib64/python3.8/site-packages/libarchive/entry.py", line
6, in <module>
from . import ffi
File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line
109, in <module>

    errno = ffi('errno', [c_archive_p], c_int)

File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line 96,
in ffi
f = getattr(libarchive, 'archive_'+name)
File "/usr/lib64/python3.8/ctypes/__init__.py", line 386, in __getattr__

    func = self.__getitem__(name)

File "/usr/lib64/python3.8/ctypes/__init__.py", line 391, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: archive_errno

Quoted 1 line
Quoted 1 line
Quoted 1 line

Thanks,

Quoted 46 lines · Joshua Watt

The libarchive python module is used by diffoscope tool to make build comparisons.

---
  meta/conf/distro/include/maintainers.inc        |  1 +
  .../python/python3-libarchive-c_2.9.bb          | 17 +++++++++++++++++
  2 files changed, 18 insertions(+)

create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb

meta/conf/distro/include/maintainers.inc +1 -0
index c68e9edf634..abbdb74a0d3 100644
@@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
  RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <[email protected]>"
  RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <[email protected]>"
  RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <[email protected]>"
+RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <[email protected]>"
  RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <[email protected]>"
  RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <[email protected]>"
  RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <[email protected]>"
meta/recipes-devtools/python/python3-libarchive-c_2.9.bb new file +17 -0
new file mode 100644
index 00000000000..4983ae527a1
@@ -0,0 +1,17 @@
+SUMMARY = "Python interface to libarchive"
+DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
+    dynamically load and access the C library."
+HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
+LICENSE = "CC0-1.0"
+LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
+
+PYPI_PACKAGE = "libarchive-c"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
+SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
+
+RDEPENDS_${PN} += "libarchive"
+
+BBCLASSEXTEND = "native"