Initial QA test for reproducible builds
cover Initial QA test for reproducible builds
Implements an initial QA check for reproducible builds. This check is sufficient for an initial implementation, and will catch a wide variety of reproducible problems, but it does have the following problems:
1) It doesn't pass. Currently, about 800 packages fail to build in a reproducible manner for core-image-minimal. I've found two major sources of non-reproducibility so far:
a) The perl-module packages don't have a consistent SOURCE_DATE_EPOCH which means when they are packaged the timestamps on all the files are different. Thankfully, this accounts for several hundred of the packages, so fixing this should remove a lot of the failures
b) Debug package strings aren't consistent. It appears that in some of the -dbg packages, the linker changes the order of the merged .debug_strings section. This trickles down into the packages that contain the executables because it changes the hash the executable contains to ensure the debug symbols match up.
2) It's not easy to debug issues when there are reproducibility problems. I had initially intended to run diffoscope on the resulting files but this takes much longer than I think we are willing to run on the autobuilder and also generates far too much output to be really useful. I think a better long term route is to have the test dump the list of non-reproducible packages and then write a helper script that can consumer this list, allow the user to select a package, then run diffoscope to examine it.
3) This test currently is incomplete and won't catch all classes of reproducibility problems. At the least, I know that it won't consistently catch the use of the __DATE__ macro in source code, since that requires the builds to be done on two separate dates (on the other hand, use of __TIME__ will be caught pretty reliably since the builds are done serially). I suspect the correct solution to this is to borrow from Debian and use something like faketime to fake out the system time to some suitable future date when doing the test build, but this will require some though to how it should be implemented.
4) It currently only tests Debian packages and core-image-minimal. The test case has support for building the other package formats and other images at the same time, the idea being that the long step in this test is building everything from scratch, and building multiple package formats and images at the same time will be much faster overall than having multiple tests that have to do from-scratch builds (although, there might be a way to serialize multiple tests and have them share the test build TMPDIR). Until at least 1 package format and image are passing, I don't see a huge motivation to enable more.
Joshua Watt (1):
oeqa: Add reproducible build selftest
meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
Signature · 1 line
2.21.0
PATCH 1/1 oeqa: Add reproducible build selftest
Adds an initial test for reproducible builds to the OE selftest. This initial test builds core-image-minimal using sstate, then does a clean build without sstate in another build directory, and finally does a binary comparison of the resulting package files between the two builds.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
meta/lib/oeqa/selftest/cases/reproducible.py new file +159 -0
@@ -0,0 +1,159 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright 2019 by Garmin Ltd. or its subsidiaries + +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars +import bisect +import functools +import multiprocessing +import textwrap + +MISSING = 'MISSING' +DIFFERENT = 'DIFFERENT' +SAME = 'SAME' + [email protected]_ordering +class CompareResult(object): + def __init__(self): + self.reference = None + self.test = None + self.status = 'UNKNOWN' + + def __eq__(self, other): + return (self.status, self.test) == (other.status, other.test) + + def __lt__(self, other): + return (self.status, self.test) < (other.status, other.test) + +class PackageCompareResults(object): + def __init__(self): + self.total = [] + self.missing = [] + self.different = [] + self.same = [] + + def add_result(self, r): + self.total.append(r) + if r.status == MISSING: + self.missing.append(r) + elif r.status == DIFFERENT: + self.different.append(r) + else: + self.same.append(r) + + def sort(self): + self.total.sort() + self.missing.sort() + self.different.sort() + self.same.sort() + + def __str__(self): + return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total)) + +def compare_file(reference, test, diffutils_sysroot): + result = CompareResult() + result.reference = reference + result.test = test + + if not os.path.exists(reference): + result.status = MISSING + return result + + r = runCmd(['cmp', '--quiet', reference, test], native_sysroot=diffutils_sysroot, ignore_status=True) + + if r.status: + result.status = DIFFERENT + return result + + result.status = SAME + return result + +class ReproducibleTests(OESelftestTestCase): + package_classes = ['deb'] + images = ['core-image-minimal'] + + def setUpLocal(self): + super().setUpLocal() + needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS'] + bb_vars = get_bb_vars(needed_vars) + for v in needed_vars: + setattr(self, v.lower(), bb_vars[v]) + + if not hasattr(self.tc, "extraresults"): + self.tc.extraresults = {} + self.extras = self.tc.extraresults + + self.extras.setdefault('reproducible.rawlogs', {})['log'] = '' + + def append_to_log(self, msg): + self.extras['reproducible.rawlogs']['log'] += msg + + def compare_packages(self, reference_dir, test_dir, diffutils_sysroot): + result = PackageCompareResults() + + old_cwd = os.getcwd() + try: + file_result = {} + os.chdir(test_dir) + with multiprocessing.Pool(processes=int(self.bb_number_threads or 0)) as p: + for root, dirs, files in os.walk('.'): + async_result = [] + for f in files: + reference_path = os.path.join(reference_dir, root, f) + test_path = os.path.join(test_dir, root, f) + async_result.append(p.apply_async(compare_file, (reference_path, test_path, diffutils_sysroot))) + + for a in async_result: + result.add_result(a.get()) + + finally: + os.chdir(old_cwd) + + result.sort() + return result + + def test_reproducible_builds(self): + capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] + + common_config = textwrap.dedent('''\ + INHERIT += "reproducible_build" + PACKAGE_CLASSES = "%s" + ''') % (' '.join('package_%s' % c for c in self.package_classes)) + + # Do an initial build. It's acceptable for this build to use sstate + self.write_config(common_config) + vars_reference = get_bb_vars(capture_vars) + bitbake(' '.join(self.images)) + + # Build native utilities + bitbake("diffutils-native -c addto_recipe_sysroot") + diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native") + + # Perform another build. This build should *not* share sstate or pull + # from any mirrors, but sharing a DL_DIR is fine + self.write_config(textwrap.dedent('''\ + TMPDIR = "${TOPDIR}/reproducible/tmp" + SSTATE_DIR = "${TMPDIR}/sstate" + SSTATE_MIRROR = "" + ''') + common_config) + vars_test = get_bb_vars(capture_vars) + bitbake(' '.join(self.images)) + + for c in self.package_classes: + package_class = 'package_' + c + + deploy_reference = vars_reference['DEPLOY_DIR_' + c.upper()] + deploy_test = vars_test['DEPLOY_DIR_' + c.upper()] + + result = self.compare_packages(deploy_reference, deploy_test, diffutils_sysroot) + + self.logger.info('Reproducibility summary for %s: %s' % (c, result)) + + self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total)) + + if result.missing or result.different: + self.fail("The following %s packages are missing or different: %s" % + (c, ' '.join(r.test for r in (result.missing + result.different)))) +
Signature · 1 line
2.21.0
Hi Joshua
Thanks for contributing this will provide some teeth to reproducible builds QA
Quoted 4 lines · Joshua Watt
Implements an initial QA check for reproducible builds. This check is sufficient for an initial implementation, and will catch a wide variety of reproducible problems, but it does have the following problems:
Quoted 8 lines
1) It doesn't pass. Currently, about 800 packages fail to build in a reproducible manner for core-image-minimal. I've found two major sources of non-reproducibility so far:
a) The perl-module packages don't have a consistentSOURCE_DATE_EPOCHwhich means when they are packaged the timestamps on all the files are different. Thankfully, this accounts for several hundred of the packages, so fixing this should remove a lot of the failures
maybe we can start with inhriting reproducible_build_simple which has hardcoded values for SOURCE_DATE_EPOCH
Quoted 6 lines
b) Debug package strings aren't consistent. It appears that in some of the -dbg packages, the linker changes the order of the merged .debug_strings section. This trickles down into the packages that contain the executables because it changes the hash the executable contains to ensure the debug symbols match up.
try adding -fno-merge-debug-strings to linker and see if that fixes this problem. If that happens then we know its an option to add when doing reproducible builds.
Quoted 8 lines
2) It's not easy to debug issues when there are reproducibility problems. I had initially intended to run diffoscope on the resulting files but this takes much longer than I think we are willing to run on the autobuilder and also generates far too much output to be really useful. I think a better long term route is to have the test dump the list of non-reproducible packages and then write a helper script that can consumer this list, allow the user to select a package, then run diffoscope to examine it.
I think that might be needed to wrap diffoscope.
Quoted 12 lines
3) This test currently is incomplete and won't catch all classes of reproducibility problems. At the least, I know that it won't consistently catch the use of the
__DATE__macro in source code, since that requires the builds to be done on two separate dates (on the other hand, use of__TIME__will be caught pretty reliably since the builds are done serially). I suspect the correct solution to this is to borrow from Debian and use something like faketime to fake out the system time to some suitable future date when doing the test build, but this will require some though to how it should be implemented.
Quoted 10 lines
4) It currently only tests Debian packages and core-image-minimal. The test case has support for building the other package formats and other images at the same time, the idea being that the long step in this test is building everything from scratch, and building multiple package formats and images at the same time will be much faster overall than having multiple tests that have to do from-scratch builds (although, there might be a way to serialize multiple tests and have them share the test build TMPDIR). Until at least 1 package format and image are passing, I don't see a huge motivation to enable more.
why does it have to depend on packaging backend ?
Quoted 8 lines
Joshua Watt (1):
oeqa: Add reproducible build selftestmeta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++ 1 file changed, 159 insertions(+)create mode 100644
meta/lib/oeqa/selftest/cases/reproducible.py
Quoted 24 lines · Khem Raj
Hi Joshua
Thanks for contributing this will provide some teeth to reproducible builds QA
Quoted 6 lines · Joshua Watt
Implements an initial QA check for reproducible builds. This check is
sufficient for an initial implementation, and will catch a wide variety
of reproducible problems, but it does have the following problems:Quoted 11 lines
1) It doesn't pass. Currently, about 800 packages fail to build in a reproducible manner for core-image-minimal. I've found
two
major sources of non-reproducibility so far:
a) The perl-module packages don't have a consistentSOURCE_DATE_EPOCHwhich means when they are packaged the timestamps on all the files are different. Thankfully,
this
accounts for several hundred of the packages, so fixing
this
should remove a lot of the failures
Quoted 3 lines
maybe we can start with inhriting
reproducible_build_simplewhich has
hardcoded values forSOURCE_DATE_EPOCH
reproducible_build.bbclass automatically inherits reproducible_build_simple if BUILD_REPRODUCIBLE_BINARIES is set, and it sets that by default.
I'm not actually sure how the value of SOURCE_DATE_EPOCH set in reproducible_build_simple.bbclass matters at all. AFAIK, reproducible_build.bbclass tries *really* hard to get a discernible date from the source code repository itself after do_unpack, and failing that uses 0. Near as I can tell, the value set in the class or by the user only matters until the end of do_unpack.
I think something happens to be broken with this heuristic in the perl recipe. I haven't tracked it down just yet.
Quoted 12 lines
Quoted 11 lines
b) Debug package strings aren't consistent. It appears that
in some
of the -dbg packages, the linker changes the order of the
merged
.debug_strings section. This trickles down into the
packages
that contain the executables because it changes the hash
the
executable contains to ensure the debug symbols match up.
Quoted 5 lines
try adding -fno-merge-debug-strings to linker and see if that fixes this
problem. If that happens then we know its an option to add when doing
reproducible builds.
Excellent. I will try that.
Quoted 14 lines
Quoted 13 lines
2) It's not easy to debug issues when there are reproducibility problems. I had initially intended to run diffoscope on the resulting files but this takes much longer than I think we are willing to run on the autobuilder and also generates far too
much
output to be really useful. I think a better long term route
is to
have the test dump the list of non-reproducible packages and
then
write a helper script that can consumer this list, allow the
user to
select a package, then run diffoscope to examine it.
Quoted 1 line
I think that might be needed to wrap diffoscope.
I'm not sure I quite follow what you are saying here?
Quoted 39 lines
Quoted 30 lines
3) This test currently is incomplete and won't catch all classes
of
reproducibility problems. At the least, I know that it won't consistently catch the use of the__DATE__macro in source
code,
since that requires the builds to be done on two separate
dates (on
the other hand, use of__TIME__will be caught pretty reliably
since
the builds are done serially). I suspect the correct solution
to
this is to borrow from Debian and use something like faketime
to
fake out the system time to some suitable future date when
doing the
test build, but this will require some though to how it should
be
implemented.4) It currently only tests Debian packages and core-image-
minimal. The
test case has support for building the other package formats
and
other images at the same time, the idea being that the long
step in
this test is building everything from scratch, and building
multiple
package formats and images at the same time will be much
fasterQuoted 8 lines
overall than having multiple tests that have to do from-
scratch
builds (although, there might be a way to serialize multiple
tests
and have them share the test build TMPDIR). Until at least 1
package
format and image are passing, I don't see a huge motivation to enable more.
Quoted 1 line
why does it have to depend on packaging backend ?
It doesn't particularly. Comparing the end packages is just the easiest way to check for reproducibility (I think?). I originally tried comparing sstate tarballs, but I quickly realized that was going to be very difficult without making a lot of changes to the way sstate tarballs are generated (e.g. timestamps and such).
On the other end of the spectrum would be to compare the final image files themselves; I like that idea and I think we *should* do that after we get the packages to be reproducible, but it's not very useful right now because it's hard(er) to track a difference in the image back to a recipe than it is to track a difference in a package back.
You could leave the test up to whatever package classes are enabled on the autobuilder, but I think this is less than ideal right now because:
1) IPKs aren't reproducible *at all*... they are basically all
different. Perhaps because of timestamps? Maybe that's easy fix?
2) The long pull on this QA is the test build from scratch... it takes
quite a while and will take even longer with a bigger image. Comparatively, generating multiple packages classes doesn't add that much more time to the build, so I *think* it might make sense in the long run to have the test build all the package formats in one go, or at least being able to share the test build TMPDIR between multiple serial tests.
Quoted 9 lines
Quoted 9 lines
Joshua Watt (1):
oeqa: Add reproducible build selftestmeta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++ 1 file changed, 159 insertions(+)create mode 100644
meta/lib/oeqa/selftest/cases/reproducible.py
Signature · 1 line
Joshua Watt <[email protected]>
PATCH v2 1/1 oeqa: Add reproducible build selftest
In reply to Joshua WattAdds an initial test for reproducible builds to the OE selftest. This initial test builds core-image-minimal using sstate, then does a clean build without sstate in another build directory, and finally does a binary comparison of the resulting package files between the two builds.
The test is currently always skipped since it doesn't pass yet, but it can easily be enabled locally
--- meta/lib/oeqa/selftest/cases/reproducible.py | 160 +++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
meta/lib/oeqa/selftest/cases/reproducible.py new file +160 -0
@@ -0,0 +1,160 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright 2019 by Garmin Ltd. or its subsidiaries + +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars +import functools +import multiprocessing +import textwrap +import unittest + +MISSING = 'MISSING' +DIFFERENT = 'DIFFERENT' +SAME = 'SAME' + [email protected]_ordering +class CompareResult(object): + def __init__(self): + self.reference = None + self.test = None + self.status = 'UNKNOWN' + + def __eq__(self, other): + return (self.status, self.test) == (other.status, other.test) + + def __lt__(self, other): + return (self.status, self.test) < (other.status, other.test) + +class PackageCompareResults(object): + def __init__(self): + self.total = [] + self.missing = [] + self.different = [] + self.same = [] + + def add_result(self, r): + self.total.append(r) + if r.status == MISSING: + self.missing.append(r) + elif r.status == DIFFERENT: + self.different.append(r) + else: + self.same.append(r) + + def sort(self): + self.total.sort() + self.missing.sort() + self.different.sort() + self.same.sort() + + def __str__(self): + return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total)) + +def compare_file(reference, test, diffutils_sysroot): + result = CompareResult() + result.reference = reference + result.test = test + + if not os.path.exists(reference): + result.status = MISSING + return result + + r = runCmd(['cmp', '--quiet', reference, test], native_sysroot=diffutils_sysroot, ignore_status=True) + + if r.status: + result.status = DIFFERENT + return result + + result.status = SAME + return result + +class ReproducibleTests(OESelftestTestCase): + package_classes = ['deb'] + images = ['core-image-minimal'] + + def setUpLocal(self): + super().setUpLocal() + needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS'] + bb_vars = get_bb_vars(needed_vars) + for v in needed_vars: + setattr(self, v.lower(), bb_vars[v]) + + if not hasattr(self.tc, "extraresults"): + self.tc.extraresults = {} + self.extras = self.tc.extraresults + + self.extras.setdefault('reproducible.rawlogs', {})['log'] = '' + + def append_to_log(self, msg): + self.extras['reproducible.rawlogs']['log'] += msg + + def compare_packages(self, reference_dir, test_dir, diffutils_sysroot): + result = PackageCompareResults() + + old_cwd = os.getcwd() + try: + file_result = {} + os.chdir(test_dir) + with multiprocessing.Pool(processes=int(self.bb_number_threads or 0)) as p: + for root, dirs, files in os.walk('.'): + async_result = [] + for f in files: + reference_path = os.path.join(reference_dir, root, f) + test_path = os.path.join(test_dir, root, f) + async_result.append(p.apply_async(compare_file, (reference_path, test_path, diffutils_sysroot))) + + for a in async_result: + result.add_result(a.get()) + + finally: + os.chdir(old_cwd) + + result.sort() + return result + + @unittest.skip("Reproducible builds do not yet pass") + def test_reproducible_builds(self): + capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] + + common_config = textwrap.dedent('''\ + INHERIT += "reproducible_build" + PACKAGE_CLASSES = "%s" + ''') % (' '.join('package_%s' % c for c in self.package_classes)) + + # Do an initial build. It's acceptable for this build to use sstate + self.write_config(common_config) + vars_reference = get_bb_vars(capture_vars) + bitbake(' '.join(self.images)) + + # Build native utilities + bitbake("diffutils-native -c addto_recipe_sysroot") + diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native") + + # Perform another build. This build should *not* share sstate or pull + # from any mirrors, but sharing a DL_DIR is fine + self.write_config(textwrap.dedent('''\ + TMPDIR = "${TOPDIR}/reproducible/tmp" + SSTATE_DIR = "${TMPDIR}/sstate" + SSTATE_MIRROR = "" + ''') + common_config) + vars_test = get_bb_vars(capture_vars) + bitbake(' '.join(self.images)) + + for c in self.package_classes: + package_class = 'package_' + c + + deploy_reference = vars_reference['DEPLOY_DIR_' + c.upper()] + deploy_test = vars_test['DEPLOY_DIR_' + c.upper()] + + result = self.compare_packages(deploy_reference, deploy_test, diffutils_sysroot) + + self.logger.info('Reproducibility summary for %s: %s' % (c, result)) + + self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total)) + + if result.missing or result.different: + self.fail("The following %s packages are missing or different: %s" % + (c, ' '.join(r.test for r in (result.missing + result.different)))) +
Signature · 1 line
2.21.0
✗ patchtest: failure for Initial QA test for reproducible builds (rev2)
In reply to Joshua Watt== Series Details ==
Series: Initial QA test for reproducible builds (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/17720/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue Errors in your Python code were encountered [test_pylint]
Suggested fix Correct the lines introduced by your patch
Output Please, fix the listed issues:
meta/lib/oeqa/selftest/cases/reproducible.py does not exist
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe